AIMOR @ Banff 2026

Reinforcement Learning as a Sequential Decision Problem using the Universal Modeling Framework

Reinforcement learning emerged in the 1980s in computer science for approximating Bellman’s equation, building on the framework of Markov decision processes introduced in the 1950s by Bellman.  This work paralleled a line of research from the optimal control community starting in 1974 for deterministic control problems, and then the work on approximate dynamic programming in operations research in the 1990s.  Ultimately, all three lines of research came to the same conclusion: approximating Bellman’s equation is really hard and generally does not work.

The lines of research emerging from these communities (computer science using “reinforcement learning”, optimal control using “adaptive dynamic programming,” and operations research using “approximate dynamic programming”) would all grow to embrace a wide range of solution methods to meet the needs of the vast range of problems being addressed.  Over a series of articles from 2014 to 2019, I formalized the idea that all RL problems are sequential decision problems: decision, information, decision, information, …, which can be modeled using what I call the “universal modeling framework,” a derivative of what is used in optimal control.

I then realized that every method for making decisions could be organized into four classes of policies, only one of which uses Bellman’s equation.  Policies in each of the four classes can be found throughout the optimal control literature, as well as in the 2018 edition of Sutton and Barto’s book Reinforcement Learning.  This laid the foundation for my 2022 volume, Reinforcement Learning and Stochastic Optimization: A unified framework for sequential decisions which is written entirely around the universal modeling framework and all four classes of policies.

I will demonstrate how this framework identifies approaches for any sequential decision problem, spanning the traditional problems with discrete actions up to the high-dimensional problems faced in operations research.  All four classes of policies can be applied to stochastic search problems known as multi-armed bandit problems (I like the name “intelligent trial-and-error”), which are arguably the most common decision problem in practice.

Warren B Powell is Professor Emeritus at Princeton University, where he taught for 39 years, and is currently a co-founder and Chief Innovation Officer at Optimal Dynamics as well as Executive-in-Residence at Rutgers Business School. He was the founder and director of CASTLE Lab, which focused on stochastic optimization with applications to freight transportation, energy systems, health, e-commerce, finance and the laboratory sciences, supported by over $60 million in funding from government and industry. He has pioneered a new universal framework that can be used to model any sequential decision problem, including the identification of four classes of policies that spans every possible method for making decisions. This is documented in his latest book with Wiley: Reinforcement Learning and Stochastic Optimization: A unified framework for sequential decisions. He published over 250 papers, five books, and produced over 60 graduate students and post-docs. He is the 2021 recipient of the Robert Herman Lifetime Achievement Award from the Society for Transportation Science and Logistics, and the 2022 Saul Gass Expository Writing Award. He is a fellow of Informs, and the recipient of numerous other awards.

Learning in Stochastic Models:  An Asymptotic Approach

We consider queueing models with reneging in which the primitive distributions are unknown and must be learned.  The question of interest is how to prioritize arriving individuals for service.  When the reneging distribution is non-exponential, the state space is very complex, because we must track the amount of time each customer in queue has been waiting in order to have a Markovian system.  As a result, learning an exact optimal policy is very challenging.  

The approach we take is to first ignore the discrete and stochastic nature of arrivals, and to instead assume that the number of individuals to arrive in a time interval of length t>0 is proportional to t.  Then, we use historical (offline) data to formulate a data-driven (fluid) optimization problem (that may incorporate machine learning predictions) whose objective is to maximize long-run average reward.  The solution to the data-driven optimization problem motivates a data-driven policy for deciding how to prioritize arriving individuals for service.  With an infinite amount of historical data available, the data-driven policy is asymptotically optimal as the arrival rate and service capacity become large.  With a finite amount of historical data available, the challenge is to establish regret bounds and/ or finite sample statistical guarantees.

Amy Ward's research focuses on the approximation and control of stochastic systems, with applications to the service industry. Much of her past work has focused on the impact of customer impatience and abandonments on performance. Her more recent work investigates the interactions between behavioral incentives and operational efficiency in service systems.

Ward is a fellow of the INFORMS Manufacturing and Service Operations Management (MSOM) Society (elected 6/2023), and is the Editor-in-Chief for the journal Operations Research (term began 1/1/2024). In the past, she was Editor-in-Chief for the journal Operations Research Letters, and earlier held the position of Chair of the Applied Probability Society.

Prior to joining Booth, Ward was Professor of Data Sciences and Operations at the University of Southern California Marshall School of Business. She has also been a Visiting Associate Professor in the Computing and Mathematical Sciences Department at Cal Tech, and an Assistant Professor in Industrial and Systems Engineering at the Georgia Institute of Technology. Outside of academia, during her doctoral studies, she spent several summers at AT&T Laboratories.