SDA video modules
This page contains a series of short video modules that started as a three-hour tutorial for the 2026 AIMOR workshop in Banff. Each module is 5 to 25 minutes long, and lists guidelines on technical level and recommended predecessor modules.
The initial modules were posted in August, 2026. I anticipate improving the presentations and adding new modules, so pay attention to the posting and updating dates.
If any faculty would like access to the PowerPoint slides for use in lectures, email me at wbpowell328@gmail.com.
My background
Summary: Experience matters, and in this brief presentation I give a peek into my 40+ years of experience thinking about, working on, and implementing systems for solving a wide range of sequential decision problems.
History of research on sequential decision problems
Summary: This is a brief history starting with "reinforcement learning" circa 1900, up through its re-introduction in the 1950s, optimal control in the 1970s, modern reinforcement learning in the 1980s, and approximate dynamic programming in the 1990s.
Introduction to sequential decision problems
Summary: A presentation of examples of the diversity of sequential decision problems, from the simple inventory problems to complex logistics problems, including an initial introduction to optimal learning problems.
Framing decision problems
Summary: This is the first step to understanding a sequential decision problem, which starts by identifying metrics, decisions and uncertainties.
The universal modeling framework (high level)
Summary: This is how we model any sequential decision problem, consisting of five elements.
Estimation methods
Summary: Modeling and solving sequential decision problems draws heavily on the use of estimation methods from machine learning. This is a high level overview.
Universal modeling framework (detailed)
Summary: I now step through the individual elements of the universal modeling framework.
State variables
Summary: This module addresses the astonishing lack of understanding of state variables in various communities (applied probability, dynamic programming). State variables are fundamental to sequential decision problems.
Designing policies
Summary: This is the first introduction to the four classes of policies.
Choosing policies
Summary: Having presented all four classes of policies, we now tackle the question of actually choosing one (or a hybrid).
Policy function approximations
Summary: These are analytical functions that map states to actions, without an imbedded optimization problem.
Cost function approximations
Summary: We introduce the powerful idea of using parameterized versions of deterministic optimization models that are tuned to work well over time.
Value function approximations I — Introduction to dynamic programming
Summary: We start our discussion by introducing Bellman's equation and describing the three curses of dimensionality.
Value function approximations II — VFA-based policies
Summary: We introduce the idea of creating a policy that uses an approximation of the state that a decision takes us to.
Value function approximations III — Reinforcement learning
Summary: We introduce approximate dynamic programming as it is typically presented under the banner of "reinforcement learning," using the language typically found in computer science.
Value function approximations IV
Summary: We present the classical techniques of forward approximate dynamic programming, with an emphasis on the use of linear models to approximate the value function.
Value function approximations V — Applications
Summary: This segment summarizes a series of applications using VFA-based policies, typically in the context of resource allocation problems.
Direct lookahead approximations I — Overview
Summary: We set up the notation for lookahead models, and identify the different ways to approximate lookahead models to make a decision now.
Direct lookahead approximations II — Deterministic lookaheads
Summary: A generalization of cost function approximations (which do not plan into the future) is to parameterize deterministic models which do plan into the future.
Direct lookahead approximations III — Stochastic lookaheads
Summary: Focuses on the issue of choosing the "policy-within-a-policy" required by stochastic lookahead models. We illustrate PFAs, CFAs and VFAs.
Direct lookahead approximations IV — Discrete actions
Summary: We introduce the idea of using decision trees for small problems, and Monte Carlo tree search for larger problems.
Direct lookahead approximations V — Vector-valued actions
Summary: We introduce two-stage stochastic programming illustrated by its use in power planning. We then introduce the idea of solving the stochastic lookahead problem using classical stochastic search.
Policy search
Summary: We address the challenge of tuning a parameterized policy using derivative-based and derivative-free stochastic search. Stochastic search is presented as a sequential decision problem, and illustrated using a mutual fund cash balance problem.
Optimal learning
Summary: One of the most important sequential decision problems involves learning over time, either in an offline setting (laboratory, computer simulation) or an online setting. We compare a CFA-based policy (interval estimation) to a DLA-based policy (the knowledge gradient).
Deterministic optimization as a sequential decision problem
Summary: There are many classical deterministic optimization problems that, in practice, are solved repeatedly over time. How a deterministic optimization models are used is often overlooked, especially when they are hard integer or nonlinear programs.
Modeling matters
Summary: This module illustrates a published example of a paper that makes the mistake of assuming that you can use information that does not arrive until the future. This type of mistake is not unusual in engineering communities.