First, we solve it exactly using dynamic programming, assuming full knowledge of

the model — the demand distribution, cost parameters, and transition dynamics.

Second, we show how a manager can learn the optimal policy from experience alone, using *Q-learning*.

Second, we show how a manager can learn the optimal policy from experience alone, using *[Q-learning](https://en.wikipedia.org/wiki/Q-learning)*.

The manager observes only the inventory level, the order placed, the resulting

profit, and the next inventory level — without knowing any of the underlying

We show that, given enough experience, the manager's learned policy converges to

the optimal one.

The lecture proceeds as follows:

1. We set up the inventory model and solve it exactly via value function iteration.

2. We introduce Q-factors and derive the Q-factor Bellman equation.

3. We implement Q-learning and show the learned policy converges to the optimal one.

A risk-sensitive extension of this model is studied in {doc}`rs_inventory_q`.

We will use the following imports:

```{code-cell} ipython3

\qquad

\text{where}

\quad

h(x,a,d) := (x - d) \vee 0 + a.

h(x,a,d) := \max(x - d, 0) + a.

$$

The term $A_t$ is units of stock ordered this period, which arrive at the start

of period $t+1$, after demand $D_{t+1}$ is realized and served.

**Timeline for period $t$:** observe $X_t$ → choose $A_t$ → demand $D_{t+1}$ arrives → profit realized → $X_{t+1}$ determined.

(We use a $t$ subscript in $A_t$ to indicate the information set: it is chosen

before $D_{t+1}$ is observed.)

Profits are given by

$$

\pi(X_t, A_t, D_{t+1}) := X_t \wedge D_{t+1} - c A_t - \kappa 1\{A_t > 0\}.

\pi(X_t, A_t, D_{t+1}) := \min(X_t, D_{t+1}) - c A_t - \kappa 1\{A_t > 0\}.

$$

Here

\sum_d \phi(d) \left[ \pi(x, a, d) + \beta \, v(h(x, a, d)) \right]

$$

When $r > 0$, the sequence $v_{k+1} = T v_k$ converges to the

When $r > 0$ (equivalently, $\beta < 1$), the sequence $v_{k+1} = T v_k$ converges to the

unique fixed point $v^*$, which is the value function of the optimal policy

(see, e.g., {cite}`Sargent_Stachurski_2025`).

We store the model primitives in a `NamedTuple`.

Demand follows a geometric distribution with parameter $p$, so $\phi(d) = (1 - p)^d \, p$ for $d = 0, 1, 2, \ldots$.

Demand follows a [geometric distribution](https://en.wikipedia.org/wiki/Geometric_distribution) with parameter $p$, so $\phi(d) = (1 - p)^d \, p$ for $d = 0, 1, 2, \ldots$.

```{code-cell} ipython3

class Model(NamedTuple):

```{code-cell} ipython3

@numba.jit(nopython=True)

def sim_inventories(ts_length, σ, p, X_init=0):

def sim_inventories(ts_length, σ, p, X_init=0, seed=0):

"""Simulate inventory dynamics under policy σ."""

np.random.seed(seed)

X = np.zeros(ts_length, dtype=np.int32)

X[0] = X_init

for t in range(ts_length - 1):

### The Q-learning update rule

Q-learning approximates the fixed point of the Q-factor Bellman equation using **stochastic approximation**.

Q-learning approximates the fixed point of the Q-factor Bellman equation using **[stochastic approximation](https://en.wikipedia.org/wiki/Stochastic_approximation)**.

At each step, the agent is in state $x$, takes action $a$, observes reward

$R_{t+1} = \pi(x, a, D_{t+1})$ and next state $X_{t+1} = h(x, a, D_{t+1})$, and

The update blends the current estimate $q_t(x, a)$ with a fresh sample of the Bellman target.

### What the manager needs to know

Notice what is **not** required to implement the update.

The manager does not need to know the demand distribution $\phi$, the unit cost $c$, the fixed cost $\kappa$, or the transition function $h$.

All the manager needs to observe at each step is:

1. the current inventory level $x$,

2. the order quantity $a$ they chose,

3. the resulting profit $R_{t+1}$ (which appears on the books), and

4. the next inventory level $X_{t+1}$ (which they can read off the warehouse).

These are all directly observable quantities — no model knowledge is required.

### The Q-table and the behavior policy

### The Q-table and the role of the max

It is important to understand how the update rule relates to the manager's

actions.

Bellman *optimality* operator, whose fixed point is $q^*$. The $\max$ in the

update target is therefore what drives convergence to $q^*$.

**The behavior policy.**

In short, the $\max$ is doing the work of finding the optimum; without it, you only evaluate a fixed policy.

### The behavior policy

The rule governing how the manager chooses actions is called the **behavior

policy**. Because the $\max$ in the update target always points toward $q^*$

profits while learning), while still occasionally experimenting to discover

better alternatives.

### What the manager needs to know

Notice what is **not** required to implement the update.

The manager does not need to know the demand distribution $\phi$, the unit cost $c$, the fixed cost $\kappa$, or the transition function $h$.

All the manager needs to observe at each step is:

1. the current inventory level $x$,

2. the order quantity $a$ they chose,

3. the resulting profit $R_{t+1}$ (which appears on the books), and

4. the next inventory level $X_{t+1}$ (which they can read off the warehouse).

These are all directly observable quantities — no model knowledge is required.

### Learning rate

We use $\alpha_t = 1 / n_t(x, a)^{0.51}$, where $n_t(x, a)$ is the number of times the pair $(x, a)$ has been visited up to time $t$.

This decays slowly enough to allow learning from later (better-informed) updates, while still satisfying the Robbins-Monro conditions for convergence.

This decays slowly enough to allow learning from later (better-informed) updates, while still satisfying the [Robbins–Monro conditions](https://en.wikipedia.org/wiki/Stochastic_approximation#Robbins%E2%80%93Monro_algorithm) for convergence.

### Exploration: epsilon-greedy

```{code-cell} ipython3

@numba.jit(nopython=True)

def q_learning_kernel(K, p, c, κ, β, n_steps, X_init,

ε_init, ε_min, ε_decay, snapshot_steps):

ε_init, ε_min, ε_decay, snapshot_steps, seed):

np.random.seed(seed)

q = np.zeros((K + 1, K + 1))

n = np.zeros((K + 1, K + 1)) # visit counts for learning rate

ε = ε_init

snapshots[snap_idx] = greedy_policy_from_q(q, K)

snap_idx += 1

# === Observe outcome ===

# === Draw D_{t+1} and observe outcome ===

d = np.random.geometric(p) - 1

reward = min(x, d) - c * a - κ * (a > 0)

x_next = max(x - d, 0) + a

```{code-cell} ipython3

def q_learning(model, n_steps=20_000_000, X_init=0,

ε_init=1.0, ε_min=0.01, ε_decay=0.999999,

snapshot_steps=None):

snapshot_steps=None, seed=1234):

x_values, d_values, ϕ_values, p, c, κ, β = model

K = len(x_values) - 1

if snapshot_steps is None:

snapshot_steps = np.array([], dtype=np.int64)

return q_learning_kernel(K, p, c, κ, β, n_steps, X_init,

ε_init, ε_min, ε_decay, snapshot_steps)

ε_init, ε_min, ε_decay, snapshot_steps, seed)

```

### Running Q-learning

We run 20 million steps and take policy snapshots at steps 10,000, 1,000,000, and at the end.

```{code-cell} ipython3

np.random.seed(1234)

snap_steps = np.array([10_000, 1_000_000, 19_999_999], dtype=np.int64)

q, snapshots = q_learning(model, snapshot_steps=snap_steps)

```

```{code-cell} ipython3

K = len(x_values) - 1

# restrict to feasible actions a ∈ {0, ..., K-x}

v_q = np.array([np.max(q[x, :K - x + 1]) for x in range(K + 1)])

σ_q = np.array([np.argmax(q[x, :K - x + 1]) for x in range(K + 1)])

```

sim_seed = 5678

# Optimal policy

np.random.seed(sim_seed)

X_opt = sim_inventories(ts_length, σ_star, p, X_init)

X_opt = sim_inventories(ts_length, σ_star, p, X_init, seed=sim_seed)

axes[0].plot(X_opt, alpha=0.7)

axes[0].set_ylabel("inventory")

axes[0].set_title("Optimal (VFI)")

# Q-learning snapshots

for i in range(n_snaps):

σ_snap = snapshots[i]

np.random.seed(sim_seed)

X = sim_inventories(ts_length, σ_snap, p, X_init)

X = sim_inventories(ts_length, σ_snap, p, X_init, seed=sim_seed)

axes[i + 1].plot(X, alpha=0.7)

axes[i + 1].set_ylabel("inventory")

axes[i + 1].set_title(f"Step {snap_steps[i]:,}")