Reinforcement Learning and Feedback Control

How an agent can learn the optimal controller online, along the plant's own trajectories, without knowing the plant's dynamics.

Notation used throughout

Every symbol is also defined where it first appears; this table is just a crib sheet you can scroll back to.

1. Two philosophies, and the gap between them

Suppose you must control a plant — a power system, a missile, a chemical reactor. Control theory gives you two classic answers, and they embody opposite trade-offs.

Optimal control designs the controller offline. You write down a performance index (minimum fuel, minimum energy, minimum risk), model the plant completely, and solve the Hamilton–Jacobi–Bellman (HJB) design equation — for linear systems this collapses to the familiar algebraic Riccati equation. The result is optimal by construction. The price: you need complete knowledge of the dynamics before you can even start, and for nonlinear systems the HJB equation is a nonlinear PDE that is often impossible to solve.

Adaptive control does the opposite. It learns online, in real time, from data measured along the actual trajectories, and it tolerates unknown plant parameters. The price: adaptive controllers are built to stabilize and track, not to optimize a user-prescribed performance index. An adaptive controller that works is not necessarily a good controller in any optimal sense.

The historical lineage, briefly, because it explains the vocabulary you will meet: reinforcement learning was developed in the computational intelligence community — from Pavlov's conditioning experiments, through animal learning models, to machine learning formulations in which an agent adjusts its actions from reward and punishment stimuli. In the human brain, dopamine in the basal ganglia plays exactly the role of the reinforcement signal. Werbos turned these ideas into actor–critic feedback control designs and named the family approximate dynamic programming (ADP); Bertsekas and Tsitsiklis developed the offline theory as neurodynamic programming; Watkins contributed Q learning. Applications reported range from missile and aircraft control to helicopter reconfiguration after rotor failure, power systems, and vehicle steering.

One language note before we start. In computational intelligence the agent maximizes reward (utility); in control engineering we minimize cost. The article (and this lesson) uses the control convention: stage costs, discounted future costs, and minimization throughout. Nothing else changes.

2. The actor–critic control loop

Why introduce an architecture at all? Because the learning problem naturally splits into two questions that need two different mechanisms: what should I do right now? and how good is what I am currently doing? Reinforcement learning answers them with two interacting components, and every algorithm in this lesson is a way of implementing them.

The actor is the feedback controller: it implements the current control policy and applies actions to the environment (in control language: the plant). The critic is an evaluator: it observes the reward/response from the environment and assesses the value of the current policy using a value function — a quantification of how close the current policy is to optimal with respect to objectives such as minimum fuel, minimum energy, or minimum risk. Learning then proceeds in two steps, forever: policy evaluation (the critic estimates the value of the current policy from observed responses), followed by policy improvement (the actor modifies the policy so the new value is better).

Two details of this picture repay attention. First, the two timescales: the actor's control action is applied in real time (fast loop), while the critic's evaluation and the resulting policy update happen more slowly, as data accumulates. Second, the relationship to classical adaptive control is exact and worth internalizing. A direct adaptive controller tunes the controller parameters; an indirect one first identifies a model, then designs a controller from the model. The actor–critic optimal adaptive controller is the logical next rung on that ladder: it identifies the performance value of the current policy first, then uses that information to update the controller. It never needs to identify the plant itself.

3. Markov decision processes: the language

Before any algorithm can be stated, we need a mathematical setting in which “control policy”, “optimal”, and “learn from experience” all have precise meanings. Markov decision processes (MDPs) provide it. They are worth the setup cost: every algorithm in the rest of the lesson — dynamic programming, policy iteration, value iteration, Q learning, integral reinforcement learning — is phrased in this language.

An MDP is a tuple $(\mathcal X, \mathcal U, P, R)$ where $\mathcal X$ is the set of states, $\mathcal U$ the set of actions (controls), $P$ the transition probabilities, and $R$ the cost function:

Here $r_k = r(x_k, u_k, x_{k+1})$ is the stage cost paid at step $k$ — think fuel burned, energy spent, risk incurred. The Markov property says the transition probabilities depend only on the current state, not on the history of how the system got there. That is exactly the property that feedback control of lumped-parameter state-space models enjoys, which is why MDPs fit control so naturally.

A policy $\pi$ is a rule for choosing actions: $\pi(x,u) = \Pr\{u \mid x\}$. If it selects a single control with probability one in each state, it is deterministic and can be written $\pi(x): \mathcal X \to \mathcal U$ — in other words, a policy is a feedback controller. A stationary policy does not depend on time. Once a stationary policy is fixed, the “closed-loop” MDP collapses to a plain Markov chain with frozen transition probabilities $p^{\pi}_{xx'} = \sum_u \pi(x,u)\, P^u_{xx'}$ (the Chapman–Kolmogorov identity). If that chain is ergodic for every policy — all states positive recurrent and aperiodic — then a stationary distribution exists and one can prove that every finite MDP has a stationary deterministic optimal policy. This is the technical license for restricting attention to feedback laws $u=\pi(x)$ from now on.

The value function of a policy is the expected value of that future cost when starting in state $x$ and following $\pi$ thereafter:

The $E_{\pi}\{\cdot\}$ is the expectation over the randomness of transitions and (possibly) of the policy itself. The basic problem of optimal sequential decision making is then:

One more piece of vocabulary that recurs later: the $Q$ (action-value) function will be introduced in Section 7 as “the value of taking a prescribed action $u$ now, then following the policy”. Keep the distinction in mind: $V$ scores states, $Q$ scores state–action pairs.

4. Dynamic programming: optimal but backward

Suppose we knew a terminal value and wanted to walk backwards from a horizon $T$. Using the Markov property and the Chapman–Kolmogorov identity, the value of following policy $\pi$ can be split into the immediate cost plus the value of continuing from the next state:

Every step of this derivation is forced: the first equality is the definition of expectation of a sum of a random immediate term and a random future term; the sum over $x'$ averages over where the state lands; the sum over $u$ averages over the policy's own randomization. Equation (1) is a backward recursion: value at time $k$ computed from value at time $k+1$.

Bellman's principle of optimality is the observation that makes DP work: “an optimal policy has the property that no matter what the previous control actions have been, the remaining controls constitute an optimal policy with regard to the state resulting from those previous controls.” In plain words: if you are on an optimal route from A to C through B, then the tail of the route, B to C, must itself be optimal. Applied to (1), the minimization over whole policies collapses to a minimization over the next single action, with the optimal value of everything that follows already priced in:

This is the dynamic programming recursion. It is exact, and it is also — for our purposes — unusable online, for two structural reasons:

(a) It runs backward in time. To evaluate $V^*_k$ you must already know $V^*_{k+1}$: the recursion starts at the horizon and walks home. Real control lives forward in time; at time $k$ the future values do not exist yet.

(b) It needs the full model. The sums over $x'$ are expectation operators over the transition probabilities $P^u_{xx'}$ and costs $R^u_{xx'}$ — that is, complete knowledge of the system dynamics. If you do not know how the plant responds, you cannot take those expectations.

5. The Bellman equation: the hinge of the whole story

5.1 From backward recursion to fixed point

The trick is to let the horizon go to infinity. When the cost is $J_k = \sum_{i=k}^{\infty} \gamma^{i-k} r_i$, the value of a policy becomes time-invariant: $V^{\pi}(x)$ no longer depends on $k$, because there is no terminal time from which to count. Applying the split (1) with $T=\infty$ and dropping the stage index gives:

Look carefully: the same $V^{\pi}$ appears on both sides. That is not a derivation slip — it is the entire point. The infinite sum of future costs, discounted, looks the same from any starting time; the value function must simply be consistent with itself under one step of the dynamics. Equation (3) is called the Bellman equation, and it is a fixed-point equation: its solution is the value defined by the infinite sum, but it can be checked — and, as we will see, iterated toward — one step at a time, forward.

For a finite MDP with $N$ states, (3) is a system of $N$ simultaneous linear equations in the $N$ unknown values $V^{\pi}(x)$ — linear because the policy $\pi$ is fixed. Solving it is called policy evaluation: it prices out the current controller.

Now set $\pi = \pi^{*}$ and let the minimization over policies act on the right-hand side. Bellman's optimality principle turns (3) into the Bellman optimality equation:

In control systems language, (4) is the discrete-time HJB equation. Unlike (3) it contains no policy — but it contains the $\min$ operator, so it is a system of $N$ nonlinear equations, and taking the expectations requires the transition probabilities, i.e. the full dynamics. The Bellman equation (3) is easier to solve and better suited to the actor–critic architecture: it evaluates a given policy, which is precisely the critic's job.

5.2 Evaluation and improvement: the two levers

Given the value $V^{\pi}$ of the current policy (evaluation), we can always construct a better policy by acting greedily with respect to it — pick, in each state, the action that minimizes immediate cost plus the discounted value of where it lands:

Why should one-step greediness help over the long run? Because $V^{\pi}(x')$ already prices all future consequences of landing in $x'$: the one-step minimization is really minimizing the total. One can show $V^{\pi'}(x) \le V^{\pi}(x)$ — the greedy policy is never worse. And the fixed points of this operation are exactly the optimal policies: if improvement changes nothing, $\pi' = \pi$, then $V^{\pi}$ satisfies the optimality equation (4) and $\pi$ is optimal. Indeed, an optimal policy, and only an optimal policy, is greedy with respect to its own value. (In computational intelligence jargon, “greedy” means optimizing a one-step horizon without regard to longer impacts — here the value function supplies those longer impacts.)

Convergence of alternating (5) with evaluation is not an accident: the pair defines a map on value functions that is a contraction under mild conditions, so by the Banach fixed-point theorem it converges to the unique solution of (4). Almost every convergence proof in this field is some incarnation of that argument.

5.3 The LQR bridge: Bellman equations become Lyapunov and Riccati equations

Consider the discrete-time LQR: $x_{k+1} = A x_k + B u_k$ with quadratic stage cost $r_k = \frac12 (x_k^{\mathsf T} Q x_k + u_k^{\mathsf T} R u_k)$, $Q \ge 0$, $R \gt 0$, and undiscounted cost $\gamma = 1$. Suppose the current policy is a linear stabilizing feedback $u_k = -K x_k$. Assume the value is quadratic in the state, $V(x_k) = \frac12 x_k^{\mathsf T} P x_k$ for some symmetric kernel matrix $P$ — reasonable, since the cost being summed is quadratic and the dynamics linear. Write the Bellman equation (3) along a trajectory:

More carefully, substituting $u_k = -Kx_k$ and $V(x_{k+1}) = \tfrac12 x_{k+1}^{\mathsf T} P\, x_{k+1} = \tfrac12 x_k^{\mathsf T}(A-BK)^{\mathsf T} P (A-BK) x_k$:

This holds for every trajectory, so the quadratic forms must match for all $x_k$ (with $V(0)=0$ anchoring the constant):

For the LQR, the Bellman equation is a Lyapunov equation. Since the derivation substituted the closed-loop matrix $A - BK$, this form requires knowing the plant. The paper is explicit that this is a trap for learning: by assuming (6) holds for all trajectories and collapsing the trajectory data into a matrix equation, you have thrown away the data form in which reinforcement learning operates. Keep the difference-equation form

— it involves no $(A,B)$ at all, only states, control, and stage cost measured at time $k$. Learning algorithms will solve (7) from data instead of solving (6) from matrices.

Now do the same for the optimality equation. Define the LQR Hamiltonian

which is the left-hand side of the Bellman equation (7) rearranged — the paper notes the Hamiltonian is exactly the temporal difference error we will meet in Section 8. Stationarity $\partial H/\partial u_k = 0$ gives $R u_k + B^{\mathsf T} P (A x_k + B u_k) = 0$, so

Substituting back into the optimality equation yields

the discrete-time algebraic Riccati equation (ARE). So the correspondence is exact:

5.4 Worked example: a three-state MDP solved by hand

To make policy evaluation and improvement concrete, here is a tiny deterministic MDP you can check with mental arithmetic. Discount $\gamma = 0.9$; the goal state $s_G$ is absorbing and free. From $s_1$ you may walk to $s_2$ (cost 1) or jump straight to $s_G$ (cost 1.6). From $s_2$ the only action is walk to $s_G$ (cost 1).

Policy evaluation. Start with policy $\pi_0$: walk everywhere. The Bellman equation (3) for this deterministic chain is $V(s) = r + \gamma V(s')$ at each state. With $V(s_G)=0$:

$V^{\pi_0}(s_2) = 1 + 0.9 \cdot 0 = 1$, then $V^{\pi_0}(s_1) = 1 + 0.9 \cdot 1 = 1.9$.

Policy improvement. In $s_1$, compare the two actions using (5): walking gives $1 + 0.9\,V^{\pi_0}(s_2) = 1.9$; jumping gives $1.6 + 0.9\cdot 0 = 1.6$. Jumping is better, so $\pi_1$: jump from $s_1$, walk from $s_2$. Re-evaluating gives $V^{\pi_1}(s_1) = 1.6$, $V^{\pi_1}(s_2) = 1$ — and a second improvement pass changes nothing: $\pi_1$ is greedy with respect to its own value, so it is optimal. Two evaluation–improvement rounds, by hand.

6. Policy iteration and value iteration

6.1 Policy iteration

Start from an initial policy $\pi_0$ and iterate on $j$ until convergence:

Policy evaluation (value update): solve the Bellman equation (3) exactly for the value of the current policy $\pi_j$:

Policy improvement (policy update): act greedily with respect to $V_j$:

Each round fully solves (9) — for a finite MDP with $N$ states, a system of $N$ linear equations — and then takes one greedy step. Note that $j$ is a policy iteration step index, not the time $k$; solving (9) requires data from many time steps, but $j$ itself counts rounds of the algorithm. For finite MDPs, policy iteration converges in at most $N$ rounds (there are only finitely many deterministic policies, and each round's policy is strictly no worse); the initialization must satisfy $V_1 \le V_0$, which a stabilizing (admissible) initial policy guarantees.

Instead of solving (9) exactly, one can iterate toward it: the iterative policy evaluation

is a contraction map (Banach fixed-point theorem again), so it converges to the solution of (9) from any nonnegative initialization — a convenient choice being the value from the previous round, $V_{j-1}$. Here $i$ is the inner iteration index. Do not confuse (11) with the backward recursion of dynamic programming: there, the index was time $k$ marching backward from a known terminal condition; here, $i$ counts sweeps of a fixed-point iteration whose target is the infinite-horizon value.

6.2 Value iteration

Value iteration simply takes one iteration of (11) per round instead of many — it never finishes evaluating the current policy, it just keeps nudging the value toward the optimum:

Value iteration is the cheapest to implement (a one-step recursion, no linear system to solve) but generally needs infinitely many rounds to converge exactly. Standard value iteration updates all states each sweep; asynchronous variants update only a subset $S_j \subseteq \mathcal X$ per sweep — in the extreme, a single state — and still converge provided every state is updated infinitely often (and, for $\gamma = 1$, a properness/absorbing-state condition). Interpreted against dynamic programming, $V_j(x')$ plays the role of an estimate of the future cost to go from $x'$; algorithms whose future-cost estimates are themselves values of some policy are called rollout algorithms, and they are the ancestors of receding-horizon (model predictive) control.

6.3 Generalized policy iteration

The two algorithms are the endpoints of a spectrum. Generalized policy iteration runs several iterations of (11) between policy improvements: policy iteration does the most evaluation work per round and converges in fewest rounds; value iteration does the least and needs the most; everything in between trades one for the other.

6.4 What these become for the LQR

Applying policy iteration to the LQR Bellman equation (7) gives the evaluation step $V_{j+1}(x_k) = \tfrac12(x_k^{\mathsf T}Qx_k + u_k^{\mathsf T}Ru_k) + V_{j+1}(x_{k+1})$, which — in kernel form — is a Lyapunov equation per round:

This is Hewer's algorithm, the classic iterative Riccati solver: it converges to the ARE solution under stabilizability and detectability, but needs a stabilizing initial gain $K_0$ and full knowledge of $(A,B)$. Value iteration instead gives the Lyapunov recursion

a one-step recursion (no Lyapunov equation solved per round) that converges to the ARE solution for any initial gain — but still requires $(A,B)$. Both are offline design methods. Similarly, with a fixed policy, the iterative policy evaluation (11) is just $P_{j+1} = (A-BK)^{\mathsf T} P_j (A-BK) + Q + K^{\mathsf T} R K$, which converges to the Lyapunov equation's solution whenever $A-BK$ is stable, from any $P_0$.

6.5 Worked example: a scalar LQR, solved completely by hand

To see the whole chain — Bellman equation, stationarity, Riccati, optimal gain — in numbers small enough to verify mentally, take an unstable scalar plant $x_{k+1} = 1.1\, x_k + u_k$ (so $a = 1.1$, $b = 1$) with cost $J = \sum_{i=k}^{\infty} \frac12 (x_i^2 + u_i^2)$ ($Q = R = 1$, $\gamma = 1$). Guess the value's shape: quadratic, $V(x) = \tfrac12 p x^2$.

Why is guessing legitimate? Because the Bellman equation is an equation in functions; we test a candidate shape and let the equation determine the unknown coefficient. Substituting $V$ into (7) with the not-yet-optimal control $u$:

The optimal control minimizes the right-hand side over $u$ — differentiate and set to zero (this is the stationarity step of the Hamiltonian):

Substituting $u^*$ back and matching the coefficients of $x^2$ gives the scalar ARE. Multiplying through by $(1+p)$ and collecting terms: $p^2 = 1.21\, p + 1$, i.e.

(The negative root is rejected: the value must be positive definite.) The optimal gain follows:

As a consistency check, insert $K$ into the Lyapunov form (6): $(1.1 - 0.703)^2 \times 1.774 - 1.774 + 1 + 0.703^2 \approx 0.279 - 1.774 + 1 + 0.495 = 0$ ✓. The closed loop $x_{k+1} = (1.1 - 0.703)\,x_k = 0.397\, x_k$ is stable, as it must be. Notice what information entered where: the value shape ($V$ quadratic) came from the cost; the gain came from minimizing; and the plant parameters $a$, $b$ entered only through the numbers in the Riccati equation. The entire game of the next three sections is to let data play the role of those numbers.

7. The Q function: minimizing without a model

Given a policy $\pi$, define the Q function (Watkins called $Q$ the “quality” function; Werbos called the resulting methods action-dependent dynamic programming) as the expected return for taking an arbitrary action $u$ at time $k$ in state $x$, and following $\pi$ thereafter:

Compare with the value function: $V^{\pi}(x) = Q^{\pi}(x, \pi(x))$ — the value of a state is the Q-value of the action the policy selects there. Substituting this identity into (15) gives the Q function its own Bellman equation,

and the optimality equation takes a striking form — compare (4): the minimum and the expectation have swapped places,

Policy iteration and value iteration go through unchanged in terms of $Q$ — evaluate (16), improve by $\arg\min_u Q$, or combine them into a single value-iteration recursion. The improvement step is now explicit: no dynamics appear in $\arg\min_u Q^{*}(x,u)$.

The Q function for the LQR

For the discrete-time LQR the Q function following a policy is $Q(x_k, u_k) = \tfrac12 (x_k^{\mathsf T} Q x_k + u_k^{\mathsf T} R u_k) + V(x_{k+1})$ — stage cost plus optimal-or-current value of the next state, with $u_k$ free. Since $V$ is quadratic, $Q$ is a quadratic form in the augmented vector $z_k = [\,x_k^{\mathsf T}\;\; u_k^{\mathsf T}\,]^{\mathsf T}$:

The kernel matrix $S$ obviously depends on the plant $(A, B)$ — but only through data will we estimate it, never through formulas like (18). The punchline is the improvement step. Setting $\partial Q/\partial u_k = 0$:

Contrast the two expressions for the optimal gain: $K = (B^{\mathsf T} P B + R)^{-1} B^{\mathsf T} P A$ contains $A$ and $B$ explicitly; $S_{uu}^{-1} S_{ux}$ contains only entries of the Q-function kernel. If we can identify $S$ from measured data — and Section 10 shows we can, with plain least squares — we obtain the optimal controller for completely unknown dynamics.

8. Learning from data: exact, Monte Carlo, temporal difference

1. Exact computation. Use $P$ and $R$ directly. This is classical dynamic programming: offline, model-based, exact. For the LQR it degenerates to repeatedly solving Lyapunov equations (policy iteration = Hewer's method) or Lyapunov recursions (value iteration). Useful as a design reference; useless without a model.

2. Monte Carlo learning. Estimate the expectations by averaging observed outcomes: run the system through complete episodes (start to termination, then restart), and average the observed returns. Convergence for finite MDPs requires every state to be visited infinitely often — the exploration problem, commonly handled by exploring starts, where every state has a nonzero chance of being chosen as the initial state. The structural drawback for control: nothing is updated until an episode terminates, so learning does not happen in real time along a trajectory but as trajectories are repeated. Monte Carlo methods are thus close relatives of iterative learning control, where the same trajectory is replayed and improved.

3. Temporal difference (TD) learning — the one that matters here. The idea, straight from stochastic approximation: replace the stochastic Bellman equation (3) by its evaluation along the single sample path the system actually produces. Along one trajectory, (3) becomes the deterministic consistency requirement

This will not hold exactly for an imperfect value estimate, and the gap is the learning signal — the temporal difference error:

Each data experience set $(x_k, x_{k+1}, r_k)$ — current state, next state, observed cost — yields one instance of (21), and the value estimate is updated so as to make $e_k$ small. Why is it legitimate to replace an expectation by one sample? Because the updates are made repeatedly along the trajectory with decreasing step sizes; stochastic approximation theory guarantees that, on average, the iterations converge to the solution of the stochastic equation being sampled. We are averaging, but online, one sample at a time, forever — no episodes, no restarts.

9. Optimal adaptive control for discrete-time systems

9.1 Setup

Consider deterministic discrete-time nonlinear dynamics in affine form (the analysis generalizes to $x_{k+1} = F(x_k,u_k)$):

A policy is a feedback controller $u_k = h(x_k)$. A policy is admissible if it stabilizes the system on a set $\mathcal X \subseteq \mathbb R^n$ (stabilizability assumed) and yields a finite cost. Take a stage cost $r(x_k, u_k) = \mathcal Q(x_k) + u_k^{\mathsf T} R u_k$, where the state-weight function $\mathcal Q$ (calligraphic, to avoid clashing with the Q function of Section 7) satisfies $\mathcal Q(x) \ge 0$ with $\mathcal Q(x) = 0 \Leftrightarrow x = 0$, and $R \gt 0$. The value of using policy $h$ is the discounted sum $V^{h}(x_k) = \sum_{i=k}^{\infty} \gamma^{i-k} \big(\mathcal Q(x_i) + u_i^{\mathsf T} R u_i\big)$, which satisfies the difference equation (the deterministic Bellman equation (20) again)

The optimal value satisfies the discrete-time HJB equation, and the optimal policy follows by minimizing over $h$:

For later use define the discrete-time Hamiltonian associated with policy $h$:

The Hamiltonian captures the energy content of the trajectories as reflected in the desired performance — and notice it is exactly the temporal difference error (21). The Bellman equation simply demands $H = 0$ for the value of a prescribed policy. No dynamics appear in (25): $V(x_{k+1})$ is measured, not predicted.

9.2 Policy iteration and value iteration, temporal-difference style

Write the two algorithms of Section 6 with deterministic dynamics and TD evaluation:

Policy iteration. Initialize with an admissible policy $h_0$; iterate on $j$:

Carrying out the minimization explicitly (differentiate $r + \gamma V_{j+1}$ with respect to $u_k$) gives the improved policy in terms of the value gradient $\nabla V = \partial V/\partial x$, taken as a column vector:

Value iteration. Same two steps, but the evaluation uses the old value on the right: $V_{j+1}(x_k) = r(x_k, h_j(x_k)) + \gamma V_j(x_{k+1})$. The reward: any initial $h_0$ may be chosen — not necessarily admissible or stabilizing.

9.3 Value function approximation: the critic network

By the Weierstrass higher-order approximation theorem, if the Bellman equation has a local smooth solution there exists a dense basis set $\{\varphi_i(x)\}$ such that, for a truncation with $L$ terms,

with basis (activation) vector $z(x) = [\varphi_1(x)\; \dots\; \varphi_L(x)]^{\mathsf T}: \mathbb R^n \to \mathbb R^L$, unknown weights $W$, and residual $\varepsilon_L(x) \to 0$ uniformly as $L \to \infty$ (bounded by a constant on compact sets for neural-network bases such as sigmoids, tanh, or Gaussians; polynomial bases are the classical Weierstrass choice). This approximator is the critic network. For the LQR the value is known to be quadratic, so the basis is the quadratic monomials in the state components — since $P$ is symmetric it has $n(n+1)/2$ independent entries, so $L = n(n+1)/2$: for $n=4$ states, ten weights.

9.4 The critic tuned by least squares

Substitute $V^h(x) \approx \hat V(x) = W^{\mathsf T} z(x)$ into the policy-iteration evaluation (26). For the current policy $h_j$ held fixed, measure along the trajectory the data set $(x_k, x_{k+1}, r(x_k, h_j(x_k)))$ and require consistency:

This is one scalar linear equation in the $L$ unknowns $W_{j+1}$ — the regression vector is $\bar z(k) = z(x_k) - \gamma z(x_{k+1})$, and it involves no dynamics whatsoever, only measured states and cost. Each time step delivers one more equation; after $L$ steps (or a batch) the least-squares solution gives $W_{j+1}$. Alternatively run recursive least squares (RLS) online, one update per time step, until convergence. The standard identifiability requirement applies: the regression vector must be persistently exciting — rich enough trajectories to pin down all $L$ parameters. A gradient-descent alternative tunes the weights by the TD error,

where the bracketed quantity is precisely the temporal difference error and the inner index $i$ advances with time while the algorithm step $j$ is held fixed.

9.5 The actor network — and why it lets us forget $f(\cdot)$

Write the control as a second network, the actor:

with activation vector $v(x): \mathbb R^n \to \mathbb R^M$ and actor weights $W_a \in \mathbb R^{M \times m}$. For the LQR the optimal control is linear in the state, so $v(x) = x$. After the critic converges to $W_{j+1}$, tune the actor by gradient descent on the one-step cost plus discounted value,

which requires observations of the data set $(x_k, x_{k+1})$ at each time. Here is the subtle and important point: the critic's equation (29) never contains the state dynamics $A$ (or $f(\cdot)$); the actor's tuning (32) uses the measured next state $x_{k+1}$, so the actor network learns the information about $A$ (or $f$) into its own weights. Only the input coupling $g(\cdot)$ (matrix $B$) is needed explicitly, and only inside the known factor of the policy update. In the LQR the actor update reproduces the gain formula $K_{j+1} = -(B^{\mathsf T} P_{j+1} B + R)^{-1} B^{\mathsf T} P_{j+1} A$, which needs full $(A,B)$ — the actor-network version is the model-free surrogate of exactly that step. And after tuning, the actor (31) is a legitimate feedback controller: it maps the current state to control; $x_{k+1}$ is only needed during learning.

9.6 Example 1 (from the paper): load frequency control of a power system

A power system is a complicated nonlinear system, but during normal operation the load varies only slightly, so a linear model about the operating point is adequate — while the parameter values in it are not precisely known. Exactly the situation optimal adaptive control is built for. The continuous-time model is $\dot x = Ax + Bu$ with state $x = [\Delta f \;\; \Delta P_g \;\; \Delta X_g \;\; \Delta E]^{\mathsf T}$ (frequency deviation, generator output, governor position, integral control) and parameters: time constants $T_G, T_T, T_P$, gains $K_P$, speed regulation $R$, integral gain $K_E$. Discretized by zero-order hold with sampling period 0.01 s it gives the discrete-time plant used for data generation, with

What the controller knows: only the governor time constant $T_G$, which appears in $B$. The entire $A$ matrix is needed only to simulate the plant — not by the control algorithm. The performance index uses $Q = I_4$, $R = 1$, $\gamma = 1$.

The critic basis is the $n(n+1)/2 = 10$ quadratic monomials; the actor basis is the state itself (linear optimal control). Value iteration runs online: every 15 measured data points $(x_k, x_{k+1}, r_k)$ (i.e. each 0.15 s) a batch least-squares problem updates the ten critic weights; the actor is then updated; the run lasts 60 s. The result, from the paper: states are regulated to zero, and the critic's kernel converges to the algebraic Riccati solution — compare the matrices:

The controller solved the Riccati equation in real time without knowing $A$. That is the paper's thesis, demonstrated on a physical system.

10. Q learning: completely unknown dynamics

10.1 Watkins' Q learning for finite MDPs

Q learning, developed by Watkins (with Werbos' earlier action-dependent designs), works for MDPs with completely unknown transition probabilities and costs. Perform an action $u_k$, measure the resulting data experience set $(x_k, x_{k+1}, r_k)$, and write the Q-function Bellman equation (16) along the sample path — the Q-learning TD form:

Based on the value-iteration form for $Q$ (Section 7), the update is

Watkins proved convergence for finite MDPs using stochastic approximation, provided all state–action pairs are visited infinitely often (exploration again) and the step sizes satisfy the standard conditions $\sum_{k} \alpha_k = \infty$, $\sum_k \alpha_k^2 \lt \infty$. On convergence the TD error (33) is approximately zero. For finite MDPs the Q function is a lookup table over all state–action pairs — and notice the structure of (34): the improvement operator $\min_u$ is applied to a stored function, exactly the model-freeness Figure 7 promised.

10.2 Q learning for dynamical systems

For discrete-time dynamical systems, parameterize the Q function directly. For the LQR it is quadratic in the augmented vector $z_k = [\,x_k^{\mathsf T} \; u_k^{\mathsf T}\,]^{\mathsf T}$ (Section 7); assume a quadratic form for nonlinear systems too (or use any suitable basis):

where $\bar z(z)$ is the quadratic basis in the components of $z$ (with redundant entries removed, $W$ collects the $(n+m)(n+m+1)/2$ independent kernel entries) and $z_{k+1} = [\,x_{k+1}^{\mathsf T} \; u_{k+1}^{\mathsf T}\,]^{\mathsf T}$ with $u_{k+1} = h_j(x_{k+1})$ computed from the current policy. Now both algorithms of Section 7 become plain linear system identification:

Q-learning policy iteration: evaluation by least squares / RLS on $W_{j+1}^{\mathsf T}\big(\bar z(z_k) - \gamma \bar z(z_{k+1})\big) = r(x_k, h_j(x_k))$, then improvement by minimizing over $u$ inside the Q function: $h_{j+1}(x_k) = \arg\min_u W_{j+1}^{\mathsf T} \bar z(x_k, u)$ — for the quadratic form this is the explicit formula $u_k = -S_{uu}^{-1} S_{ux} x_k$ of (19).

Q-learning value iteration: evaluation on $W_{j+1}^{\mathsf T} \bar z(z_k) = r(x_k, h_j(x_k)) + \gamma W_{j}^{\mathsf T} \bar z(z_{k+1})$ (old weights on the right), then the same improvement.

Three properties make this the strongest result in the discrete-time part of the paper:

(a) No dynamics at all. Neither $f(\cdot)$ nor $g(\cdot)$ appears anywhere — the algorithms solve the Bellman equation (23) and the HJB equation (24) online using only measured $(z_k, z_{k+1}, r_k)$. The LQR version converges to the algebraic Riccati solution — the ARE is solved online without knowing $A$ or $B$, which the sidebar of the paper calls an adaptive controller for the online solution of the DT LQR via Q learning; the initial gain need not be stabilizing and may even be zero.

(b) No actor network needed. Because $u_k$ is an explicit argument of $Q$, the improvement $\partial(W^{\mathsf T}\bar z(x_k,u))/\partial u = 0$ is computed directly — one network suffices. (Werbos' name for this structure: action-dependent heuristic dynamic programming, ADHDP.)

(c) Exploration is still mandatory. Probing noise must be added to the control input to guarantee persistency of excitation, so the least-squares problem identifies all kernel parameters; without it, the trajectory never visits enough state–action pairs.

11. Continuous time and integral reinforcement learning

11.1 Why continuous time is harder

Consider the continuous-time nonlinear system $\dot x = f(x) + g(x) u$, $x(0)$ an equilibrium ($f(0)=0$), $f + gu$ Lipschitz on a compact set $\mathcal X$, stabilizable. A policy $u = \mu(x)$ is admissible if it stabilizes the system with finite associated cost. The value of a policy is now an integral:

If the cost is smooth, an infinitesimal version of the Bellman equation follows from Leibniz's formula — differentiate (36) with respect to $t$:

with the continuous-time Hamiltonian

The optimal value satisfies the HJB equation $0 = \min_{\mu} H(x, \mu, \nabla V^{*})$ with optimal control $\mu^{*} = \arg\min_{\mu} H(x, \mu, \nabla V^{*})$.

11.2 Escape route 1: Euler discretization (approximate)

The quick fix is to discretize (37) by Euler's method with sample period $T_s$ — replace $\dot V$ by the forward difference $(V(x_{k+1}) - V(x_k))/T_s$ and the dynamics by their samples, writing the sampled utility as $r_s(x_k,u_k) = r(x_k,u_k)\,T_s$:

that is, $V(x_k) \approx r_s(x_k, u_k) + V(x_{k+1})$ with sampled utility $r_s(x_k, u_k) = r(x_k, u_k)\,T_s$.

The discretized Bellman equation has exactly the same form as the discrete-time Bellman equation (23), so all the discrete-time machinery of Sections 9–10 applies. The price: it is only an approximation, whose fidelity depends on the sampling period.

11.3 Escape route 2: integral reinforcement learning (exact)

The exact alternative, due to Vrabie and colleagues, observes that the cost (36) can be split over a finite interval $T \gt 0$ (not a sampling period — a learning interval, freely chosen):

Why is this the key? Because (39) is exactly the discrete-time Bellman equation (23) — immediate discounted cost replaced by a finite integral of cost, and $x(t+T)$ measured. Vrabie et al. show (39) and the Bellman equation (37) are equivalent: their positive definite solutions coincide. So the continuous-time value can be characterized by an equation that never mentions $f(\cdot)$. Define the integral reinforcement signal and the continuous-time TD error:

The measured data at each increment is the triple $(x(t), x(t+T), \xi(t\!:\!t+T))$ — and $\xi$ can be computed in real time by wiring an integrator on the stage cost, adding it as an extra continuous-time state of the controller that functions as the controller's memory. This is why the method is called integral reinforcement learning (IRL).

Policy iteration and value iteration follow immediately — evaluate $V_{j+1}$ from the integral equation with the old (PI) or same (VI) value, then improve. The improvement step minimizes the Hamiltonian (38) over $u$, which for the quadratic-in-$u$ cost gives

Look at (41): the drift dynamics $f(x)$ have vanished. IRL needs only the input coupling $g(x)$, and unlike the discrete-time actor–critic, no actor network is needed at all — the policy update is explicit in $\nabla V$. With a critic approximator $V = W_c^{\mathsf T} z(x)$, both evaluation equations become linear regressions in the data $(x(t), x(t+T), \xi)$:

solved online by RLS or batch least squares (persistency of excitation required, as always). The reinforcement learning interval $T$ need not be constant — it can vary depending on how long it takes to gather meaningful information.

11.4 The continuous-time LQR connection

For $\dot x = Ax + Bu$ with $\mathcal Q \ge 0$, $R \gt 0$ and linear policies $u = -Kx$, the Bellman equation (37) becomes the continuous-time Lyapunov equation

and the HJB equation becomes the continuous-time algebraic Riccati equation

IRL solves both online, from trajectory data, without knowing $A$. Moreover, IRL policy iteration in this case is exactly Kleinman's algorithm — the classical iterative method for the Riccati equation, which is a Newton method and therefore converges with quadratic (second-order) rate. That classical fact resurfaces in the experiments: IRL policy iteration converges dramatically faster than IRL value iteration.

11.5 What the experiments show (Examples 2–4 of the paper)

Example 2 — a DC motor. A two-state model with $Q = I$, $R = 1$, driven by the IRL value iteration algorithm with critic basis $[\,x_1^2 \;\; x_1 x_2 \;\; x_2^2\,]$ and $T = 0.04$ s. Batch least squares on three intervals of data (0.12 s) suffices for each value update, then a greedy policy update. The states are regulated to zero, and the paper's Figure 7 shows the hybrid signature: feedback gains are discontinuous, piecewise-constant in time, while the control signal itself is continuous between gain updates.

Example 3 — the power system again, now continuous-time, with a plant change. Same load frequency control model as Example 1, $T = 0.05$ s (unrelated to any sample period; it need not even be constant), 15 data points per batch update every 0.75 s. Both IRL policy iteration and value iteration converge to the continuous-time ARE solution, $P^{\mathrm{ARE}} = [\![0.4750\;\, 0.4766\;\, 0.0601\;\, 0.4751;\;\, 0.4766\;\, 0.7831\;\, 0.1237\;\, 0.3829;\;\, 0.0601\;\, 0.1237\;\, 0.0513\;\, 0.0298;\;\, 0.4751\;\, 0.3829\;\, 0.0298\;\, 2.3370]\!]$. Then the experiment changes the plant: at $t = 30$ s the $A$ matrix changes (parameters $K_P/T_P$ from 8 to 10 and $1/T_T$ from 3.663 to 5.663 — the sort of drift that operating-point changes cause). The critic parameters reconverge after 30 s to the new Riccati solution — without knowing the old or the new $A$. This is adaptivity and optimality in one experiment, and it is the paper's most convincing figure. Policy iteration again converges far faster than value iteration — the Newton (quadratic) rate at work — and needs far less computation than the discrete-time scheme of Example 1.

Example 4 — a nonlinear system with intractable HJB. Built by the converse HJB approach (construct a system from a known optimal cost): $\dot x_1 = -x_1 + x_2 + 2x_2^3$, $\dot x_2 = -\tfrac12 (x_1 + x_2) + \tfrac12 x_2 (1 + 2x_2^2)\sin^2 x_1 + \sin(x_1)\, u$, with $\mathcal Q(x) = x_1^2 + x_2^2 + 2x_2^4$, $R = 1$. One can verify by direct substitution that the HJB equation is satisfied by

Analytic solution of the HJB for this system would be intractable — here it is known only because the example was constructed backwards. IRL policy iteration with an 8-term critic basis, data along five trajectories from random initial conditions in $[-1,1]^2$ (exploration), $T = 0.1$ s, 40 points per batch: after three iteration steps (12 s) the critic weights converge to the coefficients of $V^{*}$, and the controller becomes close to $u^{*}$. The HJB equation was solved online, from data, without the drift dynamics $f(x)$.

12. Synchronous tuning, games, and H∞ control

12.1 Tuning both networks at once

Structure: a critic network $V(x) = W_1^{\mathsf T} z(x)$ and an actor built directly from the stationarity condition of the Hamiltonian (38), $u(x) = -\tfrac12 R^{-1} g^{\mathsf T}(x)\, \nabla z^{\mathsf T}(x)\, W_2$. Both tune simultaneously and continuously in time. The critic's tuning law is a normalized gradient descent on the square of the Hamiltonian — with $v = \nabla z^{\mathsf T}(x)\big(f(x) + g(x)u\big)$ the rate of change of the basis vector along the trajectory, and tuning gain $a_1 \gt 0$:

Read the bracket: it is exactly the Hamiltonian $H = r + (\nabla V)^{\mathsf T}(f + gu)$ — the continuous-time TD error — evaluated with current weights; the algorithm drives $H \to 0$, i.e. solves the Bellman equation. The normalization $v/(v^{\mathsf T}v+1)^2$ is the standard adaptive-control device that keeps the update bounded while preserving the descent direction. The actor weights are tuned simultaneously by a backpropagation-type law through the critic (equation (108) of the paper), with tuning parameters $a_2$, $F_1$, $F_2$. Unlike IRL, this scheme requires full knowledge of $f(x)$ and $g(x)$ — $v$ contains the drift — but in exchange it carries a stability guarantee throughout learning as well as convergence to the optimal value and control, under a persistency-of-excitation condition. Its real payoff is for nonlinear plants: it converges to approximate local solutions of the HJB equation online, without ever solving that PDE — which is, except for special cases, analytically intractable.

Example 5 (from the paper). On a two-state affine nonlinear system constructed by the converse HJB method, with optimal value $V^{*}(x) = 0.25\,x_1^4 + 0.5\,x_2^2$ and critic basis $[\,x_1^2\;\;x_2^2\;\;x_1^4\;\;x_2^4\,]$, the synchronous algorithm ran with exponentially decaying probing noise (exploration that fades as the weights converge). The states stayed bounded and approached zero; after roughly 80 s the critic weights converged to the coefficients of $V^{*}$, and the 3-D error plots of the paper show both the value-function error and the control error small over the whole state-space region — the HJB equation was solved approximately online, with stability throughout.

12.2 Multi-player games and H∞ control

The same machinery extends beyond single-agent optimal control, and the extension is not decorative: it reaches the most robust design in control theory. In a two-player zero-sum game one player minimizes the cost and the other maximizes it; the solution concept is the saddle point of the Hamilton–Jacobi–Isaacs equation. Vrabie and Lewis showed that adaptive critic structures converge online to that solution — and the zero-sum game is exactly the formulation of H∞ control: the disturbance is the maximizing player, and the learned policy is the worst-case-disturbance-optimal controller. The adaptive controller for the zero-sum game thus finds the H∞ controller online and in real time; a Q-learning version requires no dynamics information at all. Multi-player nonzero-sum games — coupled HJ equations, one actor–critic loop per player — follow the same pattern. Reinforcement learning also applies with output feedback (partially observable MDPs), though that direction is far less developed; one ADP design using only measured output data is cited in the paper.

13. The big map

One diagram to file the whole article away. Rows are the algorithms; columns are the questions you would ask before choosing one. The pattern to notice as you scan the table: each generation of methods moved one item from the “must know” column into the “learned from data” column.

And the through-line, in one paragraph. Optimal control gives the design equations (Bellman, HJB, Riccati) but demands the plant and a backward-in-time solution. Adaptive control gives online learning but no optimality. Reinforcement learning joins them by reorganizing the design equations around the Bellman equation as a fixed-point condition on data: the critic evaluates the current policy against that condition (its residual is the temporal difference error = the Hamiltonian), the actor improves greedily against the critic, least squares makes both implementable online, function approximation makes them scalable, and the Q function and integral reinforcement successively expel the plant model from the last places it hid. The result is a class of optimal adaptive controllers: multiloop, multi-timescale structures that learn the solutions of Hamilton–Jacobi design equations — including the Riccati equation — in real time, along the plant's own trajectories, without knowing the plant's full dynamical model.

Where to read further

• F. L. Lewis, D. Vrabie, V. Syrmos, Optimal Control, 3rd ed., Wiley, 2012 — the HJB/Riccati background (Chapter 11 situates these controllers among optimal control systems).
• R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, 1998 — the MDP/TD foundation.
• D. P. Bertsekas, J. N. Tsitsiklis, Neuro-Dynamic Programming, Athena Scientific, 1996 — convergence theory, rollout algorithms.
• C. Watkins, P. Dayan, “Q-learning,” Machine Learning, 8:279–292, 1992 — the original Q learning convergence proof.
• D. Vrabie, O. Pastravanu, M. Abu-Khalaf, F. L. Lewis, “Adaptive optimal control for continuous-time linear systems based on policy iteration,” Automatica 45(2):477–484, 2009 — integral reinforcement learning.
• K. G. Vamvoudakis, F. L. Lewis, “Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem,” Automatica 46(5):878–888, 2010 — synchronous tuning.
• G. A. Hewer, IEEE T-AC 16(4):382–384, 1971; D. L. Kleinman, IEEE T-AC 13(1):114–115, 1968 — the classical iterative Riccati algorithms behind PI.