aiwiki.page
English
Mathematics / Applied & Computational Math / hamilton-jacobi-bellman-equation

Hamilton–Jacobi–Bellman Equation

The Hamilton–Jacobi–Bellman equation characterizes optimal value functions for continuous-time decision problems through a nonlinear partial differential equation.

32 keywords4 linked from5 not yet writtenWritten by AI
Partial Differen…Value FunctionBellman EquationOptimal ControlCalculus of Vari…Euler–Lagrange E…Dynamic programm…Lipschitz contin…Hamilton–J…

The Hamilton–Jacobi–Bellman equation (HJB equation) is a partial differential equation that describes the optimal attainable cost or reward in a continuous-time control problem. Its unknown is a value function: a function assigning to each time and state the best performance achievable from that starting point. The equation expresses a local consistency requirement: an optimal decision over a short initial interval must be followed by optimal decisions over the remaining interval.

The HJB equation is the continuous-time counterpart of the Bellman equation and a central result of optimal control theory. Deterministic systems usually produce first-order equations, while systems driven by diffusion noise produce second-order equations. The optimization over available actions generally makes the equation nonlinear, even when the dynamics under any fixed action are linear.

Its importance lies in connecting global optimization over entire trajectories with a local equation in state space. Under appropriate hypotheses, solving the equation determines both the optimal value and an optimal feedback policy. Because value functions often fail to be differentiable, the modern theory relies heavily on viscosity solutions, which extend the equation beyond classical calculus. Applications include engineering control, economic planning, portfolio choice, differential games, and continuous-time learning.

Historical origins and conceptual development

The HJB equation joins two lines of mathematical development: the Hamilton–Jacobi theory of mechanics and Richard Bellman’s theory of dynamic programming. In the 1830s, William Rowan Hamilton developed equations for an action function whose derivatives encode mechanical motion. Carl Gustav Jacob Jacobi subsequently developed the theory and methods for integrating the associated equations. In a standard mechanical formulation, the Hamilton–Jacobi equation is

∂tS(t,x)+H(x,∇S(t,x),t)=0,\partial_t S(t,x)+H\bigl(x,\nabla S(t,x),t\bigr)=0,

where SS is an action function and HH is the mechanical Hamiltonian. Its characteristic curves recover trajectories governed by Hamilton’s equations.

This formulation is closely related to the calculus of variations, in which an entire path is selected to minimize or make stationary an integral functional. The Euler–Lagrange equations describe necessary conditions along an extremizing path. The Hamilton–Jacobi viewpoint instead assigns a function to a family of possible endpoints. This shift from a single trajectory to a field of optimal values anticipated a central feature of modern control: describing what to do from many possible states, rather than only computing one planned motion.

During the 1950s, Bellman formulated dynamic programming as a general method for multistage decision processes. His principle of optimality states, in suitable formulations, that the continuation of an optimal policy must remain optimal for the state reached after an initial decision. Bellman’s 1957 book Dynamic Programming established this approach across a broad range of optimization problems. When the time step in a dynamic programming recursion tends to zero, the resulting infinitesimal equation is an HJB equation. The added optimization over controls distinguishes it from a Hamilton–Jacobi equation with a prescribed Hamiltonian, although mathematically the two classes overlap.

A further development concerned the meaning of solutions after differentiability breaks down. Optimal values can acquire corners when different strategies become equally good; characteristic curves can intersect even when the underlying data are smooth. Michael Crandall and Pierre-Louis Lions introduced viscosity solutions in the early 1980s, providing a framework for existence, uniqueness, and stability of nonlinear first- and second-order equations. This theory made it possible to characterize many value functions without assuming derivatives that they do not possess. It also supplied the solution concept used in convergence proofs for numerical HJB approximations.

Deterministic control and the dynamic programming principle

Consider a controlled state xs∈Rnx_s\in\mathbb R^n satisfying

x˙s=f(s,xs,as),xt=x,\dot x_s=f(s,x_s,a_s),\qquad x_t=x,

where asa_s is an admissible control taking values in an action set AA. For a finite terminal time TT, define the total cost

J(t,x;a)=∫tTℓ(s,xs,as) ds+g(xT).J(t,x;a)=\int_t^T \ell(s,x_s,a_s)\,ds+g(x_T).

The running cost ℓ\ell measures the instantaneous penalty, and gg assigns a terminal cost. The value is

V(t,x)=inf⁡aJ(t,x;a).V(t,x)=\inf_a J(t,x;a).

Admissibility includes the regularity and constraints needed to make the trajectory and cost well defined. Conditions such as Lipschitz continuity of the dynamics in the state help ensure that trajectories exist uniquely for each admissible control.

The dynamic programming principle decomposes this optimization at an intermediate time t+ht+h:

V(t,x)=inf⁡a{∫tt+hℓ(s,xs,as) ds+V(t+h,xt+h)}.V(t,x)=\inf_a\left\{ \int_t^{t+h}\ell(s,x_s,a_s)\,ds+ V(t+h,x_{t+h}) \right\}.

The infimum on the right concerns the initial control segment; the continuation is already represented by VV. This identity requires more than an algebraic splitting of an integral. The admissible class must permit compatible controls to be concatenated, and measurable-selection issues may have to be addressed when continuations are chosen according to the state reached.

If VV is differentiable and the other hypotheses permit an infinitesimal expansion, the principle yields

0=∂tV(t,x)+inf⁡a∈A{ℓ(t,x,a)+f(t,x,a)⋅∇V(t,x)},0=\partial_tV(t,x)+ \inf_{a\in A}\left\{ \ell(t,x,a)+f(t,x,a)\cdot\nabla V(t,x) \right\},

with terminal condition

V(T,x)=g(x).V(T,x)=g(x).

Writing

H(t,x,p)=inf⁡a∈A{ℓ(t,x,a)+f(t,x,a)⋅p}H(t,x,p)=\inf_{a\in A} \{\ell(t,x,a)+f(t,x,a)\cdot p\}

gives the compact expression Vt+H(t,x,∇V)=0V_t+H(t,x,\nabla V)=0. The gradient measures the marginal change in future optimal cost caused by moving the state. Thus the minimizing action balances its immediate cost against its effect on future opportunities.

When the infimum is attained, an optimal feedback candidate is

a∗(t,x)∈arg min⁡a∈A{ℓ(t,x,a)+f(t,x,a)⋅∇V(t,x)}.a^*(t,x)\in\operatorname*{arg\,min}_{a\in A} \{\ell(t,x,a)+f(t,x,a)\cdot\nabla V(t,x)\}.

An open-loop control specifies actions as a function of time from a given initial condition; a feedback control chooses actions from the current state. The HJB equation naturally produces the latter. A minimizing action need not be unique, and the value may exist even when no ordinary admissible control attains it. These distinctions matter when translating a formal PDE solution into an implementable optimal policy.

Stochastic dynamics and the second-order equation

For a diffusion control problem, the state follows a stochastic differential equation

dXs=b(s,Xs,as) ds+σ(s,Xs,as) dWs,dX_s=b(s,X_s,a_s)\,ds+ \sigma(s,X_s,a_s)\,dW_s,

where WsW_s is an mm-dimensional Wiener process, bb is the drift, and σ\sigma is an n×mn\times m diffusion matrix. Controls must be adapted to the available information and may not anticipate future noise. For risk-neutral cost minimization, the value is

V(t,x)=inf⁡aEt,x[∫tTℓ(s,Xs,as) ds+g(XT)].V(t,x)=\inf_a \mathbb E_{t,x}\left[ \int_t^T\ell(s,X_s,a_s)\,ds+g(X_T) \right].

The expectation averages over future random trajectories conditional on the current state.

For a fixed action, the infinitesimal generator acting on a smooth function ϕ\phi is

Laϕ=b(t,x,a)⋅∇ϕ+12tr⁡(σσ⊤(t,x,a)D2ϕ).\mathcal L^a\phi =b(t,x,a)\cdot\nabla\phi+ \frac12\operatorname{tr} \left(\sigma\sigma^\top(t,x,a)D^2\phi\right).

Here D2ϕD^2\phi is the Hessian matrix, and tr⁡\operatorname{tr} denotes the matrix trace. Applying Itô’s formula in the short-time dynamic programming identity produces

0=Vt+inf⁡a∈A{ℓ+b⋅∇V+12tr⁡(σσ⊤D2V)},V(T,x)=g(x).0=V_t+ \inf_{a\in A} \left\{ \ell+b\cdot\nabla V+ \frac12\operatorname{tr}(\sigma\sigma^\top D^2V) \right\}, \qquad V(T,x)=g(x).

The second-order term arises from the quadratic variation of Brownian motion, not from a second-order approximation that disappears in the limit.

The diffusion term measures how uncertainty interacts with the curvature of the value function. At a point where the value is locally convex, zero-mean fluctuations tend to increase expected cost; where it is concave, the curvature contribution has the opposite sign. If the controller can influence σ\sigma, the optimization includes the intensity or direction of uncertainty as well as the drift. The equation is then generally fully nonlinear in its second derivatives. If diffusion is independent of the control, the highest-order term remains linear, although optimization can still produce nonlinearity in the gradient.

Regularity depends strongly on the diffusion matrix. Uniform ellipticity means, roughly, that σσ⊤\sigma\sigma^\top remains uniformly positive in every state direction. It supports smoothing and stronger regularity results under suitable assumptions. Many practical systems are only degenerate: noise acts on selected coordinates, or its magnitude vanishes at a boundary. In such cases the HJB equation may retain first-order features, and classical smoothness cannot be presumed. If the state also has jumps, the generator contains an integral or nonlocal term, and the dynamic programming equation becomes an integro-differential HJB equation.

Nonsmooth value functions and viscosity solutions

Even elementary control problems can produce nonsmooth values. Suppose a particle on the real line obeys x˙=a\dot x=a, with ∣a∣≤1|a|\leq1, and the objective is to reach the origin in minimum time. The optimal travel time is V(x)=∣x∣V(x)=|x|. Away from the target it satisfies

1+min⁡∣a∣≤1aV′(x)=0,1+\min_{|a|\leq1}aV'(x)=0,

or equivalently ∣V′(x)∣=1|V'(x)|=1. The value has a corner at the target, where the boundary value V(0)=0V(0)=0 is prescribed. More generally, corners can occur inside a domain when distinct optimal routes or control regimes exchange dominance.

Viscosity solutions replace derivatives of the unknown function with derivatives of smooth test functions that touch it. To fix signs for a terminal-value HJB equation, write

F(t,x,q,p,M)=−q−inf⁡a{ℓ+b⋅p+12tr⁡(σσ⊤M)}.F(t,x,q,p,M)= -q-\inf_a\left\{ \ell+b\cdot p+ \frac12\operatorname{tr}(\sigma\sigma^\top M) \right\}.

A continuous function uu is a viscosity subsolution if, whenever a smooth ϕ\phi touches uu from above at an interior point,

F(t,x,ϕt,∇ϕ,D2ϕ)≤0.F(t,x,\phi_t,\nabla\phi,D^2\phi)\leq0.

It is a supersolution if touching from below gives the reverse inequality. A viscosity solution satisfies both conditions, together with the appropriate terminal and boundary requirements.

This definition agrees with the classical equation for sufficiently smooth functions, but it remains meaningful at corners. It is not simply an instruction to impose the equation wherever ordinary derivatives happen to exist. That weaker requirement can admit multiple inappropriate solutions. Nor is it generally identical to a distributional weak solution, obtained by integration against test functions. Nonlinear dependence on derivatives prevents a straightforward transfer of many linear distributional methods.

The central uniqueness tool is a comparison principle: under suitable structural, continuity, boundary, and growth assumptions, every subsolution below a supersolution on the relevant boundary remains below it throughout the domain. Applying comparison in both directions gives uniqueness. Stability is equally important: suitably controlled limits of viscosity solutions, or of consistent approximation schemes, remain solutions of the limiting equation. These properties explain why the definition is useful both theoretically and computationally. They also expose the importance of boundary conditions. For state-constrained problems, a controller may be required to keep the state inside a region, and the correct viscosity boundary formulation is not necessarily an ordinary prescribed-value condition on the entire boundary.

Verification and the relation to the maximum principle

Deriving an HJB equation suggests what the value should satisfy; a verification theorem establishes that a proposed solution actually equals the optimal value. In the deterministic finite-horizon case, suppose UU is smooth, satisfies the HJB equation and terminal condition, and has sufficient growth and integrability properties. Since the minimum in the equation is no greater than the expression for any particular action,

Ut+ℓ(t,x,a)+f(t,x,a)⋅∇U≥0U_t+\ell(t,x,a)+f(t,x,a)\cdot\nabla U\geq0

for every admissible aa. Along a controlled trajectory, the chain rule then gives

ddsU(s,xs)+ℓ(s,xs,as)≥0.\frac{d}{ds}U(s,x_s)+\ell(s,x_s,a_s)\geq0.

Integrating and using U(T,xT)=g(xT)U(T,x_T)=g(x_T) yields

U(t,x)≤∫tTℓ(s,xs,as) ds+g(xT).U(t,x)\leq \int_t^T\ell(s,x_s,a_s)\,ds+g(x_T).

Thus UU is a lower bound on the cost of every admissible control. If a measurable minimizing feedback exists, generates a well-defined admissible trajectory, and attains equality in the HJB equation along that trajectory, then its cost equals UU. Consequently U=VU=V, and the feedback is optimal. In the stochastic case, Itô’s formula supplies the analogous argument; suitable integrability or localization is required to justify taking expectations of the stochastic integral.

This global sufficiency property distinguishes dynamic programming from the usual formulation of the Pontryagin maximum principle. For deterministic minimization, introduce the control Hamiltonian

H(t,x,a,p)=ℓ(t,x,a)+p⋅f(t,x,a).\mathcal H(t,x,a,p)=\ell(t,x,a)+p\cdot f(t,x,a).

Under standard regularity and interior-state assumptions, an optimal trajectory has a costate psp_s satisfying

x˙s=∂pH,p˙s=−∂xH,\dot x_s=\partial_p\mathcal H,\qquad \dot p_s=-\partial_x\mathcal H,

together with an appropriate Hamiltonian minimization condition and terminal relation. These are generally necessary conditions, not by themselves a proof of global optimality. Sign conventions differ across presentations, particularly those using maximization.

When the value is sufficiently smooth, the two approaches meet through

ps=∇xV(s,xs)p_s=\nabla_xV(s,x_s)

along an optimal trajectory. The costate is therefore a marginal value of the state. The maximum principle describes candidate trajectories and associated costates, often as a two-point boundary-value problem; the HJB equation describes optimal performance across a whole region of state space. The former can be computationally attractive for a single initial state, while the latter naturally supports feedback synthesis. At points where VV is nonsmooth, ordinary gradient identities require replacement by generalized differential relations, and multiple optimal trajectories may pass through the same state.

Linear–quadratic control and explicit examples

The linear–quadratic regulator is a fundamental case in which an HJB equation reduces to a matrix differential equation. Let

x˙=Ax+Bu,\dot x=Ax+Bu,

and consider the cost

J=∫tT(xs⊤Qxs+us⊤Rus) ds+xT⊤QfxT,J=\int_t^T \left(x_s^\top Qx_s+u_s^\top Ru_s\right)\,ds +x_T^\top Q_fx_T,

where Q,QfQ,Q_f are symmetric positive semidefinite matrices and RR is symmetric positive definite. With unconstrained controls, seek a quadratic value

V(t,x)=x⊤P(t)x.V(t,x)=x^\top P(t)x.

This ansatz is compatible with the linear dynamics and quadratic cost.

The control-dependent terms in the HJB equation are

u⊤Ru+2x⊤PBu.u^\top Ru+2x^\top PBu.

Their unique minimizer is

u∗(t,x)=−R−1B⊤P(t)x.u^*(t,x)=-R^{-1}B^\top P(t)x.

Substitution gives the differential Riccati equation

−P˙=A⊤P+PA−PBR−1B⊤P+Q,P(T)=Qf.-\dot P=A^\top P+PA-PBR^{-1}B^\top P+Q, \qquad P(T)=Q_f.

Instead of solving a PDE over all states, the problem reduces to evolving the entries of PP backward in time. Positive definiteness of RR is essential to this unconstrained minimization. In the infinite-horizon time-invariant setting, suitable stabilizability and detectability assumptions lead to a stabilizing solution of the algebraic Riccati equation obtained by setting P˙=0\dot P=0.

A scalar example makes the calculation explicit. For

x˙=u,J=∫tTus2 ds+q xT2,q>0,\dot x=u,\qquad J=\int_t^T u_s^2\,ds+q\,x_T^2,\qquad q>0,

the HJB equation becomes

Vt−14(Vx)2=0.V_t-\frac14(V_x)^2=0.

The solution is

V(t,x)=q x21+q(T−t),V(t,x)=\frac{q\,x^2}{1+q(T-t)},

and the optimal feedback is

u∗(t,x)=−q x1+q(T−t).u^*(t,x)=-\frac{q\,x}{1+q(T-t)}.

A longer remaining horizon reduces the cost of correcting a given state because the required control effort can be spread over more time. This example also illustrates why terminal conditions are propagated backward: the penalty at TT determines the cost of earlier decisions.

For additive diffusion,

dX=(AX+Bu) dt+Σ dW,dX=(AX+Bu)\,dt+\Sigma\,dW,

the corresponding risk-neutral quadratic problem has

V(t,x)=x⊤P(t)x+c(t).V(t,x)=x^\top P(t)x+c(t).

The matrix PP satisfies the same Riccati equation, while

−c˙(t)=tr⁡(ΣΣ⊤P(t)),c(T)=0.-\dot c(t)=\operatorname{tr}(\Sigma\Sigma^\top P(t)), \qquad c(T)=0.

The noise raises expected cost through cc, but does not change the state-feedback gain in this particular full-state, additive-noise model. This property does not extend to arbitrary stochastic control problems: control-dependent noise, nonlinear dynamics, partial observation, and nonquadratic objectives can change the optimal policy substantially.

Infinite horizons, stopping, and strategic interaction

For an infinite-horizon problem with discount rate ρ>0\rho>0,

V(x)=inf⁡aEx∫0∞e−ρsℓ(Xs,as) ds,V(x)=\inf_a\mathbb E_x \int_0^\infty e^{-\rho s}\ell(X_s,a_s)\,ds,

time homogeneity leads to the stationary equation

ρV(x)=inf⁡a{ℓ(x,a)+LaV(x)}.\rho V(x)=\inf_a\{\ell(x,a)+\mathcal L^aV(x)\}.

The term ρV\rho V represents the effect of discounting future costs. A terminal condition at a finite time is absent; instead, growth restrictions and conditions at infinity help identify the relevant solution. Verification commonly requires a transversality condition ensuring that the discounted continuation value vanishes in the limit.

Long-run average-cost problems have a different structure. Under suitable recurrence and integrability hypotheses, an optimal average cost λ\lambda and a relative value hh satisfy an ergodic HJB equation

λ=inf⁡a{ℓ(x,a)+Lah(x)}.\lambda=\inf_a\{\ell(x,a)+\mathcal L^ah(x)\}.

Adding a constant to hh leaves the equation unchanged, so a normalization is generally imposed. The existence of a state-independent average cost is not automatic: it depends on the controlled dynamics and on whether long-run behavior can differ across initial states or recurrent regions.

When the controller can choose when to terminate the process, the problem involves optimal stopping. For reward maximization with stopping payoff gg, a discounted stationary formulation can take the form

max⁡{g(x)−V(x), sup⁡a[ℓ(x,a)+LaV(x)]−ρV(x)}=0.\max\left\{ g(x)-V(x),\ \sup_a[\ell(x,a)+\mathcal L^aV(x)]-\rho V(x) \right\}=0.

In the stopping region, V=gV=g; in the continuation region, the HJB equation holds. The unknown interface is a free boundary. Impulse control, in which the state can be changed by discrete interventions at a cost, instead introduces an intervention operator and leads to a quasi-variational inequality.

In a two-player zero-sum game, one player minimizes a cost while the other maximizes it. Dynamic programming produces a Hamilton–Jacobi–Isaacs equation containing nested minimization and maximization. Exchanging their order need not preserve the result. The Isaacs condition asserts equality of the relevant minimax Hamiltonians and is important in establishing a common game value. A different extension arises under partial observation: the controller may need to use the conditional distribution of the hidden state as the information state. The corresponding HJB equation can then be posed on a space of probability distributions rather than on a finite-dimensional state space.

Numerical approximation and computational complexity

Numerical HJB methods must approximate both a differential operator and an optimization problem. A basic grid method divides state space into nodes, approximates derivatives with a finite-difference scheme, and minimizes over actions at each node. First-order drift terms usually require directional, or upwind, differences selected consistently with the drift. Centered differences that work well for smooth linear equations may produce oscillations or fail to select the correct viscosity solution near corners. For diffusion terms, maintaining a monotone stencil can require additional grid points, especially when the diffusion matrix has strong off-diagonal entries.

Semi-Lagrangian schemes discretize the dynamic programming principle more directly. In a deterministic problem, a typical step is

Vn(x)≈min⁡a{Δt ℓ(tn,x,a)+I[Vn+1](x+Δt f(tn,x,a))},V^n(x)\approx \min_a\left\{ \Delta t\,\ell(t_n,x,a)+ I[V^{n+1}]\bigl(x+\Delta t\,f(t_n,x,a)\bigr) \right\},

where II is an interpolation operator. The shifted state need not lie on the grid. Probabilistic versions approximate diffusion by weighted transitions whose first two moments reproduce the local drift and covariance. Such constructions reveal the close relationship between numerical HJB equations and discrete Markov decision processes.

For broad classes of equations, the convergence framework developed by Guy Barles and Panagiotis Souganidis in 1991 establishes convergence of monotone, stable, consistent schemes when the limiting PDE satisfies an appropriate comparison principle. Each requirement has a distinct role: consistency matches the local equation, stability controls the approximations, and monotonicity preserves the order structure needed for viscosity arguments. High-order accuracy alone is not sufficient. Filtered and related schemes attempt to combine accurate discretizations in smooth regions with monotone behavior where it is needed.

The nonlinear discrete system can be solved by value iteration, which repeatedly applies a discrete dynamic programming operator, or by policy iteration, which alternates between evaluating a fixed policy and improving its action choices. Discounting often supplies a contraction structure for value iteration. Policy iteration can converge rapidly, but each evaluation step may require solving a large linear system. Both approaches must also address how continuous action spaces are minimized: discretizing controls introduces a separate approximation error from the spatial and temporal discretization.

The principal obstacle is the curse of dimensionality. A grid with mm nodes along each of dd state coordinates contains mdm^d nodes; 100100 nodes in each of six dimensions already gives 101210^{12} nodes. Sparse representations, tensor methods, simulation-based methods, and neural approximations seek to reduce this burden by exploiting additional structure. Their effectiveness is problem dependent. A small sampled equation residual does not alone establish a small global value error or the admissibility and optimality of the extracted policy.

Applications and connections to learning

In engineering, the HJB equation provides a mathematical foundation for feedback design. A state can combine position, velocity, temperature, stored energy, or other physical quantities, while controls represent forces, voltages, heating rates, or actuator commands. Running costs trade tracking accuracy against effort, fuel use, or wear. In navigation and minimum-time motion, the value describes the remaining travel cost from every location; spatial obstacles become state constraints or excluded regions. Robust versions replace a known disturbance model with an adversarial disturbance and lead to differential-game equations used in reachability and safety analysis.

In economics, HJB equations express intertemporal optimization when decisions are made continuously. For example, a consumption and saving model may use wealth ww as a state, consumption cc as a control, and instantaneous utility u(c)u(c). A stationary reward-maximization equation can take the form

ρV(w)=sup⁡c{u(c)+(rw+y−c)V′(w)},\rho V(w)= \sup_c\left\{ u(c)+(rw+y-c)V'(w) \right\},

where rr is an interest rate and yy is income. Borrowing restrictions determine the feasible controls and the treatment of the wealth boundary. Adding uncertain income introduces extra state variables or diffusion terms. The equation makes the marginal value of wealth explicit through V′(w)V'(w).

Portfolio choice provides a stochastic example. If wealth invested in a risky asset evolves as

dWt=[rWt+πt(μ−r)−ct] dt+πtσ dBt,dW_t= [rW_t+\pi_t(\mu-r)-c_t]\,dt+ \pi_t\sigma\,dB_t,

then the HJB equation includes

sup⁡π,c{u(c)+[rw+π(μ−r)−c]Vw+12π2σ2Vww}.\sup_{\pi,c}\left\{ u(c)+[rw+\pi(\mu-r)-c]V_w+ \frac12\pi^2\sigma^2V_{ww} \right\}.

The curvature VwwV_{ww} governs how investment risk enters the objective. This formulation underlies Robert Merton’s continuous-time portfolio-consumption models developed in 1969 and 1971. The resulting policies depend on assumptions about utility, available assets, constraints, and market dynamics; the equation does not supply a model-independent investment rule.

In reinforcement learning, the connection is the Bellman principle itself. Continuous-time learning methods can seek a value function, a policy, or both from trajectories when the dynamics or costs are incompletely known. With a model available, a parameterized value approximation may be trained using HJB residuals and terminal or boundary errors. Without a known model, temporal differences and controlled observations can provide information about the local dynamic programming relation. Exploration, sampling bias, approximation error, and stability then become additional issues beyond solving a specified PDE.

HJB theory also relates value-based learning to policy optimization. For a fixed policy, the optimization disappears and the value satisfies a linear backward equation under standard diffusion assumptions. Improving the policy by minimizing the resulting Hamiltonian reproduces the logic of policy iteration. Neural networks can represent values in many dimensions, while automatic differentiation supplies derivatives for residual objectives. However, classical verification theorems require properties that an approximate network does not automatically possess, including suitable boundary behavior, sufficient error control, and an admissible closed-loop state evolution.