{
  "id": "hamilton-jacobi-bellman-equation",
  "title": "Hamilton–Jacobi–Bellman Equation",
  "lang": "en",
  "source_lang": "en",
  "summary": "The Hamilton–Jacobi–Bellman equation characterizes optimal value functions for continuous-time decision problems through a nonlinear partial differential equation.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "HJB equation",
    "Bellman partial differential equation",
    "dynamic programming equation"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/hamilton-jacobi-bellman-equation/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/hamilton-jacobi-bellman-equation/",
      "json": "https://aiwiki.page/zh/hamilton-jacobi-bellman-equation.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Hamilton–Jacobi–Bellman equation (HJB equation) is a partial differential equation that describes the optimal attainable cost or reward in a continuous-time control problem. Its unknown is a value function: a function assigning to each time and state the best performance achievable from that starting point. The equation expresses a local consistency requirement: an optimal decision over a short initial interval must be followed by optimal decisions over the remaining interval.\n\nThe HJB equation is the continuous-time counterpart of the Bellman equation and a central result of optimal control theory. Deterministic systems usually produce first-order equations, while systems driven by diffusion noise produce second-order equations. The optimization over available actions generally makes the equation nonlinear, even when the dynamics under any fixed action are linear.\n\nIts importance lies in connecting global optimization over entire trajectories with a local equation in state space. Under appropriate hypotheses, solving the equation determines both the optimal value and an optimal feedback policy. Because value functions often fail to be differentiable, the modern theory relies heavily on viscosity solutions, which extend the equation beyond classical calculus. Applications include engineering control, economic planning, portfolio choice, differential games, and continuous-time learning."
    },
    {
      "heading": "Historical origins and conceptual development",
      "text": "The HJB equation joins two lines of mathematical development: the Hamilton–Jacobi theory of mechanics and Richard Bellman’s theory of dynamic programming. In the 1830s, William Rowan Hamilton developed equations for an action function whose derivatives encode mechanical motion. Carl Gustav Jacob Jacobi subsequently developed the theory and methods for integrating the associated equations. In a standard mechanical formulation, the Hamilton–Jacobi equation is\n\n\\[\\partial_t S(t,x)+H\\bigl(x,\\nabla S(t,x),t\\bigr)=0,\\]\n\nwhere \\(S\\) is an action function and \\(H\\) is the mechanical Hamiltonian. Its characteristic curves recover trajectories governed by Hamilton’s equations.\n\nThis formulation is closely related to the calculus of variations, in which an entire path is selected to minimize or make stationary an integral functional. The Euler–Lagrange equations describe necessary conditions along an extremizing path. The Hamilton–Jacobi viewpoint instead assigns a function to a family of possible endpoints. This shift from a single trajectory to a field of optimal values anticipated a central feature of modern control: describing what to do from many possible states, rather than only computing one planned motion.\n\nDuring the 1950s, Bellman formulated dynamic programming as a general method for multistage decision processes. His principle of optimality states, in suitable formulations, that the continuation of an optimal policy must remain optimal for the state reached after an initial decision. Bellman’s 1957 book Dynamic Programming established this approach across a broad range of optimization problems. When the time step in a dynamic programming recursion tends to zero, the resulting infinitesimal equation is an HJB equation. The added optimization over controls distinguishes it from a Hamilton–Jacobi equation with a prescribed Hamiltonian, although mathematically the two classes overlap.\n\nA further development concerned the meaning of solutions after differentiability breaks down. Optimal values can acquire corners when different strategies become equally good; characteristic curves can intersect even when the underlying data are smooth. Michael Crandall and Pierre-Louis Lions introduced viscosity solutions in the early 1980s, providing a framework for existence, uniqueness, and stability of nonlinear first- and second-order equations. This theory made it possible to characterize many value functions without assuming derivatives that they do not possess. It also supplied the solution concept used in convergence proofs for numerical HJB approximations."
    },
    {
      "heading": "Deterministic control and the dynamic programming principle",
      "text": "Consider a controlled state \\(x_s\\in\\mathbb R^n\\) satisfying\n\n\\[\\dot x_s=f(s,x_s,a_s),\\qquad x_t=x,\\]\n\nwhere \\(a_s\\) is an admissible control taking values in an action set \\(A\\). For a finite terminal time \\(T\\), define the total cost\n\n\\[J(t,x;a)=\\int_t^T \\ell(s,x_s,a_s)\\,ds+g(x_T).\\]\n\nThe running cost \\(\\ell\\) measures the instantaneous penalty, and \\(g\\) assigns a terminal cost. The value is\n\n\\[V(t,x)=\\inf_a J(t,x;a).\\]\n\nAdmissibility includes the regularity and constraints needed to make the trajectory and cost well defined. Conditions such as Lipschitz continuity of the dynamics in the state help ensure that trajectories exist uniquely for each admissible control.\n\nThe dynamic programming principle decomposes this optimization at an intermediate time \\(t+h\\):\n\n\\[V(t,x)=\\inf_a\\left\\{ \\int_t^{t+h}\\ell(s,x_s,a_s)\\,ds+ V(t+h,x_{t+h}) \\right\\}.\\]\n\nThe infimum on the right concerns the initial control segment; the continuation is already represented by \\(V\\). This identity requires more than an algebraic splitting of an integral. The admissible class must permit compatible controls to be concatenated, and measurable-selection issues may have to be addressed when continuations are chosen according to the state reached.\n\nIf \\(V\\) is differentiable and the other hypotheses permit an infinitesimal expansion, the principle yields\n\n\\[0=\\partial_tV(t,x)+ \\inf_{a\\in A}\\left\\{ \\ell(t,x,a)+f(t,x,a)\\cdot\\nabla V(t,x) \\right\\},\\]\n\nwith terminal condition\n\n\\[V(T,x)=g(x).\\]\n\nWriting\n\n\\[H(t,x,p)=\\inf_{a\\in A} \\{\\ell(t,x,a)+f(t,x,a)\\cdot p\\}\\]\n\ngives the compact expression \\(V_t+H(t,x,\\nabla V)=0\\). The gradient measures the marginal change in future optimal cost caused by moving the state. Thus the minimizing action balances its immediate cost against its effect on future opportunities.\n\nWhen the infimum is attained, an optimal feedback candidate is\n\n\\[a^*(t,x)\\in\\operatorname*{arg\\,min}_{a\\in A} \\{\\ell(t,x,a)+f(t,x,a)\\cdot\\nabla V(t,x)\\}.\\]\n\nAn open-loop control specifies actions as a function of time from a given initial condition; a feedback control chooses actions from the current state. The HJB equation naturally produces the latter. A minimizing action need not be unique, and the value may exist even when no ordinary admissible control attains it. These distinctions matter when translating a formal PDE solution into an implementable optimal policy."
    },
    {
      "heading": "Stochastic dynamics and the second-order equation",
      "text": "For a diffusion control problem, the state follows a stochastic differential equation\n\n\\[dX_s=b(s,X_s,a_s)\\,ds+ \\sigma(s,X_s,a_s)\\,dW_s,\\]\n\nwhere \\(W_s\\) is an \\(m\\)-dimensional Wiener process, \\(b\\) is the drift, and \\(\\sigma\\) is an \\(n\\times m\\) diffusion matrix. Controls must be adapted to the available information and may not anticipate future noise. For risk-neutral cost minimization, the value is\n\n\\[V(t,x)=\\inf_a \\mathbb E_{t,x}\\left[ \\int_t^T\\ell(s,X_s,a_s)\\,ds+g(X_T) \\right].\\]\n\nThe expectation averages over future random trajectories conditional on the current state.\n\nFor a fixed action, the infinitesimal generator acting on a smooth function \\(\\phi\\) is\n\n\\[\\mathcal L^a\\phi =b(t,x,a)\\cdot\\nabla\\phi+ \\frac12\\operatorname{tr} \\left(\\sigma\\sigma^\\top(t,x,a)D^2\\phi\\right).\\]\n\nHere \\(D^2\\phi\\) is the Hessian matrix, and \\(\\operatorname{tr}\\) denotes the matrix trace. Applying Itô’s formula in the short-time dynamic programming identity produces\n\n\\[0=V_t+ \\inf_{a\\in A} \\left\\{ \\ell+b\\cdot\\nabla V+ \\frac12\\operatorname{tr}(\\sigma\\sigma^\\top D^2V) \\right\\}, \\qquad V(T,x)=g(x).\\]\n\nThe second-order term arises from the quadratic variation of Brownian motion, not from a second-order approximation that disappears in the limit.\n\nThe diffusion term measures how uncertainty interacts with the curvature of the value function. At a point where the value is locally convex, zero-mean fluctuations tend to increase expected cost; where it is concave, the curvature contribution has the opposite sign. If the controller can influence \\(\\sigma\\), the optimization includes the intensity or direction of uncertainty as well as the drift. The equation is then generally fully nonlinear in its second derivatives. If diffusion is independent of the control, the highest-order term remains linear, although optimization can still produce nonlinearity in the gradient.\n\nRegularity depends strongly on the diffusion matrix. Uniform ellipticity means, roughly, that \\(\\sigma\\sigma^\\top\\) remains uniformly positive in every state direction. It supports smoothing and stronger regularity results under suitable assumptions. Many practical systems are only degenerate: noise acts on selected coordinates, or its magnitude vanishes at a boundary. In such cases the HJB equation may retain first-order features, and classical smoothness cannot be presumed. If the state also has jumps, the generator contains an integral or nonlocal term, and the dynamic programming equation becomes an integro-differential HJB equation."
    },
    {
      "heading": "Nonsmooth value functions and viscosity solutions",
      "text": "Even elementary control problems can produce nonsmooth values. Suppose a particle on the real line obeys \\(\\dot x=a\\), with \\(|a|\\leq1\\), and the objective is to reach the origin in minimum time. The optimal travel time is \\(V(x)=|x|\\). Away from the target it satisfies\n\n\\[1+\\min_{|a|\\leq1}aV'(x)=0,\\]\n\nor equivalently \\(|V'(x)|=1\\). The value has a corner at the target, where the boundary value \\(V(0)=0\\) is prescribed. More generally, corners can occur inside a domain when distinct optimal routes or control regimes exchange dominance.\n\nViscosity solutions replace derivatives of the unknown function with derivatives of smooth test functions that touch it. To fix signs for a terminal-value HJB equation, write\n\n\\[F(t,x,q,p,M)= -q-\\inf_a\\left\\{ \\ell+b\\cdot p+ \\frac12\\operatorname{tr}(\\sigma\\sigma^\\top M) \\right\\}.\\]\n\nA continuous function \\(u\\) is a viscosity subsolution if, whenever a smooth \\(\\phi\\) touches \\(u\\) from above at an interior point,\n\n\\[F(t,x,\\phi_t,\\nabla\\phi,D^2\\phi)\\leq0.\\]\n\nIt is a supersolution if touching from below gives the reverse inequality. A viscosity solution satisfies both conditions, together with the appropriate terminal and boundary requirements.\n\nThis definition agrees with the classical equation for sufficiently smooth functions, but it remains meaningful at corners. It is not simply an instruction to impose the equation wherever ordinary derivatives happen to exist. That weaker requirement can admit multiple inappropriate solutions. Nor is it generally identical to a distributional weak solution, obtained by integration against test functions. Nonlinear dependence on derivatives prevents a straightforward transfer of many linear distributional methods.\n\nThe central uniqueness tool is a comparison principle: under suitable structural, continuity, boundary, and growth assumptions, every subsolution below a supersolution on the relevant boundary remains below it throughout the domain. Applying comparison in both directions gives uniqueness. Stability is equally important: suitably controlled limits of viscosity solutions, or of consistent approximation schemes, remain solutions of the limiting equation. These properties explain why the definition is useful both theoretically and computationally. They also expose the importance of boundary conditions. For state-constrained problems, a controller may be required to keep the state inside a region, and the correct viscosity boundary formulation is not necessarily an ordinary prescribed-value condition on the entire boundary."
    },
    {
      "heading": "Verification and the relation to the maximum principle",
      "text": "Deriving an HJB equation suggests what the value should satisfy; a verification theorem establishes that a proposed solution actually equals the optimal value. In the deterministic finite-horizon case, suppose \\(U\\) is smooth, satisfies the HJB equation and terminal condition, and has sufficient growth and integrability properties. Since the minimum in the equation is no greater than the expression for any particular action,\n\n\\[U_t+\\ell(t,x,a)+f(t,x,a)\\cdot\\nabla U\\geq0\\]\n\nfor every admissible \\(a\\). Along a controlled trajectory, the chain rule then gives\n\n\\[\\frac{d}{ds}U(s,x_s)+\\ell(s,x_s,a_s)\\geq0.\\]\n\nIntegrating and using \\(U(T,x_T)=g(x_T)\\) yields\n\n\\[U(t,x)\\leq \\int_t^T\\ell(s,x_s,a_s)\\,ds+g(x_T).\\]\n\nThus \\(U\\) is a lower bound on the cost of every admissible control. If a measurable minimizing feedback exists, generates a well-defined admissible trajectory, and attains equality in the HJB equation along that trajectory, then its cost equals \\(U\\). Consequently \\(U=V\\), and the feedback is optimal. In the stochastic case, Itô’s formula supplies the analogous argument; suitable integrability or localization is required to justify taking expectations of the stochastic integral.\n\nThis global sufficiency property distinguishes dynamic programming from the usual formulation of the Pontryagin maximum principle. For deterministic minimization, introduce the control Hamiltonian\n\n\\[\\mathcal H(t,x,a,p)=\\ell(t,x,a)+p\\cdot f(t,x,a).\\]\n\nUnder standard regularity and interior-state assumptions, an optimal trajectory has a costate \\(p_s\\) satisfying\n\n\\[\\dot x_s=\\partial_p\\mathcal H,\\qquad \\dot p_s=-\\partial_x\\mathcal H,\\]\n\ntogether with an appropriate Hamiltonian minimization condition and terminal relation. These are generally necessary conditions, not by themselves a proof of global optimality. Sign conventions differ across presentations, particularly those using maximization.\n\nWhen the value is sufficiently smooth, the two approaches meet through\n\n\\[p_s=\\nabla_xV(s,x_s)\\]\n\nalong an optimal trajectory. The costate is therefore a marginal value of the state. The maximum principle describes candidate trajectories and associated costates, often as a two-point boundary-value problem; the HJB equation describes optimal performance across a whole region of state space. The former can be computationally attractive for a single initial state, while the latter naturally supports feedback synthesis. At points where \\(V\\) is nonsmooth, ordinary gradient identities require replacement by generalized differential relations, and multiple optimal trajectories may pass through the same state."
    },
    {
      "heading": "Linear–quadratic control and explicit examples",
      "text": "The linear–quadratic regulator is a fundamental case in which an HJB equation reduces to a matrix differential equation. Let\n\n\\[\\dot x=Ax+Bu,\\]\n\nand consider the cost\n\n\\[J=\\int_t^T \\left(x_s^\\top Qx_s+u_s^\\top Ru_s\\right)\\,ds +x_T^\\top Q_fx_T,\\]\n\nwhere \\(Q,Q_f\\) are symmetric positive semidefinite matrices and \\(R\\) is symmetric positive definite. With unconstrained controls, seek a quadratic value\n\n\\[V(t,x)=x^\\top P(t)x.\\]\n\nThis ansatz is compatible with the linear dynamics and quadratic cost.\n\nThe control-dependent terms in the HJB equation are\n\n\\[u^\\top Ru+2x^\\top PBu.\\]\n\nTheir unique minimizer is\n\n\\[u^*(t,x)=-R^{-1}B^\\top P(t)x.\\]\n\nSubstitution gives the differential Riccati equation\n\n\\[-\\dot P=A^\\top P+PA-PBR^{-1}B^\\top P+Q, \\qquad P(T)=Q_f.\\]\n\nInstead of solving a PDE over all states, the problem reduces to evolving the entries of \\(P\\) backward in time. Positive definiteness of \\(R\\) is essential to this unconstrained minimization. In the infinite-horizon time-invariant setting, suitable stabilizability and detectability assumptions lead to a stabilizing solution of the algebraic Riccati equation obtained by setting \\(\\dot P=0\\).\n\nA scalar example makes the calculation explicit. For\n\n\\[\\dot x=u,\\qquad J=\\int_t^T u_s^2\\,ds+q\\,x_T^2,\\qquad q>0,\\]\n\nthe HJB equation becomes\n\n\\[V_t-\\frac14(V_x)^2=0.\\]\n\nThe solution is\n\n\\[V(t,x)=\\frac{q\\,x^2}{1+q(T-t)},\\]\n\nand the optimal feedback is\n\n\\[u^*(t,x)=-\\frac{q\\,x}{1+q(T-t)}.\\]\n\nA longer remaining horizon reduces the cost of correcting a given state because the required control effort can be spread over more time. This example also illustrates why terminal conditions are propagated backward: the penalty at \\(T\\) determines the cost of earlier decisions.\n\nFor additive diffusion,\n\n\\[dX=(AX+Bu)\\,dt+\\Sigma\\,dW,\\]\n\nthe corresponding risk-neutral quadratic problem has\n\n\\[V(t,x)=x^\\top P(t)x+c(t).\\]\n\nThe matrix \\(P\\) satisfies the same Riccati equation, while\n\n\\[-\\dot c(t)=\\operatorname{tr}(\\Sigma\\Sigma^\\top P(t)), \\qquad c(T)=0.\\]\n\nThe noise raises expected cost through \\(c\\), but does not change the state-feedback gain in this particular full-state, additive-noise model. This property does not extend to arbitrary stochastic control problems: control-dependent noise, nonlinear dynamics, partial observation, and nonquadratic objectives can change the optimal policy substantially."
    },
    {
      "heading": "Infinite horizons, stopping, and strategic interaction",
      "text": "For an infinite-horizon problem with discount rate \\(\\rho>0\\),\n\n\\[V(x)=\\inf_a\\mathbb E_x \\int_0^\\infty e^{-\\rho s}\\ell(X_s,a_s)\\,ds,\\]\n\ntime homogeneity leads to the stationary equation\n\n\\[\\rho V(x)=\\inf_a\\{\\ell(x,a)+\\mathcal L^aV(x)\\}.\\]\n\nThe term \\(\\rho V\\) represents the effect of discounting future costs. A terminal condition at a finite time is absent; instead, growth restrictions and conditions at infinity help identify the relevant solution. Verification commonly requires a transversality condition ensuring that the discounted continuation value vanishes in the limit.\n\nLong-run average-cost problems have a different structure. Under suitable recurrence and integrability hypotheses, an optimal average cost \\(\\lambda\\) and a relative value \\(h\\) satisfy an ergodic HJB equation\n\n\\[\\lambda=\\inf_a\\{\\ell(x,a)+\\mathcal L^ah(x)\\}.\\]\n\nAdding a constant to \\(h\\) leaves the equation unchanged, so a normalization is generally imposed. The existence of a state-independent average cost is not automatic: it depends on the controlled dynamics and on whether long-run behavior can differ across initial states or recurrent regions.\n\nWhen the controller can choose when to terminate the process, the problem involves optimal stopping. For reward maximization with stopping payoff \\(g\\), a discounted stationary formulation can take the form\n\n\\[\\max\\left\\{ g(x)-V(x),\\ \\sup_a[\\ell(x,a)+\\mathcal L^aV(x)]-\\rho V(x) \\right\\}=0.\\]\n\nIn the stopping region, \\(V=g\\); in the continuation region, the HJB equation holds. The unknown interface is a free boundary. Impulse control, in which the state can be changed by discrete interventions at a cost, instead introduces an intervention operator and leads to a quasi-variational inequality.\n\nIn a two-player zero-sum game, one player minimizes a cost while the other maximizes it. Dynamic programming produces a Hamilton–Jacobi–Isaacs equation containing nested minimization and maximization. Exchanging their order need not preserve the result. The Isaacs condition asserts equality of the relevant minimax Hamiltonians and is important in establishing a common game value. A different extension arises under partial observation: the controller may need to use the conditional distribution of the hidden state as the information state. The corresponding HJB equation can then be posed on a space of probability distributions rather than on a finite-dimensional state space."
    },
    {
      "heading": "Numerical approximation and computational complexity",
      "text": "Numerical HJB methods must approximate both a differential operator and an optimization problem. A basic grid method divides state space into nodes, approximates derivatives with a finite-difference scheme, and minimizes over actions at each node. First-order drift terms usually require directional, or upwind, differences selected consistently with the drift. Centered differences that work well for smooth linear equations may produce oscillations or fail to select the correct viscosity solution near corners. For diffusion terms, maintaining a monotone stencil can require additional grid points, especially when the diffusion matrix has strong off-diagonal entries.\n\nSemi-Lagrangian schemes discretize the dynamic programming principle more directly. In a deterministic problem, a typical step is\n\n\\[V^n(x)\\approx \\min_a\\left\\{ \\Delta t\\,\\ell(t_n,x,a)+ I[V^{n+1}]\\bigl(x+\\Delta t\\,f(t_n,x,a)\\bigr) \\right\\},\\]\n\nwhere \\(I\\) is an interpolation operator. The shifted state need not lie on the grid. Probabilistic versions approximate diffusion by weighted transitions whose first two moments reproduce the local drift and covariance. Such constructions reveal the close relationship between numerical HJB equations and discrete Markov decision processes.\n\nFor broad classes of equations, the convergence framework developed by Guy Barles and Panagiotis Souganidis in 1991 establishes convergence of monotone, stable, consistent schemes when the limiting PDE satisfies an appropriate comparison principle. Each requirement has a distinct role: consistency matches the local equation, stability controls the approximations, and monotonicity preserves the order structure needed for viscosity arguments. High-order accuracy alone is not sufficient. Filtered and related schemes attempt to combine accurate discretizations in smooth regions with monotone behavior where it is needed.\n\nThe nonlinear discrete system can be solved by value iteration, which repeatedly applies a discrete dynamic programming operator, or by policy iteration, which alternates between evaluating a fixed policy and improving its action choices. Discounting often supplies a contraction structure for value iteration. Policy iteration can converge rapidly, but each evaluation step may require solving a large linear system. Both approaches must also address how continuous action spaces are minimized: discretizing controls introduces a separate approximation error from the spatial and temporal discretization.\n\nThe principal obstacle is the curse of dimensionality. A grid with \\(m\\) nodes along each of \\(d\\) state coordinates contains \\(m^d\\) nodes; \\(100\\) nodes in each of six dimensions already gives \\(10^{12}\\) nodes. Sparse representations, tensor methods, simulation-based methods, and neural approximations seek to reduce this burden by exploiting additional structure. Their effectiveness is problem dependent. A small sampled equation residual does not alone establish a small global value error or the admissibility and optimality of the extracted policy."
    },
    {
      "heading": "Applications and connections to learning",
      "text": "In engineering, the HJB equation provides a mathematical foundation for feedback design. A state can combine position, velocity, temperature, stored energy, or other physical quantities, while controls represent forces, voltages, heating rates, or actuator commands. Running costs trade tracking accuracy against effort, fuel use, or wear. In navigation and minimum-time motion, the value describes the remaining travel cost from every location; spatial obstacles become state constraints or excluded regions. Robust versions replace a known disturbance model with an adversarial disturbance and lead to differential-game equations used in reachability and safety analysis.\n\nIn economics, HJB equations express intertemporal optimization when decisions are made continuously. For example, a consumption and saving model may use wealth \\(w\\) as a state, consumption \\(c\\) as a control, and instantaneous utility \\(u(c)\\). A stationary reward-maximization equation can take the form\n\n\\[\\rho V(w)= \\sup_c\\left\\{ u(c)+(rw+y-c)V'(w) \\right\\},\\]\n\nwhere \\(r\\) is an interest rate and \\(y\\) is income. Borrowing restrictions determine the feasible controls and the treatment of the wealth boundary. Adding uncertain income introduces extra state variables or diffusion terms. The equation makes the marginal value of wealth explicit through \\(V'(w)\\).\n\nPortfolio choice provides a stochastic example. If wealth invested in a risky asset evolves as\n\n\\[dW_t= [rW_t+\\pi_t(\\mu-r)-c_t]\\,dt+ \\pi_t\\sigma\\,dB_t,\\]\n\nthen the HJB equation includes\n\n\\[\\sup_{\\pi,c}\\left\\{ u(c)+[rw+\\pi(\\mu-r)-c]V_w+ \\frac12\\pi^2\\sigma^2V_{ww} \\right\\}.\\]\n\nThe curvature \\(V_{ww}\\) governs how investment risk enters the objective. This formulation underlies Robert Merton’s continuous-time portfolio-consumption models developed in 1969 and 1971. The resulting policies depend on assumptions about utility, available assets, constraints, and market dynamics; the equation does not supply a model-independent investment rule.\n\nIn reinforcement learning, the connection is the Bellman principle itself. Continuous-time learning methods can seek a value function, a policy, or both from trajectories when the dynamics or costs are incompletely known. With a model available, a parameterized value approximation may be trained using HJB residuals and terminal or boundary errors. Without a known model, temporal differences and controlled observations can provide information about the local dynamic programming relation. Exploration, sampling bias, approximation error, and stability then become additional issues beyond solving a specified PDE.\n\nHJB theory also relates value-based learning to policy optimization. For a fixed policy, the optimization disappears and the value satisfies a linear backward equation under standard diffusion assumptions. Improving the policy by minimizing the resulting Hamiltonian reproduces the logic of policy iteration. Neural networks can represent values in many dimensions, while automatic differentiation supplies derivatives for residual objectives. However, classical verification theorems require properties that an approximate network does not automatically possess, including suitable boundary behavior, sufficient error control, and an admissible closed-loop state evolution."
    }
  ],
  "links": [
    "partial-differential-equation",
    "value-function",
    "bellman-equation",
    "optimal-control",
    "calculus-of-variations",
    "euler-lagrange-equations",
    "dynamic-programming",
    "lipschitz-continuity",
    "gradient",
    "stochastic-differential-equation",
    "wiener-process",
    "expected-value",
    "hessian-matrix",
    "matrix-trace",
    "itos-formula",
    "weak-solution",
    "chain-rule",
    "zero-sum-game",
    "finite-difference-method",
    "interpolation",
    "markov-decision-process",
    "value-iteration",
    "policy-iteration",
    "curse-of-dimensionality",
    "economics",
    "reinforcement-learning",
    "automatic-differentiation"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}