{
  "id": "energy-based-model",
  "title": "Energy-Based Model",
  "lang": "en",
  "source_lang": "en",
  "summary": "An energy-based model represents compatibility or probability through a scalar energy function, with lower energies assigned to preferred configurations.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "EBM",
    "energy-based machine learning model"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/energy-based-model/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/energy-based-model/",
      "json": "https://aiwiki.page/zh/energy-based-model.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "An energy-based model (EBM) is a model in machine learning that assigns a scalar energy to a configuration of variables. Configurations consistent with the model’s learned relationships receive lower energies than incompatible configurations. Depending on the application, the configuration may be an image, a sequence, a proposed output paired with an input, or a collection of observed and hidden variables. Prediction involves finding low-energy configurations; probabilistic modeling additionally relates energy to a normalized probability distribution.\n\nThe framework is broader than a particular neural-network architecture or training procedure. It encompasses classical associative memories, probabilistic graphical models, and modern systems whose energy functions are implemented by deep networks. Its central attraction is the separation of a compatibility function from the procedure used to search, sample, or reason with it. Its central difficulty is that these procedures can be computationally expensive: evaluating an energy may be easy even when locating its minima or calculating the associated probabilities is hard.\n\nEnergy-based modeling connects ideas from statistical physics, optimization, and statistical learning. Understanding the framework requires distinguishing energy from probability, prediction from sampling, and the objective used to train a model from the energy function that the trained model represents."
    },
    {
      "heading": "Energy functions and probabilistic interpretation",
      "text": "An energy function \\(E_\\theta(z)\\) maps a configuration \\(z\\) to a real number, with parameters \\(\\theta\\) controlling the shape of the function. In a conditional model, the notation \\(E_\\theta(x,y)\\) separates an observed input \\(x\\) from a candidate output \\(y\\). Prediction is commonly defined by\n\n\\[\\hat y(x)\\in\\operatorname*{arg\\,min}_{y}E_\\theta(x,y).\\]\n\nThis definition requires neither probabilities nor a normalization constant. It specifies a preference ordering over outputs, although the search procedure may find only an approximate minimum. The energy is not generally a physical quantity measured in joules; the terminology reflects an analogy with thermodynamics and equilibrium statistical mechanics.\n\nA probabilistic EBM defines a density or mass function through the Gibbs form\n\n\\[p_\\theta(z)=\\frac{\\exp[-E_\\theta(z)/T]}{Z_\\theta(T)}, \\qquad Z_\\theta(T)=\\int \\exp[-E_\\theta(z)/T]\\,d\\mu(z),\\]\n\nwhere \\(T>0\\) is a temperature parameter and \\(\\mu\\) specifies the underlying measure. For discrete configurations, the integral becomes a sum. The partition function \\(Z_\\theta(T)\\) must be finite for this expression to define a probability distribution. Temperature is often absorbed into the energy, giving the simpler convention \\(p_\\theta(z)\\propto e^{-E_\\theta(z)}\\). Lower temperatures concentrate probability more strongly around low-energy regions.\n\nThe measure and domain matter. A bounded energy on all of \\(\\mathbb{R}^d\\), interpreted relative to ordinary volume, cannot produce an integrable density because \\(e^{-E_\\theta(z)}\\) remains bounded away from zero. A continuous EBM therefore needs suitable tail behavior, a bounded domain, or a specified reference distribution. One useful formulation is \\(p_\\theta(z)\\propto q(z)e^{-F_\\theta(z)}\\), where a reference density \\(q\\) provides part of the required tail behavior. An arbitrary scalar-output network is consequently not automatically a valid density model.\n\nEnergy differences, rather than absolute energy values, determine probability ratios:\n\n\\[\\frac{p_\\theta(z_1)}{p_\\theta(z_2)} =\\exp\\!\\left[-\\frac{E_\\theta(z_1)-E_\\theta(z_2)}{T}\\right].\\]\n\nAdding a constant to all energies leaves the distribution unchanged. In a conditional model, adding any function of the conditioning input alone likewise leaves \\(p_\\theta(y\\mid x)\\) unchanged. This non-identifiability is important when interpreting raw energy scores across different inputs or models.\n\nFinally, low energy does not necessarily imply that a region contains much probability. Probability mass depends on both density and volume. In high dimensions, a broad region of moderately low energy can contain more mass than a very narrow, deeper minimum. This distinction separates optimization, which seeks an extremum, from sampling, which must represent the distribution’s mass."
    },
    {
      "heading": "Historical development",
      "text": "Energy-based modeling grew from several related traditions rather than from a single invention. Statistical mechanics supplied the mathematical relationship between energy and equilibrium probability. The Ising model, introduced in the 1920s to describe interacting spins, provided an influential example of a global energy assembled from local interactions. Similar constructions later became useful for describing dependencies among variables in statistics, image analysis, and computation.\n\nJohn Hopfield’s 1982 work on recurrent networks established a prominent connection between neural computation and energy minimization. In a classical Hopfield network, binary units interact through symmetric weights. Updating one unit at a time according to its local field does not increase the network’s energy under the standard assumptions. Stored patterns can therefore act as attractors: a corrupted pattern evolves toward a nearby stable configuration. This made an energy landscape a concrete computational mechanism for associative memory, while also exposing limitations such as spurious attractors and restricted storage capacity.\n\nThe Boltzmann machine, developed by Geoffrey Hinton, Terrence Sejnowski, and colleagues in the 1980s, replaced deterministic state transitions with stochastic ones and introduced a probabilistic learning framework. Hidden units allowed the model to represent dependencies not directly expressed among observed units. Learning involved comparing correlations under observed data with correlations generated by the model. The conceptual formulation was powerful, but obtaining the model-generated statistics was expensive, especially when networks mixed slowly between distinct configurations.\n\nDuring the same period, probabilistic image models used local interaction energies to express assumptions such as spatial smoothness and edge preservation. Stuart and Donald Geman’s 1984 work on Bayesian image restoration connected Gibbs distributions with stochastic relaxation methods. In a separate development, Yann LeCun and collaborators promoted energy-based learning as a general framework for assigning compatibility scores to input–output configurations. Their 2006 tutorial presented probabilistic, margin-based, and other learning procedures within a common language rather than identifying EBMs with one network architecture.\n\nSeveral later developments expanded the practical toolkit. Hinton introduced contrastive divergence in 2002 as an approximate learning procedure for models such as restricted Boltzmann machines. Aapo Hyvärinen introduced score matching in 2005, and Michael Gutmann and Hyvärinen introduced noise-contrastive estimation in 2010. These methods addressed normalization or sampling difficulties in different ways. Deep convolutional networks subsequently enabled expressive image energies, while work on score-based generative models clarified connections between energy gradients, denoising, and diffusion. These developments did not eliminate the original computational difficulties, but provided distinct ways to trade exact likelihood calculations for tractable training signals."
    },
    {
      "heading": "Model structure and representative families",
      "text": "An EBM specifies an energy function, but leaves considerable freedom in how that function is constructed. A common choice is a sum of local terms,\n\n\\[E_\\theta(z)=\\sum_{a\\in\\mathcal A}\\psi_{\\theta,a}(z_a),\\]\n\nwhere each factor \\(\\psi_{\\theta,a}\\) depends on a subset \\(z_a\\) of the variables. In a Markov random field, the factor structure describes local dependencies in an undirected graph. Pairwise image models, for example, can combine terms measuring agreement with observed pixels and terms penalizing particular differences between neighboring labels. Locality can simplify computation, although it does not guarantee that global inference is easy.\n\nA classical Hopfield energy for binary units has the form\n\n\\[E(s)=-\\frac12\\sum_{i,j}w_{ij}s_i s_j-\\sum_i b_i s_i,\\]\n\nwith symmetric interactions \\(w_{ij}=w_{ji}\\) and, conventionally, no self-interactions. Stable states correspond to local energy minima. A Boltzmann machine uses a related interaction structure but defines a distribution over states. Its stochastic dynamics permit transitions that temporarily increase energy, distinguishing equilibrium sampling from purely descending toward an attractor.\n\nThe restricted Boltzmann machine (RBM) imposes a bipartite structure between visible units \\(v\\) and hidden units \\(h\\), with no within-layer connections. For binary units,\n\n\\[E_\\theta(v,h)=-a^\\top v-b^\\top h-v^\\top Wh.\\]\n\nConditioned on the visible layer, hidden units are independent; conditioned on the hidden layer, visible units are independent. This permits parallel block updates. Summing out the hidden units gives the visible free energy\n\n\\[F_\\theta(v)=-a^\\top v-\\sum_j \\log\\!\\left(1+\\exp(b_j+W_{\\cdot j}^{\\top}v)\\right).\\]\n\nThe visible distribution is proportional to \\(e^{-F_\\theta(v)}\\). Hidden configurations therefore influence visible probabilities through both their energies and their multiplicity.\n\nModern neural EBMs often replace explicitly specified pairwise interactions with a flexible neural network. A convolutional network can map an image to one scalar; a transformer can score a sequence or a paired multimodal input. This flexibility removes some restrictions imposed by handcrafted factors, but can also remove the conditional independences that made classical inference algorithms efficient. Architecture, energy parameterization, and inference method must consequently be considered together.\n\nHidden variables provide another structural choice. If \\(h\\) is unobserved, marginalization produces\n\n\\[F_\\theta(x)=-\\log\\int e^{-E_\\theta(x,h)}\\,dh.\\]\n\nMinimizing \\(E_\\theta(x,h)\\) over \\(h\\) is generally not equivalent to computing this free energy. Minimization selects the best hidden explanation, whereas marginalization aggregates all explanations. The distinction affects both training objectives and the interpretation of models that use latent-state optimization as an approximation to probabilistic inference."
    },
    {
      "heading": "Maximum-likelihood learning and the negative phase",
      "text": "For a normalized EBM with temperature absorbed into the energy, the negative log-likelihood of one observation is\n\n\\[-\\log p_\\theta(x)=E_\\theta(x)+\\log Z_\\theta.\\]\n\nDifferentiating yields\n\n\\[\\nabla_\\theta[-\\log p_\\theta(x)] = \\nabla_\\theta E_\\theta(x) - \\mathbb E_{x'\\sim p_\\theta} \\left[\\nabla_\\theta E_\\theta(x')\\right].\\]\n\nThe first term depends on observed data. The second depends on configurations distributed according to the current model. This equation explains why likelihood learning cannot generally be implemented by merely lowering the energies of training examples.\n\nUnder gradient descent, the data-dependent contribution tends to lower energy on observed configurations. The model-dependent contribution counterbalances this by raising energy where the model currently places probability, with the detailed effect mediated by shared parameters. These contributions are traditionally called the positive phase and negative phase. The terminology derives from classical Boltzmann-machine learning and should not be confused with positive and negative class labels.\n\nFor a binary Boltzmann interaction parameter, the update can be expressed as a difference between data-conditioned and model-generated correlations. This gives a precise mechanism: dependencies insufficiently represented by the model are strengthened, while dependencies overrepresented in generated states are weakened. In a neural EBM, backpropagation computes parameter derivatives of the energy at observed and generated samples, but does not itself solve the problem of generating representative model samples.\n\nContrastive divergence approximates the negative phase by starting a Markov chain at an observed example and running only a small number of transitions. In an RBM, these transitions can alternate between sampling hidden units and visible units. The resulting update is generally biased relative to the exact likelihood gradient. Its practical motivation is that short chains may provide useful learning signals even when equilibrium sampling would be prohibitively expensive.\n\nPersistent contrastive divergence, associated with Tijmen Tieleman’s 2008 work, retains chains between parameter updates instead of restarting all chains from data. If parameters change gradually and the chains explore effectively, the persistent states can track the evolving model distribution more closely. Related neural EBM implementations use replay buffers containing previously generated configurations, sometimes mixed with newly initialized samples. These techniques reduce repeated initialization costs but do not guarantee representative sampling.\n\nLikelihood training is therefore coupled to the behavior of an approximate inference algorithm. If chains remain within a subset of modes, parameter updates may neglect substantial parts of the model distribution. A decreasing approximate training objective need not imply improving normalized likelihood. The statistical target, the estimator used for its gradient, and the actual behavior of the sampler are distinct objects, even when an implementation combines them in a single training loop."
    },
    {
      "heading": "Inference, sampling, and computational cost",
      "text": "Energy-based systems perform several different computational tasks. Optimization seeks low-energy configurations. Sampling generates configurations according to a specified distribution. Marginal inference aggregates over unobserved variables. These tasks can require different algorithms even for the same energy function. A gradient descent procedure may find a plausible image reconstruction without producing a valid sample, while a sampler may intentionally visit configurations far from any local minimum.\n\nFor discrete models, Markov chain Monte Carlo provides a general sampling framework. Gibbs sampling repeatedly draws a variable or block of variables from its conditional distribution given the others. Metropolis–Hastings proposes a move and accepts or rejects it using a probability involving the ratio of target densities. For an EBM, the unknown partition function cancels from that ratio. This makes transitions possible without calculating \\(Z_\\theta\\), although it does not establish how rapidly a chain approaches equilibrium.\n\nFor differentiable continuous energies, Langevin dynamics combines movement down the energy gradient with random perturbations. A common discretization is\n\n\\[x_{k+1} =x_k-\\frac{\\epsilon}{2}\\nabla_xE_\\theta(x_k) +\\sqrt{\\epsilon}\\,\\xi_k, \\qquad \\xi_k\\sim\\mathcal N(0,I).\\]\n\nThe corresponding continuous-time process has \\(e^{-E_\\theta(x)}\\) as its stationary density under suitable conditions. A finite-step unadjusted discretization generally introduces bias. A Metropolis correction can remove discretization bias in the stationary target, subject to the usual convergence conditions, but requires additional evaluations and may reject proposed moves.\n\nThe main obstacle is often mixing: movement between regions containing substantial probability. If high-energy barriers separate modes, local proposals can remain trapped for long periods. Tempering methods introduce distributions at several temperatures, allowing flatter distributions to help exploration. Hamiltonian Monte Carlo uses momentum and gradient information to propose longer coordinated moves. Neither provides a universal solution, especially when gradients are costly or the state space has complicated geometry.\n\nStructured energies sometimes admit much faster exact inference. Chain-structured discrete models can use dynamic programming for minimum-energy decoding and partition-function computation. Tree-structured graphical models support exact message passing. Certain binary pairwise energies can be minimized using graph cuts, whereas general discrete energy minimization can be NP-hard. Expressive representations do not by themselves imply tractable inference.\n\nComputational accounting must include the inner loop. If generating one negative example requires \\(K\\) Langevin steps, each training update may require roughly \\(K\\) input-gradient evaluations per sampled configuration, in addition to parameter-gradient calculations. Inference-time refinement introduces a similar multiplicative cost. Reports of performance therefore depend not only on network size, but also on chain length, initialization, acceptance rules, stopping criteria, and whether gradients through the inference procedure are retained."
    },
    {
      "heading": "Learning without an explicit partition function",
      "text": "Several learning methods avoid direct evaluation of the partition function, but they do so by changing the estimation problem rather than making normalization universally easy. Score matching, introduced by Hyvärinen in 2005, estimates a density through its score,\n\n\\[s_\\theta(x)=\\nabla_x\\log p_\\theta(x)=-\\nabla_xE_\\theta(x).\\]\n\nBecause \\(Z_\\theta\\) is constant with respect to \\(x\\), it disappears from this derivative. This is an input gradient, not the parameter gradient used in maximum-likelihood learning.\n\nThe ideal score-matching objective compares model and data scores. Integration by parts transforms it into an expression that does not require the unknown data score:\n\n\\[\\mathcal L_{\\mathrm{SM}} = \\mathbb E_{p_{\\mathrm{data}}} \\left[ \\frac12\\|\\nabla_xE_\\theta(x)\\|^2 -\\Delta_xE_\\theta(x) \\right],\\]\n\nup to a constant independent of \\(\\theta\\). The transformation requires suitable smoothness and boundary conditions. Computing the Laplacian can be expensive in high dimensions, motivating stochastic trace estimators and related variants. Basic score matching also presumes a continuous setting in which the required derivatives are meaningful.\n\nDenoising score matching instead adds known noise to observations and trains a model to predict the score of the corrupted distribution. Pascal Vincent’s 2011 analysis connected a denoising objective with score matching. With Gaussian corruption \\(\\tilde x=x+\\sigma\\epsilon\\), the conditional score target is \\(-(\\tilde x-x)/\\sigma^2\\). Training across multiple noise levels supplies information about both fine-scale data structure and smoother, more easily explored distributions. This construction became central to score-based generative modeling.\n\nNoise-contrastive estimation (NCE), introduced by Gutmann and Hyvärinen in 2010, frames density estimation as discrimination between observed data and samples from a known noise distribution. The optimal discriminator depends on the ratio of model density to noise density. An unnormalized density and its normalization parameter can therefore be fitted through a classification objective. Statistical properties depend on assumptions about the model and noise distribution; practical efficiency depends strongly on whether noise examples provide informative comparisons.\n\nMargin-based learning offers another alternative when calibrated probabilities are unnecessary. A model can require a correct output \\(y^+\\) to have lower energy than an incorrect output \\(y^-\\) by a specified margin, for example through\n\n\\[\\max\\{0,m+E_\\theta(x,y^+)-E_\\theta(x,y^-)\\}.\\]\n\nThis is useful for ranking and structured prediction, but does not automatically estimate a normalized distribution. Likewise, contrastive learning uses relative scores among positive and negative examples, yet only under particular formulations and assumptions does it support a density-ratio interpretation. Avoiding the partition function is therefore compatible with several different learning goals, not one interchangeable training recipe."
    },
    {
      "heading": "Conditional models and structured prediction",
      "text": "A conditional EBM scores candidate outputs relative to a fixed input:\n\n\\[p_\\theta(y\\mid x) = \\frac{\\exp[-E_\\theta(x,y)]} {\\sum_{y'}\\exp[-E_\\theta(x,y')]}.\\]\n\nNormalization is over outputs \\(y'\\), not over inputs \\(x\\). This can make conditional modeling substantially easier than joint density modeling. If the output is one of a small number of classes, the denominator is directly computable, even when the input is a high-dimensional image.\n\nA multiclass classifier with logits \\(f_\\theta(x)_y\\) is a simple example: setting \\(E_\\theta(x,y)=-f_\\theta(x)_y\\) reproduces the softmax distribution. Thus, energy-based interpretation does not necessarily imply a specialized sampler or an unusual architecture. The distinction becomes more consequential when the output space is too large to enumerate, such as all possible label sequences, segmentations, or structured configurations.\n\nThe conditional random field (CRF), introduced by John Lafferty, Andrew McCallum, and Fernando Pereira in 2001, is an important probabilistic conditional energy model. A linear-chain CRF can combine local label scores with transition scores:\n\n\\[E_\\theta(x,y)= -\\sum_t u_\\theta(x,t,y_t) -\\sum_t A_{y_{t-1},y_t}.\\]\n\nThe unary term measures compatibility between the input and a label at position \\(t\\); the transition term models compatibility between neighboring labels. The forward algorithm calculates the conditional partition function, and Viterbi decoding finds the lowest-energy sequence.\n\nIn semantic segmentation, output variables assign labels to pixels or regions. An energy can combine local predictions with terms relating neighboring labels or enforcing broader consistency. For example, a local predictor may favor a particular object class at each pixel, while pairwise terms penalize label changes between visually similar neighbors. These interactions can correct isolated errors, but excessive smoothness can erase thin structures or incorrectly merge adjacent objects. The inference method and the form of the interaction terms determine whether such dependencies remain computationally manageable.\n\nConditional energies also support continuous outputs. For reconstruction or trajectory prediction, an energy can express agreement with observations together with learned structural constraints. Prediction then becomes numerical optimization over the output. Unlike a feed-forward predictor, this permits additional inference-time constraints or repeated refinement, but makes runtime and output quality depend on initialization and convergence.\n\nThe learning objective must account for how outputs will be obtained. Training may optimize a conditional likelihood, compare correct and incorrect structures through a margin, or differentiate through a finite sequence of optimization steps. These choices are not equivalent. A model trained with a particular truncated inference procedure may perform well with that procedure without defining an accurate equilibrium distribution or yielding reliable predictions under a different optimizer."
    },
    {
      "heading": "Relationships to other generative frameworks",
      "text": "Energy-based modeling overlaps with other model families, but the overlaps are not identities. An autoregressive language model, for example, factorizes a joint probability as\n\n\\[p(x_1,\\ldots,x_n)=\\prod_{t=1}^{n}p(x_t\\mid x_{<t}).\\]\n\nWhenever this normalized probability is positive, \\(E(x)=-\\log p(x)\\) provides an energy representation. Yet the autoregressive factorization already supplies tractable normalization and sequential sampling. Calling it an energy representation does not imply that the model was trained using negative-phase sampling or that its practical advantages arise from a freely parameterized unnormalized energy.\n\nA variational autoencoder specifies a latent-variable generative model and typically trains using a variational lower bound on likelihood. An EBM can also contain hidden variables, but its normalization and posterior computations need not take the same form. Hybrid constructions use energies as priors, likelihood corrections, or components of latent-variable models. Whether inference remains tractable depends on the particular combination; introducing an energy term can remove analytic simplifications available in the original model.\n\nA generative adversarial network ordinarily generates samples by transforming random inputs through a generator. Its discriminator distinguishes data from generated examples. Under idealized conditions, an optimal discriminator encodes a density ratio, which creates a connection to energy and noise-contrastive estimation. However, the usual discriminator is not automatically a normalized density model, and adversarial training does not generally produce an explicit energy whose equilibrium distribution is sampled by the generator.\n\nThe closest modern connection is with the diffusion model and score-based generation. A diffusion process progressively corrupts data, and a learned reverse process reconstructs samples. The reverse dynamics can be expressed using time-dependent scores of the corrupted distributions. If a score is represented as\n\n\\[s_\\theta(x,t)=-\\nabla_xE_\\theta(x,t),\\]\n\nthen an energy describes each noise level. Many practical score networks instead output vector fields directly. Such fields are not guaranteed to be exact gradients of scalar potentials, so a learned score network need not correspond exactly to an explicit EBM.\n\nEnergy functions can also be combined. For distributions on the same variable, multiplying unnormalized densities corresponds to adding energies:\n\n\\[e^{-E_1(x)}e^{-E_2(x)}=e^{-[E_1(x)+E_2(x)]}.\\]\n\nThis is the basis of a product of experts, in which each expert restricts acceptable configurations. The algebra is straightforward, but the combined distribution may be difficult to normalize or sample. Compatibility of the experts’ constraints also matters: individually reasonable models can jointly concentrate on undesirable regions.\n\nThese connections show why “energy-based” is best understood as a modeling formulation rather than an exclusive architectural category. The operational differences lie in what is parameterized, which quantities can be computed exactly, how learning is performed, and how predictions or samples are produced."
    },
    {
      "heading": "Applications and evaluation",
      "text": "Energy-based formulations are useful when a system must evaluate compatibility among multiple variables. In computer vision, classical applications include image restoration, stereo correspondence, and segmentation. An observed image may support several locally plausible explanations, while an energy expresses preferences among their global combinations. Modern neural energies replace or supplement manually designed terms, allowing dependencies learned from training data to shape reconstruction and generation.\n\nIn natural language processing, conditional energies support sequence labeling, structured reranking, and scoring relationships between sentences. An energy-based reranker can evaluate complete candidate translations rather than committing to the locally best next token. It can therefore represent features of an entire output, but its usefulness depends on candidate generation: a reranker cannot select a desirable answer that never enters the candidate set. Searching directly over all possible strings introduces a much larger inference problem.\n\nCompatibility scoring also appears in multimodal learning and retrieval. Image–text similarity, for example, can be converted into an energy by changing its sign and possibly applying a scale factor. Ranking candidates then amounts to finding lower-energy matches. This does not by itself establish a calibrated joint probability over all images and texts. Such systems may be more accurately characterized by their ranking objective than by an equilibrium generative interpretation.\n\nFor anomaly detection, energies or derived scores can distinguish familiar observations from some unfamiliar ones. A classifier’s logits, for instance, define the input-dependent quantity\n\n\\[F_\\theta(x)=-T\\log\\sum_y\\exp(f_\\theta(x)_y/T).\\]\n\nThis score has been investigated for out-of-distribution detection. However, shifting all logits by the same input-dependent value leaves class probabilities unchanged while changing \\(F_\\theta(x)\\). Consequently, its behavior is not determined by classification probabilities alone. More generally, high density or low energy is not a universal indicator of semantic familiarity.\n\nEvaluation must match the modeled task. Conditional prediction can be assessed using accuracy, sequence measures, or task-specific costs. Density estimation requires normalized likelihood or a justified estimator; reporting only unnormalized energies does not make different models directly comparable. Sample quality can be assessed with perceptual metrics and coverage measures, but attractive samples do not prove that a chain represents the model’s equilibrium distribution.\n\nSampling protocols are therefore part of the result. Initialization, number of updates, temperature, use of replay buffers, and postprocessing can materially change generated outputs. Evaluations involving distribution shift or adversarial examples additionally depend on the chosen shifts and attack procedures. Energy-based formulation alone supplies neither a general robustness guarantee nor a complete method for determining whether an observation lies outside the training distribution."
    },
    {
      "heading": "Limitations and unresolved questions",
      "text": "The central limitation of expressive EBMs is the gap between defining a distribution and computing with it. A neural network can evaluate \\(E_\\theta(x)\\) quickly while leaving the partition function, representative sampling, and global minimization intractable. Increasing network capacity can improve the ability to represent complex dependencies, but can also produce a more irregular energy landscape. Statistical expressiveness and computational accessibility are separate properties.\n\nA particularly important issue is the difference between an equilibrium model and a finite-run generator. Starting from a specified initial distribution and taking \\(K\\) sampling steps defines an output distribution \\(q_{\\theta,K}\\). That distribution need not equal \\(p_\\theta\\propto e^{-E_\\theta}\\). A model trained using short-run samples may generate useful outputs through those dynamics even when its long-run behavior is different. In that case, the initialization and transition procedure are essential parts of the effective generative system, rather than merely interchangeable approximations.\n\nDiagnosing convergence is difficult in high-dimensional, multimodal settings. Chains can appear stable while remaining trapped in one region. Agreement between several chains does not establish coverage if they share a restrictive initialization. Longer trajectories, alternative samplers, mode-transition statistics, and comparisons with tractable examples provide partial evidence, but none constitutes a universal convergence certificate for a large neural EBM.\n\nNormalization creates a related evaluation problem. Annealed importance sampling and other methods can estimate ratios of partition functions, but their reliability depends on intermediate distributions and effective exploration. These estimates may become expensive or unstable when distributions have poorly connected modes. Exact computations remain valuable for small systems because they allow learning rules, samplers, and estimators to be checked against known quantities.\n\nThere is also an interpretive limit to the physical analogy. Learned minima represent preferences induced by architecture, data, and training objectives; they are not automatically physical states or causal explanations. Likewise, an energy landscape can be visualized in low-dimensional projections, but those projections may conceal barriers or broad high-probability regions in the full space. Geometric descriptions such as “basin” or “attractor” require attention to the actual dynamics being used.\n\nResearch directions include learning energies with better-conditioned gradients, coupling learned proposal distributions with corrective samplers, exploiting factor structure, and representing families of distributions across noise levels. A recurring technical question is how to obtain both expressive dependencies and reliable computation without making training or inference prohibitively expensive. The answer depends on the task: an approximate minimum may suffice for structured prediction, while likelihood estimation and scientific sampling require substantially stronger control over the represented distribution."
    }
  ],
  "links": [
    "machine-learning",
    "probability-distribution",
    "thermodynamics",
    "partition-function",
    "hopfield-network",
    "markov-random-field",
    "artificial-neural-network",
    "convolutional-neural-network",
    "transformer-architecture",
    "backpropagation",
    "markov-chain-monte-carlo",
    "dynamic-programming",
    "np-hardness",
    "contrastive-learning",
    "conditional-random-field",
    "semantic-segmentation",
    "language-model",
    "variational-autoencoder",
    "generative-adversarial-network",
    "diffusion-model",
    "computer-vision",
    "training-data",
    "natural-language-processing",
    "multimodal-learning",
    "anomaly-detection",
    "distribution-shift",
    "adversarial-example"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}