Probabilistic graphical model

A probabilistic graphical model (PGM) is a mathematical representation of a joint probability distribution that combines a graph with local probabilistic functions. The graph describes relationships among random variables, while the functions specify how their values are distributed. By connecting probability theory with graph theory, graphical models provide a framework for representing uncertainty, computing predictions, and learning statistical structure. They are used in statistics, machine learning, and fields including communications and signal processing. (stat.berkeley.edu)

Representation and conditional independence

A graphical model separates qualitative structure from quantitative specification. Its graph encodes conditional independence assumptions, while its parameters determine a particular distribution compatible with those assumptions. Variables may be discrete, continuous, observed, or unobserved. Different graphical structures impose different restrictions on the distributions they can represent. (cs.cmu.edu)

Conditional independence means that, once specified conditioning variables are known, learning about one variable provides no additional information about another. It is distinct from unconditional statistical independence. These relationships permit a large distribution to be expressed through smaller components rather than an unrestricted table over every possible joint assignment. A graph therefore represents statistical assumptions, not merely a diagram of associations found in data. Graphical-model analysis is commonly organized around three tasks: representation, inference, and learning. (cs.cmu.edu)

Directed models

A Bayesian network uses a directed acyclic graph. Each node represents a variable, and its incoming neighbors are called its parents. The joint distribution factorizes as

p(x_1,\ldots,x_n) =\prod_{i=1}^{n}p(x_i\mid x_{\mathrm{pa}(i)}),

where $\mathrm{pa}(i)$ denotes the parent set of node $i$ . Each factor is a normalized conditional probability distribution. This representation expresses the assumption that a variable is independent of its nondescendants given its parents. (ftp.cs.ucla.edu)

The graphical criterion d-separation identifies conditional independences implied by the network. Importantly, an arrow does not automatically establish causation: a Bayesian network may simply encode a probabilistic factorization. A causal interpretation requires additional assumptions about the processes generating the variables and about what happens under interventions. Thus, probabilistic conditioning and causal inference are related but distinct operations. (ftp.cs.ucla.edu)

Undirected models and factor graphs

A Markov random field, also called a Markov network, uses an undirected graph. Its distribution can be written

p(x)=\frac{1}{Z}\prod_{C}\psi_C(x_C),

where $C$ ranges over specified cliques, or fully connected subsets of nodes, and $\psi_C$ is a nonnegative potential function. Potentials express compatibility among assignments and need not themselves be probabilities. The partition function $Z$ normalizes their product; for discrete variables, it is the sum of that product over all assignments. Computing it can be expensive. (cs.cmu.edu)

Undirected separation expresses conditional independence: if a set of nodes separates two other sets, their variables are conditionally independent given the separating variables. A factor graph makes factorization explicit by using two kinds of nodes—variable nodes and factor nodes—with edges connecting each factor to the variables it involves. Both directed and undirected model factorizations can be represented this way, providing a convenient basis for message-passing algorithms. (arxiv.org)

Probabilistic inference

Inference computes quantities from an already specified model. Typical queries include the distribution of an unobserved variable given evidence, the probability of observed data, or the most probable joint assignment. Marginal probabilities require summing or integrating over variables not included in the query. Finding a most probable assignment instead involves maximization; these operations answer different questions. (stat.berkeley.edu)

Exact methods include variable elimination, which combines factors and removes variables successively, and belief propagation, which passes local messages. Sum-product message passing computes exact marginals on tree-structured factor graphs. More general graphs can be processed through junction-tree methods, but their cost depends strongly on treewidth, a measure of structural complexity. For discrete models, intermediate factors can grow exponentially with the size of the variable groups involved, so a compact representation does not guarantee cheap inference. (cs.columbia.edu)

When exact computation is impractical, approximate methods include Markov chain Monte Carlo and variational inference. Sampling methods estimate quantities from generated samples. Variational methods replace difficult calculations with optimization over a more tractable family of distributions, sometimes yielding bounds on probabilities or likelihoods. Their accuracy depends on the approximation family and the particular model. (people.eecs.berkeley.edu)

Learning parameters and structure

Parameter learning estimates the local distributions or potentials from data while holding the graph fixed. Common approaches include maximum likelihood estimation and Bayesian inference. In fully observed directed models, the factorization can simplify estimation into local problems. Unobserved variables make learning harder because their possible values must be accounted for. The expectation–maximization algorithm alternates between inference about hidden variables and parameter updates. (cs.cmu.edu)

Structure learning also estimates the graph. Methods may compare candidate structures using statistical scores or examine conditional independence relationships. This introduces a combinatorial search problem in addition to parameter estimation. Learning and inference are consequently intertwined: evaluating a candidate model or updating its parameters may itself require probabilistic inference. (cs.cmu.edu)

Model families and applications

A hidden Markov model represents a sequence through hidden states and associated observations. Other graphical models describe dependencies among image regions, biological variables, or symbols transmitted through noisy communication channels. Applications include speech processing, computer vision, bioinformatics, robotics, and error-correcting codes. The choice of graph and local distributions reflects the structure of the problem; it also determines which inference and learning procedures are practical. (research.tue.nl)

References

Graphical models, exponential families, and variational inferencestat.berkeley.edu
Lecture 1: Introduction to Graphical Modelscs.cmu.edu
Bayesian Networksftp.cs.ucla.edu
Graphical Models and Inference Algorithmscs.cmu.edu
Extending Factor Graphs so as to Unify Directed and Undirected Graphical Modelsarxiv.org
Graphical Models, Exponential Families, and Variational Inferencecs.columbia.edu
An Introduction to Variational Methods for Graphical Modelspeople.eecs.berkeley.edu
Lecture 14: Inference and Learningcs.cmu.edu
Introduction to probabilistic graphical modelsresearch.tue.nl