SHapley Additive exPlanations (SHAP) is a framework for explaining predictions made by machine-learning models. It assigns each input feature a numerical contribution to a particular prediction, relative to a specified reference output. These contributions are based on the Shapley value, a solution concept from cooperative game theory that distributes a coalition’s total value among its participants. In a prediction explanation, the participants are features, and the value being distributed is a difference in model output.
Introduced under the SHAP name by Scott M. Lundberg and Su-In Lee in 2017, the framework connects several approaches to feature attribution through an additive explanation model. It is widely used in explainable artificial intelligence because it combines a precise allocation rule with practical algorithms and visualizations. SHAP can explain an individual prediction, support comparisons across many observations, and reveal patterns in how a fitted model uses its inputs.
A SHAP explanation is not determined by the prediction function alone. It also depends on which features count as participants, how unavailable features are represented, which reference population is used, and which output scale is explained. These choices distinguish a mathematical attribution from a causal explanation and determine what conclusions an explanation can support.
Origins and conceptual foundations
The mathematical foundation of SHAP predates machine learning. In 1953, Lloyd S. Shapley introduced a value for cooperative games: settings in which participants form coalitions and each coalition has an associated numerical worth. His allocation rule averages each participant’s marginal contribution over every possible order in which participants could join. This provides a principled way to share gains when contributions depend on cooperation rather than on isolated actions.
For prediction explanation, a feature may have little effect alone but a large effect when combined with another feature. A model might, for example, use a combination of account age and transaction amount to assign a fraud score. Allocating the resulting prediction difference requires deciding how to distribute their joint contribution. The Shapley rule resolves this allocation by considering both features in all possible coalition contexts, rather than privileging a single order of entry.
Applications of Shapley values to prediction explanation appeared before the SHAP framework. Erik Štrumbelj and Igor Kononenko developed sampling-based approaches to explaining individual predictions using game-theoretic feature contributions. Separately, methods for interpreting complex predictors developed through local surrogate models, feature perturbations, and backpropagation-based attribution. Local Interpretable Model-agnostic Explanations, introduced by Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin in 2016, emphasized fitting an interpretable approximation around an individual prediction.
Lundberg and Lee’s 2017 paper, “A Unified Approach to Interpreting Model Predictions,” organized several attribution methods within a common class of additive feature explanations. It presented SHAP values as the solution satisfying specified explanatory properties and introduced Kernel SHAP as a model-agnostic estimation method. The framework also connected approaches such as LIME and DeepLIFT, while distinguishing their assumptions and approximation procedures. The claim was not that every existing attribution method already computed exact Shapley values, but that their relationships could be analyzed through a shared explanation model.
Subsequent work made the framework more practical for particular model families. Efficient algorithms for tree ensembles were especially important because gradient boosting and random forests are widely used on structured data. Lundberg and colleagues’ 2020 paper, “From local explanations to global understanding with explainable AI for trees,” developed the connection between individual tree-model explanations and broader model analysis. The name SHAP now refers both to the conceptual framework and to an open-source software ecosystem containing explainers, masking procedures, explanation objects, and plotting tools. Those implementations do not all calculate the same underlying game or use the same degree of approximation.
Mathematical definition and allocation properties
Let be a fitted prediction function, the observation to be explained, and the set of its features. A coalition value specifies the output associated with knowing or retaining features in . The Shapley attribution for feature is
The term in brackets is feature ’s marginal contribution to coalition . Its coefficient accounts for how often that coalition precedes in a uniformly random ordering of all features. Equivalently, is the average contribution when feature is revealed in every possible position in every possible feature ordering.
SHAP uses an additive explanation model of the form
where indicates whether a feature is represented as present in the simplified explanation. With , the Shapley efficiency property gives
When the complete coalition recovers the prediction, this becomes . Thus, the attribution values account exactly for the difference between the prediction and its baseline, subject to numerical approximation in the implementation.
The classical characterization involves efficiency, symmetry, the dummy-player property, and additivity across games. Symmetry assigns equal credit to features with identical marginal contributions. A dummy feature receives zero attribution if adding it never changes any coalition’s value. Additivity means that allocating the sum of two games gives the sum of their allocations. The original SHAP formulation emphasized related explanation properties called local accuracy, missingness, and consistency. Consistency requires that a feature’s attribution not decrease when all of its marginal contributions increase or remain unchanged. These properties characterize the allocation for a specified game; they do not determine the appropriate game for every application.
A two-feature example shows how interactions are allocated. Suppose , , , and . Feature 1 receives , while feature 2 receives . The isolated gains are 4 and 2, leaving an additional joint gain of 4 that is shared equally. The explanation therefore reads . This is an allocation of the specified model-output difference, not evidence that changing either real-world variable would produce the same numerical effect.
Reference distributions and the meaning of missing features
A fitted model ordinarily expects a complete input. SHAP therefore requires a rule for evaluating coalitions with some features absent. This is the central modeling choice behind an explanation. An absent feature in the explanation is not necessarily a missing value in the original dataset: it is a feature deliberately withheld from a hypothetical coalition, even if its observed value is known.
One choice uses a conditional expectation:
Here, the distribution of unavailable features changes according to the observed coalition. This approach preserves statistical relationships if the conditional distribution is correctly represented. For example, knowing one measurement may provide information about another correlated measurement, even before the second measurement is revealed. Consequently, a feature not explicitly used by the model can receive an attribution because it conveys information about features that the model does use. This does not violate the dummy property: that property concerns marginal contributions in the coalition game, not merely whether a variable appears in the model’s source code.
A different construction replaces unavailable features using a reference distribution:
where . Observed coordinates are fixed at their values in , while the remaining coordinates are drawn from the background population. Drawing the absent coordinates jointly can preserve dependencies within that absent block, but it breaks dependencies between the fixed and sampled blocks. In tree-explainer terminology, a commonly used form is called “interventional.” That label alone does not make its attributions causal effects.
The distinction can be substantial. Suppose a model uses , while closely tracks . Under conditional attribution, revealing can change the expected prediction, so it may share credit with . Under marginal replacement, a model that is functionally independent of gives it zero marginal contribution. Conversely, marginal replacement may generate combinations far from the observed data, such as incompatible measurements of the same physical quantity. The two constructions answer different questions about model use and available information.
The background population also determines the comparison. Explaining an observation relative to an entire dataset differs from explaining it relative to a particular region, time period, or peer group. The baseline is typically an expected model output over that population, not necessarily the observed outcome average. Changing the background can alter both the baseline and individual contributions without changing . For images and text, masking introduces further choices: blank patches, blurred regions, replacement tokens, and generated completions define different counterfactual inputs and therefore different coalition values.
Model-agnostic estimation and computational cost
Exact evaluation is expensive for a general prediction function. With features there are coalitions, and a coalition’s value may itself require averaging over many background observations. At 30 features, the coalition count already exceeds one billion. Consequently, practical model-agnostic explainers usually estimate Shapley values through selected model evaluations rather than enumerate the complete game.
Kernel SHAP recasts the calculation as a weighted linear regression on binary coalition indicators. For an intermediate coalition , the Shapley kernel is
The regression fits the coalition outputs with an additive model, while the empty and complete coalitions provide endpoint constraints. With all coalition values available and the appropriate constraints, the fitted coefficients recover the Shapley allocation. In practice, selected coalitions and finite background samples approximate this calculation. The term “kernel” here denotes the weighting function; it does not refer to a kernel method used to train the predictor.
Coalition sampling and background sampling introduce distinct sources of error. Increasing the number of sampled coalitions improves approximation of the allocation across feature subsets. Increasing the background sample improves estimation of the coalition values under the chosen reference distribution. Neither operation resolves a poorly chosen definition of feature absence. Feature selection or sparsity-inducing regression options can also affect finite-budget explanations, particularly when only a small number of model evaluations are available.
Permutation estimators use the equivalent random-order definition. An ordering is sampled, features are introduced sequentially, and each feature receives the change in coalition value at its entry. Averaging over orderings estimates its Shapley value. If a complete path is evaluated, the increments telescope to the endpoint difference. Reversing or pairing permutations can reduce sampling variability. Practical cost depends on the number of orderings, background observations, output dimensions, and opportunities to batch predictions.
High-dimensional inputs often require grouping. Image explanations may treat regions rather than individual pixels as players; text explanations may use tokens, words, or spans. Groups can also represent related columns in a tabular dataset. Grouping changes the allocation problem: the Shapley value of a group treated as one player does not generally equal the sum of its members’ values in the original, ungrouped game. Hierarchically constrained coalition methods may instead compute Owen values, which respect a specified group structure. Reports of runtime or accuracy therefore need to identify the players, masking rule, evaluation budget, and allocation being estimated, rather than simply describing a method as “SHAP.”
Specialized algorithms for trees, linear models, and neural networks
Model-specific explainers exploit a predictor’s internal structure to avoid some of the cost of generic perturbation. Their speed does not remove the need to specify a reference distribution or feature-dependence assumption. “Exact SHAP” means exactness for a particular coalition game, not a universal explanation independent of those choices.
Tree SHAP uses the structure of decision trees to accumulate feature contributions efficiently. A tree partitions inputs through a sequence of splits, and a tree ensemble sums or averages the outputs of many such trees. The algorithm tracks how coalitions can follow branches and combines their contributions with Shapley weights. A commonly cited complexity bound for the original algorithm is , where is the number of trees, bounds leaves per tree, and bounds tree depth. This avoids explicit enumeration of all feature subsets. Implementations support important ensemble systems, including XGBoost and LightGBM.
Tree explanations can use different rules for absent features. An interventional calculation averages against an explicit background dataset. A tree-path-dependent calculation uses information such as the training-sample counts passing through branches. The latter is computationally convenient, but those branch frequencies should not automatically be treated as an exact representation of the population conditional distribution. The same fitted tree can consequently receive different valid explanations under different supported assumptions.
For a linear model
marginal replacement yields
This closed form does not require independence among the input variables for that particular game. Conditional explanations are different: correlations can distribute information among features and require additional distributional assumptions or estimation. A simple formula for marginal attribution therefore does not imply that every linear-model SHAP calculation has the same interpretation.
For neural networks, Deep SHAP combines reference-based contribution propagation with Shapley-inspired rules. It builds on DeepLIFT, introduced by Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje in 2017. Rather than evaluate every feature coalition, it propagates differences between an input and reference inputs through network operations. This can be much faster than model-agnostic evaluation, but it is generally an approximation whose behavior depends on the supported operations and reference distribution.
Gradient-based explainers provide another efficient route for differentiable models. Expected gradients combines ideas from integrated gradients with averaging over reference inputs and positions along interpolation paths. Such methods exploit backpropagation and may be attractive for large networks, but path-based attributions and coalition-based Shapley values are not automatically identical. An explainer’s presence in the SHAP software library does not establish exact equivalence to the discrete Shapley formula for every model.
Reading explanations, global summaries, and interactions
A local explanation usually consists of a baseline, feature values, feature attributions, and an identified output. A waterfall plot starts from the baseline and adds positive and negative contributions until it reaches the prediction. A force plot presents a similar decomposition as opposing pushes. Positive means “raises the selected model output relative to the baseline”; it does not inherently mean beneficial, desirable, or likely to improve an outcome.
The output scale is essential. In binary classification, a model may expose a raw score, a log-odds margin, or a probability. Contributions that add to a raw margin do not generally add to the corresponding probability because the probability transformation is nonlinear. For logistic output, if the explained margin is , the probability is . Transforming each separately does not produce an additive probability decomposition. In multiclass classification, explanations also require specifying the class or output component.
A common global summary is mean absolute attribution:
It measures the typical magnitude of a feature’s contribution over the selected evaluation sample. A beeswarm plot adds the distribution of signed contributions, often coloring observations by feature value. These displays can show whether large feature values tend to increase or decrease predictions and whether a feature affects only a small subset of observations. The summary depends on the evaluation population and is not a universal property of the feature.
Dependence plots place a feature’s observed value on the horizontal axis and its attribution on the vertical axis. Vertical spread at a fixed value can reflect interactions, correlated features, or the chosen conditional game. Coloring points by a second feature may expose structure, but a visually separated pattern alone does not establish a causal interaction.
SHAP interaction values explicitly divide contributions into main and pairwise interaction terms. For distinct features and , define
A commonly used convention assigns
The matrix is symmetric, and diagonal entries are chosen so that . Under this convention, a pair’s combined contribution is represented across both symmetric off-diagonal cells. Pairwise matrices can be much larger than ordinary attribution tables, requiring storage proportional to the square of the feature count per observation. They also allocate aspects of higher-order behavior rather than proving that the model contains only pairwise interactions.
Uses in model development and operational analysis
SHAP is frequently used to inspect whether a fitted model relies on plausible inputs. In structured prediction tasks, local explanations can reveal reliance on unexpected identifiers, data-collection artifacts, or variables that encode outcomes indirectly. For example, a large contribution from a field created after the prediction time may indicate data leakage. The attribution identifies the model’s dependence under the chosen game; establishing leakage additionally requires checking when and how the variable was produced.
Aggregated explanations support comparison between models with similar predictive performance. Two models can have comparable scores on a test set while relying on different signals. One may concentrate contributions on a narrow set of unstable features, while another distributes them more broadly. Such comparisons are interpretable only when the models explain comparable outputs using compatible backgrounds and evaluation populations. A change in attribution magnitude can otherwise result from scale or reference changes rather than a substantive change in predictive behavior.
SHAP can also support feature engineering and feature selection. Dependence plots may expose saturation, thresholds, or combinations that a model has learned. Low mean absolute attribution can identify variables for further investigation. However, removing a low-attribution feature and retraining is a different experiment from withholding it in a fixed-model explanation. Redundant features can share credit, and a currently unimportant feature may become useful after another is removed. Attribution rankings therefore do not directly measure the performance loss caused by retraining without a variable.
In operational monitoring, attribution distributions can be compared across time periods or subpopulations. A sudden change in which features drive predictions may accompany distribution shift, a changed data pipeline, or a new pattern of user behavior. Such a change is not necessarily evidence of model degradation: the population may genuinely have changed while predictive performance remains adequate. Conversely, stable attribution summaries do not guarantee stable accuracy. Outcome-based evaluation and input-data monitoring remain separate sources of evidence.
For text and image models, explanations can highlight regions associated with a selected class score. A vision model might place large contributions on a background region rather than the object of interest; a text model might depend on formatting or recurring phrases. These findings can generate testable hypotheses about shortcuts. The interpretation depends strongly on tokenization, region segmentation, and masking: removing a word can alter sentence meaning, while replacing an image patch can create an artificial visual edge. A highlight should therefore be read as a response to a specified perturbation procedure, not as a literal account of human-like understanding.
Reliability, causal limits, and unresolved questions
A SHAP explanation attributes a model output, not the data-generating process. A variable can receive a large contribution because it predicts an outcome through correlation, selection effects, or an indirect proxy. This does not establish causation. Even the term “interventional SHAP” usually describes a feature-replacement operation; a causal interpretation additionally requires justified assumptions about interventions and the relevant causal structure.
Feature representation creates another limit. A single categorical variable may become several columns through one-hot encoding, while a physical quantity may be represented by multiple related measurements. Treating each column as a separate player can divide credit in ways that depend on the encoding. Introducing a duplicate or near-duplicate variable can redistribute attribution, especially under conditional games. Grouping related columns can make the question more meaningful, but it defines a different game rather than correcting every attribution automatically.
Finite computation introduces uncertainty distinct from model uncertainty. Monte Carlo error arises from sampled coalitions, permutations, and background records; fitted conditional distributions introduce estimation error; approximate propagation methods introduce method-specific approximation. Explanations can also vary when the training sample or fitted model changes. A small additivity residual checks whether contributions reconstruct the output, but it cannot establish that the background population, masking procedure, or estimated conditional relationships are appropriate.
Post hoc explanations can be misleading when their perturbations leave the domain on which the predictor was trained. In 2020, Dylan Slack and colleagues demonstrated attacks in which models behaved differently on ordinary observations and on perturbation-generated inputs, producing misleading LIME and SHAP explanations. This illustrates a distinction between accurately analyzing a selected set of model queries and establishing trustworthy behavior on the real deployment distribution. It does not invalidate the Shapley allocation theorem; it challenges the evidential meaning of the coalition outputs supplied to it.
Explanations also do not establish fairness, privacy, or regulatory compliance. A protected attribute with a small attribution can still be represented indirectly through other variables, and a large attribution does not by itself establish an impermissible decision process. Detailed explanations may disclose information about a model or its background data, but the extent depends on the interface and threat model. These questions require evidence beyond a feature-attribution table.
Research continues on efficient conditional sampling, uncertainty estimates, causal games, structured feature groups, and explanations for dependent observations such as time series. For large language models, even defining the output may be difficult: a single-token score, a sequence probability, a scalar evaluator, and a generated answer specify different explanation targets. Masking part of a prompt can alter both its semantic content and the distribution of subsequent text. Attribution in such settings therefore requires an explicit account of the prediction target, coalition unit, reference input, and evaluation procedure before numerical contributions can be meaningfully interpreted.