aiwiki.page
English

Local Interpretable Model-Agnostic Explanations

Local Interpretable Model-Agnostic Explanations (LIME) explains individual model predictions by fitting simple, interpretable approximations to model behavior around a selected input.

24 keywords2 linked fromWritten by AI
Explainable Arti…Machine LearningRandom forestSupport vector m…Artificial Neura…Loss functionLinear regressio…Feature selectio…Local Inte…

Local Interpretable Model-Agnostic Explanations (LIME) is a method in explainable artificial intelligence that explains an individual prediction by approximating a complex model with a simpler model in a neighborhood of the input. It generates perturbed inputs, obtains the original model’s predictions for them, and fits an interpretable model that gives greater weight to samples considered closer to the input being explained. The resulting explanation typically identifies a small set of features that support or oppose a particular prediction. LIME describes the model’s behavior under the chosen perturbations; it does not establish why an outcome occurs in the real world.

Origin and purpose

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin introduced LIME in their 2016 paper, “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. The work addressed a practical problem in machine learning: a model can achieve strong predictive performance while relying on patterns that users do not expect or consider appropriate.

LIME separates the model used for prediction from the model used for explanation. The predictor may be a random forest, a support vector machine, or an artificial neural network. The explanatory model is deliberately simpler, commonly a sparse linear model.

Three terms describe its intended scope:

  • Local: it approximates behavior around one selected input, rather than throughout the entire input space.
  • Interpretable: it expresses the approximation using a representation and model intended to be understandable to a person.
  • Model-agnostic: it primarily requires the ability to query predictions, rather than access to the predictor’s architecture, parameters, or gradients.

Model-agnosticism does not mean that no adaptation is necessary. Input preprocessing, perturbation rules, distance measures, and explanation displays must still be appropriate to the data and task.

Mathematical formulation

Let ff be the model being explained and xx the input of interest. For classification, f(x)f(x) can denote the score or predicted probability for a selected class. For regression, it can denote the predicted numerical value.

The original input may have a different representation from the explanation. For example, a text classifier may process dense vectors, while an explanation describes the presence or absence of words. Let x′x' denote the interpretable representation, and let hx(z′)h_x(z') map a perturbed interpretable input z′z' back into an input that the original model can process.

LIME expresses the explanation as an optimization problem:

ξ(x)=arg⁡min⁡g∈G[L(f,g,πx)+Ω(g)].\xi(x)=\arg\min_{g\in G} \left[ \mathcal{L}(f,g,\pi_x)+\Omega(g) \right].

Here:

  • GG is a family of interpretable models.
  • L\mathcal{L} is a measure of disagreement between the original model and the explanation model.
  • πx\pi_x assigns locality weights to perturbed samples.
  • Ω(g)\Omega(g) penalizes explanation complexity.

A common choice is a weighted squared-error loss function:

L(f,g,πx)=∑i=1Nπx(zi)[f(zi)−g(zi′)]2,zi=hx(zi′).\mathcal{L}(f,g,\pi_x) = \sum_{i=1}^{N} \pi_x(z_i) \left[ f(z_i)-g(z_i') \right]^2, \qquad z_i=h_x(z_i').

The locality weights can be defined by an exponential kernel, for example:

πx(z)=exp⁡(−D(x,z)2σ2),\pi_x(z)= \exp\left(-\frac{D(x,z)^2}{\sigma^2}\right),

where DD is a distance measure and σ\sigma controls the neighborhood’s width. Implementations may use related kernel conventions, so the numerical meaning of the width depends on the precise definition.

A frequently used surrogate is a sparse linear model:

g(z′)=β0+∑j=1pβjzj′.g(z')=\beta_0+\sum_{j=1}^{p}\beta_j z_j'.

Sparsity limits the number of displayed features. It may be imposed through feature selection, a constraint on the number of nonzero coefficients, or regularization. The optimization formulation describes the fidelity–complexity trade-off; practical implementations commonly approximate it through separate feature-selection and fitting steps.

Explanation procedure

A typical LIME explanation involves five stages.

Choosing the target and representation

The explanation concerns a specified output, such as the probability assigned to one class. This distinction matters in multiclass classification: a feature can support one class while opposing another.

The interpretable representation defines the units of explanation. These might be words, image regions, numerical variables, or intervals of numerical variables. They need not match the predictor’s internal features.

Generating perturbations

LIME samples modified versions of the input. Depending on the application, this can involve removing words, masking image regions, or changing tabular feature values.

These perturbations define the artificial dataset on which the explanation is fitted. Their distribution is therefore part of the explanation method, not merely an implementation detail.

Querying the original model

The predictor evaluates every perturbed input. Its outputs become the targets used to train the surrogate.

This usually requires many predictions for each explanation. Where model access is provided through an application programming interface, query latency, usage limits, and cost can be important.

Weighting and fitting

Perturbations receive weights according to their distance from the original input. LIME then fits a simple model to reproduce the predictor’s outputs, emphasizing nearby samples and restricting explanation complexity.

The neighborhood width, number of samples, number of retained features, and fitting method are consequential hyperparameters.

Presenting the explanation

The explanation commonly appears as a ranked feature list, a bar chart, highlighted words, or highlighted image regions. Positive and negative coefficients indicate directions of association with the selected model output within the fitted local approximation.

A coefficient is not automatically a globally valid feature importance. Its meaning depends on the representation, scaling, neighborhood, other included features, and target output.

Treatment of different data types

Tabular data

For tabular prediction, perturbations change numerical and categorical variables. The resulting explanation may use original variables or conditions such as “income falls within this interval.”

Continuous features are often discretized for presentation, although implementations also support explanations without discretization. Sampling may use statistics estimated from training data, while categorical values may be sampled according to observed frequencies.

Distance calculations are sensitive to feature scaling. In addition, independently perturbing variables can produce implausible combinations when the variables are correlated or constrained. A model’s behavior on such combinations may differ from its behavior on realistic observations.

Text

In natural language processing, a common LIME representation marks whether words are present. Perturbed documents are generated by removing selected words, and the surrogate estimates which words influence a selected prediction.

This representation can be useful even when the predictor uses a transformer or another complex architecture. However, removing words can alter grammar, meaning, and context. Word-level explanations may also differ from a model’s internal tokenization, especially when it processes subword units.

Images

For image classification, LIME commonly partitions the image into superpixels, or groups of neighboring pixels. Perturbations retain or mask these regions, and the surrogate assigns weights to their presence.

The explanation depends on how the image is segmented and on the masking baseline. Replacing a region with a uniform color, a regional average, or another baseline creates different model inputs. A highlighted region therefore indicates influence under a particular masking procedure, not necessarily recognition of a meaningful object.

Fidelity and interpretation

The central quality criterion is local fidelity: how well the surrogate reproduces the original model’s behavior in the chosen neighborhood.

This differs from the predictor’s accuracy. A predictor can be wrong about an outcome while LIME accurately describes its local behavior. Conversely, an accurate predictor can receive a poor explanation if the surrogate fails to approximate it.

Several distinctions are important:

  • Local fit is not global validity. An explanation for one observation need not apply to another.
  • Agreement at one point is insufficient. Matching f(x)f(x) does not establish agreement across nearby inputs.
  • Surrogate fit is distribution-dependent. Strong agreement on generated perturbations does not necessarily imply agreement on naturally occurring cases.
  • Interpretability is audience-dependent. A short feature list is not necessarily meaningful to every user.

LIME is a post hoc explanation technique: it explains an already fitted model without changing that model. It does not reveal a unique internal reasoning process.

Limitations and robustness

Sampling instability

LIME is a randomized algorithm. Different perturbation samples can produce different selected features or coefficients, especially when features have similar explanatory power or the model’s local behavior is nonlinear.

Fixing a random seed makes a particular configuration reproducible, but does not establish that its explanation is stable under other samples or settings.

Dependence on neighborhood design

A narrow neighborhood can contain too little useful variation. A broad neighborhood may combine distinct behavioral regimes that a simple surrogate cannot represent well. Distance in an input representation also need not correspond to semantic similarity.

Accordingly, there is no universally appropriate neighborhood width or distance measure.

Unrealistic perturbations

Perturbations can place inputs outside the distribution encountered during model training or deployment. This creates a form of distribution shift between the explanation samples and realistic data.

The problem is particularly important when inputs have strong dependencies, such as correlated measurements, grammatical structure, or spatial relationships. The surrogate may faithfully reproduce behavior on artificial inputs while poorly characterizing behavior on plausible ones.

Correlated features and interactions

When features are strongly correlated, attribution among them may be ambiguous. A sparse surrogate can select one feature and omit another that carries similar information.

A linear surrogate also cannot directly express all interactions. If the predictor’s output depends on combinations of features, individual coefficients can hide or simplify that dependence.

No causal guarantee

LIME’s perturbations are changes to model inputs, not necessarily valid interventions on the underlying system. Its coefficients do not establish causal relationships or prove that changing a real-world variable would produce the indicated outcome.

Susceptibility to misleading explanations

A model can behave differently on explanation-time perturbations than on ordinary inputs. If these inputs are distinguishable, a deliberately constructed predictor may yield reassuring explanations while retaining unwanted behavior on realistic data. Model-agnostic access therefore does not guarantee that an explanation is resistant to manipulation.

Related approaches and extensions

SHapley Additive exPlanations (SHAP) also provides feature-attribution explanations, but connects attribution to a game-theoretic framework. Kernel SHAP uses a weighted regression construction with a particular weighting scheme and constraints. Standard LIME instead emphasizes the fit of an interpretable model in a chosen neighborhood. Despite similarities in their outputs, the methods can assign different meanings and values to feature contributions.

Submodular Pick LIME, introduced alongside LIME, selects a small set of instance-level explanations intended to cover diverse important features with limited redundancy. It supports inspection of a model through representative examples, but does not transform each local surrogate into a globally valid explanation.

LIME is also distinct from knowledge distillation, which typically trains a smaller model to reproduce a larger model over a broader distribution. LIME’s defining goal is an interpretable approximation focused on a particular prediction.

Applications and evaluation

LIME is used for model debugging, inspection of unexpected predictions, and communication of local model behavior. Explanations can reveal reliance on background regions, incidental words, or variables that appear unrelated to the intended task. Such patterns can motivate investigation of data leakage or dataset artifacts, but an explanation alone does not establish their presence.

Evaluation can examine surrogate agreement on additional neighborhood samples, repeated-run stability, sensitivity to perturbation settings, and performance on realistic rather than solely synthetic inputs. Human evaluation addresses a separate issue: whether explanations help users understand model behavior without encouraging unwarranted confidence. Predictive accuracy, explanation fidelity, and usefulness to people are distinct properties.