SHAP & LIME Explainability

Write the explanation contract before choosing SHAP or LIME

The method is only one part of the system. First define the question, audience, semantics, evidence threshold, and action the explanation is allowed to support.

A production contract names seven things

1. User and decision: engineer debugging a case, reviewer investigating an outcome, customer receiving a notice, or model owner monitoring a cohort
2. Explained output: model version, feature-pipeline version, output name, class index, and output scale
3. Reference semantics: background records, cohort filters, weighting, missing-feature rule, and dependence assumption
4. Interpretable representation: raw field, engineered feature, grouped concept, token, phrase, superpixel, or another human-facing unit
5. Estimator configuration: SHAP variant or LIME implementation, seed, sample count, kernel, sparsity, and numerical tolerance
6. Validation policy: fidelity, reconciliation, seed stability, background sensitivity, critical slices, and human review
7. Allowed action: investigate, monitor, communicate, contest, or propose an experiment; never silently turn attribution into causal recourse

SHAP distributes a prediction difference

For a local additive explanation, SHAP represents the explained model output as f(x) = base value + sum(feature contributions). The base value is an expected output under the chosen background and semantics. Each SHAP value is one feature's allocated share of the difference from that base.

Choose the SHAP algorithm that matches model access

• TreeSHAP: exploits tree structure and is usually the first choice for supported tree ensembles.
• Linear SHAP: uses linear-model structure; feature dependence still changes the interpretation.
• Deep or gradient variants: approximate attributions for differentiable networks and require architecture-specific validation.
• Kernel or permutation methods: query an arbitrary prediction function but can require many evaluations and produce sampling variance.
• Grouped explanations: combine related encoded features into a human concept when individual columns would split meaning or reveal internals.

LIME builds a weighted local surrogate

LIME does not decompose the black-box model directly. It samples perturbed examples, evaluates the black box, weights those examples by proximity to the case, and fits a simple interpretable model to that neighborhood.

1. 1

   Map

   Choose an interpretable representation

   Define what can be switched or varied: tabular bins, tokens, phrases, superpixels, or domain-specific groups.

2. 2

   Sample

   Generate plausible neighbors

   Perturb the case without creating impossible people, transactions, text fragments, or images that the model never encounters.

3. 3

   Approximate

   Weight and fit locally

   Query the black box, apply the proximity kernel, and fit a sparse linear model or another deliberately simple surrogate.

4. 4

   Validate

   Measure the approximation

   Report weighted local fidelity, seed stability, neighborhood coverage, coefficient signs, and the configuration that produced them.

LIME has several independent failure controls

• Neighborhood realism: perturbed records must respect types, constraints, feature relationships, and the production support.
• Kernel width: too narrow yields too little evidence; too wide makes the explanation less local.
• Representation: token deletion, category bins, and engineered features ask materially different questions.
• Sparsity: a short explanation is easier to read but can omit interacting or offsetting features.
• Sampling: low budgets or unlucky seeds can change selected features and coefficient signs.
• Fidelity: the surrogate must reproduce black-box behavior in the weighted neighborhood before its coefficients are interpreted.

Treat SHAP and LIME as different evidence contracts

Neither method wins universally. Match the estimator to model access, explanation scope, latency, and the validation burden the organization can operate.

Validate explanation payloads as production artifacts

An explanation record needs enough metadata and arithmetic evidence to be reproduced after the model, feature store, or explainer library changes.

Persist the evidence bundle

• request and case identifier under the appropriate retention and access policy;
• model, calibration, feature-pipeline, schema, and label versions;
• explainer implementation and library version;
• reference dataset manifest or cohort-query version;
• method parameters, seed, sample count, kernel, and feature groups;
• prediction, base value, contributions, local fidelity, reconciliation gap, and stability metrics;
• audience, presentation template, policy decision, reviewer action, and expiry time.

Do not use feature values as cache keys. Cache only when the model, feature pipeline, reference, explainer configuration, output scale, and authorization context are identical.

Correlation, interaction, and causality need separate treatment

Attribution methods allocate model behavior. They do not identify an intervention unless causal assumptions and evidence are added.

Use recourse language carefully

• State what the model used, not what caused the real outcome.
• Distinguish immutable facts, controllable actions, administrative corrections, and model artifacts.
• Do not suggest manipulating a proxy when the underlying decision policy would not recognize that change.
• Test whether proposed actions remain valid after model, policy, and data updates.
• Provide a contest or human-review path when an explanation affects a person.

Build an explanation service around versioned evidence

The algorithm belongs inside a controlled system. Separate online prediction from heavier explanation generation so an expensive query cannot overload the decision path.

Bound the operational path

• Rate-limit explanation requests separately from predictions and require an authorization purpose.
• Queue expensive cohort or sampled jobs; reserve synchronous work for a profiled bounded method.
• Pin model and background artifacts for the full job so a rollout cannot mix versions mid-explanation.
• Redact sensitive inputs, internal fraud rules, security features, and model details from audiences that must not see them.
• Use signed evidence manifests, immutable audit events, expiry, deletion workflows, and reproducible reruns.
• Fall back to a plain decision notice or human review when the explainer times out or fails its gate; never show an unvalidated partial ranking.

Gate explanation releases independently from model releases

A model can pass predictive evaluation while its explanations are unstable, misleading, too slow, or unsafe to disclose. The explanation policy therefore needs its own blocking criteria.

Monitor after launch

• explanation latency, queue depth, timeout rate, cache hit rate, and model-query volume;
• fidelity, reconciliation, sign agreement, rank overlap, and failed-gate rate by model and slice;
• base-value and attribution-distribution drift against input and prediction drift;
• unknown feature versions, missing references, mixed model versions, and reproducibility failures;
• sensitive-feature disclosure, unauthorized requests, scraping patterns, and suspicious probing;
• reviewer overrides, user contests, action outcomes, and cases where explanations increased confidence in a wrong decision.