• When and why to use explanations in AI systems
• The difference between faithful vs plausible explanations
• What recent research says about LLM self-explanations, SHAP, integrated gradients, and counterfactuals
• How to avoid misuse and design actionable, robust explanations for business contexts
• How to critically evaluate explanation outputs presented to you

• The following terms are often tossed around as if they’re synonymous. Let’s try to stick to these definitions:
  • Explainability: Describes how the model works and what it ‘knows’. This is usually expressed in intrinsic terms of the model itself.
    • E.g., “This model represents the notion of ownership primarily (82%) through layers 5-8”
  • Interpretability: Describes why a model made a particular decision. This is usually expressed in extrinsic terms of the data sample.
    • E.g., “This data point was classified as `reject’ because the credit score was < 660 and the annual income was < $35K”
  • Transparency: Describes the ‘openness’ of the entire process, including explicitly sharing details such as 1) training data, 2) source code, 3) training regimen, …
    • E.g., “This model was trained on the MIMIC dataset using code here”

• A model whose internal logic is opaque, even to its developers
  • e.g., deep neural networks, ensemble models like XGBoost
• Can be highly accurate but difficult to explain
  • “We know what goes in, we know what comes out — but not what happens in between”

These are not universally-acknowledged truths.

🤔 Where in your industry would you accept a small performance hit in exchange for clarity?

• Caruana et al. (2015) studied 14,199 pneumonia patients
  • ICD-9-CM principal diagnosis of pneumonia at admission
  • 10.86% died. Bagging is used to ‘avoid overfitting’.
  • A single (😳) 70:30 train:test split was used…
• 46 features, \(\color{green}{f_j}\), extracted, e.g.,
  • Patient history: chronic lung disease (+/-), admitted to ER (+/-), age (ℤ?)
  • Physical exam: heart rate (ℝ?), diastolic blood pressure (ℝ?)
  • Lab findings: potassium level (ℝ?), sodium level (ℝ?)
  • X-rays: pleural effusion, positive chest x-ray
    

• Given a data set with \(N\) instances, \(\mathcal{D}=\{(x_i, y_i)_1^N\}\), a standard GAM has the form

• where \(g(\cdot)\) is the link function, and “for each term \(\color{green}{f_j}\), \(E[\color{green}{f_j}]=0\)”.
  • Logistic regression is a special form of GAM where each \(\color{green}{𝑓_𝑗}\) is linear.
• To improve accuracy, pairwise interactions can be added:

• Sort features by ‘importance’
  • Sec 5.3: ask an expert to rank them for you, or rank by “drop in AUC when the term is removed”
  • Better way (?): filter method, i.e., statistical tests of significance.
• Plot those features in terms of their ability to predict the outcome.
  • Green bars are ±1 standard deviation of the variation in the risk score (\(y\)-axis) measured by 100 rounds of bagging.

• Local Interpretable Model-agnostic Explanations (LIME) explains a single prediction by fitting a simple, human-readable model around that instance.
• Core idea: sample perturbed neighbours of \(x_0\), weight them by proximity, and fit a sparse linear surrogate whose coefficients are shown as the “explanation.”
• Works across tabular, text, and image models via an interpretable representation (e.g., words present/absent, image superpixels on/off).
• Useful for debugging, spot-checking models, and building trust (with caveats)

📚 (Ribeiro, Singh, and Guestrin 2016)

1. Quantify fidelity: report surrogate \(R^2\) / loss on the weighted neighbourhood.
2. Stability checks: rerun with different seeds; vary kernel width; report variance of top-\(k\)
3. Neighbourhood sanity: visualize distance histogram of sampled points; ensure they’re actually near \(x_0\).
4. Feature dependence: test with/without correlated features; confirm the story doesn’t flip.
5. Don’t treat as causal: pair with counterfactual tests or anchors (here) for decision support.
6. Model comparison: use SP-LIME to compare candidates by feature-coverage, not by one screenshot.

(Ribeiro, Singh, and Guestrin 2016; Garreau 2020)

1. LLM self-explanations aren’t reliably faithful
   • treat rationales as UI, not audit evidence.
2. SHAP can mislead on feature importance under common conditions
   • require sanity/stability tests.
3. Counterfactuals must be robust/actionable
   • prefer ranges + recertification and feasibility checks.
4. Human studies favour region-based recourse
   • design explanations people can actually use.

• SHapley Additive exPlanations
• Uses game theory to assign each feature a contribution score
• Often described as fair, consistent, and model-agnostic
• Works across model families via different explainers
• 📚 (Lundberg and Lee 2017) ; 🛠️ code

For an instance \(x_0\), each feature \(i\) receives a Shapley value \(\phi_i\) summarizing its average marginal contribution across feature coalitions: \[ \phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!\,(|F|-|S|-1)!}{|F|!}\; \Big(f_{S \cup \{i\}} - f_S\Big) \]

• In practice, SHAP implements this with model-specific or model-agnostic approximations.

• TreeExplainer (trees/ensembles; fast, exact/approx)
• KernelExplainer (model-agnostic; local weighted regression around \(x_0\))
• DeepExplainer / GradientExplainer (neural nets; gradient/DeepLIFT-style)
• LinearExplainer (linear models with correlated features)

Keep in mind: Explainer choice affects speed, variance, and assumptions.

• Global: beeswarm plots (distribution of \(\phi_i\) across many points); dependence plots (feature value vs SHAP value)
• Local: horizontal bar chart for a single prediction (top-\(k\) features)

Is:

• A principled feature attribution method; helpful for debugging and communicating local/global patterns.

Isn’t:

• Causal by default. A large \(\phi_i\) does not imply that intervening on feature \(i\) will change the outcome in the real world.

• Correlation & interactions: attributions can shift among correlated features.
• Off-manifold perturbations (KernelExplainer): unrealistic samples → spurious stories.
• Parameter sensitivity: kernel width, background data, and random seeds move explanations.
• Adversarial camouflage: models can be trained to look benign under SHAP while retaining harmful logic.

Keep Your Guard Up!

• Treating SHAP bars as causal levers
• Presenting SHAP plots as regulatory compliance without sanity/stability checks
• Over-interpreting attractive visuals without robustness testing

1. Background data matters: choose a representative background; state how you chose it.
   
2. Stability checks: vary kernel width/seed/background; show variance in top-(k) features.
   
3. Sanity checks: randomize labels/features → explanations should collapse to noise.
   
4. Correlation checks: test subsets / condition on key covariates; compare with counterfactual probes.
   
5. Pair with recourse: don’t present bars alone; accompany with validated, range-based recourse.
   
6. Document assumptions: explainer, version, data slice, valid-until.

• SHAP is model-agnostic and game-theoretic; great for trees & tabular data.
  
• For deep neural networks, gradient-based methods can be more direct and efficient on the model’s native representation.
  
• Integrated Gradients (IG) attributes by integrating the gradient along a path from a baseline \(x'\)to the input \(x\).
  
• IG brings different assumptions & pitfalls (e.g., baseline choice); a useful contrast with SHAP.

Definition Sundararajan, Taly, and Yan (2017): \[ IG_i(x) \;=\; (x_i - x'_i)\, \int_{0}^{1} \frac{\partial f\!\left(x' + \alpha(x - x')\right)}{\partial x_i}\, d\alpha \]

• Pros: faithful to gradients, efficient, no coalitions.
  
• Cons: baseline-dependent; can be sensitive to path choice & saturation.
  
• Use when: deep nets on text/images/tabular embeddings; pair with the same stability/sanity habits you apply to SHAP.

• SHAP: game-theoretic, model-agnostic, but unstable with correlations
  
• IG: gradient-based, deep-learning oriented, baseline-dependent
  
• Both risk misleading attributions if applied without care

• Problem: A counterfactual \(x'\) flips the prediction for \(x\) now, but should also remain valid under model updates, input noise, and distribution shift.
• Model-set view: Let \(\mathcal{M}\) be a set of plausible future/retrained models. Robust validity can be required as:
  • Hard (for all models): \[ \forall\, M \in \mathcal{M}:\; f_M(x') = y^\star \]
  • Probabilistic (likely across models): \[ \Pr_{M \sim \mathcal{M}}\!\big[f_M(x') = y^\star\big] \;\ge\; \tau \]
• Why standard counterfactuals fail: optimized for a single snapshot (M_t); sensitive to correlated features, local boundary shifts, and off-manifold proposals.

Jiang et al. (2024) formalizes robustness, failure modes, and evaluation.

• Evaluation metrics (common in robust-CE studies):
  • Proximity of \(x'\) to \(x\): \(\ell_1\), \(\ell_2\)
  • Plausibility: outlier scores (e.g., Local Outlier Factor), data-manifold checks
  • Robust validity: fraction of retrained models \(M \in \mathcal{M}\) where \(f_M(x') = y^\star\)
• Method families highlighted by Jiang et al.:
  • Provable robustness via interval abstractions (verify \(\Delta\)-robustness; MILP-based checks) and construct CEs guaranteed to remain valid within certified perturbation sets.
  • Ensemble / model-set optimization: choose \(x'\) that maximizes worst-case or expected validity across \(M \in \mathcal{M}\).
  • Regularized objectives: multi-objective trade-off between proximity, sparsity, plausibility, and robustness.

VanNostrand et al. (2024) ran a between-subjects user study (\(N = 252\)) to examine the effects of counterfactual explanation types .

• Region-based counterfactuals improve user understanding and confidence
• Visual vs text formats affect speed and confidence, but not comprehension
• Simple interfaces reduce cognitive load

Qualitative themes (why users liked regions):

• Ranges give room to maneuver and feel more realistic; make trade-offs explicit.
  
• Users asked for justification: “Why these ranges?” → desire for provenance/version & evidence that the ranges are reliable.