Eldor Abdukhamidov
Sungkyunkwan University | South Korea
Eldor Abdukhamidov received his B.S. from Inha University in Tashkent and pursued his Master's and Ph.D. degrees in Computer Science and Engineering at Sungkyunkwan University, South Korea. His research interests include image processing, machine learning, and information security. His guest lecture, "Weaknesses in Current Interpretation Methods of Deep Learning Systems," examines the vulnerabilities of the explanation techniques we trust and illustrates them through concrete attack examples.
## Learning Objectives
After completing this workshop, you will be able to:
- Distinguish transparency, interpretability, and explainability, stating what each one describes and why they are not synonyms.
- Contrast intrinsically interpretable models (linear models, decision trees, rule lists) with post-hoc methods applied to black boxes, and explain the trade-off each represents.
- Explain how LIME produces a locally faithful, model-agnostic explanation by perturbing an input and fitting a sparse linear surrogate to the black box's responses.
- Explain how SHAP attributes a prediction to features using Shapley values, and state the properties (local accuracy, missingness, consistency) that make its attribution unique.
- Describe gradient- and activation-based explanation signals for neural networks, saliency maps, Class Activation Mapping, and Integrated Gradients, and note where each is faithful and where it misleads.
- Explain why explanations are not automatically trustworthy: that they can be unstable under tiny input changes, and that explanation methods themselves can be fooled by an adversary.
- Connect explanation attacks to the adversarial perturbations of Workshop 3, and argue why explanation robustness is a design requirement for trustworthy AI systems.
## Theoretical Background
### Three Words That Are Not Synonyms
Discussions of trustworthy AI move quickly between "transparent," "interpretable," and "explainable," and treating them as one idea causes real confusion. They describe different things.
**Transparency** is the degree to which a system's workings are open and inspectable. It is a property of the *process and its documentation* as much as of the model: what data was used, how the model was trained, what its known limitations are, who is accountable, and whether outsiders can examine any of this. A fully documented pipeline around an opaque model can still be transparent in this sense, even if the model's internal computation is not itself legible.
**Interpretability** is the extent to which a human can understand the model's *mechanism*, the actual mapping from inputs to outputs. It is often an intrinsic property of the model class. A linear model is interpretable because its coefficients directly state how each feature moves the prediction; a short decision tree is interpretable because you can read the decision path; a rule list is interpretable because the rules are the model. Interpretability is understanding *how the model works*.
**Explainability** is the ability to produce a faithful account of *why an otherwise opaque model made a particular decision*, usually after the fact. When the model is a deep network whose mechanism no human can follow directly, we reconstruct a post-hoc explanation, an attribution over features, a heatmap over pixels, that approximates the reason for a specific output. Explainability is understanding *why a decision was made*, without necessarily understanding the mechanism that made it.
The three are related, better transparency and interpretability generally make trustworthy explanation easier, but they are not equivalent, and a system can have one without the others.
### Intrinsic Interpretability vs. Post-hoc Explanation
There are two broad routes to a legible model, and the choice between them is one of the central design decisions in this area.
**Intrinsically interpretable models** are transparent by construction. Linear and logistic regression, decision trees, and rule lists can be read directly: the model *is* its own explanation. When the stakes are high and the data permits, preferring such a model is often the most defensible choice, because there is no gap between the explanation and the mechanism.
**Post-hoc methods** take an already-trained black box, a deep network, a gradient-boosted ensemble, and construct explanations around it. They are what make otherwise opaque, high-performing models usable in settings that demand accountability. The catch, which the rest of this workshop develops, is that a post-hoc explanation is an *approximation* of the model's reasoning, not the reasoning itself, and the quality and honesty of that approximation cannot be assumed.
The methods below fall into two families by *scope*: **local** methods explain a single prediction (why *this* input got *this* label), while **global** methods describe overall model behavior across a dataset.
### LIME: Local, Model-Agnostic Surrogates
LIME, Local Interpretable Model-agnostic Explanations (Ribeiro, Singh & Guestrin, 2016), explains one prediction at a time and treats the underlying model as a black box it can only query. To explain a single instance, LIME perturbs that input many times, feeds each perturbed version to the black box, records the outputs, and then fits a simple, interpretable surrogate, typically a sparse linear model, to those local input-output pairs, weighting each sample by how close it is to the original instance. The surrogate is not meant to be globally accurate; it only has to be *locally faithful*, capturing the black box's behavior in the neighborhood of the instance being explained.
Formally, LIME chooses the explanation $g$ that balances local fidelity against simplicity:
$$
\text{explanation}(x) = \arg\min_{g \in G} \; \mathcal{L}(f, g, \pi_x) + \Omega(g)
$$
where $f$ is the original model, $g$ is the interpretable surrogate drawn from a class $G$ (for example, sparse linear models), $\pi_x$ weights samples by proximity to $x$, $\mathcal{L}$ measures how unfaithful $g$ is to $f$ in that neighborhood, and $\Omega(g)$ penalizes complexity so the explanation stays readable. Because it only needs to query outputs, LIME works with any model and any modality, images, text, or tabular data. Its weaknesses follow from the same design: the explanation depends on how the perturbations are drawn, and two similar inputs can yield noticeably different explanations, a fragility that becomes a security problem later in this workshop.
### SHAP: Attribution with Game-Theoretic Guarantees
SHAP, SHapley Additive exPlanations (Lundberg & Lee, 2017), also attributes a prediction to its features, but grounds the attribution in cooperative game theory. Treat the features as players in a game whose payoff is the model's output; the **Shapley value** of a feature is its average marginal contribution across all possible orderings in which features are added to the coalition. SHAP assigns each feature its Shapley value, producing an *additive* explanation in which the feature contributions sum to the difference between the prediction and a baseline.
$$
\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!\,(|F| - |S| - 1)!}{|F|!} \left[ f\big(S \cup \{i\}\big) - f(S) \right]
$$
Here $\phi_i$ is the attribution for feature $i$, $F$ is the full feature set, $S$ ranges over subsets that exclude $i$, and $f(S)$ is the model's output using only the features in $S$. What distinguishes SHAP is that Shapley values are the *unique* attribution satisfying three properties simultaneously: **local accuracy** (the attributions add up to the actual prediction), **missingness** (a feature that is absent gets zero credit), and **consistency** (if a model changes so that a feature contributes more, its attribution does not decrease). That uniqueness is SHAP's strength; its cost is computational, since exact Shapley values require evaluating exponentially many feature subsets, and practical implementations rely on approximations and a chosen background distribution.
### Explanation Signals for Neural Networks
For image models in particular, several methods explain a decision by pointing at the input regions responsible for it.
- **Saliency (gradient) maps** take the gradient of the class score with respect to the input pixels: pixels with large gradients are the ones whose change would most affect the score. They are fast and high-resolution, but often noisy and prone to saturation.
- **Class Activation Mapping (CAM)** produces a heatmap by combining the final convolutional feature maps with the classifier weights for a target class, localizing the regions that drove the prediction. It is intuitive and cheap but limited to architectures with global pooling.
- **Integrated Gradients** accumulates gradients along a straight path from a baseline input to the actual input, which satisfies a *completeness* axiom: the attributions sum exactly to the difference between the model's output on the input and on the baseline, so no contribution is lost or double-counted.
- **Attention weights** in transformer models are sometimes read as explanations, but attention is an imperfect signal: high attention does not reliably mean high importance, so it should be treated as suggestive rather than authoritative.
Every method in this section is an *approximation* of the model's reasoning. That gap between the explanation and the true mechanism is exactly what an adversary exploits.
### The Security Angle: Explanations Can Be Attacked
This is the part that sets a security-focused treatment of interpretability apart from a purely usability-focused one. It is tempting to treat an explanation as ground truth, if the heatmap highlights the dog's face, the model must be looking at the dog. That inference is not safe. Explanations have their own attack surface, and a growing body of work shows they can be manipulated independently of the prediction.
**Explanations are unstable.** Ghorbani, Abid & Zou (2019), "Interpretation of Neural Networks is Fragile," showed that a perturbation too small to change a network's prediction, or even to be perceptible, can nonetheless produce a completely different explanation. The feature-importance map is not a robust function of the input, so two visually identical images can be assigned very different "reasons," which undermines any auditor who relies on the map's stability.
**Explanation methods can be fooled by design.** Slack, Hilgard, Jia, Singh & Lakkaraju (2020), "Fooling LIME and SHAP," went further: an adversary can *construct* a model that is blatantly biased on real inputs yet returns innocuous, unbiased-looking explanations from LIME and SHAP. The trick exploits the very mechanism these methods depend on, they probe the model with *perturbed, out-of-distribution* samples. The adversarial model learns to detect whether an input is a genuine instance or one of these synthetic probes, behaving discriminatorily on real data while switching to a benign surrogate whenever it senses it is being explained. The explanation looks clean precisely because the model games the explainer. Manipulated saliency maps are the visual analogue: an image can be perturbed so the model is fooled while its saliency map still looks perfectly reasonable, so the explanation reassures the user about a decision that is in fact compromised.
The lesson is blunt: **an explanation is not automatically trustworthy.** These attacks are close cousins of the adversarial perturbations from [Workshop 3](../Workshop03/Adversarial_Attacks_-_White-Box_Attacks.md), small, targeted input changes exploiting the sensitivity of a learned function, aimed here at the explanation rather than (or in addition to) the label. Explanation *robustness*, the stability and faithfulness of an explanation under adversarial pressure, therefore belongs in the threat model, and it should be tested rather than assumed.
### Trust as a System Property
Because explanations can be gamed, trust in an AI system cannot rest on explanation alone; it is a property of the whole sociotechnical system. Several dimensions matter together:
- **User acceptance:** people adopt a system they believe behaves safely and can be questioned, and explanations tailored to the audience support that belief, when the explanations are faithful.
- **Organizational adoption:** institutions need assurance that a system meets operational and compliance standards, which increasingly requires documented, auditable explanations.
- **Accountability:** clear ownership of decisions and their consequences, which transparency and honest explanation make possible.
- **Transparency by design:** building interpretability and documentation in from the start, preferring intrinsically interpretable models where they suffice, and validating explanations for fidelity and robustness rather than shipping them as unverified reassurance.
These themes connect the workshop outward: the fairness auditing of [Workshop 7](../Workshop07/Ethics_in_AI_-_Bias_and_Fairness.md) depends on the explanation tools introduced here, and the accountability such tools enable is exactly what the regulation and standards of Workshop 10 increasingly require.
## Hands-on Lab
The activity is implemented in **PyTorch** with **torchvision**, using a pre-trained **ResNet-18** classifier on **ImageNet** images, and image handling via OpenCV, scikit-image, PIL, NumPy, and Matplotlib. Rather than calling a packaged explainer, the notebook builds the interpretation methods from first principles: it implements **Class Activation Mapping (CAM)** by registering a forward hook on the last convolutional block and combining the captured feature maps with the classifier weights, and it implements **Integrated Gradients** by accumulating gradients along a path from a baseline to the input. It then turns these tools toward security, using CAM to inspect adversarial inputs and show how interpretation exposes the "focus displacement" a successful attack produces. The notebook also includes a taxonomy that situates LIME, SHAP, Grad-CAM, saliency maps, and attention against CAM and Integrated Gradients, so the conceptual methods from the theory section are mapped onto what the code actually runs.
### Activity 01: Model Interpreters
Generate and compare local explanations for an image classifier's predictions, implementing CAM and Integrated Gradients on a pre-trained ResNet-18, then use the resulting heatmaps to examine how an adversarial input shifts the model's apparent focus.
- Open on GitHub: [SecAI_Workshop08_Activity01_Interpreters.ipynb](https://github.com/SecureAI-luc/SecureAI-Lab/blob/main/Workshop08/SecAI_Workshop08_Activity01_Interpreters.ipynb) | Open in Colab: [SecAI_Workshop08_Activity01_Interpreters.ipynb](https://colab.research.google.com/github/SecureAI-luc/SecureAI-Lab/blob/main/Workshop08/SecAI_Workshop08_Activity01_Interpreters.ipynb?authuser=1)
To run the activity, open its Colab link and use **File → Save a copy in Drive** to get your own editable copy, then execute it there.