Ryan Sheatsley
University of Wisconsin-Madison | Postdoctoral Research Associate
Ryan is a Postdoctoral Research Associate at the University of Wisconsin-Madison, under the direction of Prof. Patrick McDaniel. His research is at the intersection of computer security and machine learning, where he investigates the risks of deploying machine learning systems in security-centric domains, such as network intrusion and malware detection. He also applies computer security principles to radiation detection, internet measurement, and Internet of Things.
## Learning Objectives
After completing this workshop, you will be able to:
- Distinguish robustness (correct behavior under bounded, worst-case perturbation) from resilience (acceptable operation under faults, distribution shift, and failures), and say which mechanisms serve which goal.
- Name the three dimensions of robustness, evasion, poisoning, and distributional, and match each to the failure mode it addresses.
- Write the adversarial-training objective as a saddle-point (minimax) problem and explain why the inner maximization is solved with PGD, making it the strongest empirical defense.
- Explain what a certified defense guarantees that an empirical one cannot, and contrast the two main certification families: probabilistic randomized smoothing (an $L_2$ radius) and deterministic bound propagation such as zonotope/DeepZ analysis (an $L_\infty$ radius, used in this workshop's lab).
- State the robustness-accuracy trade-off precisely: stronger robustness generally lowers clean accuracy, and certification adds inference-time cost on top.
- Describe how domain adaptation and transfer learning keep models reliable under distribution and covariate shift.
- Outline the systems-level resilience mechanisms, redundancy, graceful degradation, health monitoring, and failure recovery, that keep an ML service running through faults.
- Evaluate a defense honestly: report robust accuracy under a stated attack and budget, report certified radius and certified accuracy for certified models, and recognize obfuscated gradients as a false sense of robustness.
## Theoretical Background
### From Attacking to Defending
Everything in Workshops 3 and 4 was an attacker's tool. The reason to build those tools first is that a defense can only be judged against the attacks it must survive, and the strongest defenses in this session are constructed directly from the strongest attack you have already built. Adversarial training uses PGD as a subroutine. The evaluation pitfalls at the end of this session are exactly the ones an attacker exploits when a defense is tested lazily. Keep the attacker in mind throughout: a defense is a claim about what an adversary cannot do, and a claim is only as good as the adversary you tested it against.
### Three Dimensions of Robustness
"Robust" is not a single property. It is useful to separate three, because each is defended differently and each maps to a different stage of the ML pipeline.
| Dimension | Threat | When it strikes | Defense in this session |
|-----------|--------|-----------------|-------------------------|
| **Evasion robustness** | Adversarial examples: bounded perturbations of an input at inference | Deployment / test time | Adversarial training, certified defenses |
| **Poisoning robustness** | Corrupted or manipulated training data | Training time | Robust training that tolerates a fraction of adversarial data |
| **Distributional robustness** | Distribution / covariate shift between training and deployment data | Deployment, without any adversary | Domain adaptation, transfer learning |
Workshops 3 and 4 were entirely about the first row. This workshop's defenses concentrate there too, but resilience forces attention to the other two: a model that is provably robust to adversarial noise is still useless if the deployment distribution has drifted away from what it was trained on.
### Adversarial Training: the Strongest Empirical Defense
Adversarial training is the leading practical defense against evasion attacks, and its formulation (Madry et al., 2018) is worth stating exactly, because the structure explains both its strength and its cost. It is a **saddle-point (minimax)** problem:
$$
\min_{\theta} \; \mathbb{E}_{(x,y)} \Big[ \max_{\|\delta\| \leq \epsilon} L\big(f_\theta(x + \delta),\, y\big) \Big].
$$
Read it from the inside out. The **inner maximization** is an attacker: for each training example it searches the $\epsilon$-ball for the perturbation $\delta$ that does the most damage. The **outer minimization** is the defender: it updates the parameters $\theta$ so that even that worst-case input is classified correctly. Training therefore means, at every step, attacking your own model and then learning from the attack.
The inner maximization is intractable exactly, so it is approximated, and the standard choice is **PGD**, the iterative projected attack from Workshop 3. This is the direct payoff of building the attack first: adversarial training is a loop that calls PGD on every batch and minimizes the loss on what it returns.
- **Strong**: it substantially raises robust accuracy and remains the empirical benchmark other defenses are measured against.
- **Costly**: each training step runs a multi-step attack, so training is several times more expensive than standard training.
- **Empirical, not guaranteed**: it hardens the model against the perturbations PGD finds, but offers no proof that some other perturbation inside the budget will not still succeed.
- **Accuracy cost**: it typically lowers clean-data accuracy in exchange for robustness (see the trade-off below).
### Certified Defenses: Trading the Guarantee In
An empirical defense answers "the attacks I tried failed." A **certified defense** answers a strictly stronger question: "no perturbation within this radius can change the prediction, whether or not anyone has tried it." That is a mathematical guarantee, not an experimental result, which is why certified defenses matter for high-consequence applications where an unseen future attack is unacceptable. Two families deliver it.
**Randomized smoothing (Cohen, Rosenfeld & Kolter, 2019)** is the probabilistic route. From a base classifier it constructs a **smoothed classifier** $g(x)$ that returns the class most likely to be predicted when the input is corrupted with Gaussian noise, $x + \mathcal{N}(0, \sigma^2 I)$. For each input this yields a **certified $L_2$ radius**: a distance within which the smoothed prediction provably cannot flip. The guarantee is probabilistic (it holds with high confidence from many noise samples) and it is naturally an $L_2$ certificate. Its costs are the two you would expect from a certificate: the noise lowers clean accuracy, and each prediction requires many noisy forward passes at inference time.
**Deterministic bound propagation** is the exact-analysis route, and it is the one this workshop's lab uses. Instead of sampling, it propagates the entire $\epsilon$-ball of possible inputs through the network as an abstract region and checks whether the whole region lands in a single class. Representing that region as a **zonotope** and propagating it with the **DeepZ** analysis (Singh et al., 2018) gives a deterministic **$L_\infty$ certificate**: if the over-approximated output region cannot reach any competing class, the prediction is certified robust. Training a model to make these certified regions large, certified adversarial training, optimizes directly for the certification rate rather than merely for resistance to sampled attacks. The trade-off is again clean accuracy and, especially here, heavy computation.
| | Empirical (adversarial training) | Certified: randomized smoothing | Certified: bound propagation (DeepZ) |
|---|---|---|---|
| Guarantee | None; resists tested attacks | Probabilistic, high-confidence | Deterministic (sound over-approximation) |
| Norm | Whatever the attack uses (often $L_\infty$) | $L_2$ radius | $L_\infty$ radius |
| Main cost | Training-time (PGD per step) | Inference-time (many noisy passes) | Training- and certification-time |
| Used in the lab | Yes (PGD training) | No | Yes (certified training) |
### The Robustness-Accuracy Trade-off
State it plainly: more robustness generally means lower clean accuracy. Widening decision boundaries to absorb worst-case perturbations, whether by training on adversarial examples or by optimizing for a certified radius, pulls the boundary away from where clean-data accuracy alone would place it. Certification adds a second cost on top, inference-time sampling for smoothing, or expensive analysis for bound propagation. There is no free robustness; every defense in this session buys its guarantee with accuracy, compute, or both, and the right balance depends on the application.
### Distribution Shift, Domain Adaptation, and Transfer Learning
Not every failure has an adversary behind it. A model trained on one distribution and deployed on another degrades even when no one is attacking it, and handling that is the distributional-robustness half of the story.
- **Covariate shift**: the input distribution changes while the input-to-label relationship stays fixed.
- **Label shift**: the class proportions change in deployment.
- **Concept drift**: the relationship itself changes over time, so yesterday's decision boundary is wrong today.
**Domain adaptation** narrows the gap between a training (source) domain and a deployment (target) domain, for example by learning domain-invariant features or by adapting on unlabeled target data. **Transfer learning** reuses representations learned on a large source task and fine-tunes them on the target, so a model reaches acceptable performance on the new distribution without training from scratch. Both keep a model reliable as deployment conditions drift away from the lab.
### Resilience: the Systems View
Robustness is a property of the model; resilience is a property of the system that serves it. Even a well-defended model runs on infrastructure that can fail, and resilience engineering keeps the service acceptable when it does.
- **Redundancy**: run multiple models or replicas and fail over or vote across them, so no single failure takes the service down.
- **Graceful degradation**: fall back to a simpler, more reliable model and reduce functionality rather than returning nothing when the primary path fails.
- **Health monitoring**: track live performance and input statistics to detect drift or degradation, and alert or roll back before users are affected.
- **Failure recovery**: have tested procedures to restore or refresh models quickly after a failure.
These mechanisms recur in the secure-deployment material of Workshop 11, where operating AI systems safely in production is the whole subject.
### Evaluating a Defense Honestly
A defense is a claim, and the history of this field is littered with claims that did not survive scrutiny. Evaluate carefully.
- **Robust accuracy** is accuracy measured under a specified attack at a specified budget. A robustness number is meaningless without both: robust to which attack, within what $\epsilon$?
- **Certified accuracy** and **certified radius** are the honest metrics for certified models: the fraction of inputs provably robust, and the radius over which the guarantee holds.
- **Obfuscated gradients are a trap.** A defense that merely masks or breaks the gradient can post excellent numbers against a gradient-based attack while giving a completely false sense of robustness, because a stronger or adaptive attacker sidesteps the masking. This is the same lesson as the empirical detectors of Workshop 3, which looked reliable at one attack strength and failed at another, and it is why proper evaluation uses adaptive attacks that know the defense, and leans on transferability (Workshop 3) to probe defenses that resist direct gradients.
The through-line: an empirical defense is only as trustworthy as the adversary it was tested against, which is precisely why certified defenses, tested against all perturbations in a radius by construction, are worth their cost where the stakes justify it.
### Where This Leads
The defensive posture built here recurs throughout the rest of the program. Workshop 6 turns to privacy defenses, differential privacy and federated learning, which protect training data rather than predictions. Workshop 11 returns to the resilience mechanisms above under the banner of secure deployment and operation. Robustness and resilience are not a single workshop's topic so much as a stance the remaining sessions keep applying.
## Hands-on Lab
The activity is implemented with the **Adversarial Robustness Toolbox (ART)** over a **PyTorch** classifier, the same ART estimator pattern used in the attack workshops, now driving ART's defensive trainers instead of its attacks. On a compact MNIST CNN it trains and compares three strategies, standard training, empirical PGD adversarial training (`AdversarialTrainer` with ART's `ProjectedGradientDescent`), and certified adversarial training (`AdversarialTrainerCertifiedPytorch`, zonotope/DeepZ), then measures each model's clean accuracy against its certified $L_\infty$ robustness. Because certified training is very compute-heavy, the notebook loads a pre-trained certified model for the final evaluation rather than training it from scratch.
### Activity 01: Certified Adversarial Training
Train and evaluate a robust MNIST classifier with ART, comparing standard, PGD-adversarially-trained, and certified (zonotope/DeepZ) models to see the gap between clean accuracy and provable, certified $L_\infty$ robustness.
- Open on GitHub: [SecAI_Workshop05_Activity01_Certified_Adversarial_Training.ipynb](https://github.com/SecureAI-luc/SecureAI-Lab/blob/main/Workshop05/SecAI_Workshop05_Activity01_Certified_Adversarial_Training.ipynb) | Open in Colab: [SecAI_Workshop05_Activity01_Certified_Adversarial_Training.ipynb](https://colab.research.google.com/github/SecureAI-luc/SecureAI-Lab/blob/main/Workshop05/SecAI_Workshop05_Activity01_Certified_Adversarial_Training.ipynb?authuser=1)
To run the activity, open its Colab link and use **File → Save a copy in Drive** to get your own editable copy, then execute it there.