Mohammed Abuhamad
Loyola University Chicago | Assistant Professor
Mohammed Abuhamad is an assistant professor of Computer Science at Loyola University Chicago. He received a Ph.D. degree in Computer Science from the University of Central Florida (UCF) in 2020. He also received a Ph.D. degree in Electrical and Computer Engineering from INHA University, (Incheon, Republic of Korea) in 2020. He received a Master degree in Information Technology (Artificial Intelligence) from the National University of Malaysia, (Bangi, Malaysia) in 2013.
He is interested in AI/Deep-Learning-based Information Security, especially Software and Mobile/IoT Security. He is also interested in Machine Learning-based Applications and Adversarial Machine Learning. He has published several peer-reviewed research papers in top-tier conferences and journals such as ACM CCS, PoPETS, IEEE ICDCS, and IEEE IoT-J.
## Learning Objectives
After completing this workshop, you will be able to:
- Explain what a threat model is and why every AI security claim depends on the threat model it assumes.
- Characterize any adversary using the five components of a threat model: learning stage, model type, adversarial objective, adversarial capabilities, and adversarial knowledge.
- Distinguish the major classes of adversarial attacks, evasion, poisoning, backdoor/trojan, extraction, and privacy inference, and locate each one within the five-component framework.
- Place an attacker on the knowledge spectrum from white-box to black-box, and explain why gray-box (partial) knowledge enables adaptive attacks against a known defense.
- Describe how the Adversarial Robustness Toolbox (ART) wraps a model in a uniform estimator so the same attack and defense code runs across different frameworks.
- Read an adversarial ML research paper and articulate the threat model its results assume.
## Theoretical Background
### What a Threat Model Is and Why It Matters
A threat model answers a small set of pointed questions about an adversary: At what point in the system's lifecycle can they act? What kind of model are they attacking? What are they trying to accomplish? What can they control? What do they already know? Answering these questions turns a vague worry ("could this be attacked?") into a concrete, testable specification ("this system resists an attacker who can only query the deployed API and has no access to its parameters").
Threat models serve several functions in AI security. They provide a common language for discussing risk across technical and non-technical stakeholders. They let teams prioritize by focusing on realistic threats rather than theoretical ones. And they create the baseline against which any proposed defense is judged, because a defense is only meaningful relative to the adversary it is meant to stop. Unstated assumptions are dangerous: they lead to false confidence, where a system that is "secure" against a weak assumed attacker collapses against a stronger real one.
### The Five Components of a Threat Model
Any adversary in AI security can be described along five independent axes. Keeping them separate is what makes the framework precise; conflating them (for example, treating "model type" as a synonym for "white-box vs. black-box") is a common source of confusion.
| Component | The question it answers | Typical values |
|-----------|-------------------------|----------------|
| **1. Learning stage** | When in the lifecycle can the attacker act? | Training time vs. inference/deployment time |
| **2. Model type** | What kind of model is under attack? | Discriminative/predictive vs. generative |
| **3. Adversarial objective** | What is the attacker trying to achieve? | Availability, integrity, privacy (confidentiality), abuse |
| **4. Adversarial capabilities** | What can the attacker actually control? | Training data, input data, model, labels, source code, query access |
| **5. Adversarial knowledge** | What does the attacker already know? | White-box, gray-box, black-box |
**1. Learning stage.** The lifecycle of a model splits cleanly into a training phase and a deployment (inference) phase, and an attacker with access at one is not necessarily present at the other. Attacks that corrupt the model while it is being trained are fundamentally different from attacks that manipulate inputs to an already-trained, deployed model. This single axis is the first fork in almost every attack taxonomy.
**2. Model type.** This axis is about the *kind* of model, not the attacker's access to it. Discriminative (predictive) models learn a mapping from inputs to labels or values, classifiers and regressors, and are the target of most classic adversarial-example work. Generative models learn the data distribution itself and produce new samples; large language models, diffusion models, and GANs fall here. Model type matters because it changes which objectives are even meaningful: an "abuse" objective, for instance, only makes sense against a generative system that can be steered into producing harmful output.
**3. Adversarial objective.** What the attacker wants maps onto the classic CIA security triad, extended for AI:
- **Availability** - degrade overall performance so the system becomes unreliable or unusable for everyone (for example, poisoning that wrecks a recommender's quality).
- **Integrity** - cause specific wrong outputs while the system otherwise looks healthy (for example, making a classifier misread a particular input).
- **Privacy (confidentiality)** - extract information that should stay hidden, either the private training data or the proprietary model itself.
- **Abuse** - coerce a generative model into producing content it should refuse, an objective specific to generative systems.
**4. Adversarial capabilities.** Independent of what the attacker knows, this axis captures what they can physically do: alter training data, control the inputs supplied at inference time, modify the model or its parameters, change labels, read source code, or simply send queries and observe responses. Capabilities constrain which attacks are even possible; an attacker who can only query a public API cannot poison its training set.
**5. Adversarial knowledge.** This is where the white-box / gray-box / black-box distinction lives, and it belongs here and nowhere else. It measures how much of the system's internals the attacker can see, and it is detailed in its own section below because it so strongly shapes both attacks and evaluation.
### The Knowledge Spectrum: White-Box, Gray-Box, Black-Box
An attacker's knowledge of the target ranges along a spectrum, and where they sit determines both which techniques they can use and how a defender should interpret the result.
| Knowledge level | What the attacker sees | Why it matters |
|-----------------|------------------------|----------------|
| **White-box** | Full access: architecture, parameters, gradients | Worst case; used as the upper bound for robustness evaluation |
| **Gray-box** | Partial knowledge (e.g., the architecture or the defense in place) | Knowing the defense enables adaptive attacks tailored to defeat it |
| **Black-box** | Query access only: inputs in, predictions out | Most realistic for public APIs and deployed services |
**White-box** access is the strongest assumption. Because the attacker has the model's gradients, they can craft the most powerful attacks possible, which is precisely why white-box settings are the standard for *evaluating* robustness: if a defense survives a white-box attack, it will likely survive weaker ones. A white-box result is a worst-case upper bound on how bad things can get.
**Black-box** access is the most realistic assumption for systems exposed as public APIs, where an outsider can send inputs and read predictions but cannot see inside. Black-box attacks must therefore work through queries alone, without gradients.
**Gray-box** sits in between and is where adaptive attacks arise. An attacker who knows *which defense* a system uses, even without full internals, can design an attack specifically to circumvent that defense. This is why "security through obscurity" is fragile: a defense should be evaluated under the assumption that the attacker knows it is there.
### Classes of Adversarial Attacks
The major attack families each occupy a distinct region of the five-component framework. Understanding where each one lives makes it clear which threat models it is relevant to.
| Attack class | Learning stage | Primary objective | Typical knowledge |
|--------------|----------------|-------------------|-------------------|
| **Evasion** | Inference time | Integrity | White-box or black-box |
| **Poisoning** | Training time | Availability or integrity | Requires data control |
| **Backdoor / trojan** | Training time | Integrity (trigger-based) | Requires data or model control |
| **Extraction / model stealing** | Inference time | Confidentiality (of the model) | Black-box (query access) |
| **Privacy / inference** | Inference time | Confidentiality (of the data) | Black-box or white-box |
**Evasion.** An inference-time integrity attack: the attacker perturbs an input so a deployed model misclassifies it, without touching the model itself. In the white-box setting, gradient-based methods such as FGSM, PGD, and Carlini & Wagner (C&W) craft minimal perturbations directly from the model's gradients. In the black-box setting, where gradients are unavailable, query-based methods such as SimBA, ZOO, and HopSkipJump estimate the needed perturbation from predictions alone. Workshops 3 and 4 implement both families in code.
**Poisoning.** A training-time attack: the attacker injects or alters training samples so the learned model is compromised from the start. Poisoning is relevant whenever an attacker can influence the training data, such as models retrained on user-submitted content. Depending on the goal, poisoning can target availability (degrading overall accuracy) or integrity (teaching specific wrong behavior).
**Backdoor / trojan.** A subclass of poisoning in which the attacker plants a hidden trigger during training. The model behaves normally on ordinary inputs but produces attacker-chosen output whenever the trigger (a specific pixel pattern, phrase, or signal) is present. Backdoors are hard to detect precisely because normal-case performance is unaffected.
**Extraction / model stealing.** An inference-time attack on the confidentiality of the *model*. By querying a deployed model and training a substitute on its responses, an attacker can replicate proprietary behavior, an intellectual-property concern that requires only black-box query access.
**Privacy / inference.** Inference-time attacks on the confidentiality of the *training data*. Membership inference determines whether a specific record was in the training set; model inversion reconstructs representative training inputs; property inference reveals aggregate characteristics of the training data. These exploit the fact that models often behave measurably differently on data they were trained on. Workshop 6 studies these alongside privacy defenses.
### Threat Models Across Deployment Contexts
The right threat model depends on what a system does and what a failure would cost. A movie recommender and a medical-diagnosis model do not warrant the same assumptions. Systems in adversarial environments, fraud detection, autonomous vehicles, biometric authentication, must assume intelligent, well-resourced, adaptive opponents. Systems facing mostly accidental misuse can adopt lighter assumptions. Consumer-facing systems should further assume that some users will themselves try to game the system for convenience or gain. Calibrating the threat model to the stakes is what keeps security investment proportionate: strong assumptions and rigorous defenses where harm is high, lighter ones where it is not.
### Tooling: The Adversarial Robustness Toolbox (ART)
The Adversarial Robustness Toolbox (ART), maintained by the Trusted-AI project (originally IBM), is the library the hands-on labs in later workshops build on. Its central idea is the **estimator**: ART wraps a model from any supported framework, TensorFlow, PyTorch, scikit-learn, and others, in a uniform estimator interface, so the same attack and defense code runs unchanged across backends.
This wrapper design connects directly to the knowledge spectrum above. **White-box attacks** need the estimator's gradients, so ART's estimator exposes them for gradient-based methods like FGSM, PGD, and C&W. **Black-box attacks** need only the estimator's `predict()` method, matching a query-only attacker such as SimBA or ZOO. Because both kinds of attack are expressed against the same estimator abstraction, ART lets you swap models, attacks, and defenses without rewriting the surrounding code, which is exactly why the program standardizes on it.
## Hands-on Lab
This session has two parts: an analytical exercise you can do with pen and paper, and a companion reference document that maps the landscape above onto the structure of the Adversarial Robustness Toolbox (ART). Workshop 2 has no coded lab of its own; the hands-on coded labs begin in [Workshop 3](../Workshop03/Adversarial_Attacks_-_White-Box_Attacks.md), where you implement these attacks against real image classifiers.
### Activity 1: Extracting the Threat Model (analytical, no code)
Read the PGD paper, "Towards Deep Learning Models Resistant to Adversarial Attacks" (Madry et al., 2017, [arXiv:1706.06083](https://arxiv.org/abs/1706.06083)), and articulate the threat model the authors assume using the five components from this workshop: learning stage, model type, adversarial objective, adversarial capabilities, and adversarial knowledge. Write one or two sentences per component. This exercise trains the core skill of the session, reading security research and pinning down exactly which adversary a result applies to.
### Activity 2: ART and the Threat-Model Landscape (companion reference)
Work through the companion reference document below. It is a self-contained, self-guided reading resource that walks through the adversarial-attack landscape, the white/black/gray-box knowledge spectrum, model robustness and the accuracy-robustness trade-off, the key attack and defense parameters, and how ART's estimator abstraction organizes attacks and defenses across frameworks. It expands on the material above and previews the attacks you will implement in Workshops 3 and 4.
- Companion reference: [Adversarial Attacks and Robustness in AI Systems (ART)](https://github.com/SecureAI-luc/SecureAI-Lab/blob/main/Workshop02/SecAI_Workshop02_ART.md)