Blaine Hoak
University of Wisconsin-Madison & Visa Research | Ph.D. Student & Research Intern
Blaine Hoak is a Ph.D. student at the University of Wisconsin-Madison and a Ph.D. Research Intern at Visa Research on the Trustworthy AI team. Her research centers on evaluating and advancing the security of machine learning models, with a primary focus on adversarial robustness: understanding, designing, and defending against sophisticated attacks that expose worst-case failures of machine learning models. Such attacks not only test the resiliency of models but also highlight discrepancies between human and machine perception, and her work aims to explain and bridge that gap to build safer, better-aligned models.
Aside from AI trustworthiness, she has collaborated on work broadly within security and privacy, with the goal of identifying and protecting against threats to user privacy and security across a variety of systems. Her work has been published in top-tier security and machine learning venues including CCS, USENIX Security, IEEE S&P, and ICLR.
## Learning Objectives
After completing this workshop, you will be able to:
- State the black-box threat model precisely: the attacker has only query access to the model's outputs and no access to weights, architecture, or gradients.
- Distinguish score-based attacks (the attacker sees output probabilities or confidences) from decision-based attacks (the attacker sees only the top predicted label), and place both ZOO and SimBA as score-based.
- Explain how ZOO estimates gradients from queries using symmetric finite differences and then optimizes with a C&W-style objective, and why this is accurate but query-expensive.
- Explain how SimBA succeeds with almost no machinery by sampling directions from an orthonormal basis and keeping only the steps that reduce the true-class probability.
- Reason about query budget as the central black-box cost, and read a black-box result as queries versus success rate rather than iterations.
- Relate black-box attacks back to Workshop 3: use transferability to explain substitute-model attacks, and contrast query-based black-box attacks with the white-box upper bound.
- Describe the practical defenses that blunt black-box attacks: rate limiting, query monitoring and anomaly detection, and returning coarse outputs.
## Theoretical Background
### The Black-Box Threat Model
In the black-box setting the attacker can send inputs to the model and observe what comes back, and nothing more. The weights, the architecture, and above all the gradient of the loss with respect to the input, which was the entire engine of the Workshop 3 attacks, are all hidden. This is the setting that describes almost every model a real attacker meets: a classification API, a content-moderation endpoint, a fraud-scoring service. The defender is no longer conceding total knowledge; they are conceding only that the model can be queried.
Because the attacker cannot compute $\nabla_x L$ directly, black-box attacks either estimate it from queries or search input space without it. That reframes the entire cost model. White-box attacks are measured in iterations, because each iteration is one cheap gradient computation. Black-box attacks are measured in **queries**, because information about the model can only be bought one query at a time, and queries are the resource a defender can meter, price, or cut off.
### Score-Based vs. Decision-Based
What the model returns decides what kind of black-box attack is possible, and the two regimes are quite different.
- **Score-based.** The output includes the model's confidences: a probability vector, logits, or at least the score of the top classes. This is a rich signal. The attacker can watch the true-class probability move by a fraction of a percent in response to a tiny input change, which is exactly the feedback a gradient estimate or a greedy search needs.
- **Decision-based.** The output is only the final predicted label, with no confidence attached. This is a far harsher setting: most small input changes leave the label unchanged, so the attacker gets no gradient of feedback at all and must instead walk carefully along the decision boundary itself.
**Both attacks in this workshop, ZOO and SimBA, are score-based:** each one reads the model's output probabilities and steers by how the true-class probability responds. They would not work against a model that returned only a bare label, which is precisely why returning coarse outputs is a defense worth discussing at the end.
### From White-Box to Black-Box: Two Routes
There are two broad ways to attack a model you cannot see inside, and it is worth naming both even though the labs focus on the second.
**Substitute-model (transfer) attacks.** Train a local surrogate model on data you collected by querying the target, craft ordinary white-box adversarial examples against that surrogate, and transfer them to the target. This works because of **transferability**, the property from Workshop 3 that adversarial examples crafted on one model frequently fool other models trained on similar data. The appeal is that once the surrogate is trained, the attack itself queries only the surrogate, not the target, so it sidesteps query monitoring on the real system. The cost is that transfer is never guaranteed and the surrogate must be similar enough to the target for its adversarial examples to carry over.
**Query-based attacks.** Skip the surrogate and attack the target directly, using its own outputs as the only guide. This is what ZOO and SimBA do. There is nothing to train and nothing to transfer, but every step of the attack spends queries against the very system you are trying to fool, which is exactly where rate limiting and anomaly detection bite.
### ZOO - Zeroth-Order Optimization
ZOO (Chen et al., 2017) takes the direct route and asks: if the only thing stopping a white-box attack is the missing gradient, why not estimate the gradient from queries and then run a white-box optimizer on the estimate? "Zeroth-order" refers exactly to this, optimizing using only function values (zeroth-order information) rather than analytic derivatives.
The estimate comes from a symmetric difference quotient. To approximate the partial derivative of the loss along input coordinate $i$, query the model at the input nudged slightly in the $+e_i$ and $-e_i$ directions and take the difference:
$$
\frac{\partial L}{\partial x_i} \approx \frac{L(x + h\,e_i) - L(x - h\,e_i)}{2h},
$$
where $e_i$ is the unit vector along coordinate $i$ and $h$ is a small step. Assemble these coordinate-wise estimates into an approximate gradient, then feed it into a **C&W-style optimization**, the same margin-based min-norm objective from Workshop 3, driven now by estimated rather than exact gradients.
The trade-off is stark and defines the attack. Each coordinate costs two queries per gradient estimate, so a single approximate gradient over a $d$-dimensional image costs on the order of $2d$ queries, and the optimization needs many such gradients. ZOO is therefore **accurate**, often rivaling white-box success, but **query-expensive**, which is why the labs run it on small images and why coordinate-wise or stochastic tricks exist to keep the query count manageable. It is score-based throughout: the loss it differences is computed from the model's output probabilities.
### SimBA - Simple Black-Box Attack
SimBA (Guo et al., 2019) is the opposite design philosophy: instead of reconstructing a gradient, do the least possible and let the model's own score tell you whether each move helped. Its full name is the **Simple Black-box Attack**, and the name is the method.
Fix an orthonormal basis for input space. Then repeat a single, almost trivial loop:
1. Sample a direction $q$ from the basis that has not been used yet.
2. Try adding a fixed step, $x + \epsilon q$, and query the model. If the true-class probability drops, keep the change.
3. Otherwise try the opposite step, $x - \epsilon q$, and query again. If that drops the true-class probability, keep it instead.
4. If neither helps, discard both and move on to the next direction.
Because the basis is orthonormal, every accepted step is in a fresh, non-interfering direction, and the accumulated perturbation grows in a controlled way. The basis can be the raw **pixel basis**, or, more query-efficiently, a low-frequency **DCT (discrete cosine transform) basis**, which concentrates the perturbation in the smooth, low-frequency components that classifiers are most sensitive to. The workshop labs use the DCT variant. For all its simplicity, SimBA is surprisingly **query-efficient**, and it is again score-based: the whole decision rule is "did the true-class probability go down," which requires reading the output probabilities.
### ZOO and SimBA Side by Side
| | ZOO | SimBA |
|---|---|---|
| Core idea | Estimate the gradient, then optimize | Greedily keep steps that lower the true-class probability |
| Uses gradients? | Yes, estimated by finite differences | No |
| Feedback signal | Output probabilities (score-based) | Output probabilities (score-based) |
| Query cost | High (about $2d$ queries per gradient) | Low (one or two queries per direction) |
| Strength | Accurate, close to white-box success | Simple and query-efficient |
| Basis / search | Coordinate-wise finite differences | Orthonormal basis (pixel or low-frequency DCT) |
Both land in the same score-based black-box regime, but from opposite ends: ZOO spends queries to recover white-box precision, while SimBA spends as few queries as it can and accepts a cruder search.
### Query Budget and Practical Constraints
Because queries are the currency of a black-box attack, the honest way to report one is as a curve of **queries versus success rate**: how many inputs were fooled, at what query cost. This also points directly at how a real deployment defends itself, and none of these defenses require making the model itself more robust.
- **Rate limiting.** Capping how fast a client can query stretches a high-query attack like ZOO from minutes into days, often past the point of being worthwhile.
- **Monitoring and anomaly detection.** Query-based attacks emit a telltale pattern, long runs of near-identical inputs that differ by tiny perturbations. Monitoring for that signature can flag an attack in progress.
- **Coarse outputs.** Returning only the top label instead of a full probability vector removes the exact signal both ZOO and SimBA rely on, pushing an attacker from the tractable score-based regime into the much harder decision-based one.
### Where Black-Box Sits Relative to White-Box
It is worth stating the relationship to Workshop 3 plainly, because it is the reason both workshops exist. White-box attacks are the **worst-case upper bound**: they assume an attacker who knows everything, and a defense that survives them has cleared the highest bar. Black-box attacks are the **realistic case**: they assume only what a public API actually exposes, and they are the threat most deployed models will actually face. A model can look secure against a casual probe and still fall to a determined white-box adversary; conversely, the fact that a capable black-box attacker needs thousands of queries is itself a meaningful, if empirical, layer of protection. Reading the two workshops together gives both the ceiling and the floor of the evasion threat.
## Hands-on Lab
The four activities implement both black-box attacks on both a handwritten-digit model (MNIST-10) and a color-image model (CIFAR-10), each using the Adversarial Robustness Toolbox (ART). ART provides `SimBA` and `ZooAttack` as drop-in evasion classes over the same `KerasClassifier` estimator used in Workshop 3, and the notebooks reuse ART's `BinaryInputDetector` to explore detecting the resulting perturbations. Because both attacks are score-based, each notebook wraps a model that returns output probabilities and tracks how the true-class probability falls as queries accumulate.
### Activity 01 - SimBA on MNIST-10
The Simple Black-box Attack against a 10-class MNIST digit classifier, keeping only the basis-direction steps that lower the true-class probability.
- Open on GitHub: [SecAI_Workshop04_Activity01_SimBA_MNIST10.ipynb](https://github.com/SecureAI-luc/SecureAI-Lab/blob/main/Workshop04/SecAI_Workshop04_Activity01_SimBA_MNIST10.ipynb) | Open in Colab: [SecAI_Workshop04_Activity01_SimBA_MNIST10.ipynb](https://colab.research.google.com/github/SecureAI-luc/SecureAI-Lab/blob/main/Workshop04/SecAI_Workshop04_Activity01_SimBA_MNIST10.ipynb?authuser=1)