Kai Yue
North Carolina State University | Ph.D. Candidate in Electrical Engineering
Kai Yue is a Ph.D. candidate in Electrical Engineering at North Carolina State University. His research spans federated machine learning, image processing, and computer vision. His work has been recognized at the International Conference on Machine Learning (ICML) 2022 and the USENIX Security Symposium 2023, and published in journals including IEEE Transactions on Neural Networks and Learning Systems. Prior to his doctoral studies, Kai earned his B.E. degree with honors in Electrical Engineering from the University of Science and Technology of China, and served as a Research Assistant at UCLA focusing on artificial intelligence. He has served as a reviewer for top-tier conferences including CVPR, ICCV, and ECCV, and has mentored in the GEARS Research Program at NC State.
## Learning Objectives
After completing this workshop, you will be able to:
- Name the four main privacy threats against trained models, membership inference, model inversion, attribute or property inference, and training-data memorization, and explain why overfitting and overconfidence are what make them possible.
- State the definition of $(\epsilon, \delta)$-differential privacy precisely and read it: what the privacy budget $\epsilon$ controls, what the failure probability $\delta$ allows, and why composition consumes budget across repeated queries.
- Describe DP-SGD as per-example gradient clipping plus Gaussian noise on the aggregated gradient, tracked by a privacy accountant, and identify where each step enters the training loop.
- Explain the FedAvg protocol, local training on private data, transmission of updates only, and weighted server aggregation, and say exactly what does and does not leave the client.
- Explain why shared gradients are not private by demonstrating the Deep Leakage from Gradients (DLG) reconstruction attack, and why this motivates combining federated learning with differential privacy or secure aggregation.
- Describe a backdoor (model-poisoning) attack in the federated setting, in which a malicious client injects a trigger the global model learns to obey.
- Distinguish central from local differential privacy (LDP) and state the trust-model and utility difference between them.
- State the privacy-utility trade-off explicitly: more noise or a smaller $\epsilon$ buys more privacy at the cost of accuracy.
## Theoretical Background
### Why Models Leak
A trained model is a compressed function of its training data, and compression is not the same as forgetting. When a model overfits, it does not just learn the general pattern; it learns particular examples, and it becomes overconfident on inputs it has seen before. Both effects are directly measurable from the outside, which is why privacy leakage is possible at all. An attacker who can query a model, and in the federated setting can also see gradients, can turn that residual information about individuals back into knowledge about the training set.
Four attack families make this concrete. They were introduced as threats in Workshop 2; here they set the problem that differential privacy and federated learning are built to solve.
| Threat | The attacker's question | What it exploits |
|--------|-------------------------|------------------|
| **Membership inference** | Was this specific record in the training set? | Higher confidence / lower loss on training examples than on unseen ones |
| **Model inversion** | What did a training input look like? | The model or its gradients encode reconstructable features of individual inputs |
| **Attribute / property inference** | What is a hidden attribute of this individual, or of the dataset? | Correlations the model learned between visible and sensitive attributes |
| **Training-data memorization** | Can I extract a training record verbatim? | Overfit models retain exact examples, later recovered by extraction |
The common root is the same across all four: a model that fits its training data too tightly leaks that data. This is why privacy defenses and generalization are related, and why the mechanisms below work by deliberately blurring the model's dependence on any single record.
### Differential Privacy
Differential privacy (Dwork) is not a technique but a **definition** of privacy, one that a mechanism either provably satisfies or does not. A randomized mechanism $\mathcal{M}$ is $(\epsilon, \delta)$-differentially private if, for every pair of **adjacent** datasets $D$ and $D'$ that differ in a single record, and for every set of outcomes $S$:
$$
\Pr[\mathcal{M}(D) \in S] \;\leq\; e^{\epsilon} \cdot \Pr[\mathcal{M}(D') \in S] \;+\; \delta .
$$
Read it as a guarantee about individuals. Adding or removing any one person's record changes the distribution of outputs by at most a factor of $e^{\epsilon}$ (plus a small slack $\delta$). Because the output barely moves when your record is present versus absent, an adversary who sees the output cannot confidently tell whether you were in the dataset, and therefore cannot learn much that is specific to you.
- **Privacy budget $\epsilon$**: the core knob. A smaller $\epsilon$ forces the two distributions closer together, which means stronger privacy and, in practice, more added noise and lower accuracy. A larger $\epsilon$ is a weaker guarantee.
- **Failure probability $\delta$**: the small probability that the clean $\epsilon$-bound does not hold. It should be kept very small, conventionally smaller than the inverse of the dataset size.
- **Noise mechanisms**: the guarantee is achieved by adding calibrated random noise, from a Laplace distribution for pure $\epsilon$-DP or a Gaussian distribution for the $(\epsilon, \delta)$ variant that suits the many-query setting of model training.
- **Composition**: privacy is a budget that is spent, not a switch. Every query or training step that touches the data consumes some $\epsilon$, and the guarantees add up, so repeated access erodes protection. This is why DP training must track cumulative spend.
### DP-SGD: Differential Privacy for Training
Applying differential privacy to deep learning means making the training process itself a DP mechanism. **DP-SGD** (Abadi et al., 2016) does this by modifying the gradient step so that no single example can move the parameters too much and its contribution is masked by noise. Each step:
1. **Clip per-example gradients.** Compute the gradient for each individual example and rescale it so its norm is at most a bound $C$. This caps the sensitivity of the update to any one record, the quantity differential privacy needs to control.
2. **Add Gaussian noise** to the summed, clipped gradients, so the aggregate that drives the update is noised rather than exact.
3. **Update** the parameters with the noisy, clipped gradient as usual.
A **privacy accountant** tracks how much of the $\epsilon$ budget the training run has spent across all steps, using tight composition analysis, so the final model comes with a stated $(\epsilon, \delta)$ guarantee. In the lab this is provided by **Opacus**, the differential-privacy library for PyTorch, which attaches to an ordinary optimizer and handles the per-example clipping, noise, and accounting.
### Federated Learning
Differential privacy limits what a released model reveals; **federated learning** attacks the problem earlier, by never gathering the raw data in the first place. Introduced as FedAvg (McMahan et al., 2017), it replaces centralized training on a pooled dataset with a coordination protocol over many clients that each keep their own data.
**The FedAvg protocol:**
1. **Distribute.** The server sends the current global model $w_t$ to a set of participating clients.
2. **Train locally.** Each client $k$ runs several steps of SGD on its own private data, producing an updated local model $w_t^{k}$. The data never leaves the device.
3. **Send updates only.** Each client returns its model update (weights or gradients), not any training examples.
4. **Aggregate.** The server combines the updates into a new global model by a **weighted average**, weighting each client by its number of examples $n_k$:
$$
w_{t+1} \;=\; \sum_{k} \frac{n_k}{n}\, w_t^{k} .
$$
5. **Repeat** for many rounds until the global model converges.
Because only updates cross the network and raw data stays local, federated learning offers a real structural privacy benefit and is the setup behind this workshop's per-client weight sets (the repo ships 50-client and 100-client federated model collections). But the benefit has limits that motivate the rest of the session:
- **It is not a formal guarantee.** "Data stayed on the device" is an architectural property, not a bound on leakage. The updates themselves can reveal the data, as the next section shows.
- **Communication cost.** Model updates are large and rounds are many, so bandwidth, not computation, is often the bottleneck.
- **Non-IID data.** Clients hold different, skewed distributions, which slows and destabilizes convergence relative to centralized training.
- **A new attack surface.** A distributed protocol invites malicious participants, which is exactly what the backdoor attack below exploits.
### Deep Leakage from Gradients: Why Updates Are Not Safe
The central cautionary result of the session is that the gradients a federated client shares are enough to reconstruct its private training data. **Deep Leakage from Gradients** (DLG, Zhu et al., 2019) makes an honest-but-curious server the attacker. It observes the true gradient $\nabla_w L(w; x, y)$ that a client computed on a private example, then optimizes a pair of **dummy** inputs and labels $(x', y')$ so that the gradient they produce matches the observed one:
$$
\min_{x',\, y'} \; \big\| \nabla_w L(w; x', y') - \nabla_w L(w; x, y) \big\|^2 .
$$
As the dummy gradient is driven to match the real one, the dummy input converges to the client's actual training image, pixel by pixel. Gradients work as a reconstruction target because backpropagation encodes precise, example-specific information through the chain rule; the update is not an anonymous summary of the data but a near-invertible function of it. The lesson is direct: federated learning's "only updates leave the device" is not privacy on its own, which is why it is paired with differential privacy (noising the gradients, as in DP-SGD, degrades the DLG match) or with secure aggregation (so the server never sees any individual client's update). The lab demonstrates both the attack and the noise-based defense.
### Backdoor Attacks in Federated Learning
DLG is an attack by a curious server against clients. A backdoor attack reverses the direction: a **malicious client** attacks the global model. Because the server aggregates whatever updates clients submit, a participant can poison its update so that the assembled global model learns a hidden rule: classify inputs normally, except misclassify any input that carries a specific **trigger** pattern into an attacker-chosen target class. The repo includes such a trigger (a small `devil.png` image stamped onto inputs) and the resulting poisoned model artifacts.
This is a **model-poisoning** threat specific to the distributed setting, and it is insidious because the backdoored model behaves correctly on all clean inputs, so accuracy metrics look normal while the model obeys the attacker on triggered inputs. It connects the privacy track back to the poisoning-robustness dimension of Workshop 5: federated learning widens the poisoning attack surface because the server trusts client updates it cannot inspect.
### Local Differential Privacy
The DP-SGD story above is **central** differential privacy: a trusted party (the server, or whoever trains the model) sees the real data or real gradients and adds noise centrally. **Local differential privacy (LDP)** removes that trusted party. Each client adds noise to its own update **on-device**, before anything is shared, so the server only ever receives already-privatized data. The repo's `FL_LDP_models` set (a 100-client collection) corresponds to this stronger trust model.
The trade-off is trust for utility. LDP protects against a compromised or curious server because no one ever holds an un-noised update, but because every client noises independently, the aggregate carries far more total noise than a single central injection would, so the accuracy cost is larger for the same nominal privacy level.
### The Privacy-Utility Trade-off
State it plainly, because it governs every choice in this workshop: more privacy costs accuracy. A smaller $\epsilon$, a larger noise multiplier, or a shift from central to local DP all tighten the privacy guarantee and all lower model utility, because the same noise that hides individuals also blurs the signal the model is trying to learn. There is no setting that maximizes both; privacy engineering is choosing an operating point on this curve that the application can live with. This mirrors the robustness-accuracy trade-off of Workshop 5, another case where a stronger guarantee is bought with clean-data performance.
### Where This Connects
These ideas run forward and back through the program. The membership-inference and model-extraction threats named in Workshop 2 are the concrete attacks that differential privacy defends against. The regulatory drivers, GDPR, CCPA, and related regimes that legally require this kind of data protection, are the subject of Workshop 10 on data governance and standards; the mechanisms built here are how organizations meet those obligations in practice.
## Hands-on Lab
The labs are built on **PyTorch**, using **Opacus** for the differential-privacy machinery (per-example clipping, Gaussian noise, and the privacy accountant) and the standard MNIST-scale vision stack (torchvision, torchmetrics, matplotlib). They move from building a federated system to attacking it, so the privacy failures are demonstrated on the very system the first activity constructs. The per-client weight sets, test/train shards, and poisoned-model artifacts the notebooks load are the federated collections shipped directly under each workshop's root-level `datasets/` folder.
### Activity 01: Federated Learning
Build a federated learning system with FedAvg: train local models across clients on their own data, aggregate the updates into a global model on the server, and add differential privacy to the training so the process carries a formal guarantee. The primary notebook is the PyTorch and Opacus implementation covering FedAvg, the federated optimization objective, and DP in federated training; a companion notebook walks through the communication protocol and aggregation-in-practice from a systems angle.
- Open on GitHub: [SecAI_Workshop06_Activity01_FL.ipynb](https://github.com/SecureAI-luc/SecureAI-Lab/blob/main/Workshop06/SecAI_Workshop06_Activity01_FL.ipynb) | Open in Colab: [SecAI_Workshop06_Activity01_FL.ipynb](https://colab.research.google.com/github/SecureAI-luc/SecureAI-Lab/blob/main/Workshop06/SecAI_Workshop06_Activity01_FL.ipynb?authuser=1)
- Open on GitHub: [SecAI_Workshop06_Activity01_FL2.ipynb](https://github.com/SecureAI-luc/SecureAI-Lab/blob/main/Workshop06/SecAI_Workshop06_Activity01_FL2.ipynb) | Open in Colab: [SecAI_Workshop06_Activity01_FL2.ipynb](https://colab.research.google.com/github/SecureAI-luc/SecureAI-Lab/blob/main/Workshop06/SecAI_Workshop06_Activity01_FL2.ipynb?authuser=1)