Neophytos Christou
Brown University | PhD Student, Secure Systems Lab (SSL)
Neophytos Christou is a PhD student at the Secure Systems Lab (SSL) at Brown University, working on systems security and software hardening under the supervision of Professor Vasileios Kemerlis. His research spans uncovering security vulnerabilities in deep-learning (DL) frameworks, mitigating deserialization vulnerabilities in PHP, and hardening software against novel attacks that combine memory corruption with speculative execution. His work has been published and presented at top-tier security venues, including the USENIX Security Symposium and the Network and Distributed System Security (NDSS) Symposium. His USENIX Security 2023 paper, IvySyn: Automated Vulnerability Discovery for Deep Learning Frameworks, is the basis for this lecture.
## Learning Objectives
After completing this workshop, you will be able to:
- Explain how the model-level attacks introduced earlier in the program (adversarial examples, data poisoning, model extraction, model inversion, and membership inference) apply to a live, exposed model endpoint, and which deployment control blunts each one.
- Describe why deep-learning frameworks such as TensorFlow and PyTorch carry native-code vulnerabilities, name the common bug classes (buffer overflow, integer overflow, type confusion, use-after-free), and explain what an attacker can achieve by triggering them.
- Explain the supply-chain risks specific to ML: malicious model files that execute code on load, untrusted model hubs, and dependency CVEs, and how to manage them by pinning, scanning, and patching.
- Apply secure deployment practices: container hardening, authenticated and authorized API endpoints over TLS, rate limiting and quotas, strict input validation, and secrets management.
- Design a monitoring strategy that detects both attacks and degradation in production, distinguishing the signals for data drift, concept drift, performance loss, and abuse.
- Outline an incident-response process adapted to ML-specific incidents such as a poisoned model, an extraction campaign, or silent drift, from preparation through post-incident review.
## Theoretical Background
### Model-Level Attacks Against a Live Endpoint
Earlier workshops studied model-level attacks largely in the laboratory, against models the attacker could inspect at leisure. Deployment removes that convenience for the defender and, for several attack classes, hands it to the adversary: a public prediction endpoint is a standing invitation to probe the model with real queries. The operational question is therefore not only "is this attack possible?" but "what does my endpoint expose that makes it easier, and what control reduces that exposure?"
| Attack (see prior workshop) | What it looks like against a deployed endpoint | Primary deployment control |
|---|---|---|
| Adversarial examples (WS2-4) | Crafted inputs that force an attacker-chosen misclassification at inference time | Robust models; input validation and anomaly detection; monitor for abnormal input distributions |
| Data poisoning / backdoors (WS6) | A trigger pattern learned during training fires in production; or a compromised retraining pipeline reintroduces poison | Provenance and integrity checks on training data and models; guard the retraining loop; monitor accuracy and per-class behavior |
| Model extraction / stealing (WS4) | Many black-box queries reconstruct a functional copy of a proprietary model | Rate limiting and per-client quotas; authentication; reduce output granularity (labels over full probability vectors); query-pattern monitoring |
| Model inversion (WS6) | Model outputs are used to reconstruct or infer sensitive training data | Limit output detail; access control; differential-privacy training; monitor for reconstruction-style query patterns |
| Membership inference (WS6) | An adversary determines whether a specific record was in the training set | Regularization and DP during training; limit confidence-score exposure; rate limiting |
The recurring theme is that several of these attacks are *query-driven*: extraction, inversion, and membership inference all depend on issuing large numbers of black-box queries and mining the answers. That makes rate limiting, quotas, authentication, and query-pattern monitoring, discussed below, not just abuse controls but genuine defenses against model-level attacks.
### Framework and Supply-Chain Vulnerabilities
The guest lecture's central message is that the model is only the visible tip of the deployed system. Underneath it sits a large stack of third-party software, and the framework at the bottom of that stack is written in memory-unsafe languages.
**Why frameworks are vulnerable.** A deep-learning framework has a layered architecture: a high-level Python API and Python-to-C++ bindings on top, and a core implementation of the operator kernels, the code that actually performs the math operations and tensor manipulations, written in C and C++ underneath. Programs written in memory-unsafe languages are prone to *memory errors*: bugs that let memory be corrupted or leaked in unintended ways. When an attacker can influence the inputs that reach a vulnerable kernel, a memory error can be abused to crash the process (denial of service), leak sensitive data, corrupt model weights to induce attacker-chosen behavior, or, in the worst case, achieve arbitrary code execution on the serving host. The scale is not hypothetical: the lecture notes that TensorFlow has had more than 400 CVEs issued, many of them memory-safety related, and that PyTorch has on the order of 500 GitHub issues describing memory-safety-related crashes.
**Common bug classes in operator kernels.** The vulnerabilities cluster into a handful of familiar low-level categories, usually reachable by passing a kernel malformed shapes, dtypes, or attribute values that its validation fails to reject.
| Bug class | What goes wrong | Typical consequence |
|---|---|---|
| Buffer overflow | Data written past the bounds of an allocated buffer | Memory corruption; crash; potential code execution |
| Integer overflow | An arithmetic operation (often a size calculation) exceeds the integer range and wraps | Undersized allocations, out-of-bounds access, "bad alloc" crashes |
| Type confusion | An object is interpreted as the wrong type | Memory corruption; unpredictable behavior |
| Use-after-free | Memory is accessed after it has been freed and possibly reused | Corruption; crash; potential code execution |
| Null-pointer dereference / missing validation | Missing input checks let a null or malformed value reach the kernel | Crash (denial of service); information disclosure |
**IvySyn: finding these bugs automatically.** The lecture's research contribution, IvySyn (USENIX Security 2023), attacks this problem at the source. Rather than testing the high-level Python API, IvySyn *fuzzes the native (C/C++) implementation* of the framework directly, systematically feeding kernels malformed inputs to uncover memory-safety and fatal runtime errors. When it finds a crashing input, it automatically synthesizes a *Proof-of-Vulnerability* (PoV): a short, high-level Python snippet that reproduces the crash through the public API, so framework developers can identify and fix the bug. Applied to TensorFlow and PyTorch, IvySyn uncovered 61 previously unknown vulnerabilities, 39 of which were assigned CVEs. The takeaway for a deployer is twofold: the framework you depend on is an attack surface that ongoing research is actively probing, and keeping that framework patched is a security control, not just a maintenance chore.
**Malicious model files.** A second supply-chain risk is specific to ML and easy to overlook: the model file itself can be malicious. Many checkpoint formats serialize with Python's `pickle`, and `pickle` can execute arbitrary code during deserialization. Because `torch.load` is pickle-based, loading a checkpoint downloaded from an untrusted source or an unvetted model hub can run attacker-controlled code on your machine, a supply-chain compromise disguised as "just loading a model" (this ties directly to Workshop 9's treatment of unsafe deserialization). Treat a third-party checkpoint with the same suspicion as a third-party executable: prefer a format that stores only tensor data and cannot execute code, such as **safetensors**, and verify the provenance and integrity of any model you did not train yourself.
**Dependency CVEs and vulnerability management.** Beneath the framework sits a deep tree of transitive dependencies, GPU libraries (CUDA, cuDNN), numerical libraries (NumPy, SciPy), serialization libraries (protocol buffers), and hundreds more, each of which is attack surface and each of which may carry a known CVE. Outdated libraries carry published, exploitable flaws; compromised or maliciously inserted packages carry injected code; insecure default configurations ship open ports or default credentials. The discipline that manages this is unglamorous but decisive: **pin** versions with a lockfile for reproducible builds, **scan** regularly against vulnerability advisories, and **patch** promptly when advisories land. Tools introduced in Workshop 9 apply directly here, pip-audit and Bandit for Python code and dependencies, and Trivy for container images and filesystems.
### Secure Deployment Practices
Deployment turns a model into a running service, and a running service has an operational attack surface that must be hardened deliberately.
**Container hardening.** Package the service in a minimal image and give a compromised container as little to work with as possible. Build from a minimal base image (for example `python:3.11-slim` rather than a full OS image) to shrink the attack surface; install only what is needed and remove package caches; run as a **non-root** user so a compromise does not immediately hold full container privileges; use a **read-only filesystem** where possible; keep secrets out of the image entirely; and scan the final image with a tool such as Trivy before it ships. A minimal, non-root pattern looks like:
```dockerfile
FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN useradd --create-home appuser
COPY --chown=appuser:appuser . .
USER appuser
EXPOSE 8080
CMD ["python", "app.py"]
```
**Secure API endpoints.** An exposed, unauthenticated prediction endpoint is the single most common deployment mistake, and it enables everything from model extraction to denial of service. Three controls are baseline:
- **Authentication** establishes *who* is calling (for example, per-client API tokens), so anonymous mass querying is not possible.
- **Authorization** checks that the authenticated caller is *permitted* to perform the requested operation, on the principle of least privilege.
- **TLS** encrypts all traffic so inputs and predictions cannot be intercepted or tampered with in transit.
**Rate limiting and quotas.** Cap how many queries a client can issue per unit time, and enforce longer-horizon quotas. This is both an availability control and a genuine defense against the query-driven model-level attacks above: extraction, inversion, and membership inference all need many queries, and a client throttled to a modest rate cannot cheaply mount them. This is the operational counterpart to the black-box attacks studied in Workshop 4, where the attacker's power scales with query budget.
**Strict input validation.** Never trust input arriving over the API. Validate that each request has the expected shape, dtype, and value range before it reaches the model or, crucially, the framework kernels beneath it, and reject anything malformed. Bounding input size prevents resource-exhaustion denial of service, and strict shape and range checks are a first line of defense against the malformed inputs that trigger framework memory-safety bugs.
**Secrets management.** Credentials, API keys, and tokens must never be hardcoded in source or baked into images, where they leak into version-control history and container layers and become permanently exposed. Keep them external, injected at runtime through environment variables loaded from protected files or, better, held in a dedicated secrets vault with access control and audit logging.
| Deployment weakness | Risk it creates | Control |
|---|---|---|
| Exposed, unauthenticated API | Model extraction, abuse, denial of service | Authentication, authorization, TLS |
| No rate limiting | Query-driven extraction/inversion; brute force; DoS | Rate limits and per-client quotas |
| Hardcoded credentials | Unauthorized access if image or repo leaks | External secrets management |
| Verbose logging / debug in production | Information disclosure of internals, keys, user data | Configuration review; disable debug; scoped logging |
| Unpatched framework or dependencies | Known exploits, including memory-safety RCE | Pin, scan, and patch; subscribe to advisories |
| Containers running as root, writable FS | Larger blast radius on compromise | Non-root user, read-only FS, minimal image, image scanning |
### Monitoring and Operations
A hardened deployment is not a finished deployment. Attacks and degradation both unfold *after* release, and without monitoring they unfold unseen. Effective operations wire observability into the service from day one so that both security incidents and quality decay are detected quickly and can trigger alerts.
The signals worth watching fall into two overlapping groups: those that indicate an *attack* and those that indicate *degradation*. The two are related, because a poisoning or evasion campaign often shows up first as an unexplained accuracy or distribution change.
| Signal | What it detects |
|---|---|
| Prediction latency | Performance degradation or resource-exhaustion / DoS attacks |
| Model accuracy / quality against ground truth | Concept drift, silent degradation, or the effects of poisoning |
| Input distribution shift (data drift) | Out-of-distribution or adversarial inputs; upstream data changes |
| Error rates and API response codes | System failures; probing and attacks (spikes in 4xx/5xx) |
| Authentication failures | Credential brute-forcing and unauthorized-access attempts |
| Query volume and patterns per client | Extraction, inversion, and membership-inference campaigns |
| Resource utilization | Resource exhaustion and denial-of-service conditions |
Two distinctions matter here. **Data drift** is a change in the *inputs* the model receives (the world, or an attacker, sends it different data than it was trained on); **concept drift** is a change in the *relationship* between inputs and the correct output (the same input should now be labeled differently). Both erode accuracy, but they call for different responses, and both can be either benign (the world moved on) or adversarial (someone is manipulating the pipeline). Logging, anomaly and intrusion detection, drift detection, and model-performance monitoring together give the operations team the evidence to tell these cases apart and to alert before damage compounds.
### Incident Response for AI Systems
When monitoring surfaces a genuine incident, a prepared team responds along a well-worn structure, adapted here to the ML-specific incidents this workshop has described. The classic phases, preparation, detection, containment, eradication, recovery, and post-incident review, map cleanly onto AI operations.
- **Preparation.** Before anything happens, define roles and an on-call path, decide in advance what "known-good" means (a signed, provenance-tracked previous model and a clean dataset version to roll back to), and ensure logging is rich enough to reconstruct events.
- **Detection and triage.** An alert fires from the monitoring layer. Triage separates false positives from real incidents and classifies the incident: is this an extraction campaign (query-volume anomaly), a poisoned or backdoored model (accuracy or per-class anomaly), drift (distribution shift), or a framework-level exploit (crashes, unexpected process behavior)?
- **Containment.** Stop the bleeding. ML-specific containment includes **rolling back to a known-good model**, **disabling or isolating the endpoint**, revoking compromised credentials or API tokens, and tightening rate limits for suspect clients, all of which depend on having prepared those known-good artifacts in advance.
- **Eradication.** Remove the root cause: patch the vulnerable framework or dependency, purge poisoned data from the training set and pipeline, retrain from a trusted checkpoint, or fix the misconfiguration that exposed the endpoint.
- **Recovery.** Restore normal service in a controlled way, validating the replacement model's accuracy and security before returning it to full traffic, and watch the relevant monitoring signals closely for recurrence.
- **Post-incident review.** Conduct a blameless post-mortem: what happened, why detection was fast or slow, what control failed, and what change (a new monitor, a new validation rule, a pipeline fix) prevents a repeat. A learning culture turns each incident into stronger future posture.
The connective idea across all six phases is that incident response for ML is only as good as the preparation that precedes it: versioned models and data, provenance you can trust, rich logs, and monitoring that detects the incident in the first place.
## Key Takeaways
- Deployment expands the threat model twice over: the live model becomes a queryable target for adversarial, extraction, inversion, and membership-inference attacks, and the framework it runs on becomes a target in its own right.
- Query-driven model-level attacks (extraction, inversion, membership inference) are blunted by the same operational controls that stop abuse: authentication, rate limiting, quotas, and query-pattern monitoring.
- Deep-learning frameworks are large C/C++ codebases with hundreds of memory-safety bugs (TensorFlow alone has 400-plus CVEs); a malformed input to a vulnerable kernel can crash, leak, corrupt weights, or execute code. Research like IvySyn finds these automatically, and patching the framework is a security control.
- Model files are executables in disguise: pickle-based checkpoints (`torch.load`) run arbitrary code on load, so verify provenance and prefer safetensors, and pin, scan, and patch the whole dependency tree.
- The deployment baseline is hardened containers (minimal, non-root, read-only, scanned), authenticated and authorized endpoints over TLS, rate limiting, strict input validation, and externalized secrets.
- Monitoring must catch both attacks and degradation; distinguish data drift from concept drift, and wire alerting in from day one.
- Incident response for AI works when preparation exists: known-good models and data to roll back to, provenance you trust, and rich logs, running through detection, containment, eradication, recovery, and a blameless post-mortem.
## Additional Resources
- **Slide deck - Neophytos Christou, IvySyn: Automated Vulnerability Discovery for Deep Learning Frameworks (PDF):** [`SecAI_Workshop11_NeophytosChristou.pdf`](https://github.com/SecureAI-luc/SecureAI-Lab/blob/main/Workshop11/slides/SecAI_Workshop11_NeophytosChristou.pdf) - the guest lecture deck on memory-safety errors in the native code of TensorFlow and PyTorch and how fuzzing uncovers them.
- **Reference notes - Vulnerabilities in AI Models and Build Tools (PDF):** [`document-1.pdf`](https://github.com/SecureAI-luc/SecureAI-Lab/blob/main/Workshop11/docs/document-1.pdf) - a structured catalog of model-level, development/deployment, and data-governance vulnerabilities that maps to this workshop's threat model.
- **TensorFlow security advisories:** [github.com/tensorflow/tensorflow/tree/master/tensorflow/security](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/security) - the published advisories and CVE list for the framework.
- **PyTorch security policy:** [pytorch.org/docs/stable/notes/serialization.html](https://pytorch.org/docs/stable/notes/serialization.html) - guidance on serialization and the risks of loading untrusted checkpoints.
- **safetensors:** [github.com/huggingface/safetensors](https://github.com/huggingface/safetensors) - a safe tensor-serialization format that stores data without executing code, unlike pickle-based checkpoints.
- **OWASP Machine Learning Security Top 10:** [mltop10.info](https://mltop10.info/) - the most significant security risks specific to machine-learning systems.
- **OWASP Top 10:** [owasp.org/www-project-top-ten](https://owasp.org/www-project-top-ten/) - the baseline web-application risks an ML service inherits once it is exposed over an API.
- **NIST Cybersecurity Framework:** [nist.gov/cyberframework](https://www.nist.gov/cyberframework) - the Identify/Protect/Detect/Respond/Recover structure that organizes deployment and operations controls.
- **CVE / NVD:** [nvd.nist.gov](https://nvd.nist.gov/) - the National Vulnerability Database for checking known CVEs in frameworks and dependencies.
- **pip-audit:** [github.com/pypa/pip-audit](https://github.com/pypa/pip-audit) - scans Python dependencies for known vulnerabilities.
- **Bandit:** [bandit.readthedocs.io](https://bandit.readthedocs.io/) - static analysis for common security issues in Python code.
- **Trivy:** [trivy.dev](https://trivy.dev/) - vulnerability scanning for container images and filesystems.
- **Docker security best practices:** [docs.docker.com/develop/security-best-practices](https://docs.docker.com/develop/security-best-practices/) - hardening guidance for building and running containers.
- **[Program Resource Library](../resources.md)** - shared papers, tools, and datasets for the full workshop series.
## Next Steps
Continue to [Workshop 12: Case Studies and Real-World Applications - AIShield](../Workshop12/Case_Studies_RealWorld_Applications_AIShield.md), which brings the program's development, governance, and deployment threads together in real-world case studies and a look at how a commercial platform hardens AI systems in practice.