AI/ML & LLM Supply-Chain Security

Everything this course has already taught you about supply-chain risk — a build pulling in something it didn't write, an artifact that might not be what its name claims, a dependency nobody re-checked after the day it was added — has a direct, largely unglamorous analogue in machine learning. A trained model is a binary artifact from somewhere else. A training set is a pile of dependencies you didn't audit. A prompt is user input, and user input that reaches a system prompt without a trust boundary in between is the newest shape of an old bug class. This page takes the four hardest parts of that analogy — model provenance and signing, training-data poisoning, prompt injection, and the OWASP Top 10 for LLM Applications — and works through each one at the depth an engineer actually shipping this stuff needs, not the depth of a keynote slide.

The ML supply chain is a supply chain, with three new artifact types

☺ Like you're 10: Same swap-meet problem as always — just three new kinds of sealed box: the model itself, the pile of data it learned from, and the libraries that loaded both.

Nothing about machine learning invents a new category of risk. It reuses the exact one this course has spent every other page on — you are assembling something from parts you didn't build, and any one of those parts can be swapped, poisoned, or malicious before it reaches you — and adds three artifact types that don't have a clean precedent in a normal application's dependency tree:

• A trained model — a large, functionally opaque binary of weights, usually pulled from a public hub or an internal registry, that you cannot read the way you can read a diff.
• A training or fine-tuning dataset — frequently scraped or aggregated from many sources you don't control, sometimes from the open web itself, with no equivalent of a package manager's checksum pinning it to what it was when someone decided to trust it.
• An ML-specific dependency stack — PyTorch, transformers, langchain, and dozens of smaller packages, which are ordinary PyPI packages subject to the exact typosquatting and dependency-confusion attacks covered in software composition analysis in depth, except now the code they pull in runs with training- or inference-time privilege over real data.

Map each one against the software supply chain this course already secures, and the shape doesn't change — only the noun does:

Hold that table in mind for the rest of the page. Every section below is one row of it, taken apart.

Model provenance and signing

☺ Like you're 10: Anyone can put a file named "Castle Set" on a table. A signature is how you prove which table it actually came from.

Public model hubs work like a package registry with none of the ecosystem-level trust signals package registries have spent a decade building. A repository named acme-org/support-model on Hugging Face Hub is a string, not a verified identity — nothing stops a look-alike repo from mimicking a well-known organization's name, and nothing about downloading a .bin file cryptographically proves it was produced by the training run its model card claims. The fix is the one this course already teaches for container images: don't trust the name, verify a signature bound to a specific artifact digest.

Signing weights the way you sign images

The OpenSSF's model-signing effort extends Sigstore's keyless-signing model — the same cosign/Sigstore machinery covered in Sigstore & cosign — to model files: a signature and an in-toto attestation get bound to the model's content hash, verifiable later against Sigstore's transparency log without either party managing a long-lived private key. The tooling and exact CLI surface are still moving quickly as of this writing, so treat the commands below as illustrative of the pattern rather than a copy-paste reference, and check the project's current documentation before wiring this into a real gate.

# Pattern: sign a model artifact the same way you'd sign a container image —
# a digest-bound signature plus a transparency-log entry, not a long-lived key.
model_signing sign ./checkpoints/support-model.safetensors \
  --signature ./checkpoints/support-model.sig

# Verify before the model is allowed to load in any pipeline stage —
# a deployment gate should refuse to serve an unsigned or re-signed-by-someone-else model.
model_signing verify ./checkpoints/support-model.safetensors \
  --signature ./checkpoints/support-model.sig \
  --identity "ci@acme.io" --identity-provider "https://token.actions.githubusercontent.com"

Pin by revision, never by "main"

The single most common ML supply-chain mistake looks exactly like the container mistake it's copying: pulling a model by a mutable reference. from_pretrained("acme-org/support-model") with no revision resolves to whatever the repo's default branch currently points to — identical in spirit to FROM node:latest, and just as silently able to hand you a different artifact next Tuesday than the one your team reviewed.

# Bad: "main" is a moving target — whoever controls the repo can repoint it
model = AutoModelForCausalLM.from_pretrained("acme-org/support-model")

# Good: pin to the exact commit SHA of the revision your team actually reviewed and signed off on
model = AutoModelForCausalLM.from_pretrained(
    "acme-org/support-model",
    revision="a1b2c3d4e5f6789...",
)

The model card isn't the same thing as an ML-BOM

A model card — the documentation format Margaret Mitchell and co-authors proposed in 2019 ("Model Cards for Model Reporting") — is prose: intended use, training data description, evaluation results, known limitations. It's genuinely useful, and it's also unsigned, unstructured, and not something a pipeline gate can parse and enforce. CycloneDX's 1.5 specification closed that gap with a machine-learning-model component type carrying a structured modelCard field — the same SBOM machinery covered in software bills of materials, extended with fields an ML pipeline actually needs to check automatically.

// A CycloneDX ML-BOM fragment — check CycloneDX's current schema before treating
// field names below as exhaustive; the ML-BOM extension is newer and evolves faster
// than the core SBOM spec it's built on.
{
  "type": "machine-learning-model",
  "bom-ref": "pkg:huggingface/acme-org/support-model@a1b2c3d",
  "name": "support-model",
  "modelCard": {
    "modelParameters": {
      "task": "text-generation",
      "architectureFamily": "transformer",
      "datasets": [{ "ref": "internal-support-tickets-v4" }]
    },
    "considerations": {
      "technicalLimitations": ["Not evaluated for medical or legal advice"]
    }
  }
}

The practical rule: generate the ML-BOM at the moment you register a model to the registry — the same "generate at build time, not from memory six months later" discipline that software bills of materials argues for a container image — and a "does this model have a current, signed ML-BOM" check becomes exactly as gate-able as "does this image have a current, signed SBOM."

The pickle problem — a checkpoint that can run code just by loading

☺ Like you're 10: Some file formats are just data. Some can secretly also be a tiny robot that does something the instant you open the box — and a lot of the internet's most popular model files are the second kind by default.

This is the most concrete, most exploited AI-specific vulnerability in this whole page, and it has nothing to do with the model's behavior — it's a deserialization bug wearing a machine-learning costume. torch.save() and torch.load() use Python's pickle format by default. Pickle is not a data format the way JSON is; it's a serialized sequence of bytecode-like instructions for reconstructing a Python object, and one of the objects it can reconstruct is the result of calling an arbitrary callable with arbitrary arguments — via an object's __reduce__ method. Loading a pickle file doesn't just parse data; it can execute code, on your machine, at load time, before your model ever produces a single token.

# Illustrative only — this shows *why* torch.load() on an untrusted checkpoint is
# equivalent to running someone else's binary. It is not a working exploit to reuse.
import pickle, os

class Payload:
    def __reduce__(self):
        # pickle.load() invokes whatever __reduce__ returns as part of reconstruction —
        # the object doesn't need to deserialize successfully for this to already have run.
        return (os.system, ("id",))

# torch.save()/torch.load() are pickle underneath. A ".bin" or ".pt" checkpoint from
# an untrusted source can carry this instead of — or hidden alongside — real tensors.

Keras' native H5 format has an analogous risk through Lambda layers, which can embed and execute arbitrary Python at load time; joblib and dill inherit the same pickle-family risk directly. None of this is theoretical — it's the exact reason model-hosting platforms now scan for it by default.

The fix that removes the vulnerable class, not the fix that patches it

Safetensors, Hugging Face's alternative serialization format, stores only tensor data and a JSON header describing shapes and dtypes — there is no reconstruction step and nothing in the format can express a callable. It doesn't make pickle loading safer; it makes the vulnerability class structurally impossible for anything serialized this way, the same relationship a memory-safe language has to a buffer overflow. Prefer safetensors for anything you distribute or accept from outside your organization, and treat a .bin/.pt checkpoint from an untrusted source the way you'd treat an unsigned binary someone emailed you.

# Scan a checkpoint before it's allowed into the registry — catches unsafe pickle
# opcodes (e.g. GLOBAL importing os.system) without ever executing torch.load() against it.
modelscan scan -p ./checkpoints/support-model.bin

# Hugging Face Hub runs an equivalent scan automatically on upload and labels
# each file in a repo's Files tab with a pickle-scan status — check it, don't assume it.

Training-data poisoning

☺ Like you're 10: If someone can sneak one bad ingredient into the flour before the bakery even starts mixing, they don't need to break in later — every cake made from that batch is already compromised.

Model provenance and signing protect the artifact after training finishes. Training-data poisoning attacks the step before that — corrupting what the model learns from, so the resulting weights are compromised no matter how carefully you sign and scan the file they end up in. Three shapes of the attack come up repeatedly:

The research that made this concrete and practical rather than theoretical is Nicholas Carlini and co-authors' 2023 paper, "Poisoning Web-Scale Training Datasets is Practical", which demonstrated two attacks against real, widely-used web-scale datasets:

• Split-view poisoning — large datasets like LAION are frequently distributed as a list of URLs, not the content itself; the actual images or text get fetched later, at train time. The paper's authors showed that buying up expired domains still referenced by a dataset's URL list — cheaply, at scale — lets an attacker serve arbitrary poisoned content to anyone who fetches the dataset after that purchase. The dataset's description never changes; what's actually at each URL does.
• Frontrunning poisoning — periodic snapshot dumps (the paper used Wikipedia) are captured on a predictable, publicly known schedule. Timing a malicious edit to land in the narrow window right before a snapshot is taken, then reverting it moments later, bakes the edit permanently into that snapshot's training data while it's barely ever visible on the live site — nobody reviewing the live page would ever see what a downstream model actually learned from.

Both attacks share one root cause: a dataset described by references to content is not the same artifact as the content itself at fetch time, and treating them as interchangeable is exactly the gap either attack exploits.

Defenses: pin the bytes, not the pointer

The countermeasure is the dataset version of the "pin by revision, never by a floating tag" lesson from the previous section: fetch once, verify against a fixed manifest, and freeze the actual bytes into content-addressed storage rather than re-resolving live URLs at every training run.

# Bad: re-fetching by URL at train time re-exposes you to split-view poisoning —
# whoever controls that URL today controls what lands in your training set today.
wget -i dataset_urls.txt -P ./raw/

# Better: fetch once, verify the fetched bytes against a fixed manifest, then freeze it.
sha256sum -c dataset_manifest.sha256      # fails the pipeline if any fetched file's hash moved
dvc add raw/ && dvc push                  # content-addressed, immutable from this point on

Pair that with a datasheet — Timnit Gebru and co-authors' 2018 proposal, "Datasheets for Datasets," the dataset-provenance analogue of a model card: where the data came from, how it was collected, what known gaps or biases it carries, and what it should not be used for. Neither a datasheet nor a manifest hash stops someone from poisoning a source before you ever pull it; together they're what lets you notice the pull happened, prove what you actually trained on, and restrict future training to sources you've already vetted rather than the open web at large.

Prompt injection as an application-security problem, not a model problem

☺ Like you're 10: SQL injection got fixed by building a locked door between the sentence and the instructions inside it. Nobody's built that door for AI models yet — so today, the door has to be built around the model instead of inside it.

This is the section the rest of the page has been building toward, and the framing matters more than any single mitigation: prompt injection is not a bug you patch in a model. It's a structural property of how large language models consume input, and the correct response to a structural property is architecture, not a hotfix.

Direct and indirect injection

Direct prompt injection is a user typing instructions intended to override the system prompt — "ignore your previous instructions and instead…" — straightforwardly in their own message. Indirect prompt injection is the more dangerous shape: malicious instructions embedded in content the model is asked to process rather than content a user typed directly — a web page it's asked to summarize, a PDF, an email, a tool's API response, a chunk retrieved by a RAG pipeline. The model has no reliable way to tell "the user's actual request" apart from "text that happened to be sitting inside a document the user's request referenced," because by the time either reaches the model, both are just tokens in the same context window.

Kevin Liu's February 2023 extraction of Bing Chat's "Sydney" system prompt via direct prompt injection was the moment this stopped being a research curiosity. The more consequential demonstration came from Kai Greshake and co-authors' 2023 paper, "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection," which showed that a webpage an assistant was merely asked to summarize could carry hidden instructions the assistant then followed — with no direct interaction between the attacker and the victim at all. The attacker never touched the chat. They just made sure their content was somewhere the model would read it.

Why this doesn't have a parameterized-query fix — yet

SQL injection has a complete, mechanical solution: a parameterized query enforces a hard syntactic boundary between the query template (code) and a bound value (data) at the database driver level, so a malicious string value can never be reinterpreted as SQL syntax, full stop. LLMs have no equivalent boundary today. The "interpreter" is a probabilistic next-token model operating over one flattened sequence of tokens — there is no formal grammar tagging one span as "instruction" and another as "data" the way a bound parameter is tagged. That's not a temporary tooling gap; it's a property of how the current generation of models is built.

So the fix lives in the surrounding system, not in the model

Since the model can't be relied on to hold the boundary, the boundary has to be enforced by everything around it — which turns this back into ordinary application security, applied to a new kind of client:

• Least privilege for anything the model can do. An agent or tool-calling LLM should hold exactly the scoped credentials its task needs, never a standing broad credential — the same discipline covered in workload identity & pipeline IAM and zero trust for pipelines, applied to a model instead of a CI job.
• Human-in-the-loop before anything consequential. Sending an email, making a purchase, deleting a resource, or pushing code triggered by model output should require explicit confirmation, not silent execution — the same principle behind a manual approval gate anywhere else in a pipeline.
• Treat retrieved content and tool output as untrusted, always. A RAG chunk or an API response should never silently gain instruction-level trust just because it arrived inside the same context window as the system prompt.
• Encode and validate model output before it reaches anything else. Rendering model output as raw HTML, or piping it into a shell, without the same escaping discipline you'd apply to any other untrusted string, is structurally the same defect as reflected XSS — OWASP's LLM Top 10 (next section) names this improper output handling, and SAST, DAST & SCA already covers the injection-class defenses it borrows from.
• Sandbox anything the model can execute, and log the full prompt-response chain so an injection attempt is visible after the fact even on the runs where it wasn't blocked beforehand.

The OWASP Top 10 for LLM Applications

☺ Like you're 10: Same idea as the original OWASP Top 10 for web apps — a ranked list of the mistakes that keep showing up, so you can check your work against it instead of guessing what to worry about.

The OWASP GenAI Security Project publishes a Top 10 for LLM Applications, first released in 2023 and substantially revised for 2025. The exact names, ordering, and even count have shifted across releases — the table below reflects the 2025 naming, and you should treat it as the current shape rather than a permanent one; check the project's own page at owasp.org before citing a specific item in an audit or a security review.

Read across that table and the pattern from the opening section repeats: roughly a third of it is the supply-chain risk this page already dissected, a third is an AppSec concept wearing a new label (output handling is XSS, excessive agency is least privilege, system prompt leakage is "don't hardcode a secret"), and the rest — embedding weaknesses, misinformation, unbounded consumption — is genuinely specific to how these systems fail. Knowing which third you're looking at tells you which existing control to reach for first.

Wiring it into the pipeline — MLSecOps in one diagram

☺ Like you're 10: Every checkpoint from earlier in the page, lined up in the order a model actually travels through — the same line-up a container image already walks through, just carrying different boxes.

None of the preceding sections require inventing a new discipline. They require running the gates this course already trusts — signing, an SBOM-equivalent inventory, admission control, least-privilege identity, runtime guardrails — against an artifact type that happens to be a directory of floating-point numbers instead of a container image.

The gate step is where all of the earlier sections cash out into one enforceable decision, and it's the exact same pattern as the admission-control policy this course already teaches for infrastructure-as-code in IaC security & policy as code — a policy-as-code check that runs before deployment, not a person remembering to eyeball a checklist.

package devsecops.model_deploy

default allow = false

allow {
    input.model.signature_verified == true
    input.model.scan_status == "clean"
    input.model.format != "pickle"
    input.model.mlbom_present == true
}

deny[msg] {
    input.model.format == "pickle"
    msg := sprintf("model %s is a pickle checkpoint — convert to safetensors before this gate passes", [input.model.name])
}

deny[msg] {
    input.model.signature_verified == false
    msg := sprintf("model %s has no verified signature — refusing to deploy an unverified artifact", [input.model.name])
}

Runtime is the last checkpoint, not the only one: content-safety classifiers like Llama Guard or a configurable rules engine like NVIDIA NeMo Guardrails screen prompts and completions in flight, purpose-built prompt-injection detectors like Rebuff or commercial equivalents add a second opinion specifically on injection attempts, and rate limiting addresses the unbounded-consumption risk from the Top 10 table above. None of these are a substitute for the least-privilege and human-in-the-loop controls covered earlier — they're the same defense-in-depth idea as a WAF sitting in front of an application that also has its own input validation: useful, and not the whole plan by itself.

You now have the same supply-chain instincts this course built for containers and dependencies, retargeted at models, datasets, and prompts. From here, software bills of materials covers the SBOM machinery this page's ML-BOM section builds on in full depth, vulnerability management & triage covers how a scan finding — model or otherwise — actually gets triaged and closed, and zero trust for pipelines goes deeper on the least-privilege identity model this page leaned on for agents and tool-calling. For the tools mentioned above in practice, see Sigstore & cosign.