AI Supply Chain Security: Where the New Risks Actually Come From

Key Takeaways

• AI systems introduce a supply chain that traditional scanners can't see - the risk lives in model weights, datasets, and training pipelines, not just code and packages.
• A poisoned model or dataset is nothing like a CVE: there's no signature, no version bump, and often no visible symptom until the model behaves badly in production.
• Effective AI supply chain security depends on provenance - knowing where every model and dataset came from, how it was built, and whether it can be trusted.

For a decade, supply chain security has meant one thing: knowing what's in your code. Which packages, which versions, which transitive dependencies, which known CVEs. That model works well - right up until an AI model enters the pipeline. Suddenly the most important artifact in your system is a multi-gigabyte blob of learned weights that no package scanner can meaningfully inspect. AI supply chain security is the discipline of closing that gap.

Why AI Systems Have a Supply Chain Traditional Tools Don't Cover

Traditional supply chain tooling is built around a simple assumption: risk lives in code and the packages that code depends on. Scanners parse manifests, match dependencies against vulnerability databases, and flag known issues. It's a mature, well-understood workflow.

AI systems break that assumption. The critical components aren't just libraries - they're models and data:

• Pretrained models pulled from public hubs, often with opaque origins and no meaningful "version history."
• Training and fine-tuning datasets that may contain poisoned, mislabeled, or copyrighted samples.
• Model weights - dense numerical artifacts that encode behavior in a way no scanner can read like source code.
• Serialization formats that can smuggle executable code inside what looks like a data file.

None of these fit neatly into a dependency graph. A scanner can tell you a Python package has a known CVE, but it can't tell you that a downloaded model was fine-tuned on manipulated data or that its weights were tampered with. This is the core of ai model security: the artifact carrying the most risk is precisely the one existing tools understand the least.

Mapping the AI Pipeline From Training Data to Deployed Model

To secure the AI supply chain, you first have to see it. Unlike a code pipeline, the AI pipeline moves risk through several stages that each introduce distinct attack surface.

• Raw training data. Data is collected or scraped, often from public or third-party sources. Poisoning here is subtle and early - a small fraction of manipulated samples can shape downstream behavior.
• Data preparation and labeling. Cleaning, filtering, and labeling steps can introduce bias or malicious samples, sometimes through compromised labeling vendors.
• Training and fine-tuning. The model learns from the prepared data. A tampered base model or a poisoned fine-tuning set bakes risk directly into the weights.
• Packaging and serialization. The trained model is saved to a file format and often published to a registry or hub. Unsafe formats can carry embedded code that executes on load.
• Distribution. Models are pulled from internal or public repositories, frequently with weak integrity guarantees about who built them and how.
• Deployment and inference. The model runs behind an endpoint, where its behavior - not its code - is what attackers ultimately exploit.

Every handoff between these stages is a trust boundary. Without ml model provenance - a verifiable record of what happened at each step - teams inherit risk they can neither see nor prove. For a broader look at hardening the systems around these endpoints, our guide on how to protect against attacks covers the runtime side of the picture.

How a Compromised Model or Dataset Differs From a Vulnerable Package

The instinct is to treat a bad model like a bad package - find it, flag it, patch it. But the two are fundamentally different kinds of risk, and that difference is why detection looks so unfamiliar.

A vulnerable package has a known identity. It has a name, a version, and a CVE with a published description. You can match it against a database, and once a fix ships, you upgrade and the problem is gone. The vulnerability is legible.

A compromised model or dataset has none of that:

‍No signature to match. A poisoned model isn't a "known bad version" in any database. It looks statistically like a healthy model until it encounters the trigger it was trained to respond to.

• No visible symptom. A backdoored model passes standard evaluations and serves correct answers almost all the time - the malicious behavior only surfaces under specific, attacker-chosen inputs.
• No clean patch. You can't upgrade a poisoned dataset. Remediation often means retraining from a trusted baseline, which is expensive and slow.
• Opaque internals. Weights aren't human-readable. You cannot audit a model's "logic" the way you can read a suspicious function in source code.

This is why llm supply chain risk demands different controls. The question shifts from "does this artifact match a known vulnerability?" to "can I trust where this artifact came from and how it was built?" Detection becomes a question of provenance and integrity rather than signature matching.

What Existing Supply Chain Controls Miss Once AI Enters the Picture

Many teams assume their existing supply chain practices - SBOMs, dependency scanning, signing - extend naturally to AI. They partially do, but critical gaps open up the moment models and data enter the picture.

• SBOMs stop at the code boundary. A standard software bill of materials inventories packages and libraries, but typically has no representation for a model file, its training data, or its lineage. The most important artifact goes uncatalogued. Our overview of the best SBOM generation tools is a good starting point, but AI components need an extended bill of materials to be covered.
• Vulnerability databases don't cover models. There's no equivalent CVE feed for "this pretrained model contains a backdoor." Scanners have nothing to match against.
• Integrity checks verify files, not behavior. A checksum confirms a model downloaded intact - it says nothing about whether the model was poisoned before it was ever published.
• Signing proves the sender, not the safety. A signed model tells you who uploaded it, but not whether their training pipeline was compromised upstream.
• Serialization risk is invisible. Code-focused tooling rarely inspects model file formats for embedded executable payloads that run on load.

The through-line is that traditional controls answer "is this the file I expected?" while AI risk demands answering "can I trust how this model came to exist?" Closing that gap is exactly what a mature ai supply chain security program is built to do. You can find more on securing modern build and deployment pipelines across the Echo blog.

FAQ

Can a compromised AI model pass normal scans?

Yes, easily. Standard scans check code and dependencies for known vulnerabilities, but a poisoned or backdoored model isn't a known bad version in any database. Its weights look statistically normal, and it passes typical evaluations while behaving correctly on most inputs. The malicious behavior only triggers under specific, attacker-chosen conditions - so conventional scanning gives a clean bill of health it can't actually guarantee.

Does an SBOM cover AI models and training data?

Usually not. A traditional software bill of materials inventories packages, libraries, and their versions, but has no native representation for model weights, datasets, or training lineage. That leaves the highest-risk AI artifacts uncatalogued. Teams increasingly extend the concept into an AI-specific bill of materials that records model origin, datasets, and pipeline steps - but a standard code-focused SBOM alone does not cover them.

How do teams verify a pretrained model's provenance?

Provenance verification means establishing a trustworthy record of where a model came from and how it was built. Teams pull models only from trusted or internal registries, verify cryptographic signatures and checksums, and require documented lineage covering the base model, training data, and fine-tuning steps. Emerging frameworks record this metadata as attestations, so a model's origin and build history can be audited rather than assumed.