Supply-Chain Attacks on AI Apps: How a pip install Steals Your Keys

What happened

Theory

LiteLLM's release pipeline was compromised and published 1.107.0 to PyPI and Docker. The GitHub source tag was clean; the wheels and images were built from a lookalike repository (berri-ai instead of berriai). Unpinned pip installs in the window and the floating Docker tags (latest and others) were affected.

Practical example: Who got hit

requirements.txt with a bare 'litellm' installed 1.107.0 between 11:39 and 13:23 UTC; a pinned litellm==1.106.2 did not.

Code that runs without an import

Theory

Python imports the site module at startup. It reads .pth files in site-packages, which normally add folders to sys.path, but executes lines that start with import. It also imports sitecustomize. The bad wheel shipped cook.pth (one line: import cook) and a sitecustomize.py, so every Python process ran the stealer, pip included.

Practical example: See it safely

Write a one-line .pth file into a throwaway virtual environment and start Python: the Zen of Python prints before your code runs.

python3 -m venv /tmp/demo && cd /tmp/demo
echo 'import this' > "$(bin/python -c 'import site; print(site.getsitepackages()[0])')/zen.pth"
bin/python -c 'print(1)'

How the release was compromised

Theory

The attacker held GitHub CLI tokens from three developers, which LiteLLM believes came from the logs of an internet-exposed internal deployment. Branch protection blocked the workflow change, but an organization owner's token disabled protection and force pushed it. A second token approved the release, a third deleted the workflow logs, and the pipeline signed and published through trusted publishing.

Practical example: What would have stopped it

Enable 'Do not allow bypassing the above settings' so admins and owners are bound by branch protection, require a second person for release-workflow changes, and keep tokens out of logs.

Pin exact versions

Theory

A floating requirement takes whatever is newest at install time. Pin exact versions for every package, including transitive dependencies, install only from the lockfile, and upgrade in a pull request. Pin Docker images by digest, not a tag like latest.

Practical example: Floating vs pinned

litellm>=1.100 picked up 1.107.0 in the window; litellm==1.106.2 did not. For images, use name@sha256:<digest>.

litellm==1.106.2

Hash the files

Theory

pip's hash-checking mode requires every requirement pinned with == and hashed, and refuses to install a file that doesn't match. Hashes catch a swapped or tampered file behind a version you already locked, but if you lock while a bad release is live, you hash the malware.

Practical example: Generate hashes

Let a tool write the hashes, then install in hash-checking mode.

uv pip compile requirements.in --generate-hashes -o requirements.txt
pip install --require-hashes -r requirements.txt

Wait a few days

Theory

The poisoned release was removed in under two hours. A cooldown tells the installer to ignore anything uploaded in the last few days, so short-lived bad releases never become candidates. Allow a per-package exception when you need an urgent fix.

Practical example: Cooldown settings

pip only considers packages uploaded at least three days ago; uv excludes anything newer than seven days.

pip install --uploaded-prior-to P3D -r requirements.txt
# pyproject.toml
[tool.uv]
exclude-newer = "P7D"

Assume it ran

Theory

The stealer read SSH keys, cloud credentials, CI secrets, AI tool tokens and the instance's role credentials. Keep secrets out of the build step, give each service only the keys it needs, block outbound traffic it doesn't need, and check the source repository in a release's provenance before deploying it.

Practical example: If you ran 1.107.0

Don't start with pip install (it runs the payload again). Delete the files and kill the daemon, or better, rebuild from scratch, then rotate every key the machine could reach.