Red-team lab / Failure patterns
A downloaded model or artefact carried code
Model files are not just weights: several common formats can run code when loaded, and a model can carry behaviour its publisher didn't document.
Design it out
Load only safe formats, from a mirror you control, with provenance checked, in a sandbox; evaluate behaviour before use.
- Safe weight formats only Accept only model formats that cannot run code when loaded, and reject the rest.
- Sandboxed loading and scanning Open and scan every new model or package in an isolated sandbox before it reaches production.
- Private registry mirror Install models, packages and tools only from a reviewed internal mirror with an allowlist.
- Pre-deployment behaviour evaluation Test a model against safety, trigger and red-team suites before it is adopted for a use.
The agent design check flags it as: Downloaded model files can carry code
.
Practise it
Where it happened
- Models that ran code on loadLoading a model file can run code. Use safe formats, scan, and load untrusted models in a sandbox.
Risk codes
MITRE ATLAS AML.T0010AML.T0109AML.T0115AML.T0020AML.T0018.000AML.T0070
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama