Red-team lab / Fix the architecture / The model download

The model download

Your team downloads an open-weight model and two community plugins from public hubs to run in production. Anyone can publish there, and nothing is checked before it is loaded.

Attacks to bring to 10% or less

Malicious model file: A downloaded model file runs the attacker's code the moment it is loaded.
Backdoored weights: A model behaves normally until a hidden trigger makes it misbehave.
Look-alike package: A model, package or tool with a name close to a real one is installed instead.

Budget: 5 cost points.

Your mission

Pick controls for this system within the budget of 5 cost points.
Submit the design. Until it passes you see only the remaining risk, how many attacks are still too likely, and the weakest point.
Bring every attack to 10% or less. Then the full breakdown and notes on every control you chose unlock.

Scoring: Each submission counts as an attempt: passing on the first try earns 100 points on the main leaderboard, 10 fewer for each extra submission (never below 10). The Best designs board ranks your best design: the share of risk removed (up to 1000), minus 20 per cost point.

Hints

Where to look
The danger is in the files, before the model says a word.
Why the obvious fix falls short opens after 2 attempts
Some formats run code when opened; public hubs let anyone publish look-alikes; and a backdoor sits in the weights, where no file scan sees it.
What kind of control holds opens after 4 attempts
One control for what a file can do when loaded, one that keeps look-alike names out at the door (a reviewed mirror), one that tests the model's behaviour.

Likelihoods are a teaching model, not measurements: they encode common judgement about how much each control helps.

How is this challenge?

Safe weight formats only cost 1 · engineeringAccept only model formats that cannot run code when loaded, and reject the rest.Sandboxed loading and scanning cost 1 · latencyOpen and scan every new model or package in an isolated sandbox before it reaches production.Signed artefacts and provenance cost 2 · engineeringVerify signatures, hashes and the publisher's identity before anything is installed.Private registry mirror cost 2 · opsInstall models, packages and tools only from a reviewed internal mirror with an allowlist.Pre-deployment behaviour evaluation cost 2 · latencyTest a model against safety, trigger and red-team suites before it is adopted for a use.Pinned versions and an SBOM cost 1 · engineeringPin exact versions and hashes, and keep a bill of materials of every model and package in use.Logging and anomaly alerts cost 1 · engineeringLog prompts, tool calls and outputs; alert on unusual patterns.Hardened system prompt cost 1 · UXDetailed rules in the prompt about secrets, untrusted content and tool use.

Cost 0 / 5

Loading…

Score = share of risk removed (up to 1000) minus 20 per cost point. Your best design counts.

Built with Llama

The model download

What your controls do, and don't

Joseph's walkthrough

What your controls do, and don't

Joseph's walkthrough

Share how you solved it (optional)

Other players' solutions