Red-team lab / Learn / Output, rendering and trust
Output, rendering and trust
What happens to a model's answer: rendered, executed, or believed. · 3 min read
Output is input to something else
A model's answer is rarely the end. A browser renders it, a shell runs it, another agent reads it, or a person acts on it. Each of those is a place where a manipulated answer turns into harm.
Rendering is the classic case: a chat that displays images makes the browser fetch whatever URL the model wrote, with no click. Put data in that URL and rendering becomes a leak.
When the target is the person
People trust tools that are usually right. An instruction that reaches someone through their assistant borrows its credibility, so a human check only helps if the human gets independent information.
What holds
Treat model output as untrusted: sanitise rendered Markdown, never execute it unreviewed, and show where each claim or recommendation comes from. Keep verification steps (a bank change, a policy answer) outside the assistant.
Practise it
Further reading
Built with Llama