Prompt injection & AI security
Prompt injection is what happens when untrusted text - a webpage, an email, a document, a message from someone else - contains hidden instructions that an AI system reads and mistakenly follows as if they came from you, the actual user. It's the AI-era version of the classic security rule "never trust user input," except here the "input" can be anything the model reads, not just what you typed.
This becomes a real risk specifically once a model has tool use (E2) or reads content it didn't generate itself - an AI agent that reads emails, browses the web, or opens documents on your behalf can be quietly told, by the content itself, to exfiltrate data, take unintended actions, or ignore its real instructions. Defenses are still evolving: treating fetched content as data rather than commands, requiring human approval before irreversible actions (tying back to the autonomy ladder, E4), and sandboxing what tools an agent is allowed to call are the main mitigations today - there's no fully solved fix yet.
Directly relevant to what you build: next time you design or evaluate a feature where an AI reads content from outside the user (an inbox, a scraped webpage, an uploaded document), ask explicitly - "if this content contained hidden instructions, what's the worst it could make the AI do, and what actually stops that?" That single question is most of what security-conscious AI product design really is.
What is prompt injection?