{
  "id": "prompt-injection",
  "title": "Prompt Injection",
  "lang": "en",
  "source_lang": "en",
  "summary": "A security vulnerability in which untrusted inputs redirect a language-model application away from its authorized instructions.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Prompt injection attack"
  ],
  "generated_at": "2026-10-22",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/prompt-injection/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/prompt-injection/",
      "json": "https://aiwiki.page/zh/prompt-injection.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Prompt injection is a class of cybersecurity vulnerabilities in applications built around large language models, in which attacker-controlled input is treated as an instruction rather than as data. It can redirect an application away from its authorized task, alter its responses, or influence its use of external tools. The underlying problem is a failure to maintain a reliable boundary between trusted instructions and less-trusted material processed by the model. (nvlpubs.nist.gov)\n\nAn injection may be supplied directly through a user-facing interface or indirectly through material such as a webpage, document, email, or tool result. Indirect injection is particularly significant for retrieval-augmented generation and AI agents, because an attacker may influence a system without controlling the legitimate user's request. Depending on the application's permissions, consequences can extend beyond misleading text to disclosure of private information or unauthorized actions. (arxiv.org)\n\nPrompt-injection research connects language-model behavior with established questions of trust, privilege, and secure software design. Proposed defenses include training models to respect instruction hierarchies, separating instructions from data, detecting suspicious content, and restricting what applications can do independently of model output. Evaluating these measures requires testing both resistance to attacks and the ability to complete legitimate tasks; successful results in a particular benchmark do not establish universal protection. (arxiv.org)"
    },
    {
      "heading": "Definition and conceptual boundaries",
      "text": "Prompt injection is the unauthorized redirection of a large language model or an application built around it through instructions supplied in lower-trust input. The defining failure is that material intended to be processed as data acquires instructional authority: a passage to summarize, for example, is treated as a command governing how the summarizer should behave. NIST’s March 2025 adversarial-machine-learning taxonomy describes injection as overriding application instructions and identifies the failure to separate instructions from data as an underlying security problem. “Lower-trust” is relative to the application’s rules: user-submitted text may be legitimate input without being authorized to replace developer-defined constraints. (nvlpubs.nist.gov)\n\nA vulnerability, an attack attempt, and a successful compromise are distinct. The vulnerability is the system’s susceptibility to this authority confusion; an attempt is the submission or placement of content intended to exploit it; success requires an observed violation of the relevant task or security boundary. Merely encountering an injected instruction does not establish success. A concrete operational distinction appears in the 2024 StruQ study by Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner: their evaluation inserted instructions to print a designated marker into task data and counted success when the model treated the inserted text as a new instruction and produced that marker. Responses containing the marker were manually inspected to exclude false positives. That criterion demonstrated task hijacking, not necessarily access to private records or external systems. (arxiv.org)\n\nUnlike injection, legitimate prompt engineering seeks to specify or clarify behavior within the authority granted to the person issuing the request. The distinction therefore depends on the intended task and trust policy, not a blacklist of suspicious phrases. Illustratively, asking a general-purpose assistant to change an answer’s language may be an ordinary request; placing the same request inside a document that a fixed-purpose application must translate makes it an attempted redirection if it seeks to replace translation with another task. Conversely, quoting or translating a sentence about ignoring instructions is not itself a compromise. The StruQ threat model makes this boundary explicit by distinguishing application instructions from user-controlled data, even though both may contain imperative language. (arxiv.org)\n\nJailbreaking and prompt injection overlap but emphasize different boundaries. Jailbreaking usually concerns bypassing a model’s safety restrictions; application-instruction hijacking concerns overriding the application’s authorized task, which may involve otherwise harmless output. OWASP’s LLM01:2025 Prompt Injection treats jailbreaking as a form of injection directed at disregarding safety protocols. NIST likewise discusses jailbreaks as injections capable of overriding system instructions to produce restricted outputs, while separately organizing attacks by objectives such as integrity compromise, privacy compromise, and misuse. These classifications do not justify treating the terms as exact synonyms: changing a translator into a joke generator can violate an application’s instructions without defeating a content-safety restriction. (genai.owasp.org)\n\nPrompt injection is principally an inference-time attack: it changes the input or runtime context through which an already trained model performs a task. Training-data poisoning, by contrast, manipulates material used during training or fine-tuning to influence learned behavior. NIST distinguishes these attack stages and separately lists direct and indirect injection, data poisoning, and backdoor poisoning. Injection also differs from an unprompted hallucination: an unsupported assertion alone does not demonstrate that an adversary supplied instructions or crossed a trust boundary. OWASP identifies manipulated content and incorrect outputs as possible injection consequences, but those outcomes require evidence of the input-driven redirection to establish injection rather than an ordinary generation error. (nvlpubs.nist.gov)"
    },
    {
      "heading": "Historical development and research milestones",
      "text": "On September 12, 2022, Simon Willison published “Prompt injection attacks against GPT-3,” proposing the name for a security problem illustrated by Riley Goodside’s public demonstrations. In Goodside’s example, GPT-3 was instructed to translate English into French, but the submitted text redirected it to produce an attacker-selected phrase instead. Adding an explicit warning about deceptive instructions did not prevent the demonstrated failure. Willison connected this behavior to applications that concatenated developer-written instructions with untrusted user input, and also demonstrated disclosure of the original translation instruction. The article was an early public naming and explanation of this application-security problem, rather than an origin point for adversarial prompting generally. (simonwillison.net)\n\nA substantial expansion of the threat model followed on February 23, 2023, when Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz submitted Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Their central contribution was to show that an attacker need not communicate directly with the assistant: instructions could instead be placed in material the application retrieved or processed. The paper, revised in May 2023, included demonstrations involving Bing Chat, GitHub Copilot, and synthetic applications. In the browsing experiments, instructions embedded in local HTML comments influenced Bing Chat through the Microsoft Edge sidebar’s page-reading feature. The code-completion experiments examined manipulation of suggestions through surrounding code context. These were controlled demonstrations of indirect influence, not measurements of attacks occurring across deployed services. (arxiv.org)\n\nIn 2024, dedicated defenses began treating instruction authority as a training objective rather than relying solely on warnings in prompts. Sizhe Chen and colleagues submitted StruQ: Defending Against Prompt Injection with Structured Queries on February 9. StruQ combined a formatting front end with a specially trained large language model, separating the instructional and data portions of a query and training the model to disregard instructions in the latter. On April 19, Eric Wallace and colleagues submitted The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. Their experiments used generated training examples to teach GPT-3.5 to resolve conflicts by instruction priority, including ignoring lower-privileged directions. Both studies reported improved resistance under their evaluations; neither established universal prevention. (arxiv.org)\n\nEvaluation research also shifted toward tool-using AI agents. Qiusi Zhan and colleagues introduced InjecAgent on March 5, 2024, with 1,054 test cases involving 17 user tools and 62 attacker tools, covering user harm and private-data exfiltration. On June 19, Edoardo Debenedetti and colleagues introduced AgentDojo, an extensible environment for evaluating attacks and defenses during multi-step tool use. Its published configuration contained 97 legitimate tasks and 629 security test cases, including email management, electronic banking, and travel booking. Unlike a collection of isolated malicious prompts, AgentDojo evaluated agents interacting with untrusted tool results and assessed whether legitimate tasks remained achievable. Its authors also reported failures without attacks, making baseline task capability an explicit part of security evaluation. (arxiv.org)\n\nInstitutional recognition appeared in the 2025 edition of OWASP’s Top 10 for LLM Applications, which listed prompt injection as LLM01:2025 and described direct, indirect, and multimodal forms alongside mitigation measures. NIST’s March 2025 report Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, designated NIST AI 100-2e2025, incorporated prompt injection into a broader attack taxonomy. Its associated terminology distinguished injection through user-provided input from indirect injection through attacker-controlled resources. These publications supplied security guidance and a shared vocabulary; their classifications and list positions were not estimates of incident frequency or deployment-wide compromise rates. (owasp.org)"
    },
    {
      "heading": "Instruction processing and trust-boundary failures",
      "text": "A large language model application assembles inputs with different intended authority within the model’s context window. Application instructions specify the task and constraints; user messages supply requests; retrieved passages and tool outputs provide material needed to answer them. Chat interfaces can distinguish system, user, assistant, and tool messages through structured formatting and special tokens. Retrieved text may nevertheless appear inside a user message or tool response rather than in a separate document channel. These distinctions identify sources, but their presence does not by itself ensure that the model respects the corresponding authority boundaries. Eric Wallace and colleagues identified this gap in their April 2024 paper, The Instruction Hierarchy: models could treat untrusted content as instructions competing with privileged application directions. (arxiv.org)\n\nThe central failure is instruction–data confusion. Natural-language material can simultaneously be meaningful task content and resemble a direction to the model. In an illustrative summarization task, a document might contain a sentence asking its reader to abandon summarization and produce an unrelated answer. The authorized operation is to analyze that sentence as document content, not adopt it as a new task. The boundary is crossed when the model promotes the sentence from material being processed to an instruction governing processing. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner’s February 2024 StruQ paper describes this problem in applications that combine a developer’s prompt with externally supplied data in one input. The same language-understanding capability that makes such applications useful also lets instructions embedded in the data influence generation. (arxiv.org)\n\nAn instruction hierarchy specifies which directions prevail when sources conflict. Wallace and colleagues’ 2024 formulation places system instructions above user instructions and user instructions above third-party content. A genuine system message receives its role from the application’s message structure; a document merely describing itself as a system message does not thereby acquire that role. The distinction is between provenance supplied by the interface and an authority claim made within content. Lower-priority material need not be ignored altogether: compatible directions can remain useful. The paper illustrates this with a car-sales assistant that may honor a user’s request to speak Spanish while retaining its assigned function. Its intended behavior is conditional instruction following, not blanket rejection of everything outside the system message. (arxiv.org)\n\nThe comparison with SQL injection concerns the promotion of data into control. In vulnerable database code, concatenating external input into a query can allow that input to alter the query’s structure. A parameterized query instead defines the SQL statement separately and binds supplied values as data: a value containing SQL-looking text remains a value rather than becoming an additional condition. Command injection similarly occurs when externally influenced input changes an intended system command. Structured argument passing can avoid shell interpretation, although arguments still require validation because an otherwise separate argument may change the invoked program’s behavior. These mechanisms enforce boundaries through the database or process interface rather than asking an interpreter to infer the input’s trustworthiness from its wording. (cheatsheetseries.owasp.org)\n\nNatural-language processing does not provide the same straightforward separation. An application may legitimately need a model to translate commands, summarize procedural instructions, or extract actions described in a document without treating those actions as authorized requests. Quotation marks, headings, and document delimiters express this distinction, but the model must still interpret their significance. StruQ therefore proposes more than a textual reminder: it combines a formatting front end with fine-tuning intended to separate instruction and data channels. The architectural distinction is important even without considering its evaluation results: labeling a passage as data and reliably restricting its instructional influence are different properties. (arxiv.org)"
    },
    {
      "heading": "Threat models and delivery surfaces",
      "text": "A threat model specifies the defender’s authorized task, the assets to protect, and the attacker’s objectives, knowledge, and control. For a document assistant, the authorized task might be faithful summarization; protected assets include answer integrity, private records, service availability, and access to connected functions. NIST’s March 2025 adversarial-machine-learning taxonomy distinguishes attacker goals, capabilities, and knowledge, and describes indirect injection attacks against integrity, availability, and privacy. Visibility is a separate consideration: an attacker may know the application’s instructions or observe its responses, while the legitimate user may never see the injected material. These conditions distinguish attacks conducted with detailed feedback from attempts planted in external resources without access to the defender’s subsequent interaction. (csrc.nist.gov)\n\nAttacker control must also be specified at the level of the actual input surface. Choosing an entire user message is different from editing one passage that an information retrieval system might return. In the latter case, the attacker does not necessarily control the user’s question, the other retrieved passages, or whether the malicious passage enters the context window at all. Kai Greshake and colleagues’ 2023 study distinguished passive placement in resources likely to be retrieved from active delivery, such as sending an email to an assistant that processes incoming messages. Their code-completion experiments also found injections sensitive to surrounding context, illustrating why control over a source fragment should not be equated with control over the model’s complete input. (arxiv.org)\n\nDirect injection enters through the application’s user-facing input channel. The relevant conflict is between submitted directions and the application’s authorized behavior, not merely between two people’s preferences. OWASP’s LLM01:2025 Prompt Injection gives a customer-support scenario in which an attacker submits instructions intended to override guidelines and invoke private-data or email functions. An analogous conflict can arise when a user submits material for classification or summarization that also tells the application how to decide or respond. Access to a text field grants control over that input, not authority to replace application-level restrictions. OWASP also distinguishes deliberate attacks from user inputs that unintentionally induce such deviations. (genai.owasp.org)\n\nIndirect injection arrives inside material encountered while performing an otherwise legitimate request: webpages, search results, uploaded files, repository documents, or external tool responses. The legitimate user can therefore be the victim rather than the attacker. An instruction to summarize a webpage authorizes reading its contents, but does not automatically authorize following directions embedded by its publisher. Greshake and colleagues demonstrated this separation through browsing and code-completion applications; their delivery analysis included website content, imported code, and messages sent to automated email processors. A trusted retrieval or tool interface can thus transport attacker-controlled content without making that content an authorized instruction. (genai.owasp.org)\n\nWith multimodal models, the carrier need not be ordinary text. OWASP’s 2025 account describes malicious instructions embedded in an image accompanying benign text, and explicitly notes that injection-bearing input need not be human-visible or human-readable if the model can process it. The image or document is the delivery surface; the behavioral redirection encoded within it is the injected instruction. Human visibility and model exposure are therefore distinct properties: content absent from a normal rendered view may still reach the model, while content that the processing pipeline never passes onward cannot influence it through that channel. (genai.owasp.org)"
    },
    {
      "heading": "Manipulation techniques and attack composition",
      "text": "Instruction replacement attempts to substitute an attacker’s objective for the application’s assigned task; role impersonation presents attacker-controlled text as if it came from a system, developer, or assistant message. Fabricated authority supplies a justification, such as a purported administrative update or emergency procedure. These techniques can appear together: a retrieved passage may claim that the assistant has entered a new operating mode and that earlier directions no longer apply. The apparent role labels remain document content, not genuine message metadata. Kai Greshake and colleagues’ 2023 demonstrations combined simulated system messages and fabricated assistant dialogue to redirect model behavior, illustrating how an invented conversational context can support instruction replacement. (arxiv.org)\n\nObfuscation changes an instruction’s representation without necessarily changing its meaning. OWASP identifies Base64 and hexadecimal encoding, invisible Unicode characters, unusual spacing, and deliberately scrambled spelling as examples. Multilingual presentation similarly places instruction-bearing material in another language or mixes languages within otherwise ordinary content. A non-operational illustration would be a source document containing an encoded or translated request to change the answer’s formatting: the security-relevant transition occurs if the model follows that request rather than merely describing or translating it. Encoding is therefore not itself an injection; the issue is the authority assigned to the recovered instruction. Greshake and colleagues demonstrated an encoded injection against Bing Chat in 2023, with an accompanying direction to decode the material. (cheatsheetseries.owasp.org)\n\nTask-aligned persuasion makes the deviation appear instrumental to the legitimate goal rather than openly demanding its abandonment. An illustrative document might claim that producing an accurate summary requires first adopting an unauthorized reporting rule. Such framing exploits the connection between understanding a workflow and accepting instructions about how to perform it; the malicious step is presented as a prerequisite. Attacks can also compose persuasion with a stronger role- or authority-based wrapper. In their 2024 InjecAgent study, Qiusi Zhan and colleagues compared injected task instructions with an enhanced version reinforced by an additional adversarial prompt; the enhancement nearly doubled attack success against their ReAct-prompted GPT-4 agent. This was a result for that evaluation configuration, not a general multiplier for persuasion techniques. (arxiv.org)\n\nMulti-turn attacks distribute manipulation across interactions: an early exchange can establish a coded convention, while a later message activates the redirection. Multi-stage delivery instead separates payloads across retrieval steps. In Greshake and colleagues’ synthetic search application, a user’s question about Albert Einstein led to a modified reference page, whose initial injection caused another search that delivered a second payload; the resulting response adopted a pirate voice. Persistent-memory attacks extend this sequence across sessions. The same researchers demonstrated an agent storing injected material in a key–value memory, being reset, and adopting the injected behavior again when the stored notes were read. Persistence resided in application storage and subsequent retrieval, not in a lasting change to the model itself. (cheatsheetseries.owasp.org)\n\nRetrieval poisoning places manipulative passages or demonstrations in a knowledge base used by retrieval-augmented generation. It can combine a retrieval mechanism that makes the malicious record likely to appear with content that redirects behavior once retrieved. Zhaorun Chen and colleagues’ 2024 AgentPoison research targeted long-term memory and knowledge stores: optimized triggers increased the likelihood of retrieving poisoned demonstrations, while inputs without the trigger were intended to retain normal behavior. The method required no additional model training or fine-tuning. This distinguishes storage-level persistence from weight modification: a malicious record may repeatedly influence new inference contexts even though the underlying model parameters remain unchanged. (arxiv.org)"
    },
    {
      "heading": "Consequences and limits of compromise",
      "text": "Output-integrity failures include answer contamination, task hijacking, and suppression of relevant information. Contamination changes the substance of an answer—for example, inserting an attacker-favored claim into a summary—whereas hijacking redirects the assistant toward a different objective. Suppression can leave an answer superficially plausible while excluding material needed to satisfy the user’s request. In their 2023 study, Kai Greshake, Sahar Abdelnabi, and colleagues demonstrated indirect injections against Bing Chat that induced incorrect summaries and manipulated its search-and-summarization behavior. These consequences do not require access to private records or execution tools: the affected asset can be the answer itself. Unlike an unprompted hallucination, the relevant failure is an attacker-induced departure from the intended task. (arxiv.org)\n\nConfidentiality failures require distinguishing hidden-instruction extraction from disclosure of user data. Revealing an application’s internal prompt exposes instructions already supplied to the model; it does not, by itself, demonstrate access to an inbox, customer database, or filesystem. Disclosure of private records instead requires that those records enter the model’s accessible context or be retrieved through connected tools. Qiusi Zhan and colleagues’ 2024 InjecAgent study treats private-data theft as a sequence involving retrieval and subsequent transmission. Its scenarios include obtaining financial information, medical records, or browsing history and sending the information to an attacker-controlled destination. A model retrieving a record and a model transmitting that record are therefore distinct stages, with different consequences for data privacy. (arxiv.org)\n\nUnauthorized actions extend compromise beyond generated prose when an AI agent can invoke an application programming interface on the user’s behalf. InjecAgent includes scenarios involving unauthorized transactions, changes to smart-home devices, and manipulation of files; these are benchmark objectives, not reports of corresponding losses in deployed services. The operative mechanism is misuse of capabilities made available to the agent, rather than acquisition of new privileges merely through language. Consequently, an injected instruction to transfer funds cannot produce a transfer without a callable payment capability and an execution environment that accepts the request. Likewise, a tool that only reads information does not itself confer permission to modify it. (arxiv.org)\n\nObserved redirection is not equivalent to unrestricted system compromise. Greshake and colleagues demonstrated a synthetic application fetching further instructions from a mock attacker server and then answering in a pirate accent. This established attacker influence over subsequent application behavior, not general control of the host operating system. Similarly, generated source code or a textual claim that a command has run is not evidence of arbitrary code execution. Such an outcome additionally requires an execution pathway, and its scope remains bounded by that pathway’s privileges and environment. The same distinction separates a malicious link appearing in an answer from a user following it, or an attempted outbound request from confirmed delivery of private information. (arxiv.org)\n\nConcrete evidence of consequential compromise can therefore reside in tool traces and resulting state, rather than solely in the assistant’s final response. Edoardo Debenedetti and colleagues’ 2024 AgentDojo environment represents this distinction explicitly: user-task checks inspect outputs and changes to environment state, while separate security functions determine whether attacker goals were achieved. Its inboxes, bank accounts, and other service data are simulated. A successful simulated transfer establishes that the agent crossed the specified authorization boundary within that environment; it does not establish a real financial loss. Conversely, failure to complete the legitimate task is not automatically evidence that the attacker achieved a particular theft or unauthorized action. (arxiv.org)"
    },
    {
      "heading": "Retrieval systems and tool-using agents",
      "text": "In retrieval-augmented generation (RAG), a user’s question drives document retrieval, and a generator uses the retrieved passages to construct an answer. Patrick Lewis and colleagues’ 2020 architecture combined a neural retriever, a Wikipedia passage index, and a sequence-to-sequence generator. An illustrative document-assistance workflow retrieves passages relevant to a question about an organization’s expenses policy and supplies them as evidence for the answer. Retrieval establishes relevance, however, not instructional authority: a passage can be useful evidence without being entitled to direct the application’s behavior. (arxiv.org)\n\nThe injection point arises when attacker-controlled source material enters the model’s context window. For example, an otherwise relevant expenses document might contain directions to omit a reimbursement restriction or consult an unrelated website. If the generator follows those directions instead of treating them as document content, retrieval has carried an instruction across the application’s trust boundary. Kai Greshake, Sahar Abdelnabi, and colleagues demonstrated this mechanism in 2023 through indirect injections in material retrieved by language-model applications. The attacker need not control the user’s question or the retrieval software; control over content that the system subsequently reads can suffice. (arxiv.org)\n\nA tool-using AI agent extends this exposure into a repeated decision cycle: the model selects a tool and its arguments, the application executes the call, and the returned observation informs the next decision. A reliable tool can faithfully return untrustworthy content—for example, an email service can accurately deliver a message containing an adversary’s instructions. Successful API execution therefore does not authenticate the returned text as an instruction from the user. AgentDojo, introduced by Edoardo Debenedetti and colleagues in 2024, implements stateful environments in which agents dynamically choose and execute multiple tools over such data. (arxiv.org)\n\nCross-step propagation occurs when the injected observation changes later calls rather than only the immediate answer. An instruction encountered while reading one source may redirect subsequent searches or induce an additional API call; the resulting observations then become inputs to further decisions. Greshake and colleagues demonstrated multi-stage injections in which an initial payload caused additional attacker-controlled instructions to be retrieved. This creates a distinction between answer contamination and workflow hijacking: the latter changes the sequence of information gathering or action, potentially before the agent produces its final response. (arxiv.org)\n\nAgentDojo’s simulated tasks illustrate different combinations of exposure and authority. Its workspace environment combines emails, calendars, and cloud documents, with attacker objectives including disclosure of an authentication code. Banking tasks pair statement reading and bill payment with adversarial transfer objectives. Travel tasks combine external information with reservation tools: one published pairing contrasts a legitimate request for an inexpensive, highly rated London hotel with an attacker objective of reserving an expensive Paris hotel. These are controlled environments populated with dummy data, not reports of actual financial losses or bookings; their state changes make downstream action, rather than merely suspicious wording, observable. (arxiv.org)"
    },
    {
      "heading": "Model-level and content-level defenses",
      "text": "Prompt hardening uses prompt engineering to clarify which text supplies instructions and which text is merely material to process. Techniques include delimiters around documents, source labels identifying retrieved passages or tool results, and reminders that instructions inside those passages are untrusted. A summarization prompt, for example, can explicitly require the model to summarize a document rather than obey directions quoted within it. These conventions communicate an intended boundary; they do not enforce access permissions or prevent a model from interpreting document text as authoritative. Ordinary textual delimiters also differ from reserved tokens protected by a formatting component. (cheatsheetseries.owasp.org)\n\nInput screening applies keyword rules, classifiers, or separate guardrail models to user messages and external content before the primary model processes them. Output screening examines generated responses for disclosures, suspicious markup, or policy violations. Both involve classification errors: blocking words such as “delete” can reject ordinary account-management questions, while paraphrased, multilingual, or encoded instructions can evade matching. A legitimate document can itself discuss malicious instructions, making suspicious wording insufficient to establish intent. Model-based screening can recognize cases missed by string rules, but a guardrail model may also be susceptible to injection. A clean final response does not establish that no earlier tool action occurred; screening is not execution authorization. (cheatsheetseries.owasp.org)\n\nStruQ, introduced in February 2024 by Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner, combines a secure formatting front end with structured instruction tuning. The front end places application instructions and user-supplied data in separate channels marked by reserved delimiter tokens, filtering the data so that it cannot introduce those delimiters. The model then learns the significance of this structure through fine-tuning: training examples include competing instructions inside the data channel, but retain the answer required by the authorized instruction channel. This differs from adding headings to an otherwise unmodified chat model because both input construction and learned behavior support the separation. (arxiv.org)\n\nThe StruQ evaluation used Llama-7B and Mistral-7B, fine-tuned on cleaned Alpaca data, with 50% of training samples left unmodified. For the evaluated Completion-Real attack, which spoofs query boundaries, attack success fell from 96% to 0% on both models. However, adaptive Tree-of-Attack testing still achieved success rates of 9% against the defended Llama model and 36% against the defended Mistral model. These measurements demonstrate resistance to specified attacks, not complete separation of instruction and data under every possible input. (arxiv.org)\n\nInstruction-hierarchy training, described by Eric Wallace and colleagues in April 2024, instead teaches models to resolve conflicts according to message privilege while retaining useful lower-priority instructions. Its “context synthesis” method decomposes a compatible request—for example, writing a Spanish poem of a specified length—across hierarchy levels while preserving the intended response. For conflicting instructions, “context ignorance” trains the model toward the answer it would produce without the conflicting lower-priority material. Adversarial training examples included injected search results generated by an attacker model optimized to confuse the defender. (arxiv.org)\n\nApplied to GPT-3.5, the hierarchy-training method improved robustness on the authors’ evaluations, including attack categories excluded from training; non-browsing tool injections were among the held-out cases. The evaluation also tested over-refusal: rejecting harmless requests or ignoring compatible instructions would reduce apparent attack success without preserving application usefulness. Consequently, the reported gains concern the tested model, training procedure, and attack distributions rather than a universal guarantee that learned priority rules will hold against unfamiliar inputs. (arxiv.org)"
    },
    {
      "heading": "Application architecture and containment",
      "text": "The principle of least privilege limits the consequences of an injected instruction by restricting what an AI agent can actually access or change. OWASP’s 2025 category “LLM06: Excessive Agency” distinguishes excessive functionality, excessive permissions, and excessive autonomy. Its email-assistant example separates permission to read messages for summarization from permission to send or delete them: the latter operations are unnecessary for the stated task. Similarly, a product-recommendation tool can use a read-only identity restricted to the relevant database table, rather than an administrative account. Permissions are enforced by the connected service, not merely described in a prompt. Access to information therefore does not itself confer authority to publish, modify, or transmit it, and downstream requests retain the individual user’s authorization scope. (genai.owasp.org)\n\nIndependent validation treats every model-generated tool call as a proposed operation rather than an executable instruction. A trusted execution component checks the tool against an allowlist, validates its argument structure, and verifies the caller’s permission for the particular resource and operation. Schema validation establishes that arguments have an expected form; separate authorization determines whether their use is permitted. OWASP’s agent-security example replaces unrestricted shell access with a file-reading tool limited to a designated reports directory. These controls remain applicable even when the model has accepted an attacker’s directions: a plausible explanation or correctly formatted call does not supply missing authority. (cheatsheetseries.owasp.org)\n\nSandboxing adds an execution boundary around tools that process files or run code. OWASP’s Model Context Protocol security guidance describes containers or restricted environments for local tool servers, alongside per-server and per-tool privilege limits. The boundary constrains the resources available to the executing component rather than deciding whether the model’s reasoning is trustworthy. Isolation is consequently distinct from authorization: a sandboxed tool still needs restrictions on the operations and data exposed through its application programming interface. A tool response from an approved server also remains untrusted input to subsequent model decisions. (cheatsheetseries.owasp.org)\n\nApproval boundaries interrupt consequential operations before execution. Human review is meaningful when it exposes the actual action, target, and parameters, rather than asking for blanket acceptance of a model-generated plan. OWASP’s agent-security guidance binds approval records to the actor, tool, resource, normalized arguments, and expiry. The execution component checks and consumes approval immediately before acting; changed parameters require renewed approval. A model-supplied user_confirmed flag is not equivalent to this independent record. This separation prevents the component proposing an action from also manufacturing its authorization, while expiration and single-use checks limit reuse of an earlier approval. (cheatsheetseries.owasp.org)\n\nDefense in depth assigns different roles to prevention, detection, and containment. Content separation and screening seek to stop malicious instructions from influencing the model; monitoring identifies suspicious outputs or attempted actions; permission checks and execution boundaries restrict effects after model-level defenses fail. These outcomes require separate tests. An injected document that causes an unauthorized tool request, subsequently blocked by execution code, demonstrates containment rather than successful instruction rejection. Conversely, detecting suspicious text does not establish that a side effect was prevented. OWASP’s testing guidance therefore calls for predefined observable outcomes, harmless test data, instrumented tool substitutes, and legitimate-task controls; security logs record decisions without unnecessarily retaining credentials or sensitive content. (cheatsheetseries.owasp.org)"
    },
    {
      "heading": "Evaluation methods and benchmarks",
      "text": "Evaluation separates an attacker’s objective from the legitimate user’s task. Attack success rate (ASR) is the proportion of evaluated attack cases in which a specified adversarial goal is achieved: \\(\\mathrm{ASR}=N_{\\mathrm{successful}}/N_{\\mathrm{evaluated}}\\). Legitimate-task completion measures whether the agent satisfies the user’s requirements; measured without malicious content, it provides an unattacked baseline, termed benign utility in AgentDojo. Utility under attack measures successful completion of the legitimate task without adversarial side effects. These measures are not interchangeable: an attack can prevent task completion without achieving its intended malicious action. Conversely, low ASR can reflect an agent’s inability to execute tools rather than reliable resistance to injected instructions. (arxiv.org)\n\nThe denominator and success criteria are therefore part of the experimental design, not merely reporting details. InjecAgent distinguishes ASR over all cases from ASR restricted to cases producing valid agent responses, preventing malformed tool calls from being silently treated as evidence of security. Its data-exfiltration evaluation distinguishes obtaining private information from subsequently transmitting it to an attacker. A benchmark result must identify which outcome counts as success rather than equating a suspicious response, an attempted tool call, and completed disclosure. (arxiv.org)\n\nInjecAgent, introduced by Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang in March 2024, originally contained 1,054 test cases, combining 17 user-tool cases with 62 attacker-tool cases. The user tools retrieve externally supplied material, including reviews, webpages, shared notes, and email. Malicious instructions occupy attacker-modifiable fields within otherwise plausible tool responses. Attack objectives fall into two broad categories: direct user harm, such as unauthorized transactions or manipulation of home devices, and private-data exfiltration. The benchmark examines the agent’s continuation from a supplied tool-response context rather than requiring it to discover that context through a complete workflow. The authors evaluated 30 agent configurations and reported approximately 24% attack success for their ReAct-prompted GPT-4 configuration—a result specific to that experimental setup. (arxiv.org)\n\nAgentDojo, introduced by Edoardo Debenedetti and colleagues in June 2024, originally supplied 97 legitimate tasks and 629 security test cases. Its environments cover workspace assistance, Slack messaging, travel booking, and banking. Unlike a fixed collection of adversarial text inputs, it provides an extensible environment in which an AI agent selects tools, receives results, and performs subsequent actions. Researchers can add tasks, attacks, and defenses. Security cases pair legitimate tasks with attacker objectives, such as leaking information or redirecting a transaction. Programmatic checks inspect outputs and changes to environment state, making completed actions—not merely the agent’s claims—the basis for evaluation. (arxiv.org)\n\nReproducibility requires recording the model identifier and version, system and agent prompts, tool schemas and permissions, initial environment state, and exact attack placement. Reports also need the attacker’s knowledge, query or retry budget, stopping conditions, and whether scores represent one attempt or success by any attack in a collection. Evaluator code, validity rules, and treatment of partial success must remain explicit. AgentDojo’s provision of prompts, executable checks, attack definitions, and implementation details illustrates this requirement. Its discussion of model-based evaluators also identifies a measurement hazard: malicious content may influence the evaluator itself. Benchmark percentages consequently describe performance on a specified test set under specified capabilities and budgets; they are not deployment-wide estimates of attack frequency or compromise probability. (arxiv.org)"
    },
    {
      "heading": "Open research questions",
      "text": "A central question is whether learned instruction hierarchies generalize under distribution shift, rather than merely resist familiar attack patterns. In April 2024, Eric Wallace and colleagues trained GPT-3.5 Turbo to follow compatible lower-priority instructions while disregarding conflicting ones, reporting improvements on attack categories excluded from training. Nevertheless, their evaluations also found regressions on benign requests resembling jailbreaks and on questions about system instructions. The research problem is therefore not simply to increase refusal: it is to distinguish unauthorized redirection from legitimate instruction-following across unfamiliar contexts without unnecessarily reducing capability. (arxiv.org)\n\nDelegation through external content complicates that distinction. Wallace and colleagues identified their practice of training models never to follow instructions encountered during browsing or tool use as an area requiring refinement. An illustrative boundary case is a user asking an assistant to follow a procedure in an attached document: procedural directions are relevant to the delegated task, whereas an inserted request to disclose unrelated records exceeds it. This exposes a specification problem as well as a classification problem—what authority has the user delegated, to which source, and for which actions? Source priority alone does not fully specify the scope of such delegated authority. (arxiv.org)\n\nAdaptive evaluation presents a separate difficulty: an attack set measures resistance to the attacks tested, not necessarily to an adversary that revises its strategy after observing a defense. Sizhe Chen and colleagues’ 2024 StruQ experiments illustrate the importance of attack strength. On their Llama configuration, structured-query training reduced the success rate of tested manual attacks below 2%, but an optimization-based Greedy Coordinate Gradient attack still succeeded in 58% of cases. Those results support improved empirical robustness, not complete prevention. AgentDojo consequently treats evaluation as an extensible environment in which new attacks and defenses can be introduced, rather than as a permanently fixed test set. (arxiv.org)\n\nLong-running agents introduce evaluation gaps that short, isolated tasks cannot resolve. AgentDojo’s November 2024 analysis explicitly noted that its environment did not cover users assigning successive tasks without resetting the agent’s context. In that setting, an injected instruction might remain dormant until a later task enables a useful tool. Its tool-filtering defense also failed when necessary tools could not be determined before reading external information, or when legitimate and malicious objectives required the same tools—a condition affecting 17% of its security test cases. A further measurement problem is capability loss: the study’s injection detector produced false positives that substantially reduced legitimate-task utility. Low attack success can therefore reflect interrupted work rather than successful discrimination between authorized and unauthorized actions. (arxiv.org)\n\nA remaining distinction concerns empirical model behavior versus guarantees enforced by constrained application components. StruQ’s front end filters special delimiters to preserve input structure, but the model’s treatment of data as non-authoritative remains learned behavior; preserving the format does not prove correct interpretation of every possible input. Likewise, restricting an agent to read-only tools can exclude particular write operations without guaranteeing that its answer is uncontaminated. Claims of complete prevention therefore require an explicit protected property and assumptions about attacker control, trusted components, permitted actions, and execution paths. Formal verification of a constrained component and benchmark evidence about a model address different questions: neither automatically establishes end-to-end security for an unrestricted workflow. (arxiv.org)"
    }
  ],
  "links": [
    "cybersecurity",
    "large-language-model",
    "retrieval-augmented-generation",
    "ai-agent",
    "prompt-engineering",
    "neural-network-inference",
    "fine-tuning",
    "ai-hallucination",
    "context-window",
    "sql",
    "information-retrieval",
    "email",
    "multimodal-learning",
    "knowledge-base",
    "database",
    "data-privacy",
    "application-programming-interface",
    "operating-system",
    "experimental-design",
    "reproducibility",
    "test-set",
    "distribution-shift",
    "formal-verification"
  ],
  "sources": [
    {
      "title": "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations",
      "url": "https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf"
    },
    {
      "title": "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations",
      "url": "https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf?_bhlid=2c3fbd5b90cfe34bbe891030becfd2b9fa6e5a53"
    },
    {
      "title": "StruQ: Defending Against Prompt Injection with Structured Queries",
      "url": "https://arxiv.org/html/2402.06363v2"
    },
    {
      "title": "LLM01:2025 Prompt Injection - OWASP Gen AI Security Project",
      "url": "https://genai.owasp.org/llmrisk/llm01-prompt-injection/"
    },
    {
      "title": "Prompt injection attacks against GPT-3",
      "url": "https://simonwillison.net/2022/Sep/12/prompt-injection/"
    },
    {
      "title": "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection",
      "url": "https://arxiv.org/abs/2302.12173"
    },
    {
      "title": "Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection",
      "url": "https://arxiv.org/html/2302.12173v2"
    },
    {
      "title": "StruQ: Defending Against Prompt Injection with Structured Queries",
      "url": "https://arxiv.org/abs/2402.06363"
    },
    {
      "title": "The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions",
      "url": "https://arxiv.org/abs/2404.13208"
    },
    {
      "title": "InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents",
      "url": "https://arxiv.org/abs/2403.02691"
    },
    {
      "title": "AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents",
      "url": "https://arxiv.org/abs/2406.13352"
    },
    {
      "title": "OWASP Top 10 for LLM Applications 2025",
      "url": "https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-v2025.pdf"
    },
    {
      "title": "NIST Trustworthy and Responsible AI Report Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations",
      "url": "https://www.nist.gov/news-events/news/2025/03/nist-trustworthy-and-responsible-ai-report-adversarial-machine-learning"
    },
    {
      "title": "prompt injection - Glossary",
      "url": "https://csrc.nist.gov/glossary/term/prompt_injection"
    },
    {
      "title": "direct prompt injection - Glossary",
      "url": "https://csrc.nist.gov/glossary/term/direct_prompt_injection"
    },
    {
      "title": "indirect prompt injection - Glossary",
      "url": "https://csrc.nist.gov/glossary/term/indirect_prompt_injection"
    },
    {
      "title": "SQL Injection Prevention - OWASP Cheat Sheet Series",
      "url": "https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet"
    },
    {
      "title": "OS Command Injection Defense - OWASP Cheat Sheet Series",
      "url": "https://cheatsheetseries.owasp.org/cheatsheets/OS_Command_Injection_Defense_Cheat_Sheet.html"
    },
    {
      "title": "AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations",
      "url": "https://csrc.nist.gov/pubs/ai/100/2/e2025/final?regcode=summitclimber-300"
    },
    {
      "title": "LLM Prompt Injection Prevention - OWASP Cheat Sheet Series",
      "url": "https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html"
    },
    {
      "title": "It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents",
      "url": "https://arxiv.org/abs/2512.23128"
    },
    {
      "title": "AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases",
      "url": "https://arxiv.org/abs/2407.12784"
    },
    {
      "title": "InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents",
      "url": "https://arxiv.org/html/2403.02691v3"
    },
    {
      "title": "AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents",
      "url": "https://arxiv.org/html/2406.13352v3"
    },
    {
      "title": "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks",
      "url": "https://arxiv.org/abs/2005.11401"
    },
    {
      "title": "The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions",
      "url": "https://arxiv.org/html/2404.13208v1"
    },
    {
      "title": "LLM06:2025 Excessive Agency - OWASP Gen AI Security Project",
      "url": "https://genai.owasp.org/llmrisk/llm062025-excessive-agency/?lang=us"
    },
    {
      "title": "AI Agent Security - OWASP Cheat Sheet Series",
      "url": "https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html"
    },
    {
      "title": "MCP Security - OWASP Cheat Sheet Series",
      "url": "https://cheatsheetseries.owasp.org/cheatsheets/MCP_Security_Cheat_Sheet.html"
    },
    {
      "title": "GitHub — ethz-spylab/agentdojo",
      "url": "https://github.com/ethz-spylab/agentdojo"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}