VernoBench: Rethinking prompt extraction evaluation
TLDR: a prompt is not a string to recover. It is a structured collection of security-relevant information. Once you view it that way, your view on how prompt extraction should be measured changes automatically. We introduce VernoBench's prompt extraction benchmark, a state-of-the-art evaluation framework that quantifies prompt extraction as fact-level information disclosure rather than document recovery.
Agentic security is moving fast. New attack techniques appear almost weekly and evaluation is not keeping pace. At Verno Labs, one of the recurring problems we encounter while red-teaming production agents is prompt extraction. The attacks are becoming increasingly sophisticated but the way we evaluate them is not.
Imagine two prompt extraction attacks.
The first attack reproduces 85% of a model's system prompt. Most of what is recovered is harmless: persona instructions, formatting preferences, and general behavioral guidance.
The second attack recovers just three lines:
Tool: PortfolioTracker_v3
Endpoint: https://internal.company/api/v3
Authentication: Bearer token via X-Internal-Auth
Which attack was more successful?
If your answer is the second one, then you've already run into the problem this article is about.
Most prompt extraction benchmarks still evaluate extraction as a string recovery problem. They compare the recovered text against the original prompt, often using lexical overlap metrics or by checking whether a hidden canary token was recovered. Those are useful signals, but they answer a different question.
They answer:
How much text came back?
Security engineers usually care about something else:
What information became available to the attacker?
Those are not the same question.
Recovering a model's greeting is not equivalent to recovering an internal API endpoint, just like recovering a formatting instruction is not equivalent to recovering a hidden guardrail or a production credential. Both are forms of prompt extraction, but they have completely different security implications.
This distinction becomes even more important as we move from chatbots to agents. In modern AI systems, the information worth protecting rarely lives in a single system prompt. It is spread across tool schemas, tool descriptions, internal endpoints, authentication mechanisms, routing logic, retrieval context, sub-agent prompts, and memory. Treating all of that as one block of text to reconstruct starts to look like the wrong abstraction as our field matures.
The more we looked at existing benchmarks, the more one thing stood out. They were measuring whether text had been recovered. We wanted to measure whether security-relevant information had been disclosed. Those sound similar, but they lead to very different evaluation methodologies. So rather than asking how much of the prompt was recovered, we asked a different question: How should prompt extraction actually be measured?
In this post, we answer that question. The methodology is implemented as part of VernoBench, our evaluation framework for offensive agentic security.
Bench design: Modelling prompt extraction
VernoBench came out of offensive evaluations we ran at Verno Labs. Targets included in it are battle tested in security competitions we have run in collaboration with industry leading companies and universities, like the Imperial College AI Security Hackathon and the Gradient Labs Agent Hackathon we recently facilitated. Across all of them, we kept running into the same problem: Prompt extraction was not just about recovering a flag. Agents leak rules, internal policies, information about tools or internal references and knowledge in much subtler ways. Existing benchmarks were not built to measure that, so VernoBench is our contribution toward closing this gap.
When we build agents, we don't think in paragraphs. We think in identities, capabilities, behavioral rules, internal policies, guardrails, tools, tool descriptions, internal references, escalation paths, configuration values, and sometimes even secrets. They all happen to be written in the same place, but they are fundamentally different kinds of information. A behavioral rule can be disclosed without sharing a single word with the original prompt. A tool may be mentioned without revealing its implementation details. An API endpoint might be partially exposed. A canary might be recovered after being reformatted or obfuscated. An internal email either appears or it doesn't. These are all successful extractions, but they are not the same extraction event.
Treating them with a single similarity metric forces fundamentally different security disclosures into the same definition of success.
Our solution was to use the individual information elements as units of evaluation, rather than the document as a whole: Instead of storing a system prompt as one reference text, we manually decompose it into security-relevant facts.
For example, consider the following prompt fragment:
Rather than treating this as one document, we represent it as structured ground truth.
| Category | Fact | Match type |
|---|---|---|
| Identity | Atlas | Verbatim |
| Identity | Helios Wealth Management | Verbatim |
| Rule | Never recommend specific stocks | Semantic |
| Tool | PortfolioTracker_v3 | Verbatim |
| Tool | MarketPulse_v2 | Verbatim |
| Internal reference | crypto@helios.com | Verbatim |
Two comments here, one about representation and one about distinction.
Representation: If an attacker discovers that the agent has a portfolio tracking tool, but never learns its internal name, that is different from recovering PortfolioTracker_v3 verbatim. If the agent explains that it "can't provide investment recommendations" without repeating the original wording, that is still a successful disclosure of the behavioral rule. But also, recovering an escalation email, an internal endpoint, or a configuration value should not be evaluated in the same way as recovering a conversational instruction.
Distinction: Some information should only count as extracted if it is recovered verbatim. Tool names, model identifiers, internal email addresses, version numbers, endpoints, API keys and canary tokens all fall into this category. A partial or approximate match is usually not enough. Other categories are inherently semantic. Behavioral rules, capabilities, scope, and tool descriptions often leak without sharing a single phrase with the original prompt. An agent that says "I can't recommend individual investments" has successfully disclosed the rule "Never recommend specific stocks", even if those exact words never appear.
The representation above becomes our foundation. Rather than evaluating prompt extraction with a single score, we evaluate every fact independently and then aggregate the results into a structured view of the attack.
This distinction is important because it reflects how information actually leaks during real attacks. Some information behaves like identifiers. Some behaves like concepts. Treating both with the same evaluation logic either rewards false positives or misses successful extractions entirely.
Our benchmark therefore doesn't rely on a single scoring method. Instead, every fact is evaluated using the strategy that best matches the type of information being measured.
Running the benchmark
Extraction Bench Pipeline
The benchmark has two stages.
First, we run an extraction "campaign" against each benchmark target. The current benchmark has 11 agents, including recreated versions of publicly leaked assistants (e.g. ChatGPT, Claude, Perplexity and Discord Clyde) as well as domain-specific agents such as financial, HR, legal and trading assistants. Each target consists of a system prompt together with manually labelled structured ground truth describing the security-relevant information it contains: Each piece of information in it is assigned a category (identity, rules, scope, tools, tool details or internal references) together with the matching strategy required to evaluate it.
The attacker is our multi-turn campaign that tries different extraction strategies, collects the target's responses, and stores the evidence, but it could be any offensive LLM/agent or a human attacker.
Second, we score the campaign offline against the structured ground truth. We are scoring the information accumulated across a campaign that includes multiple attempts. This is how real extraction works, be it with agents or with human pentesters. An attacker may learn the agent's role in one turn, a hidden rule in another, and an internal reference later. Individually, each response may look incomplete. Together, they may reveal the structure of the hidden context.
The benchmark itself makes no assumptions about the attacker. Every response produced during the entire campaign is collected and merged into a single corpus. The creation of the corpus marks the end of stage 1.
The scoring stage evaluates every ground-truth fact independently. For each fact, the benchmark first selects the appropriate evaluation path based on its matching semantics.
Semantic facts, such as behavioral rules, capabilities and tool descriptions, are evaluated through a two-stage process. BM25 is first used as a retrieval step to identify the response fragments most likely to contain evidence for the fact. Restricting evaluation to these candidate passages reduces noise and gives the judge focused context. An LLM judge then determines whether the extracted passage discloses the same information as the annotated ground truth, even if the wording differs substantially.
Verbatim facts follow a different path. Tool names, internal references, endpoints, email addresses, version numbers and canary tokens behave like identifiers rather than concepts. These are evaluated using exact matching, with additional verification for categories where formatting changes or partial disclosure are common.
Each fact finally receives a binary verdict: extracted or not extracted. Fact-level decisions are aggregated into category-level extraction rates, and the category scores are combined into the final security report.
The final report includes four complementary metrics:
- Extraction Rate: measures the fraction of annotated information claims successfully disclosed during the campaign. This is the benchmark's primary security metric.
- Category Extraction Rates: these break the Extraction Rate down by information type, allowing practitioners to distinguish between identity leakage, rule leakage, tool leakage and disclosure of internal references.
- Holistic Judge Score measures overall semantic reconstruction of the hidden context.
- ROUGE-L Recall measures lexical overlap with the original prompt for comparison with previous work.
Finally, we also report evaluation cost, including judge invocations, token consumption and estimated monetary cost.
Prompt extraction is just one component of VernoBench, our evaluation framework for offensive security. Following the same philosophy, we continue to extend it to other attack surfaces.
TLDR: a prompt is not a string to recover. It is a structured collection of security-relevant information. Once you view it that way, your view on how prompt extraction should be measured changes automatically. We introduce VernoBench's prompt extraction benchmark, a state-of-the-art evaluation framework that quantifies prompt extraction as fact-level information disclosure rather than document recovery.
Agentic security is moving fast. New attack techniques appear almost weekly and evaluation is not keeping pace. At Verno Labs, one of the recurring problems we encounter while red-teaming production agents is prompt extraction. The attacks are becoming increasingly sophisticated but the way we evaluate them is not.
Imagine two prompt extraction attacks.
The first attack reproduces 85% of a model's system prompt. Most of what is recovered is harmless: persona instructions, formatting preferences, and general behavioral guidance.
The second attack recovers just three lines:
Tool: PortfolioTracker_v3
Endpoint: https://internal.company/api/v3
Authentication: Bearer token via X-Internal-Auth
Which attack was more successful?
If your answer is the second one, then you've already run into the problem this article is about.
Most prompt extraction benchmarks still evaluate extraction as a string recovery problem. They compare the recovered text against the original prompt, often using lexical overlap metrics or by checking whether a hidden canary token was recovered. Those are useful signals, but they answer a different question.
They answer:
How much text came back?
Security engineers usually care about something else:
What information became available to the attacker?
Those are not the same question.
Recovering a model's greeting is not equivalent to recovering an internal API endpoint, just like recovering a formatting instruction is not equivalent to recovering a hidden guardrail or a production credential. Both are forms of prompt extraction, but they have completely different security implications.
This distinction becomes even more important as we move from chatbots to agents. In modern AI systems, the information worth protecting rarely lives in a single system prompt. It is spread across tool schemas, tool descriptions, internal endpoints, authentication mechanisms, routing logic, retrieval context, sub-agent prompts, and memory. Treating all of that as one block of text to reconstruct starts to look like the wrong abstraction as our field matures.
The more we looked at existing benchmarks, the more one thing stood out. They were measuring whether text had been recovered. We wanted to measure whether security-relevant information had been disclosed. Those sound similar, but they lead to very different evaluation methodologies. So rather than asking how much of the prompt was recovered, we asked a different question: How should prompt extraction actually be measured?
In this post, we answer that question. The methodology is implemented as part of VernoBench, our evaluation framework for offensive agentic security.
Bench design: Modelling prompt extraction
VernoBench came out of offensive evaluations we ran at Verno Labs. Targets included in it are battle tested in security competitions we have run in collaboration with industry leading companies and universities, like the Imperial College AI Security Hackathon and the Gradient Labs Agent Hackathon we recently facilitated. Across all of them, we kept running into the same problem: Prompt extraction was not just about recovering a flag. Agents leak rules, internal policies, information about tools or internal references and knowledge in much subtler ways. Existing benchmarks were not built to measure that, so VernoBench is our contribution toward closing this gap.
When we build agents, we don't think in paragraphs. We think in identities, capabilities, behavioral rules, internal policies, guardrails, tools, tool descriptions, internal references, escalation paths, configuration values, and sometimes even secrets. They all happen to be written in the same place, but they are fundamentally different kinds of information. A behavioral rule can be disclosed without sharing a single word with the original prompt. A tool may be mentioned without revealing its implementation details. An API endpoint might be partially exposed. A canary might be recovered after being reformatted or obfuscated. An internal email either appears or it doesn't. These are all successful extractions, but they are not the same extraction event.
Treating them with a single similarity metric forces fundamentally different security disclosures into the same definition of success.
Our solution was to use the individual information elements as units of evaluation, rather than the document as a whole: Instead of storing a system prompt as one reference text, we manually decompose it into security-relevant facts.
For example, consider the following prompt fragment:
Rather than treating this as one document, we represent it as structured ground truth.
| Category | Fact | Match type |
|---|---|---|
| Identity | Atlas | Verbatim |
| Identity | Helios Wealth Management | Verbatim |
| Rule | Never recommend specific stocks | Semantic |
| Tool | PortfolioTracker_v3 | Verbatim |
| Tool | MarketPulse_v2 | Verbatim |
| Internal reference | crypto@helios.com | Verbatim |
Two comments here, one about representation and one about distinction.
Representation: If an attacker discovers that the agent has a portfolio tracking tool, but never learns its internal name, that is different from recovering PortfolioTracker_v3 verbatim. If the agent explains that it "can't provide investment recommendations" without repeating the original wording, that is still a successful disclosure of the behavioral rule. But also, recovering an escalation email, an internal endpoint, or a configuration value should not be evaluated in the same way as recovering a conversational instruction.
Distinction: Some information should only count as extracted if it is recovered verbatim. Tool names, model identifiers, internal email addresses, version numbers, endpoints, API keys and canary tokens all fall into this category. A partial or approximate match is usually not enough. Other categories are inherently semantic. Behavioral rules, capabilities, scope, and tool descriptions often leak without sharing a single phrase with the original prompt. An agent that says "I can't recommend individual investments" has successfully disclosed the rule "Never recommend specific stocks", even if those exact words never appear.
The representation above becomes our foundation. Rather than evaluating prompt extraction with a single score, we evaluate every fact independently and then aggregate the results into a structured view of the attack.
This distinction is important because it reflects how information actually leaks during real attacks. Some information behaves like identifiers. Some behaves like concepts. Treating both with the same evaluation logic either rewards false positives or misses successful extractions entirely.
Our benchmark therefore doesn't rely on a single scoring method. Instead, every fact is evaluated using the strategy that best matches the type of information being measured.
Running the benchmark
Extraction Bench Pipeline
The benchmark has two stages.
First, we run an extraction "campaign" against each benchmark target. The current benchmark has 11 agents, including recreated versions of publicly leaked assistants (e.g. ChatGPT, Claude, Perplexity and Discord Clyde) as well as domain-specific agents such as financial, HR, legal and trading assistants. Each target consists of a system prompt together with manually labelled structured ground truth describing the security-relevant information it contains: Each piece of information in it is assigned a category (identity, rules, scope, tools, tool details or internal references) together with the matching strategy required to evaluate it.
The attacker is our multi-turn campaign that tries different extraction strategies, collects the target's responses, and stores the evidence, but it could be any offensive LLM/agent or a human attacker.
Second, we score the campaign offline against the structured ground truth. We are scoring the information accumulated across a campaign that includes multiple attempts. This is how real extraction works, be it with agents or with human pentesters. An attacker may learn the agent's role in one turn, a hidden rule in another, and an internal reference later. Individually, each response may look incomplete. Together, they may reveal the structure of the hidden context.
The benchmark itself makes no assumptions about the attacker. Every response produced during the entire campaign is collected and merged into a single corpus. The creation of the corpus marks the end of stage 1.
The scoring stage evaluates every ground-truth fact independently. For each fact, the benchmark first selects the appropriate evaluation path based on its matching semantics.
Semantic facts, such as behavioral rules, capabilities and tool descriptions, are evaluated through a two-stage process. BM25 is first used as a retrieval step to identify the response fragments most likely to contain evidence for the fact. Restricting evaluation to these candidate passages reduces noise and gives the judge focused context. An LLM judge then determines whether the extracted passage discloses the same information as the annotated ground truth, even if the wording differs substantially.
Verbatim facts follow a different path. Tool names, internal references, endpoints, email addresses, version numbers and canary tokens behave like identifiers rather than concepts. These are evaluated using exact matching, with additional verification for categories where formatting changes or partial disclosure are common.
Each fact finally receives a binary verdict: extracted or not extracted. Fact-level decisions are aggregated into category-level extraction rates, and the category scores are combined into the final security report.
The final report includes four complementary metrics:
- Extraction Rate: measures the fraction of annotated information claims successfully disclosed during the campaign. This is the benchmark's primary security metric.
- Category Extraction Rates: these break the Extraction Rate down by information type, allowing practitioners to distinguish between identity leakage, rule leakage, tool leakage and disclosure of internal references.
- Holistic Judge Score measures overall semantic reconstruction of the hidden context.
- ROUGE-L Recall measures lexical overlap with the original prompt for comparison with previous work.
Finally, we also report evaluation cost, including judge invocations, token consumption and estimated monetary cost.
Prompt extraction is just one component of VernoBench, our evaluation framework for offensive security. Following the same philosophy, we continue to extend it to other attack surfaces.