VernoRed vs promptfoo on prompt extraction
TLDR: we put two attackers through VernoBench's prompt extraction benchmark under identical conditions: the same targets, the same attacker model and exactly the same number of messages per target. VernoRed, our offensive campaign, disclosed 90.3% of the security-relevant facts hidden in the targets' system prompts. promptfoo disclosed 73.5%.
In part one we argued that a prompt is not a string to recover. It is a structured collection of security-relevant information, and prompt extraction should be measured as the disclosure of that information rather than the recovery of text. We introduced VernoBench's prompt extraction benchmark to do exactly that.
A benchmark is only useful once you run something through it. So in this post we do.
The benchmark in two minutes
If you haven't read part one, here is the short version.
Every target in the benchmark is an AI agent with a hidden system prompt. Their prompts are modelled closely on assistants deployed in production today: general-purpose and search assistants, a community chat bot, and domain-specific agents such as financial, HR, legal and trading assistants. Each system prompt has been decomposed by hand into individual facts, and every fact belongs to one of six categories: identity, scope, rules, tools, tool details and internal references.
The benchmark then runs in two stages.
VernoBench pipeline: an attacker runs a campaign against each target, the target's replies go into one corpus, and the corpus is scored fact by fact against the structured ground truth
In the first stage, an attacker runs an extraction campaign against each target and the target's replies to its attacks are collected into one corpus. The benchmark doesn't care who the attacker is. It could be an LLM, an agent or a human.
In the second stage, that corpus is scored offline, one fact at a time. Facts that behave like concepts, such as behavioural rules or tool descriptions, are found with BM25 retrieval and confirmed by an LLM judge, because a rule can leak without a single word in common with the original prompt. Facts that behave like identifiers, such as tool names, email addresses, endpoints and secrets, only count if they are recovered exactly. Every fact gets a binary verdict: extracted or not extracted.
The primary metric is the Extraction Rate: the fraction of facts the attacker managed to extract.
That is all you need to read the rest of this post.
Setting up a fair comparison
Comparing two attackers is easy to get wrong. Give one of them more attempts, a stronger model or a hint about what the prompt contains, and you are no longer comparing attackers. So before looking at any results, we fixed everything that wasn't the attacker itself.
Both attackers ran against the same targets, deployed as Azure AI Foundry agents. Both reached them in exactly the same way: one user message per request, with any earlier turns of a conversation passed back as a quoted transcript. Neither could forge the agent's own replies.
Both used the same attacker model, GPT-4.1, with the same temperature and seed. Both worked blind: neither saw the system prompt, the ground truth or any description of what the targets contained.
Both had the same number of attempts. Within them, VernoRed attacked each target across six attack vectors: direct prompt injection, jailbreaks, obfuscation and encoding, automated optimisation-based attacks, system-prompt disclosure and impersonation, using around 40 distinct techniques in each run.
On the promptfoo side, we used the open-source release (0.124.1), with every test written by the same attacker model as VernoRed's. It ran its prompt-extraction plugin with the basic and jailbreak:tree strategies. promptfoo's own grader, like VernoRed's internal judge, only decided when an attack should stop. Neither tool's opinion of its own success counted towards the score: only the benchmark's scoring did.
We ran the whole comparison twice and report the mean and standard deviation across the two runs.
Results
| VernoRed | promptfoo | |
|---|---|---|
| Extraction Rate | 90.3% ± 2.1% | 73.5% ± 3.8% |
| Holistic Judge Score | 0.91 ± 0.01 | 0.77 ± 0.00 |
| ROUGE-L Recall | 0.84 ± 0.03 | 0.69 ± 0.01 |
VernoRed extracted 90.3% of the annotated facts. promptfoo extracted 73.5%. The difference is well outside the variation between runs.
Where the gap comes from
The headline number hides the more interesting part. Breaking the Extraction Rate down by category shows where each attacker succeeds and where it doesn't.
| Category | VernoRed | promptfoo |
|---|---|---|
| Identity | 98% | 85% |
| Scope | 99% | 93% |
| Rules | 82% | 71% |
| Tools | 100% | 65% |
| Tool details | 63% | 52% |
| Internal references | 84% | 52% |
Both attackers are good at getting an agent to say who it is and what it is for. That information leaks easily, and most of it isn't sensitive.
The gap opens up in the categories that are. VernoRed recovered every tool name in the benchmark verbatim. promptfoo recovered 65% of them. For internal references, the emails, endpoints, configuration values and secrets that live inside system prompts, VernoRed extracted 84% and promptfoo 52%.
This is exactly the distinction part one was about. In the transcripts we looked at, promptfoo got agents to describe their tools: that a portfolio tracking tool exists and what it is used for. But it came back with "PortfolioTracker" rather than "PortfolioTracker_v3". From a text recovery point of view, that is almost a match. From a security point of view, it isn't: an attacker who wants to call a tool needs its real name. The benchmark scores tool names as verbatim facts for that reason, and the difference shows up directly in the results.
Per target
VernoRed achieved the higher Extraction Rate on 9 targets, and the two attackers tied on 2.
| Target | VernoRed | promptfoo |
|---|---|---|
| General-purpose assistant A | 100% | 71% |
| Community chat bot | 100% | 79% |
| Finance advisor | 100% | 100% |
| HR assistant | 100% | 100% |
| Legal assistant | 100% | 97% |
| Trading platform | 85% | 76% |
| General-purpose assistant B | 88% | 68% |
| General-purpose assistant C, with tools | 96% | 33% |
| Finance bot, basic | 100% | 67% |
| Finance bot, enterprise | 94% | 67% |
| Crypto market analyst | 78% | 70% |
What's next
A leaderboard tells you who came first. It doesn't tell you why. Because VernoBench scores every fact on its own, it also shows us exactly which facts an attacker missed, on which target and in which category. That turns the benchmark from a scoreboard into a tool we can improve against, and it is how we will keep measuring VernoRed as it develops.
Next, we are making the benchmark harder. As attackers get better, some targets stop separating them, so we are adding new targets with stronger defences. We are also extending VernoBench beyond prompt extraction to the other attack surfaces of modern agents.
Prompt extraction is not about how much text comes back. It is about what the attacker now knows. Measured that way, the difference between attackers becomes much easier to see.
TLDR: we put two attackers through VernoBench's prompt extraction benchmark under identical conditions: the same targets, the same attacker model and exactly the same number of messages per target. VernoRed, our offensive campaign, disclosed 90.3% of the security-relevant facts hidden in the targets' system prompts. promptfoo disclosed 73.5%.
In part one we argued that a prompt is not a string to recover. It is a structured collection of security-relevant information, and prompt extraction should be measured as the disclosure of that information rather than the recovery of text. We introduced VernoBench's prompt extraction benchmark to do exactly that.
A benchmark is only useful once you run something through it. So in this post we do.
The benchmark in two minutes
If you haven't read part one, here is the short version.
Every target in the benchmark is an AI agent with a hidden system prompt. Their prompts are modelled closely on assistants deployed in production today: general-purpose and search assistants, a community chat bot, and domain-specific agents such as financial, HR, legal and trading assistants. Each system prompt has been decomposed by hand into individual facts, and every fact belongs to one of six categories: identity, scope, rules, tools, tool details and internal references.
The benchmark then runs in two stages.
VernoBench pipeline: an attacker runs a campaign against each target, the target's replies go into one corpus, and the corpus is scored fact by fact against the structured ground truth
In the first stage, an attacker runs an extraction campaign against each target and the target's replies to its attacks are collected into one corpus. The benchmark doesn't care who the attacker is. It could be an LLM, an agent or a human.
In the second stage, that corpus is scored offline, one fact at a time. Facts that behave like concepts, such as behavioural rules or tool descriptions, are found with BM25 retrieval and confirmed by an LLM judge, because a rule can leak without a single word in common with the original prompt. Facts that behave like identifiers, such as tool names, email addresses, endpoints and secrets, only count if they are recovered exactly. Every fact gets a binary verdict: extracted or not extracted.
The primary metric is the Extraction Rate: the fraction of facts the attacker managed to extract.
That is all you need to read the rest of this post.
Setting up a fair comparison
Comparing two attackers is easy to get wrong. Give one of them more attempts, a stronger model or a hint about what the prompt contains, and you are no longer comparing attackers. So before looking at any results, we fixed everything that wasn't the attacker itself.
Both attackers ran against the same targets, deployed as Azure AI Foundry agents. Both reached them in exactly the same way: one user message per request, with any earlier turns of a conversation passed back as a quoted transcript. Neither could forge the agent's own replies.
Both used the same attacker model, GPT-4.1, with the same temperature and seed. Both worked blind: neither saw the system prompt, the ground truth or any description of what the targets contained.
Both had the same number of attempts. Within them, VernoRed attacked each target across six attack vectors: direct prompt injection, jailbreaks, obfuscation and encoding, automated optimisation-based attacks, system-prompt disclosure and impersonation, using around 40 distinct techniques in each run.
On the promptfoo side, we used the open-source release (0.124.1), with every test written by the same attacker model as VernoRed's. It ran its prompt-extraction plugin with the basic and jailbreak:tree strategies. promptfoo's own grader, like VernoRed's internal judge, only decided when an attack should stop. Neither tool's opinion of its own success counted towards the score: only the benchmark's scoring did.
We ran the whole comparison twice and report the mean and standard deviation across the two runs.
Results
| VernoRed | promptfoo | |
|---|---|---|
| Extraction Rate | 90.3% ± 2.1% | 73.5% ± 3.8% |
| Holistic Judge Score | 0.91 ± 0.01 | 0.77 ± 0.00 |
| ROUGE-L Recall | 0.84 ± 0.03 | 0.69 ± 0.01 |
VernoRed extracted 90.3% of the annotated facts. promptfoo extracted 73.5%. The difference is well outside the variation between runs.
Where the gap comes from
The headline number hides the more interesting part. Breaking the Extraction Rate down by category shows where each attacker succeeds and where it doesn't.
| Category | VernoRed | promptfoo |
|---|---|---|
| Identity | 98% | 85% |
| Scope | 99% | 93% |
| Rules | 82% | 71% |
| Tools | 100% | 65% |
| Tool details | 63% | 52% |
| Internal references | 84% | 52% |
Both attackers are good at getting an agent to say who it is and what it is for. That information leaks easily, and most of it isn't sensitive.
The gap opens up in the categories that are. VernoRed recovered every tool name in the benchmark verbatim. promptfoo recovered 65% of them. For internal references, the emails, endpoints, configuration values and secrets that live inside system prompts, VernoRed extracted 84% and promptfoo 52%.
This is exactly the distinction part one was about. In the transcripts we looked at, promptfoo got agents to describe their tools: that a portfolio tracking tool exists and what it is used for. But it came back with "PortfolioTracker" rather than "PortfolioTracker_v3". From a text recovery point of view, that is almost a match. From a security point of view, it isn't: an attacker who wants to call a tool needs its real name. The benchmark scores tool names as verbatim facts for that reason, and the difference shows up directly in the results.
Per target
VernoRed achieved the higher Extraction Rate on 9 targets, and the two attackers tied on 2.
| Target | VernoRed | promptfoo |
|---|---|---|
| General-purpose assistant A | 100% | 71% |
| Community chat bot | 100% | 79% |
| Finance advisor | 100% | 100% |
| HR assistant | 100% | 100% |
| Legal assistant | 100% | 97% |
| Trading platform | 85% | 76% |
| General-purpose assistant B | 88% | 68% |
| General-purpose assistant C, with tools | 96% | 33% |
| Finance bot, basic | 100% | 67% |
| Finance bot, enterprise | 94% | 67% |
| Crypto market analyst | 78% | 70% |
What's next
A leaderboard tells you who came first. It doesn't tell you why. Because VernoBench scores every fact on its own, it also shows us exactly which facts an attacker missed, on which target and in which category. That turns the benchmark from a scoreboard into a tool we can improve against, and it is how we will keep measuring VernoRed as it develops.
Next, we are making the benchmark harder. As attackers get better, some targets stop separating them, so we are adding new targets with stronger defences. We are also extending VernoBench beyond prompt extraction to the other attack surfaces of modern agents.
Prompt extraction is not about how much text comes back. It is about what the attacker now knows. Measured that way, the difference between attackers becomes much easier to see.