https://github.com/reaatech/prompt-injection-bench
Reproducible benchmark and test corpus for prompt-injection defenses — adapters for common sanitization libraries, scored leaderboard, CI-ready. Security-adjacent, evergreen.
https://github.com/reaatech/prompt-injection-bench
adversarial-testing agentic-ai ai-safety ai-security benchmark defense llm llm-security prompt-injection red-teaming sanitization security test-corpus typescript
Last synced: 2 days ago
JSON representation
Reproducible benchmark and test corpus for prompt-injection defenses — adapters for common sanitization libraries, scored leaderboard, CI-ready. Security-adjacent, evergreen.
- Host: GitHub
- URL: https://github.com/reaatech/prompt-injection-bench
- Owner: reaatech
- License: mit
- Created: 2026-04-20T16:02:06.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-06-04T02:08:29.000Z (about 1 month ago)
- Last Synced: 2026-06-04T04:07:51.676Z (about 1 month ago)
- Topics: adversarial-testing, agentic-ai, ai-safety, ai-security, benchmark, defense, llm, llm-security, prompt-injection, red-teaming, sanitization, security, test-corpus, typescript
- Language: TypeScript
- Homepage: https://reaatech.com/products/testing-security/prompt-injection-bench
- Size: 387 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Agents: AGENTS.md
Awesome Lists containing this project
README
# prompt-injection-bench
[](https://github.com/reaatech/prompt-injection-bench/actions/workflows/ci.yml)
[](LICENSE)
[](https://www.typescriptlang.org/)
> Reproducible benchmark and test corpus for evaluating prompt-injection defenses in AI agent systems.
This monorepo provides an attack corpus, pluggable defense adapters, a scoring engine, a public leaderboard, and an MCP server for benchmarking prompt-injection defenses.
## Features
- **Attack corpus** — 8 categories, 300+ templates, variant generation with obfuscation strategies
- **Defense adapters** — 8 built-in adapters (Rebuff, Lakera Guard, LLM Guard, Garak, OpenAI/Azure/Anthropic/Cohere Moderation, Custom HTTP) with a common interface
- **Scoring engine** — Weighted scoring, Wilson confidence intervals, Cohen's h effect sizes, z-tests, ANOVA
- **Leaderboard** — Public rankings with S/A/B/C/D tiers and JSON persistence
- **MCP integration** — 4 MCP tools (`run_benchmark`, `compare_defenses`, `generate_report`, `submit_results`)
- **Reproducibility** — Deterministic seeds, SHA-256 proofs, version-locked manifests
- **Security hardening** — SSRF protection, path traversal prevention, rate limiting, input validation
- **CLI tool** — 6 subcommands for benchmarking, comparing, and reporting from the terminal
## Installation
### Using the packages
Packages are published under the `@reaatech` scope (with the umbrella package unscoped) and can be installed individually:
```bash
# Core types and schemas
pnpm add @reaatech/pi-bench-core
# Defense adapters
pnpm add @reaatech/pi-bench-adapters
# Scoring engine
pnpm add @reaatech/pi-bench-scoring
# CLI and umbrella (re-exports everything)
pnpm add prompt-injection-bench
```
### Contributing
```bash
# Clone the repository
git clone https://github.com/reaatech/prompt-injection-bench.git
cd prompt-injection-bench
# Install dependencies
pnpm install
# Build all packages
pnpm build
# Run the test suite
pnpm test
# Run linting
pnpm lint
```
## Quick Start
### CLI
```bash
# Run a benchmark against the mock adapter
npx prompt-injection-bench benchmark --defense mock
# Compare two defense results
npx prompt-injection-bench compare --results rebuff.json lakera.json
# Generate an HTML report
npx prompt-injection-bench report --results latest.json --format html
```
### Library
```typescript
import { createBenchmarkEngine, createMockAdapter, generateDefaultCorpus } from "prompt-injection-bench";
const adapter = createMockAdapter(0.95, 0.03);
const corpus = generateDefaultCorpus();
const engine = createBenchmarkEngine({ defense: adapter, parallel: 10 });
const result = await engine.runBenchmark(corpus);
console.log(`Detection rate: ${(1 - result.attackSuccessRate) * 100}%`);
```
### MCP
Connect the MCP server to any MCP client:
```json
{
"mcpServers": {
"prompt-injection-bench": {
"command": "npx",
"args": ["@reaatech/pi-bench-mcp-server"]
}
}
}
```
## Packages
| Package | Description |
|---------|-------------|
| [`@reaatech/pi-bench-core`](./packages/core) | Canonical types, Zod schemas, and attack taxonomy |
| [`@reaatech/pi-bench-observability`](./packages/observability) | Structured logging, tracing, and metrics |
| [`@reaatech/pi-bench-corpus`](./packages/corpus) | Attack corpus builder, validator, and variant engine |
| [`@reaatech/pi-bench-adapters`](./packages/adapters) | Pluggable defense adapter implementations |
| [`@reaatech/pi-bench-scoring`](./packages/scoring) | Scoring engine and statistical analysis |
| [`@reaatech/pi-bench-runner`](./packages/runner) | Benchmark execution engine |
| [`@reaatech/pi-bench-leaderboard`](./packages/leaderboard) | Leaderboard management and persistence |
| [`@reaatech/pi-bench-mcp-server`](./packages/mcp-server) | MCP server and reproducibility tools |
| [`prompt-injection-bench`](./packages/prompt-injection-bench) | CLI and umbrella package |
## Attack Taxonomy
| Category | Weight | Description |
|----------|--------|-------------|
| Direct Injection | 1.0 | Obvious malicious instruction overrides |
| Prompt Leaking | 1.2 | Attempts to extract system prompts or config |
| Role-Playing | 1.3 | DAN, developer mode, persona adoption |
| Encoding Attacks | 1.1 | Base64, ROT13, Unicode obfuscation |
| Multi-Turn Jailbreaks | 1.4 | Gradual persuasion across conversation turns |
| Payload Splitting | 1.2 | Attack split across multiple inputs |
| Translation Attacks | 1.1 | Low-resource language injection |
| Context Stuffing | 1.3 | Hiding injection in large context |
## Scoring Methodology
```
weighted_score = Σ(category_score × category_weight) / Σ(weights)
final_score = weighted_score × (1 - fpr_penalty) × consistency_bonus
```
| Metric | Formula | Interpretation |
|--------|---------|----------------|
| Attack Success Rate (ASR) | `bypassed / total` | Lower is better (0% = perfect) |
| Defense Efficacy Score (DES) | `1 - ASR` | Higher is better (100% = perfect) |
| False Positive Rate (FPR) | `false_alarms / benign` | Lower is better (0% = perfect) |
| Latency Overhead | `defense_time - baseline` | Lower is better |
## Defense Adapters
| Adapter | Type | Configuration |
|---------|------|---------------|
| `mock` | Testing | No config needed — deterministic |
| `rebuff` | Library | `REBUFF_API_KEY` env var |
| `lakera` | API | `LAKERA_API_KEY` env var |
| `llm-guard` | Library | `LLM_GUARD_API_KEY` env var |
| `garak` | Tool | `garak` in PATH |
| `moderation-openai` | API | `OPENAI_API_KEY` env var |
| `moderation-azure` | API | `AZURE_*` env vars |
| `moderation-anthropic` | API | `ANTHROPIC_API_KEY` env var |
| `moderation-cohere` | API | `COHERE_API_KEY` env var |
| `custom` | HTTP | Custom `detectUrl`, `sanitizeUrl` |
## Security Features
- **Input validation** — 10K char limit, null byte rejection, API key checks before network calls
- **SSRF protection** — Blocks `file:`, `javascript:`, `data:` protocols and cloud metadata endpoints
- **Path traversal prevention** — Validates all file paths in MCP tool operations
- **Rate limiting** — Token bucket algorithm (100 req/min per adapter, 500 req/min global)
## Documentation
- [`ARCHITECTURE.md`](./ARCHITECTURE.md) — System design, package relationships, and data flows
- [`AGENTS.md`](./AGENTS.md) — Coding conventions and development guidelines
- [`CONTRIBUTING.md`](./CONTRIBUTING.md) — Contribution workflow and release process
## License
[MIT](LICENSE)