{"id":51702346,"url":"https://github.com/reaatech/prompt-injection-bench","last_synced_at":"2026-07-16T12:30:49.928Z","repository":{"id":354535033,"uuid":"1216145826","full_name":"reaatech/prompt-injection-bench","owner":"reaatech","description":"Reproducible benchmark and test corpus for prompt-injection defenses — adapters for common sanitization libraries, scored leaderboard, CI-ready. Security-adjacent, evergreen.","archived":false,"fork":false,"pushed_at":"2026-06-04T02:08:29.000Z","size":396,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-04T04:07:51.676Z","etag":null,"topics":["adversarial-testing","agentic-ai","ai-safety","ai-security","benchmark","defense","llm","llm-security","prompt-injection","red-teaming","sanitization","security","test-corpus","typescript"],"latest_commit_sha":null,"homepage":"https://reaatech.com/products/testing-security/prompt-injection-bench","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reaatech.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-20T16:02:06.000Z","updated_at":"2026-06-04T02:08:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/reaatech/prompt-injection-bench","commit_stats":null,"previous_names":["reaatech/prompt-injection-bench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/reaatech/prompt-injection-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reaatech%2Fprompt-injection-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reaatech%2Fprompt-injection-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reaatech%2Fprompt-injection-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reaatech%2Fprompt-injection-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reaatech","download_url":"https://codeload.github.com/reaatech/prompt-injection-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reaatech%2Fprompt-injection-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35544514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-16T02:00:06.687Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-testing","agentic-ai","ai-safety","ai-security","benchmark","defense","llm","llm-security","prompt-injection","red-teaming","sanitization","security","test-corpus","typescript"],"created_at":"2026-07-16T12:30:49.476Z","updated_at":"2026-07-16T12:30:49.923Z","avatar_url":"https://github.com/reaatech.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# prompt-injection-bench\n\n[![CI](https://github.com/reaatech/prompt-injection-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/reaatech/prompt-injection-bench/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![TypeScript](https://img.shields.io/badge/TypeScript-5.8-blue)](https://www.typescriptlang.org/)\n\n\u003e Reproducible benchmark and test corpus for evaluating prompt-injection defenses in AI agent systems.\n\nThis monorepo provides an attack corpus, pluggable defense adapters, a scoring engine, a public leaderboard, and an MCP server for benchmarking prompt-injection defenses.\n\n## Features\n\n- **Attack corpus** — 8 categories, 300+ templates, variant generation with obfuscation strategies\n- **Defense adapters** — 8 built-in adapters (Rebuff, Lakera Guard, LLM Guard, Garak, OpenAI/Azure/Anthropic/Cohere Moderation, Custom HTTP) with a common interface\n- **Scoring engine** — Weighted scoring, Wilson confidence intervals, Cohen's h effect sizes, z-tests, ANOVA\n- **Leaderboard** — Public rankings with S/A/B/C/D tiers and JSON persistence\n- **MCP integration** — 4 MCP tools (`run_benchmark`, `compare_defenses`, `generate_report`, `submit_results`)\n- **Reproducibility** — Deterministic seeds, SHA-256 proofs, version-locked manifests\n- **Security hardening** — SSRF protection, path traversal prevention, rate limiting, input validation\n- **CLI tool** — 6 subcommands for benchmarking, comparing, and reporting from the terminal\n\n## Installation\n\n### Using the packages\n\nPackages are published under the `@reaatech` scope (with the umbrella package unscoped) and can be installed individually:\n\n```bash\n# Core types and schemas\npnpm add @reaatech/pi-bench-core\n\n# Defense adapters\npnpm add @reaatech/pi-bench-adapters\n\n# Scoring engine\npnpm add @reaatech/pi-bench-scoring\n\n# CLI and umbrella (re-exports everything)\npnpm add prompt-injection-bench\n```\n\n### Contributing\n\n```bash\n# Clone the repository\ngit clone https://github.com/reaatech/prompt-injection-bench.git\ncd prompt-injection-bench\n\n# Install dependencies\npnpm install\n\n# Build all packages\npnpm build\n\n# Run the test suite\npnpm test\n\n# Run linting\npnpm lint\n```\n\n## Quick Start\n\n### CLI\n\n```bash\n# Run a benchmark against the mock adapter\nnpx prompt-injection-bench benchmark --defense mock\n\n# Compare two defense results\nnpx prompt-injection-bench compare --results rebuff.json lakera.json\n\n# Generate an HTML report\nnpx prompt-injection-bench report --results latest.json --format html\n```\n\n### Library\n\n```typescript\nimport { createBenchmarkEngine, createMockAdapter, generateDefaultCorpus } from \"prompt-injection-bench\";\n\nconst adapter = createMockAdapter(0.95, 0.03);\nconst corpus = generateDefaultCorpus();\nconst engine = createBenchmarkEngine({ defense: adapter, parallel: 10 });\n\nconst result = await engine.runBenchmark(corpus);\nconsole.log(`Detection rate: ${(1 - result.attackSuccessRate) * 100}%`);\n```\n\n### MCP\n\nConnect the MCP server to any MCP client:\n\n```json\n{\n  \"mcpServers\": {\n    \"prompt-injection-bench\": {\n      \"command\": \"npx\",\n      \"args\": [\"@reaatech/pi-bench-mcp-server\"]\n    }\n  }\n}\n```\n\n## Packages\n\n| Package | Description |\n|---------|-------------|\n| [`@reaatech/pi-bench-core`](./packages/core) | Canonical types, Zod schemas, and attack taxonomy |\n| [`@reaatech/pi-bench-observability`](./packages/observability) | Structured logging, tracing, and metrics |\n| [`@reaatech/pi-bench-corpus`](./packages/corpus) | Attack corpus builder, validator, and variant engine |\n| [`@reaatech/pi-bench-adapters`](./packages/adapters) | Pluggable defense adapter implementations |\n| [`@reaatech/pi-bench-scoring`](./packages/scoring) | Scoring engine and statistical analysis |\n| [`@reaatech/pi-bench-runner`](./packages/runner) | Benchmark execution engine |\n| [`@reaatech/pi-bench-leaderboard`](./packages/leaderboard) | Leaderboard management and persistence |\n| [`@reaatech/pi-bench-mcp-server`](./packages/mcp-server) | MCP server and reproducibility tools |\n| [`prompt-injection-bench`](./packages/prompt-injection-bench) | CLI and umbrella package |\n\n## Attack Taxonomy\n\n| Category | Weight | Description |\n|----------|--------|-------------|\n| Direct Injection | 1.0 | Obvious malicious instruction overrides |\n| Prompt Leaking | 1.2 | Attempts to extract system prompts or config |\n| Role-Playing | 1.3 | DAN, developer mode, persona adoption |\n| Encoding Attacks | 1.1 | Base64, ROT13, Unicode obfuscation |\n| Multi-Turn Jailbreaks | 1.4 | Gradual persuasion across conversation turns |\n| Payload Splitting | 1.2 | Attack split across multiple inputs |\n| Translation Attacks | 1.1 | Low-resource language injection |\n| Context Stuffing | 1.3 | Hiding injection in large context |\n\n## Scoring Methodology\n\n```\nweighted_score = Σ(category_score × category_weight) / Σ(weights)\nfinal_score = weighted_score × (1 - fpr_penalty) × consistency_bonus\n```\n\n| Metric | Formula | Interpretation |\n|--------|---------|----------------|\n| Attack Success Rate (ASR) | `bypassed / total` | Lower is better (0% = perfect) |\n| Defense Efficacy Score (DES) | `1 - ASR` | Higher is better (100% = perfect) |\n| False Positive Rate (FPR) | `false_alarms / benign` | Lower is better (0% = perfect) |\n| Latency Overhead | `defense_time - baseline` | Lower is better |\n\n## Defense Adapters\n\n| Adapter | Type | Configuration |\n|---------|------|---------------|\n| `mock` | Testing | No config needed — deterministic |\n| `rebuff` | Library | `REBUFF_API_KEY` env var |\n| `lakera` | API | `LAKERA_API_KEY` env var |\n| `llm-guard` | Library | `LLM_GUARD_API_KEY` env var |\n| `garak` | Tool | `garak` in PATH |\n| `moderation-openai` | API | `OPENAI_API_KEY` env var |\n| `moderation-azure` | API | `AZURE_*` env vars |\n| `moderation-anthropic` | API | `ANTHROPIC_API_KEY` env var |\n| `moderation-cohere` | API | `COHERE_API_KEY` env var |\n| `custom` | HTTP | Custom `detectUrl`, `sanitizeUrl` |\n\n## Security Features\n\n- **Input validation** — 10K char limit, null byte rejection, API key checks before network calls\n- **SSRF protection** — Blocks `file:`, `javascript:`, `data:` protocols and cloud metadata endpoints\n- **Path traversal prevention** — Validates all file paths in MCP tool operations\n- **Rate limiting** — Token bucket algorithm (100 req/min per adapter, 500 req/min global)\n\n## Documentation\n\n- [`ARCHITECTURE.md`](./ARCHITECTURE.md) — System design, package relationships, and data flows\n- [`AGENTS.md`](./AGENTS.md) — Coding conventions and development guidelines\n- [`CONTRIBUTING.md`](./CONTRIBUTING.md) — Contribution workflow and release process\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freaatech%2Fprompt-injection-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freaatech%2Fprompt-injection-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freaatech%2Fprompt-injection-bench/lists"}