{"id":47319553,"url":"https://github.com/Privatris/AgentLeak","last_synced_at":"2026-03-31T22:00:33.018Z","repository":{"id":330495698,"uuid":"1122458704","full_name":"Privatris/AgentLeak","owner":"Privatris","description":"AgentLeak: Open benchmark for privacy leakage in LLM agents — 7 channels, multi-agent, multi-framework.","archived":false,"fork":false,"pushed_at":"2026-02-01T08:56:26.000Z","size":24025,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-01T19:14:35.736Z","etag":null,"topics":["agentic-ai","agents","benchmark","crewai","llm","multi-agent","privacy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Privatris.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-24T19:18:23.000Z","updated_at":"2026-02-01T08:56:29.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Privatris/AgentLeak","commit_stats":null,"previous_names":["privatris/agentleak"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Privatris/AgentLeak","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Privatris%2FAgentLeak","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Privatris%2FAgentLeak/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Privatris%2FAgentLeak/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Privatris%2FAgentLeak/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Privatris","download_url":"https://codeload.github.com/Privatris/AgentLeak/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Privatris%2FAgentLeak/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31257008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-31T18:32:52.363Z","status":"ssl_error","status_checked_at":"2026-03-31T18:32:51.507Z","response_time":111,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-ai","agents","benchmark","crewai","llm","multi-agent","privacy"],"created_at":"2026-03-17T17:00:28.311Z","updated_at":"2026-03-31T22:00:32.961Z","avatar_url":"https://github.com/Privatris.png","language":"Python","funding_links":[],"categories":["Agentic security"],"sub_categories":[],"readme":"# AgentLeak\n\nBenchmark for privacy leakage in multi-agent LLM systems.\n\nThis repository accompanies the IEEE Access paper: *AgentLeak: A Full-Stack Benchmark for Privacy Leakage in Multi-Agent LLM Systems*.\n\nPaper https://arxiv.org/abs/2602.11510\n\n## Key Results (5,694 traces across 5 models)\n\n| Model | C1 (Output) | C2 (Internal) | H1 (Audit Gap) | Total Leak |\n|-------|-------------|---------------|----------------|------------|\n| **Claude-3.5-Sonnet** | 8.2% | 53.9% | 45.7% | 55.2% |\n| GPT-4o | 17.2% | 76.8% | 59.6% | 77.6% |\n| GPT-4o-mini | 41.2% | 75.3% | 34.2% | 76.3% |\n| Llama-3.3-70B | 26.9% | 67.8% | 41.3% | 89.9% |\n| Mistral-Large | 47.5% | 96.2% | 48.7% | 99.3% |\n| **Average** | **28.2%** | **74.0%** | **45.9%** | **79.7%** |\n\n### Key Findings\n\n- **Internal channels leak 2.6× more** than external (74.0% vs 28.2%)\n- **Output-only audits miss 45.9%** of violations\n- **Claude 3.5 Sonnet paradox**: Lowest C1 leakage (8.2%) but 6.6× internal/external ratio—the highest among all models\n- **Finding 7 (Tool Leakage)**: Tool inputs (C3) and system logs (C6) exhibit extremely high leakage rates (up to **85%** on Claude 3.5), even when the final agent output (C1) is perfectly sanitized.\n- Pattern C2 \u003e C1 holds **across all 5 models** tested\n\n## Scope\n\n- 1,000 scenarios (healthcare, finance, legal, corporate)\n- 7 channels: C1 output, C2 inter-agent, C3-C4 tools, C5 memory, C6 logs, C7 artifacts\n- 32 attack classes, 6 families\n- SDK: CrewAI, LangChain, AutoGPT, MetaGPT\n\n## Reproduction\n\n### Main Benchmark (C1, C2, C5)\nTo reproduce the main results (Output, Internal, Memory):\n```bash\ncd benchmarks/ieee_repro\npython benchmark.py --n 1000 --traces --model openai/gpt-4o\n```\n\n### Advanced Tools \u0026 Logs Benchmark (C3, C6)\nTargets \"Secondary Channel\" leakage where sensitive data is sent to external tools or dumped in logs.\n```bash\ncd benchmarks/ieee_repro\n# Run for a specific model (e.g., Claude 3.5)\npython benchmark_tools.py --n 100 --model anthropic/claude-3.5-sonnet\n\n# Or run the automated multi-model test suite\n./run_tools_benchmark.sh\n```\nResults are saved in `benchmarks/ieee_repro/results/tools/`.\n\n## Structure\n\n- `agentleak/`: The core framework SDK\n- `agentleak_data/`: The dataset of 1000 scenarios\n- `benchmarks/ieee_repro/`: Scripts to reproduce the paper's findings, including Finding 7 (Tools \u0026 Logs).\n- `benchmarks/showcase/`: Real-world CrewAI integration demo showing the SDK in action.\n- `paper/`: The LaTeX source of the IEEE Access paper\n\n## Setup\n\n```bash\ngit clone https://github.com/Privatris/AgentLeak\ncd AgentLeak\npip install -e .\npytest tests/ -v\n```\n\n## Usage\n\n```python\nfrom agentleak import AgentLeakTester, DetectionMode\n\ntester = AgentLeakTester(mode=DetectionMode.HYBRID)\nresult = tester.check(\n    vault={\"ssn\": \"123-45-6789\"},\n    output=\"The SSN is 123-45-6789\",\n    channel=\"C1\"\n)\nprint(f\"Leak: {result.leaked}, Confidence: {result.confidence}\")\n```\n\nCLI:\n```bash\npython -m agentleak run --quick --dry-run\npython -m agentleak run --full\n```\n\n## Reproduction\n\n```bash\ncd benchmarks/ieee_repro\npython benchmark.py --n 100 --traces --model openai/gpt-4o-mini\n```\n\nTraces are in `benchmarks/ieee_repro/results/traces/`.\n\n## Citation\n\n```bibtex\n@article{el2026agentleak,\n  title        = {AgentLeak: A Full-Stack Benchmark for Privacy Leakage in Multi-Agent LLM Systems},\n  author       = {El Yagoubi, Faouzi and Badu-Marfo, Godwin and Al Mallah, Ranwa},\n  journal      = {arXiv preprint arXiv:2602.11510},\n  year         = {2026},\n  url          = {https://arxiv.org/abs/2602.11510},\n  abstract     = {Multi-agent Large Language Model (LLM) systems create privacy risks that current benchmarks cannot measure. When agents coordinate on tasks, sensitive data passes through inter-agent messages, shared memory, and tool arguments, pathways that output-only audits never inspect. We introduce AgentLeak, the first full-stack benchmark for privacy leakage covering internal channels, spanning 1,000 scenarios across healthcare, finance, legal, and corporate domains, paired with a 32-class attack taxonomy and a three-tier detection pipeline. Testing several models across thousands of traces shows that internal channels in multi-agent configurations are the primary privacy vulnerability and that output-only audits miss a large fraction of violations, underscoring the need for coordinated privacy protections on inter-agent communication.},\n  note         = {Submitted to arXiv on 12 Feb 2026.},\n}\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPrivatris%2FAgentLeak","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPrivatris%2FAgentLeak","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPrivatris%2FAgentLeak/lists"}