{"id":13406013,"url":"https://github.com/promptfoo/promptfoo","last_synced_at":"2026-03-03T02:11:39.384Z","repository":{"id":161226306,"uuid":"633927609","full_name":"promptfoo/promptfoo","owner":"promptfoo","description":"Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.","archived":false,"fork":false,"pushed_at":"2025-03-08T00:53:33.000Z","size":323228,"stargazers_count":5756,"open_issues_count":188,"forks_count":475,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-03-08T01:24:43.599Z","etag":null,"topics":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"latest_commit_sha":null,"homepage":"https://promptfoo.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/promptfoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["typpo"]}},"created_at":"2023-04-28T15:48:49.000Z","updated_at":"2025-03-07T21:56:31.000Z","dependencies_parsed_at":"2023-09-26T23:40:46.827Z","dependency_job_id":"5f651e5a-0f03-4d24-887d-7e05e94a5e10","html_url":"https://github.com/promptfoo/promptfoo","commit_stats":null,"previous_names":["promptfoo/promptfoo","typpo/promptfoo"],"tags_count":278,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/promptfoo","download_url":"https://codeload.github.com/promptfoo/promptfoo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243562171,"owners_count":20311233,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"created_at":"2024-07-30T19:02:18.587Z","updated_at":"2025-10-15T17:39:08.781Z","avatar_url":"https://github.com/promptfoo.png","language":"TypeScript","funding_links":["https://github.com/sponsors/typpo"],"categories":["AI Hacking Resources \u0026 Links","llm","TypeScript","Agentic AI ![](https://img.shields.io/badge/_-AGENTIC-22d3ee?style=flat-square\u0026logo=openai\u0026logoColor=white)","🤖 LLM \u0026 Chatbot Testing","Evaluation \u0026 Observability","Tools of Trade","Evaluation \u0026 Quality Control","Prompts","\u003cimg src=\"./assets/satellite.svg\" width=\"16\" height=\"16\" style=\"vertical-align: middle;\"\u003e Satellites","Tools \u0026 Frameworks","A01_文本生成_文本对话","Evaluation Tools","Tooling \u0026 Dev Experience","✍ Prompt Engineering","[↑](#table-of-contents)Tools \u003ca name=\"tools\"\u003e\u003c/a\u003e","Repos","Testing \u0026 Evaluation","Papers","Design Primitives","Open Source Security Tools","\u003ca id=\"tools\"\u003e\u003c/a\u003e🛠️ Tools","Tools \u0026 Code","vulnerability-scanners","testing","其他LLM框架","Tools","Evaluation and Observability","📊 At a glance","AI Frameworks and SDKs","Evaluation and Monitoring","💉 Prompt Injection and Agent Threats","开源工具","资源列表","Red Teaming \u0026 Evaluation","Prompt and Behaviour Optimization","Tools \u0026 Platforms","What's New","Software","Frameworks","Evaluation \u0026 Testing","🤖 AI \u0026 Machine Learning","Don't forget to give a :star: to make the project popular","Testing \u0026 Security","AI Red Teaming (Testing AI Targets) (41)","5 · Evaluation infrastructure (the eval stack: datasets, scorers, online/offline, tracing, CI)","GPT Security","Securing AI SaaS","3）参考实现与开源工具（GitHub）","⚔️ LLM And GenAI Security Testing Tools","Security Tools","3. Prompt Optimization","LLM-as-Judge Evaluation","*Ops for AI","Evaluation, Observability \u0026 Safety","Prompt Engineering","提示工程与优化 (Prompt Engineering)","Observability / evals","LLM and Agent Observability","Prompt Engineering \u0026 Management","Other LLM Frameworks","Attack Techniques \u0026 Red Teaming","🧰 Tools","Table of Contents","8. MLOps / LLMOps \u0026 Production","LLM Testing / Monitoring","LLMs Eval","Supporting Infrastructure","Catalog","Evaluation Frameworks","Agent Security","How to run these benchmarks","LLMOps"],"sub_categories":["Shannon: Autonomous Web Application Exploitation","Agent gateways and LLMOps","Security Testing","大语言对话模型及数据","Datasets","Development Tools","Testing \u0026 Optimization","Red-Teaming Harnesses \u0026 Automated Security Testing","Jailbreak Defense","Verification \u0026 CI Integration","Offensive / Red Teaming","Bleeding Edge ⚗️","文章","LLM Evaluations and Benchmarks","Tools and Frameworks","Prompting","项目","General-Purpose Machine Learning","Automated Red Teaming","Open Source Frameworks","🆕 Recently Added (January 2026)","AI \u0026 LLM Testing","Eval \u0026 Testing","Prompt Testing \u0026 Optimization","AI Products with UI Generation","5a · Eval frameworks \u0026 harnesses (code-first test-runners)","Standard","Application Security","沙箱、可观测与评测","Evaluators and Test Harnesses","Vulnerability Scanners (Red Teaming)","Rust","LLMOps","Evaluation \u0026 Observability","Codex Resources","推理网关 (Inference Gateways)","Videos Playlists","LLM \u0026 GenAI Red Teaming","Guardrail Frameworks","🤖 AI Security / AI Red Teaming","Evaluation","Evaluation Harnesses \u0026 Benchmarks","Start with a unified harness","LLM Evaluation \u0026 Testing"],"readme":"# Promptfoo: LLM evals \u0026 red teaming\n\n[![npm](https://img.shields.io/npm/v/promptfoo)](https://npmjs.com/package/promptfoo)\n[![npm](https://img.shields.io/npm/dm/promptfoo)](https://npmjs.com/package/promptfoo)\n[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/typpo/promptfoo/main.yml)](https://github.com/promptfoo/promptfoo/actions/workflows/main.yml)\n![MIT license](https://img.shields.io/github/license/promptfoo/promptfoo)\n[![Discord](https://github.com/user-attachments/assets/2092591a-ccc5-42a7-aeb6-24a2808950fd)](https://discord.gg/promptfoo)\n\n`promptfoo` is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.\n\n## Quick Start\n\n```sh\n# Install and initialize project\nnpx promptfoo@latest init\n\n# Run your first evaluation\nnpx promptfoo eval\n```\n\nSee [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.\n\n## What can you do with Promptfoo?\n\n- **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)\n- **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning\n- **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))\n- **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)\n- **Share results** with your team\n\nHere's what it looks like in action:\n\n![prompt evaluation matrix - web viewer](https://www.promptfoo.dev/img/claude-vs-gpt-example@2x.png)\n\nIt works on the command line too:\n\n![prompt evaluation matrix - command line](https://github.com/promptfoo/promptfoo/assets/310310/480e1114-d049-40b9-bd5f-f81c15060284)\n\nIt also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):\n\n![gen ai red team](https://www.promptfoo.dev/img/riskreport-1@2x.png)\n\n## Why promptfoo?\n\n- 🚀 **Developer-first**: Fast, with features like live reload and caching\n- 🔒 **Private**: Runs 100% locally - your prompts never leave your machine\n- 🔧 **Flexible**: Works with any LLM API or programming language\n- 💪 **Battle-tested**: Powers LLM apps serving 10M+ users in production\n- 📊 **Data-driven**: Make decisions based on metrics, not gut feel\n- 🤝 **Open source**: MIT licensed, with an active community\n\n## Learn More\n\n- 📚 [Full Documentation](https://www.promptfoo.dev/docs/intro/)\n- 🔐 [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)\n- 🎯 [Getting Started](https://www.promptfoo.dev/docs/getting-started/)\n- 💻 [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)\n- 📦 [Node.js Package](https://www.promptfoo.dev/docs/usage/node-package/)\n- 🤖 [Supported Models](https://www.promptfoo.dev/docs/providers/)\n\n## Contributing\n\nWe welcome contributions! Check out our [contributing guide](https://www.promptfoo.dev/docs/contributing/) to get started.\n\nJoin our [Discord community](https://discord.gg/promptfoo) for help and discussion.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpromptfoo%2Fpromptfoo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo/lists"}