{"id":13406013,"url":"https://github.com/promptfoo/promptfoo","last_synced_at":"2026-03-03T02:11:39.384Z","repository":{"id":161226306,"uuid":"633927609","full_name":"promptfoo/promptfoo","owner":"promptfoo","description":"Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.","archived":false,"fork":false,"pushed_at":"2025-03-08T00:53:33.000Z","size":323228,"stargazers_count":5756,"open_issues_count":188,"forks_count":475,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-03-08T01:24:43.599Z","etag":null,"topics":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"latest_commit_sha":null,"homepage":"https://promptfoo.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/promptfoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["typpo"]}},"created_at":"2023-04-28T15:48:49.000Z","updated_at":"2025-03-07T21:56:31.000Z","dependencies_parsed_at":"2023-09-26T23:40:46.827Z","dependency_job_id":"5f651e5a-0f03-4d24-887d-7e05e94a5e10","html_url":"https://github.com/promptfoo/promptfoo","commit_stats":null,"previous_names":["promptfoo/promptfoo","typpo/promptfoo"],"tags_count":278,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/promptfoo","download_url":"https://codeload.github.com/promptfoo/promptfoo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243562171,"owners_count":20311233,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"created_at":"2024-07-30T19:02:18.587Z","updated_at":"2025-10-15T17:39:08.781Z","avatar_url":"https://github.com/promptfoo.png","language":"TypeScript","funding_links":["https://github.com/sponsors/typpo"],"categories":["AI Hacking Resources \u0026 Links","TypeScript","🤖 LLM \u0026 Chatbot Testing","*Ops for AI","Prompts","✍ Prompt Engineering","\u003cimg src=\"./assets/satellite.svg\" width=\"16\" height=\"16\" style=\"vertical-align: middle;\"\u003e Satellites","A01_文本生成_文本对话","Tooling \u0026 Dev Experience","Agent Security","[↑](#table-of-contents)Tools \u003ca name=\"tools\"\u003e\u003c/a\u003e","llm","Repos","Evaluation \u0026 Quality Control","Evaluation \u0026 Observability","Testing \u0026 Evaluation","Papers","Design Primitives","Open Source Security Tools","\u003ca id=\"tools\"\u003e\u003c/a\u003e🛠️ Tools","Tools \u0026 Code","vulnerability-scanners","GPT Security","testing","其他LLM框架","LLM-as-Judge Evaluation","Tools","📊 At a glance","AI Frameworks and SDKs","Evaluation and Monitoring","💉 Prompt Injection and Agent Threats","开源工具","资源列表","Red Teaming \u0026 Evaluation","Tools \u0026 Platforms","What's New","Software","Frameworks","Tools \u0026 Frameworks","Don't forget to give a :star: to make the project popular","Securing AI SaaS","3）参考实现与开源工具（GitHub）","🤖 AI \u0026 Machine Learning","⚔️ LLM And GenAI Security Testing Tools","Security Tools","3. Prompt Optimization","Evaluation, Observability \u0026 Safety","Prompt Engineering","提示工程与优化 (Prompt Engineering)","LLM and Agent Observability","Prompt Engineering \u0026 Management","Other LLM Frameworks","Attack Techniques \u0026 Red Teaming","Table of Contents","8. MLOps / LLMOps \u0026 Production","LLM Testing / Monitoring","LLMs Eval","Supporting Infrastructure","AI Red Teaming (Testing AI Targets)"],"sub_categories":["Shannon: Autonomous Web Application Exploitation","LLMOps","Testing \u0026 Optimization","大语言对话模型及数据","Development Tools","Red-Teaming Harnesses \u0026 Automated Security Testing","Jailbreak Defense","Verification \u0026 CI Integration","Bleeding Edge ⚗️","Standard","文章","LLM Evaluations and Benchmarks","Tools and Frameworks","Prompting","项目","General-Purpose Machine Learning","Automated Red Teaming","Open Source Frameworks","🆕 Recently Added (January 2026)","AI \u0026 LLM Testing","Eval \u0026 Testing","Prompt Testing \u0026 Optimization","Application Security","沙箱、可观测与评测","Evaluators and Test Harnesses","Vulnerability Scanners (Red Teaming)","Rust","Evaluation \u0026 Observability","Codex Resources","推理网关 (Inference Gateways)","Security Testing","Videos Playlists","LLM \u0026 GenAI Red Teaming","🤖 AI Security / AI Red Teaming","Evaluation"],"readme":"# Promptfoo: LLM evals \u0026 red teaming\n\n[![npm](https://img.shields.io/npm/v/promptfoo)](https://npmjs.com/package/promptfoo)\n[![npm](https://img.shields.io/npm/dm/promptfoo)](https://npmjs.com/package/promptfoo)\n[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/typpo/promptfoo/main.yml)](https://github.com/promptfoo/promptfoo/actions/workflows/main.yml)\n![MIT license](https://img.shields.io/github/license/promptfoo/promptfoo)\n[![Discord](https://github.com/user-attachments/assets/2092591a-ccc5-42a7-aeb6-24a2808950fd)](https://discord.gg/promptfoo)\n\n`promptfoo` is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.\n\n## Quick Start\n\n```sh\n# Install and initialize project\nnpx promptfoo@latest init\n\n# Run your first evaluation\nnpx promptfoo eval\n```\n\nSee [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.\n\n## What can you do with Promptfoo?\n\n- **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)\n- **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning\n- **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))\n- **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)\n- **Share results** with your team\n\nHere's what it looks like in action:\n\n![prompt evaluation matrix - web viewer](https://www.promptfoo.dev/img/claude-vs-gpt-example@2x.png)\n\nIt works on the command line too:\n\n![prompt evaluation matrix - command line](https://github.com/promptfoo/promptfoo/assets/310310/480e1114-d049-40b9-bd5f-f81c15060284)\n\nIt also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):\n\n![gen ai red team](https://www.promptfoo.dev/img/riskreport-1@2x.png)\n\n## Why promptfoo?\n\n- 🚀 **Developer-first**: Fast, with features like live reload and caching\n- 🔒 **Private**: Runs 100% locally - your prompts never leave your machine\n- 🔧 **Flexible**: Works with any LLM API or programming language\n- 💪 **Battle-tested**: Powers LLM apps serving 10M+ users in production\n- 📊 **Data-driven**: Make decisions based on metrics, not gut feel\n- 🤝 **Open source**: MIT licensed, with an active community\n\n## Learn More\n\n- 📚 [Full Documentation](https://www.promptfoo.dev/docs/intro/)\n- 🔐 [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)\n- 🎯 [Getting Started](https://www.promptfoo.dev/docs/getting-started/)\n- 💻 [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)\n- 📦 [Node.js Package](https://www.promptfoo.dev/docs/usage/node-package/)\n- 🤖 [Supported Models](https://www.promptfoo.dev/docs/providers/)\n\n## Contributing\n\nWe welcome contributions! Check out our [contributing guide](https://www.promptfoo.dev/docs/contributing/) to get started.\n\nJoin our [Discord community](https://discord.gg/promptfoo) for help and discussion.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpromptfoo%2Fpromptfoo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo/lists"}