{"id":13406013,"url":"https://github.com/promptfoo/promptfoo","last_synced_at":"2026-03-03T02:11:39.384Z","repository":{"id":161226306,"uuid":"633927609","full_name":"promptfoo/promptfoo","owner":"promptfoo","description":"Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.","archived":false,"fork":false,"pushed_at":"2025-03-08T00:53:33.000Z","size":323228,"stargazers_count":5756,"open_issues_count":188,"forks_count":475,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-03-08T01:24:43.599Z","etag":null,"topics":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"latest_commit_sha":null,"homepage":"https://promptfoo.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/promptfoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["typpo"]}},"created_at":"2023-04-28T15:48:49.000Z","updated_at":"2025-03-07T21:56:31.000Z","dependencies_parsed_at":"2023-09-26T23:40:46.827Z","dependency_job_id":"5f651e5a-0f03-4d24-887d-7e05e94a5e10","html_url":"https://github.com/promptfoo/promptfoo","commit_stats":null,"previous_names":["promptfoo/promptfoo","typpo/promptfoo"],"tags_count":278,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/promptfoo","download_url":"https://codeload.github.com/promptfoo/promptfoo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243562171,"owners_count":20311233,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"created_at":"2024-07-30T19:02:18.587Z","updated_at":"2025-10-15T17:39:08.781Z","avatar_url":"https://github.com/promptfoo.png","language":"TypeScript","readme":"# Promptfoo: LLM evals \u0026 red teaming\n\n[![npm](https://img.shields.io/npm/v/promptfoo)](https://npmjs.com/package/promptfoo)\n[![npm](https://img.shields.io/npm/dm/promptfoo)](https://npmjs.com/package/promptfoo)\n[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/typpo/promptfoo/main.yml)](https://github.com/promptfoo/promptfoo/actions/workflows/main.yml)\n![MIT license](https://img.shields.io/github/license/promptfoo/promptfoo)\n[![Discord](https://github.com/user-attachments/assets/2092591a-ccc5-42a7-aeb6-24a2808950fd)](https://discord.gg/promptfoo)\n\n`promptfoo` is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.\n\n## Quick Start\n\n```sh\n# Install and initialize project\nnpx promptfoo@latest init\n\n# Run your first evaluation\nnpx promptfoo eval\n```\n\nSee [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.\n\n## What can you do with Promptfoo?\n\n- **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)\n- **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning\n- **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))\n- **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)\n- **Share results** with your team\n\nHere's what it looks like in action:\n\n![prompt evaluation matrix - web viewer](https://www.promptfoo.dev/img/claude-vs-gpt-example@2x.png)\n\nIt works on the command line too:\n\n![prompt evaluation matrix - command line](https://github.com/promptfoo/promptfoo/assets/310310/480e1114-d049-40b9-bd5f-f81c15060284)\n\nIt also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):\n\n![gen ai red team](https://www.promptfoo.dev/img/riskreport-1@2x.png)\n\n## Why promptfoo?\n\n- 🚀 **Developer-first**: Fast, with features like live reload and caching\n- 🔒 **Private**: Runs 100% locally - your prompts never leave your machine\n- 🔧 **Flexible**: Works with any LLM API or programming language\n- 💪 **Battle-tested**: Powers LLM apps serving 10M+ users in production\n- 📊 **Data-driven**: Make decisions based on metrics, not gut feel\n- 🤝 **Open source**: MIT licensed, with an active community\n\n## Learn More\n\n- 📚 [Full Documentation](https://www.promptfoo.dev/docs/intro/)\n- 🔐 [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)\n- 🎯 [Getting Started](https://www.promptfoo.dev/docs/getting-started/)\n- 💻 [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)\n- 📦 [Node.js Package](https://www.promptfoo.dev/docs/usage/node-package/)\n- 🤖 [Supported Models](https://www.promptfoo.dev/docs/providers/)\n\n## Contributing\n\nWe welcome contributions! Check out our [contributing guide](https://www.promptfoo.dev/docs/contributing/) to get started.\n\nJoin our [Discord community](https://discord.gg/promptfoo) for help and discussion.\n","funding_links":["https://github.com/sponsors/typpo"],"categories":["🤖 LLM \u0026 Chatbot Testing","TypeScript","Prompts","✍ Prompt Engineering","\u003cimg src=\"./assets/satellite.svg\" width=\"16\" height=\"16\" style=\"vertical-align: middle;\"\u003e Satellites","A01_文本生成_文本对话","Tooling \u0026 Dev Experience","⚔️ LLM And GenAI Security Testing Tools","Other LLM Frameworks","[↑](#table-of-contents)Tools \u003ca name=\"tools\"\u003e\u003c/a\u003e","llm","Repos","Papers","Open Source Security Tools","\u003ca id=\"tools\"\u003e\u003c/a\u003e🛠️ Tools","Tools \u0026 Code","vulnerability-scanners","testing","其他LLM框架","Tools","AI Frameworks and SDKs","Evaluation and Monitoring","开源工具","资源列表","Tools \u0026 Platforms","What's New","Software","Tools \u0026 Frameworks","Don't forget to give a :star: to make the project popular","📋 Contents","Securing AI SaaS","LLMs Eval","LLM Testing / Monitoring","Security Tools","3. Prompt Optimization","GPT Security","Agent Security","Evaluation, Observability \u0026 Safety","Prompt Engineering","提示工程与优化 (Prompt Engineering)","AI Red Teaming (Testing AI Targets)","3）参考实现与开源工具（GitHub）","Prompt Engineering \u0026 Management","Attack Techniques \u0026 Red Teaming"],"sub_categories":["Testing \u0026 Optimization","大语言对话模型及数据","Development Tools","Videos Playlists","Red-Teaming Harnesses \u0026 Automated Security Testing","Jailbreak Defense","Bleeding Edge ⚗️","文章","LLM Evaluations and Benchmarks","Prompting","项目","General-Purpose Machine Learning","Open Source Frameworks","🆕 Recently Added (January 2026)","AI \u0026 LLM Testing","Prompt Testing \u0026 Optimization","📊 8. MLOps / LLMOps \u0026 Production","Application Security","Evaluators and Test Harnesses","Vulnerability Scanners (Red Teaming)","Rust","Standard","Evaluation \u0026 Observability","Codex Resources","推理网关 (Inference Gateways)","Security Testing","沙箱、可观测与评测","LLM \u0026 GenAI Red Teaming"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpromptfoo%2Fpromptfoo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo/lists"}