{"id":13406013,"url":"https://github.com/promptfoo/promptfoo","last_synced_at":"2026-03-03T02:11:39.384Z","repository":{"id":161226306,"uuid":"633927609","full_name":"promptfoo/promptfoo","owner":"promptfoo","description":"Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.","archived":false,"fork":false,"pushed_at":"2025-03-08T00:53:33.000Z","size":323228,"stargazers_count":5756,"open_issues_count":188,"forks_count":475,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-03-08T01:24:43.599Z","etag":null,"topics":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"latest_commit_sha":null,"homepage":"https://promptfoo.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/promptfoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["typpo"]}},"created_at":"2023-04-28T15:48:49.000Z","updated_at":"2025-03-07T21:56:31.000Z","dependencies_parsed_at":"2023-09-26T23:40:46.827Z","dependency_job_id":"5f651e5a-0f03-4d24-887d-7e05e94a5e10","html_url":"https://github.com/promptfoo/promptfoo","commit_stats":null,"previous_names":["promptfoo/promptfoo","typpo/promptfoo"],"tags_count":278,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/promptfoo","download_url":"https://codeload.github.com/promptfoo/promptfoo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243562171,"owners_count":20311233,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"created_at":"2024-07-30T19:02:18.587Z","updated_at":"2025-10-15T17:39:08.781Z","avatar_url":"https://github.com/promptfoo.png","language":"TypeScript","funding_links":["https://github.com/sponsors/typpo"],"categories":["AI Hacking Resources \u0026 Links","🤖 LLM \u0026 Chatbot Testing","TypeScript","Prompts","✍ Prompt Engineering","\u003cimg src=\"./assets/satellite.svg\" width=\"16\" height=\"16\" style=\"vertical-align: middle;\"\u003e Satellites","A01_文本生成_文本对话","🤖 AI \u0026 Machine Learning","Tooling \u0026 Dev Experience","[↑](#table-of-contents)Tools \u003ca name=\"tools\"\u003e\u003c/a\u003e","llm","Repos","Papers","Open Source Security Tools","\u003ca id=\"tools\"\u003e\u003c/a\u003e🛠️ Tools","Tools \u0026 Code","vulnerability-scanners","testing","其他LLM框架","LLM-as-Judge Evaluation","Tools","AI Frameworks and SDKs","Evaluation and Monitoring","开源工具","资源列表","Red Teaming \u0026 Evaluation","Tools \u0026 Platforms","What's New","Software","Frameworks","Tools \u0026 Frameworks","Don't forget to give a :star: to make the project popular","📋 Contents","AI Red Teaming (Testing AI Targets)","Securing AI SaaS","3）参考实现与开源工具（GitHub）","LLMs Eval","LLM Testing / Monitoring","⚔️ LLM And GenAI Security Testing Tools","Security Tools","3. Prompt Optimization","GPT Security","Agent Security","Evaluation, Observability \u0026 Safety","Prompt Engineering","提示工程与优化 (Prompt Engineering)","Prompt Engineering \u0026 Management","Attack Techniques \u0026 Red Teaming","Other LLM Frameworks"],"sub_categories":["Shannon: Autonomous Web Application Exploitation","Testing \u0026 Optimization","大语言对话模型及数据","Development Tools","Red-Teaming Harnesses \u0026 Automated Security Testing","Jailbreak Defense","Bleeding Edge ⚗️","文章","LLM Evaluations and Benchmarks","Prompting","项目","General-Purpose Machine Learning","Automated Red Teaming","Open Source Frameworks","🆕 Recently Added (January 2026)","AI \u0026 LLM Testing","Eval \u0026 Testing","Prompt Testing \u0026 Optimization","📊 8. MLOps / LLMOps \u0026 Production","Application Security","沙箱、可观测与评测","Evaluators and Test Harnesses","Vulnerability Scanners (Red Teaming)","Rust","Standard","Evaluation \u0026 Observability","Codex Resources","推理网关 (Inference Gateways)","Security Testing","LLM \u0026 GenAI Red Teaming","Videos Playlists"],"readme":"# Promptfoo: LLM evals \u0026 red teaming\n\n[![npm](https://img.shields.io/npm/v/promptfoo)](https://npmjs.com/package/promptfoo)\n[![npm](https://img.shields.io/npm/dm/promptfoo)](https://npmjs.com/package/promptfoo)\n[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/typpo/promptfoo/main.yml)](https://github.com/promptfoo/promptfoo/actions/workflows/main.yml)\n![MIT license](https://img.shields.io/github/license/promptfoo/promptfoo)\n[![Discord](https://github.com/user-attachments/assets/2092591a-ccc5-42a7-aeb6-24a2808950fd)](https://discord.gg/promptfoo)\n\n`promptfoo` is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.\n\n## Quick Start\n\n```sh\n# Install and initialize project\nnpx promptfoo@latest init\n\n# Run your first evaluation\nnpx promptfoo eval\n```\n\nSee [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.\n\n## What can you do with Promptfoo?\n\n- **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)\n- **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning\n- **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))\n- **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)\n- **Share results** with your team\n\nHere's what it looks like in action:\n\n![prompt evaluation matrix - web viewer](https://www.promptfoo.dev/img/claude-vs-gpt-example@2x.png)\n\nIt works on the command line too:\n\n![prompt evaluation matrix - command line](https://github.com/promptfoo/promptfoo/assets/310310/480e1114-d049-40b9-bd5f-f81c15060284)\n\nIt also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):\n\n![gen ai red team](https://www.promptfoo.dev/img/riskreport-1@2x.png)\n\n## Why promptfoo?\n\n- 🚀 **Developer-first**: Fast, with features like live reload and caching\n- 🔒 **Private**: Runs 100% locally - your prompts never leave your machine\n- 🔧 **Flexible**: Works with any LLM API or programming language\n- 💪 **Battle-tested**: Powers LLM apps serving 10M+ users in production\n- 📊 **Data-driven**: Make decisions based on metrics, not gut feel\n- 🤝 **Open source**: MIT licensed, with an active community\n\n## Learn More\n\n- 📚 [Full Documentation](https://www.promptfoo.dev/docs/intro/)\n- 🔐 [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)\n- 🎯 [Getting Started](https://www.promptfoo.dev/docs/getting-started/)\n- 💻 [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)\n- 📦 [Node.js Package](https://www.promptfoo.dev/docs/usage/node-package/)\n- 🤖 [Supported Models](https://www.promptfoo.dev/docs/providers/)\n\n## Contributing\n\nWe welcome contributions! Check out our [contributing guide](https://www.promptfoo.dev/docs/contributing/) to get started.\n\nJoin our [Discord community](https://discord.gg/promptfoo) for help and discussion.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpromptfoo%2Fpromptfoo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo/lists"}