{"id":13406013,"url":"https://github.com/promptfoo/promptfoo","last_synced_at":"2026-03-03T02:11:39.384Z","repository":{"id":161226306,"uuid":"633927609","full_name":"promptfoo/promptfoo","owner":"promptfoo","description":"Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.","archived":false,"fork":false,"pushed_at":"2025-03-08T00:53:33.000Z","size":323228,"stargazers_count":5756,"open_issues_count":188,"forks_count":475,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-03-08T01:24:43.599Z","etag":null,"topics":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"latest_commit_sha":null,"homepage":"https://promptfoo.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/promptfoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["typpo"]}},"created_at":"2023-04-28T15:48:49.000Z","updated_at":"2025-03-07T21:56:31.000Z","dependencies_parsed_at":"2023-09-26T23:40:46.827Z","dependency_job_id":"5f651e5a-0f03-4d24-887d-7e05e94a5e10","html_url":"https://github.com/promptfoo/promptfoo","commit_stats":null,"previous_names":["promptfoo/promptfoo","typpo/promptfoo"],"tags_count":278,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/promptfoo","download_url":"https://codeload.github.com/promptfoo/promptfoo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243562171,"owners_count":20311233,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"created_at":"2024-07-30T19:02:18.587Z","updated_at":"2025-10-15T17:39:08.781Z","avatar_url":"https://github.com/promptfoo.png","language":"TypeScript","funding_links":["https://github.com/sponsors/typpo"],"categories":["LLMOps","AI Hacking Resources \u0026 Links","Agent Security","TypeScript","🤖 LLM \u0026 Chatbot Testing","Evaluation \u0026 Observability","Evaluation \u0026 Quality Control","Prompts","\u003cimg src=\"./assets/satellite.svg\" width=\"16\" height=\"16\" style=\"vertical-align: middle;\"\u003e Satellites","A01_文本生成_文本对话","Evaluation Tools","Tooling \u0026 Dev Experience","✍ Prompt Engineering","[↑](#table-of-contents)Tools \u003ca name=\"tools\"\u003e\u003c/a\u003e","llm","Repos","Testing \u0026 Evaluation","Papers","Design Primitives","Open Source Security Tools","Tools of Trade","\u003ca id=\"tools\"\u003e\u003c/a\u003e🛠️ Tools","Tools \u0026 Code","vulnerability-scanners","GPT Security","testing","其他LLM框架","Tools","Agentic AI ![](https://img.shields.io/badge/_-AGENTIC-22d3ee?style=flat-square\u0026logo=openai\u0026logoColor=white)","📊 At a glance","AI Frameworks and SDKs","Evaluation and Monitoring","💉 Prompt Injection and Agent Threats","开源工具","资源列表","Red Teaming \u0026 Evaluation","Tools \u0026 Platforms","What's New","Software","Frameworks","Tools \u0026 Frameworks","Evaluation \u0026 Testing","🤖 AI \u0026 Machine Learning","Don't forget to give a :star: to make the project popular","Testing \u0026 Security","5 · Evaluation infrastructure (the eval stack: datasets, scorers, online/offline, tracing, CI)","Securing AI SaaS","3）参考实现与开源工具（GitHub）","⚔️ LLM And GenAI Security Testing Tools","Security Tools","3. Prompt Optimization","LLM-as-Judge Evaluation","Evaluation, Observability \u0026 Safety","Prompt Engineering","提示工程与优化 (Prompt Engineering)","Observability / evals","LLM and Agent Observability","Prompt Engineering \u0026 Management","Evaluation and Observability","Other LLM Frameworks","Attack Techniques \u0026 Red Teaming","🧰 Tools","Table of Contents","8. MLOps / LLMOps \u0026 Production","LLM Testing / Monitoring","LLMs Eval","AI Red Teaming (Testing AI Targets)","Supporting Infrastructure","Prompt and Behaviour Optimization","Catalog","Evaluation Frameworks","*Ops for AI","How to run these benchmarks"],"sub_categories":["LLM Evaluation \u0026 Testing","Shannon: Autonomous Web Application Exploitation","大语言对话模型及数据","Datasets","Development Tools","Testing \u0026 Optimization","Red-Teaming Harnesses \u0026 Automated Security Testing","Jailbreak Defense","Verification \u0026 CI Integration","Offensive / Red Teaming","Bleeding Edge ⚗️","Standard","文章","LLM Evaluations and Benchmarks","Agent gateways and LLMOps","Tools and Frameworks","Prompting","项目","General-Purpose Machine Learning","Automated Red Teaming","Open Source Frameworks","🆕 Recently Added (January 2026)","AI \u0026 LLM Testing","Eval \u0026 Testing","Prompt Testing \u0026 Optimization","AI Products with UI Generation","5a · Eval frameworks \u0026 harnesses (code-first test-runners)","Application Security","沙箱、可观测与评测","Evaluators and Test Harnesses","Vulnerability Scanners (Red Teaming)","Rust","Evaluation \u0026 Observability","Codex Resources","推理网关 (Inference Gateways)","Security Testing","Videos Playlists","LLM \u0026 GenAI Red Teaming","Guardrail Frameworks","🤖 AI Security / AI Red Teaming","Evaluation","Evaluation Harnesses \u0026 Benchmarks","LLMOps","Start with a unified harness"],"readme":"# Promptfoo: LLM evals \u0026 red teaming\n\n[![npm](https://img.shields.io/npm/v/promptfoo)](https://npmjs.com/package/promptfoo)\n[![npm](https://img.shields.io/npm/dm/promptfoo)](https://npmjs.com/package/promptfoo)\n[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/typpo/promptfoo/main.yml)](https://github.com/promptfoo/promptfoo/actions/workflows/main.yml)\n![MIT license](https://img.shields.io/github/license/promptfoo/promptfoo)\n[![Discord](https://github.com/user-attachments/assets/2092591a-ccc5-42a7-aeb6-24a2808950fd)](https://discord.gg/promptfoo)\n\n`promptfoo` is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.\n\n## Quick Start\n\n```sh\n# Install and initialize project\nnpx promptfoo@latest init\n\n# Run your first evaluation\nnpx promptfoo eval\n```\n\nSee [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.\n\n## What can you do with Promptfoo?\n\n- **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)\n- **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning\n- **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))\n- **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)\n- **Share results** with your team\n\nHere's what it looks like in action:\n\n![prompt evaluation matrix - web viewer](https://www.promptfoo.dev/img/claude-vs-gpt-example@2x.png)\n\nIt works on the command line too:\n\n![prompt evaluation matrix - command line](https://github.com/promptfoo/promptfoo/assets/310310/480e1114-d049-40b9-bd5f-f81c15060284)\n\nIt also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):\n\n![gen ai red team](https://www.promptfoo.dev/img/riskreport-1@2x.png)\n\n## Why promptfoo?\n\n- 🚀 **Developer-first**: Fast, with features like live reload and caching\n- 🔒 **Private**: Runs 100% locally - your prompts never leave your machine\n- 🔧 **Flexible**: Works with any LLM API or programming language\n- 💪 **Battle-tested**: Powers LLM apps serving 10M+ users in production\n- 📊 **Data-driven**: Make decisions based on metrics, not gut feel\n- 🤝 **Open source**: MIT licensed, with an active community\n\n## Learn More\n\n- 📚 [Full Documentation](https://www.promptfoo.dev/docs/intro/)\n- 🔐 [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)\n- 🎯 [Getting Started](https://www.promptfoo.dev/docs/getting-started/)\n- 💻 [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)\n- 📦 [Node.js Package](https://www.promptfoo.dev/docs/usage/node-package/)\n- 🤖 [Supported Models](https://www.promptfoo.dev/docs/providers/)\n\n## Contributing\n\nWe welcome contributions! Check out our [contributing guide](https://www.promptfoo.dev/docs/contributing/) to get started.\n\nJoin our [Discord community](https://discord.gg/promptfoo) for help and discussion.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpromptfoo%2Fpromptfoo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo/lists"}