https://github.com/promptfoo/promptfoo

Test your prompts, models, and RAGs. Catch regressions and improve prompt quality. LLM evals for OpenAI, Azure, Anthropic, Gemini, Mistral, Llama, Bedrock, Ollama, and other local & private models with CI/CD integration.
https://github.com/promptfoo/promptfoo

ci ci-cd cicd evaluation evaluation-framework llm llm-eval llm-evaluation llm-evaluation-framework llmops prompt-engineering prompt-testing prompts rag testing

Last synced: 3 months ago
JSON representation

Test your prompts, models, and RAGs. Catch regressions and improve prompt quality. LLM evals for OpenAI, Azure, Anthropic, Gemini, Mistral, Llama, Bedrock, Ollama, and other local & private models with CI/CD integration.

• Host: GitHub
• URL: https://github.com/promptfoo/promptfoo
• Owner: promptfoo
• License: mit
• Created: 2023-04-28T15:48:49.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-05-27T05:04:36.000Z (8 months ago)
• Last Synced: 2024-05-27T19:24:38.096Z (8 months ago)
• Topics: ci, ci-cd, cicd, evaluation, evaluation-framework, llm, llm-eval, llm-evaluation, llm-evaluation-framework, llmops, prompt-engineering, prompt-testing, prompts, rag, testing
• Language: TypeScript
• Homepage: https://www.promptfoo.dev/
• Size: 14.7 MB
• Stars: 3,018
• Watchers: 16
• Forks: 201
• Open Issues: 68
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

• awesome-langchain-zh - Promptfoo
• awesome-ChatGPT-repositories - promptfoo - Test your prompts, models, RAGs. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality. LLM evals for OpenAI/Azure GPT, Anthropic Claude, VertexAI Gemini, Ollama, Local & private models like Mistral/Mixtral/Llama with CI/CD (Prompts)
• awesome-langchain - Promptfoo
• StarryDivineSky - promptfoo/promptfoo
• awesome - promptfoo/promptfoo - Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration. (TypeScript)
• awesome - promptfoo/promptfoo - Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration. (TypeScript)
• jimsghstars - promptfoo/promptfoo - Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command (TypeScript)