{"id":48060851,"url":"https://github.com/promptfoo/promptfoo-python","last_synced_at":"2026-04-04T14:32:32.771Z","repository":{"id":340907264,"uuid":"1128135929","full_name":"promptfoo/promptfoo-python","owner":"promptfoo","description":"Python wrapper for the promptfoo CLI - LLM testing, red teaming, and security evaluation","archived":false,"fork":false,"pushed_at":"2026-04-03T02:34:25.000Z","size":191,"stargazers_count":1,"open_issues_count":2,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-03T05:50:38.193Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/promptfoo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-01-05T07:35:57.000Z","updated_at":"2026-04-02T18:46:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/promptfoo/promptfoo-python","commit_stats":null,"previous_names":["promptfoo/promptfoo-python"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/promptfoo/promptfoo-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/promptfoo","download_url":"https://codeload.github.com/promptfoo/promptfoo-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptfoo%2Fpromptfoo-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31402739,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-04T14:32:32.643Z","updated_at":"2026-04-04T14:32:32.749Z","avatar_url":"https://github.com/promptfoo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Promptfoo: LLM evals \u0026 red teaming\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/promptfoo/\"\u003e\u003cimg src=\"https://badge.fury.io/py/promptfoo.svg\" alt=\"PyPI version\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/promptfoo/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/pyversions/promptfoo.svg\" alt=\"Python versions\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://npmjs.com/package/promptfoo\"\u003e\u003cimg src=\"https://img.shields.io/npm/dm/promptfoo\" alt=\"npm downloads\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/promptfoo/promptfoo/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/github/license/promptfoo/promptfoo\" alt=\"MIT license\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://discord.gg/promptfoo\"\u003e\u003cimg src=\"https://github.com/user-attachments/assets/2092591a-ccc5-42a7-aeb6-24a2808950fd\" alt=\"Discord\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n\u003e **📦 About this Python package**\n\u003e\n\u003e This is a lightweight wrapper that installs promptfoo via `pip`. It requires **Node.js 20+** and executes `npx promptfoo@latest` under the hood.\n\u003e\n\u003e **💡 If you have Node.js installed**, we recommend using `npx promptfoo@latest` directly for better performance:\n\u003e\n\u003e ```bash\n\u003e npx promptfoo@latest init\n\u003e npx promptfoo@latest eval\n\u003e ```\n\u003e\n\u003e See the [main project](https://github.com/promptfoo/promptfoo) for the official npm package.\n\u003e\n\u003e **🐍 Use this pip wrapper when you:**\n\u003e\n\u003e - Need to install via `pip` for Python-only CI/CD environments\n\u003e - Want to manage promptfoo with poetry/pipenv/pip alongside Python dependencies\n\u003e - Work in environments where pip packages are easier to approve than npm\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003ccode\u003epromptfoo\u003c/code\u003e is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://www.promptfoo.dev\"\u003eWebsite\u003c/a\u003e ·\n  \u003ca href=\"https://www.promptfoo.dev/docs/getting-started/\"\u003eGetting Started\u003c/a\u003e ·\n  \u003ca href=\"https://www.promptfoo.dev/docs/red-team/\"\u003eRed Teaming\u003c/a\u003e ·\n  \u003ca href=\"https://www.promptfoo.dev/docs/\"\u003eDocumentation\u003c/a\u003e ·\n  \u003ca href=\"https://discord.gg/promptfoo\"\u003eDiscord\u003c/a\u003e\n\u003c/p\u003e\n\n## Installation\n\n### Requirements\n\n- **Python 3.9+** (for this wrapper)\n- **Node.js 20+** (required to run promptfoo)\n\n### Install from PyPI\n\n```bash\npip install promptfoo\n```\n\n### Alternative: Use npx (Recommended)\n\nIf you have Node.js installed, you can skip the wrapper and use npx directly:\n\n```bash\nnpx promptfoo@latest init\nnpx promptfoo@latest eval\n```\n\nThis is faster and gives you direct access to the latest version.\n\n## Quick Start\n\n```bash\n# Install\npip install promptfoo\n\n# Initialize project\npromptfoo init\n\n# Run your first evaluation\npromptfoo eval\n```\n\nSee [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.\n\n## What can you do with Promptfoo?\n\n- **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)\n- **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning\n- **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))\n- **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)\n- **Review pull requests** for LLM-related security and compliance issues with [code scanning](https://www.promptfoo.dev/docs/code-scanning/)\n- **Share results** with your team\n\nHere's what it looks like in action:\n\n![prompt evaluation matrix - web viewer](https://www.promptfoo.dev/img/claude-vs-gpt-example@2x.png)\n\nIt works on the command line too:\n\n![prompt evaluation matrix - command line](https://github.com/promptfoo/promptfoo/assets/310310/480e1114-d049-40b9-bd5f-f81c15060284)\n\nIt also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):\n\n![gen ai red team](https://www.promptfoo.dev/img/riskreport-1@2x.png)\n\n## Why Promptfoo?\n\n- 🚀 **Developer-first**: Fast, with features like live reload and caching\n- 🔒 **Private**: LLM evals run 100% locally - your prompts never leave your machine\n- 🔧 **Flexible**: Works with any LLM API or programming language\n- 💪 **Battle-tested**: Powers LLM apps serving 10M+ users in production\n- 📊 **Data-driven**: Make decisions based on metrics, not gut feel\n- 🤝 **Open source**: MIT licensed, with an active community\n\n## How This Wrapper Works\n\nThis Python package is a thin wrapper that:\n\n1. Checks if Node.js is installed\n2. Executes `npx promptfoo@latest \u003cyour-args\u003e` (or uses globally installed promptfoo if available)\n3. Passes through all arguments and environment variables\n4. Returns the same exit code\n\nThe actual promptfoo logic runs via the official TypeScript package from npm. All features and commands work identically.\n\n## Python-Specific Usage\n\n### With pip\n\n```bash\npip install promptfoo\npromptfoo eval\n```\n\n### With poetry\n\n```bash\npoetry add --group dev promptfoo\npoetry run promptfoo eval\n```\n\n### With requirements.txt\n\n```bash\necho \"promptfoo\" \u003e\u003e requirements.txt\npip install -r requirements.txt\npromptfoo eval\n```\n\n### In CI/CD (GitHub Actions example)\n\n```yaml\n- name: Setup Node.js\n  uses: actions/setup-node@v4\n  with:\n    node-version: \"20\"\n\n- name: Install promptfoo\n  run: pip install promptfoo\n\n- name: Run red team tests\n  run: promptfoo redteam run\n```\n\n## Learn More\n\n- 📚 [Full Documentation](https://www.promptfoo.dev/docs/intro/)\n- 🔐 [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)\n- 🎯 [Getting Started](https://www.promptfoo.dev/docs/getting-started/)\n- 💻 [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)\n- 📦 [Main Project (npm)](https://github.com/promptfoo/promptfoo)\n- 🤖 [Supported Models](https://www.promptfoo.dev/docs/providers/)\n- 🔬 [Code Scanning Guide](https://www.promptfoo.dev/docs/code-scanning/)\n\n## Troubleshooting\n\n### \"ERROR: promptfoo requires Node.js\"\n\nThe wrapper needs Node.js to run. Install it:\n\n- **macOS**: `brew install node`\n- **Ubuntu/Debian**: `sudo apt install nodejs npm`\n- **Windows**: Download from [nodejs.org](https://nodejs.org/)\n- **Any OS**: Use [nvm](https://github.com/nvm-sh/nvm)\n\n### Slow First Run\n\nThe first time you run `promptfoo`, npx downloads the latest version from npm (typically ~50MB). Subsequent runs use the cached version and are fast.\n\nTo speed this up, install promptfoo globally:\n\n```bash\nnpm install -g promptfoo\n```\n\nThe Python wrapper will automatically use the global installation when available.\n\n### Version Pinning\n\nBy default, this wrapper uses `npx promptfoo@latest`. To pin a specific version:\n\n```bash\nexport PROMPTFOO_VERSION=0.95.0\npromptfoo --version\n```\n\nOr install a specific version globally:\n\n```bash\nnpm install -g promptfoo@0.95.0\n```\n\n### Telemetry\n\nThis wrapper collects anonymous usage telemetry to help improve the package, mirroring the telemetry in the main promptfoo project.\n\n**What is collected:** A random anonymous user ID, wrapper version, Python version, and whether you're running in CI. If you've previously logged into promptfoo and your email is stored in `~/.promptfoo/promptfoo.yaml`, that is also sent.\n\n**To opt out:**\n\n```bash\nexport PROMPTFOO_DISABLE_TELEMETRY=1\n```\n\n## Contributing\n\n**For promptfoo features, docs, and bugs:** contribute to [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo).\n\n**For pip wrapper issues** (installation, Python shim, CI integration): open issues here. See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup.\n\nJoin our [Discord community](https://discord.gg/promptfoo) for help and discussion.\n\n\u003ca href=\"https://github.com/promptfoo/promptfoo/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=promptfoo/promptfoo\" /\u003e\n\u003c/a\u003e\n\n## License\n\nMIT License - Same as promptfoo\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpromptfoo%2Fpromptfoo-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptfoo%2Fpromptfoo-python/lists"}