https://github.com/promptfoo/promptfoo-python
Python wrapper for the promptfoo CLI - LLM testing, red teaming, and security evaluation
https://github.com/promptfoo/promptfoo-python
Last synced: 3 months ago
JSON representation
Python wrapper for the promptfoo CLI - LLM testing, red teaming, and security evaluation
- Host: GitHub
- URL: https://github.com/promptfoo/promptfoo-python
- Owner: promptfoo
- License: mit
- Created: 2026-01-05T07:35:57.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2026-04-03T02:34:25.000Z (3 months ago)
- Last Synced: 2026-04-03T05:50:38.193Z (3 months ago)
- Language: Python
- Size: 187 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
# Promptfoo: LLM evals & red teaming
---
> **📦 About this Python package**
>
> This is a lightweight wrapper that installs promptfoo via `pip`. It requires **Node.js 20+** and executes `npx promptfoo@latest` under the hood.
>
> **💡 If you have Node.js installed**, we recommend using `npx promptfoo@latest` directly for better performance:
>
> ```bash
> npx promptfoo@latest init
> npx promptfoo@latest eval
> ```
>
> See the [main project](https://github.com/promptfoo/promptfoo) for the official npm package.
>
> **🐍 Use this pip wrapper when you:**
>
> - Need to install via `pip` for Python-only CI/CD environments
> - Want to manage promptfoo with poetry/pipenv/pip alongside Python dependencies
> - Work in environments where pip packages are easier to approve than npm
---
promptfoo is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.
Website ·
Getting Started ·
Red Teaming ·
Documentation ·
Discord
## Installation
### Requirements
- **Python 3.9+** (for this wrapper)
- **Node.js 20+** (required to run promptfoo)
### Install from PyPI
```bash
pip install promptfoo
```
### Alternative: Use npx (Recommended)
If you have Node.js installed, you can skip the wrapper and use npx directly:
```bash
npx promptfoo@latest init
npx promptfoo@latest eval
```
This is faster and gives you direct access to the latest version.
## Quick Start
```bash
# Install
pip install promptfoo
# Initialize project
promptfoo init
# Run your first evaluation
promptfoo eval
```
See [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.
## What can you do with Promptfoo?
- **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)
- **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning
- **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))
- **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)
- **Review pull requests** for LLM-related security and compliance issues with [code scanning](https://www.promptfoo.dev/docs/code-scanning/)
- **Share results** with your team
Here's what it looks like in action:
It works on the command line too:
It also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):
## Why Promptfoo?
- 🚀 **Developer-first**: Fast, with features like live reload and caching
- 🔒 **Private**: LLM evals run 100% locally - your prompts never leave your machine
- 🔧 **Flexible**: Works with any LLM API or programming language
- 💪 **Battle-tested**: Powers LLM apps serving 10M+ users in production
- 📊 **Data-driven**: Make decisions based on metrics, not gut feel
- 🤝 **Open source**: MIT licensed, with an active community
## How This Wrapper Works
This Python package is a thin wrapper that:
1. Checks if Node.js is installed
2. Executes `npx promptfoo@latest ` (or uses globally installed promptfoo if available)
3. Passes through all arguments and environment variables
4. Returns the same exit code
The actual promptfoo logic runs via the official TypeScript package from npm. All features and commands work identically.
## Python-Specific Usage
### With pip
```bash
pip install promptfoo
promptfoo eval
```
### With poetry
```bash
poetry add --group dev promptfoo
poetry run promptfoo eval
```
### With requirements.txt
```bash
echo "promptfoo" >> requirements.txt
pip install -r requirements.txt
promptfoo eval
```
### In CI/CD (GitHub Actions example)
```yaml
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "20"
- name: Install promptfoo
run: pip install promptfoo
- name: Run red team tests
run: promptfoo redteam run
```
## Learn More
- 📚 [Full Documentation](https://www.promptfoo.dev/docs/intro/)
- 🔐 [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)
- 🎯 [Getting Started](https://www.promptfoo.dev/docs/getting-started/)
- 💻 [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)
- 📦 [Main Project (npm)](https://github.com/promptfoo/promptfoo)
- 🤖 [Supported Models](https://www.promptfoo.dev/docs/providers/)
- 🔬 [Code Scanning Guide](https://www.promptfoo.dev/docs/code-scanning/)
## Troubleshooting
### "ERROR: promptfoo requires Node.js"
The wrapper needs Node.js to run. Install it:
- **macOS**: `brew install node`
- **Ubuntu/Debian**: `sudo apt install nodejs npm`
- **Windows**: Download from [nodejs.org](https://nodejs.org/)
- **Any OS**: Use [nvm](https://github.com/nvm-sh/nvm)
### Slow First Run
The first time you run `promptfoo`, npx downloads the latest version from npm (typically ~50MB). Subsequent runs use the cached version and are fast.
To speed this up, install promptfoo globally:
```bash
npm install -g promptfoo
```
The Python wrapper will automatically use the global installation when available.
### Version Pinning
By default, this wrapper uses `npx promptfoo@latest`. To pin a specific version:
```bash
export PROMPTFOO_VERSION=0.95.0
promptfoo --version
```
Or install a specific version globally:
```bash
npm install -g promptfoo@0.95.0
```
### Telemetry
This wrapper collects anonymous usage telemetry to help improve the package, mirroring the telemetry in the main promptfoo project.
**What is collected:** A random anonymous user ID, wrapper version, Python version, and whether you're running in CI. If you've previously logged into promptfoo and your email is stored in `~/.promptfoo/promptfoo.yaml`, that is also sent.
**To opt out:**
```bash
export PROMPTFOO_DISABLE_TELEMETRY=1
```
## Contributing
**For promptfoo features, docs, and bugs:** contribute to [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo).
**For pip wrapper issues** (installation, Python shim, CI integration): open issues here. See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup.
Join our [Discord community](https://discord.gg/promptfoo) for help and discussion.
## License
MIT License - Same as promptfoo