https://github.com/voicetestdev/voicetest
Test harness for voice agents. Import from Retell, VAPI, Bland, LiveKit. Run autonomous simulations. Evaluate with LLM judges.
https://github.com/voicetestdev/voicetest
bland dspy livekit llm retell testing vapi voice-ai
Last synced: 2 months ago
JSON representation
Test harness for voice agents. Import from Retell, VAPI, Bland, LiveKit. Run autonomous simulations. Evaluate with LLM judges.
- Host: GitHub
- URL: https://github.com/voicetestdev/voicetest
- Owner: voicetestdev
- License: apache-2.0
- Created: 2026-01-14T16:53:52.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2026-02-25T01:19:13.000Z (4 months ago)
- Last Synced: 2026-02-25T07:21:04.274Z (4 months ago)
- Topics: bland, dspy, livekit, llm, retell, testing, vapi, voice-ai
- Language: Python
- Homepage: https://voicetest.dev
- Size: 17.8 MB
- Stars: 8
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-voice-agents - voicetest
- awesome-agents - voicetest - source test harness for voice agents with support for Retell, VAPI, Bland, and LiveKit. Run autonomous simulations and evaluate with LLM judges.  (Testing and Evaluation)
- awesome-testing - voicetest - Open-source test harness for voice AI agents supporting Retell, VAPI, LiveKit, and Bland with autonomous simulations and LLM-based evaluation. (Software / AI & LLM Testing)
README
[](https://pypi.org/project/voicetest/) [](https://github.com/voicetestdev/voicetest/releases) [](https://opensource.org/licenses/Apache-2.0) [](https://github.com/voicetestdev/voicetest/actions/workflows/test.yml)
Open-source test harness for voice agent workflows.
- **Simulate conversations** — LLM-powered users talk to your agent, LLM judges score the results
- **Test any platform** — Retell, VAPI, LiveKit, Bland, Telnyx, or custom agents
- **Convert between formats** — Import from one platform, export to another via a unified graph IR
- **Diagnose failures** — Auto-fix broken prompts with an LLM-powered repair loop
- **Run anywhere** — CLI, Web UI, REST API, CI/CD
## Installation
```bash
uv tool install voicetest
```
Or add to a project (`uv run voicetest` to run):
```bash
uv add voicetest
```
Or with pip:
```bash
pip install voicetest
```
## Quick start
Try voicetest with a sample healthcare receptionist agent and 8 test cases:
```bash
# Set up an API key (free, no credit card at https://console.groq.com)
export GROQ_API_KEY=gsk_...
# Load demo and start web UI
voicetest demo --serve
```
> **Tip:** If you have [Claude Code](https://claude.ai/claude-code) installed, skip API key setup and use `claudecode/sonnet` as your model. See [Claude Code Passthrough](https://voicetest.dev/docs/configuration/#claude-code-passthrough).
## Web UI
```bash
voicetest serve
```
Agent import, graph visualization, test execution with real-time streaming transcripts, run history, diagnosis, and more at http://localhost:8000.
## Platform support
Import from any supported format, convert through the unified AgentGraph, and export to any other:
```
Retell CF ─────┐ ┌───▶ Retell LLM
│ │
Retell LLM ────┼ ├───▶ Retell CF
│ │
VAPI ──────────┼ ├───▶ VAPI
│ │
Bland ─────────┼───▶ AgentGraph ──┼───▶ Bland
│ │
Telnyx ────────┤ ├───▶ Telnyx
│ │
LiveKit ───────┤ ├───▶ LiveKit
│ │
XLSForm ───────┤ ├───▶ Mermaid
│ │
Custom ────────┘ └───▶ Voicetest JSON
```
| Platform | Import | Push | Sync | API Key Env Var |
| -------- | ------ | ---- | ---- | ---------------------------------------- |
| Retell | ✓ | ✓ | ✓ | `RETELL_API_KEY` |
| VAPI | ✓ | ✓ | ✓ | `VAPI_API_KEY` |
| Bland | ✓ | ✓ | | `BLAND_API_KEY` |
| Telnyx | ✓ | ✓ | ✓ | `TELNYX_API_KEY` |
| LiveKit | ✓ | ✓ | ✓ | `LIVEKIT_API_KEY` + `LIVEKIT_API_SECRET` |
## CI/CD
Run voice agent tests in GitHub Actions to catch regressions before production:
```yaml
name: Voice Agent Tests
on:
push:
paths: ["agents/**"]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
- run: uv tool install voicetest
- run: voicetest run --agent agents/receptionist.json --tests agents/tests.json --all
env:
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
```
## Configuration
Settings are stored in `.voicetest/settings.toml`:
```toml
[models]
agent = "groq/llama-3.1-8b-instant"
simulator = "groq/llama-3.1-8b-instant"
judge = "groq/llama-3.1-8b-instant"
[run]
max_turns = 20
audio_eval = false
streaming = false
```
Any [LiteLLM-compatible model](https://docs.litellm.ai/docs/providers) works — OpenAI, Anthropic, Google, Ollama, and more. See the [full configuration reference](https://voicetest.dev/docs/configuration/).
## Documentation
Full documentation is at [voicetest.dev/docs](https://voicetest.dev/docs/).
| Topic | Description |
| -------------------------------------------------------------- | ------------------------------------------------------------ |
| [Getting Started](https://voicetest.dev/docs/getting-started/) | Install, demo, first test walkthrough |
| [Core Concepts](https://voicetest.dev/docs/concepts/) | Agent graphs, node types, test cases |
| [CLI Reference](https://voicetest.dev/docs/cli/) | All commands and options |
| [Features](https://voicetest.dev/docs/features/) | Format conversion, diagnosis, audio eval, snippets, and more |
| [Configuration](https://voicetest.dev/docs/configuration/) | Models, settings, Claude Code, platform credentials |
| [Architecture](https://voicetest.dev/docs/architecture/) | Engine internals, DI, storage |
| [Development](https://voicetest.dev/docs/development/) | Contributing, Docker setup, code quality |
## Contact
Questions, feedback, or partnerships: [hello@voicetest.dev](mailto:hello@voicetest.dev)
## License
Apache 2.0