https://github.com/voiceeval/voiceeval

VoiceEval: World-Class Open-source Voice AI Testing Framework
https://github.com/voiceeval/voiceeval

open-source pytest python testing testing-framework voice-agents voice-ai voice-testing

Last synced: 6 months ago
JSON representation

VoiceEval: World-Class Open-source Voice AI Testing Framework

• Host: GitHub
• URL: https://github.com/voiceeval/voiceeval
• Owner: voiceeval
• License: apache-2.0
• Created: 2025-11-24T18:10:29.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2025-11-24T18:40:34.000Z (7 months ago)
• Last Synced: 2025-11-27T10:11:49.389Z (7 months ago)
• Topics: open-source, pytest, python, testing, testing-framework, voice-agents, voice-ai, voice-testing
• Homepage:
• Size: 10.7 KB
• Stars: 1
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# VoiceEval



**The Open-Source Voice AI Testing Framework**

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)[![PyPI version](https://badge.fury.io/py/voiceeval.svg)](https://pypi.org/project/voiceeval/)



---

## 🚧 Under Active Development

VoiceEval is currently in **early development**. We're building the world's most comprehensive voice AI testing framework—**making voice agents reliable, testable, and production-ready**.

**⭐ Star this repo** to follow our progress and get notified when we launch!

---

## What is VoiceEval?

VoiceEval is an open-source testing framework built specifically for voice AI agents. As voice AI systems become critical business infrastructure, testing them manually doesn't scale—edge cases slip through, latency issues compound across the ASR-LLM-TTS pipeline, and there's no standardized way to measure quality.

VoiceEval solves this by providing developers with comprehensive automated testing tools: measure transcription accuracy (WER), validate conversation flow, test task completion, monitor latency, and evaluate response quality using LLM-as-judge validation. All with familiar pytest-style workflows and seamless CI/CD integration.

### Key Features (Coming Soon)

- ✅ **Comprehensive Metrics**: WER, latency, task completion, conversation flow

- ✅ **LLM-as-Judge**: Intelligent evaluation using GPT-4, Claude, local LLMs

- ✅ **Pytest Integration**: Familiar testing workflow developers already know

- ✅ **CI/CD Native**: GitHub Actions, GitLab CI, Jenkins ready

- ✅ **Enterprise Ready**: SOC2 compliant, self-hosted options

---

## Roadmap

Coming Soon...

## Contributing

We welcome contributions! More details coming soon.

For now:

- 🐛 [Report bugs](https://github.com/voiceeval/voiceeval/issues)

- 💡 [Request features](https://github.com/voiceeval/voiceeval/issues)

- ⭐ Star the repo to show support!

---

## License

Apache 2.0 - See [LICENSE](LICENSE) for details.

---



**Built with ❤️ by developers who believe voice AI should just work.**