{"id":29031626,"url":"https://github.com/51bitquant/openai_voice_agent_twilio","last_synced_at":"2025-06-26T10:04:49.632Z","repository":{"id":300878319,"uuid":"1007451339","full_name":"51bitquant/openai_voice_agent_twilio","owner":"51bitquant","description":"Twilio Streaming Voice Assistant using OpenAI Realtime Agent","archived":false,"fork":false,"pushed_at":"2025-06-24T03:19:48.000Z","size":240,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-24T04:27:11.195Z","etag":null,"topics":["agent","fastapi","openai","openai-realtime","realtime","realtime-api","twilio","twilio-voice"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/51bitquant.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-24T02:54:39.000Z","updated_at":"2025-06-24T03:19:51.000Z","dependencies_parsed_at":"2025-06-24T04:27:34.384Z","dependency_job_id":"8d56f0d6-8e6c-4490-803e-76c48e4f03a7","html_url":"https://github.com/51bitquant/openai_voice_agent_twilio","commit_stats":null,"previous_names":["51bitquant/openai_voice_agent_twilio"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/51bitquant/openai_voice_agent_twilio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/51bitquant%2Fopenai_voice_agent_twilio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/51bitquant%2Fopenai_voice_agent_twilio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/51bitquant%2Fopenai_voice_agent_twilio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/51bitquant%2Fopenai_voice_agent_twilio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/51bitquant","download_url":"https://codeload.github.com/51bitquant/openai_voice_agent_twilio/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/51bitquant%2Fopenai_voice_agent_twilio/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262044447,"owners_count":23249750,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","fastapi","openai","openai-realtime","realtime","realtime-api","twilio","twilio-voice"],"created_at":"2025-06-26T10:04:48.987Z","updated_at":"2025-06-26T10:04:49.616Z","avatar_url":"https://github.com/51bitquant.png","language":"Python","readme":"# Twilio OpenAI Realtime API Demo - Python FastAPI Implementation\n\nA comprehensive FastAPI reimplementation of the Twilio + OpenAI Realtime voice calling system, featuring enhanced reliability, modularity, and production-ready capabilities.\nIf you’d like to explore the TypeScript/JavaScript version, check it out here: https://github.com/openai/openai-realtime-twilio-demo\n## 🚀 Features\n\n- 🎙️ **Real-time Voice Calls**: Audio streaming through Twilio WebSocket connections\n- 🤖 **OpenAI Integration**: Voice recognition and generation using OpenAI Realtime API\n- 🔧 **Function Calling**: Support for OpenAI custom function execution (weather queries, etc.)\n- 📱 **Web Monitoring**: Real-time call logs and status monitoring interface\n- 🏗️ **Modular Architecture**: Clean separation of concerns with layered design\n- 🔄 **Auto-Reconnection**: Intelligent reconnection with exponential backoff strategy\n- 🛡️ **Enhanced Error Handling**: Comprehensive exception handling and recovery mechanisms\n- 📊 **Health Monitoring**: Built-in system resource monitoring and health checks\n- 🧹 **Memory Management**: Automatic session cleanup and connection management\n\n## 🏗️ Architecture Overview\n\n![img.png](img.png)\n\nThis FastAPI implementation features a modern, production-ready architecture:\n\n### Design Patterns\n- **Layered Architecture**: Clear separation between presentation, business, and data layers\n- **Dependency Injection**: Global service lifecycle management\n- **Factory Pattern**: Dynamic function registration and execution\n- **Observer Pattern**: WebSocket connection management and event distribution\n- **Singleton Pattern**: Global service managers\n\n### Project Structure\n\n```\nopenai_voice_agent_twilio/\n├── 📱 app/                          # Main application package\n│   ├── 📄 main.py                   # FastAPI application entry point\n│   ├── ⚙️ config.py                 # Configuration management\n│   ├── 📊 models/                   # Data model layer\n│   │   └── schemas.py               # Pydantic data models\n│   ├── 🔧 services/                 # Business logic layer\n│   │   ├── session_manager.py       # Core session management\n│   │   ├── openai_client.py         # OpenAI WebSocket client\n│   │   ├── function_handlers.py     # Function execution handlers\n│   │   └── session_cleanup.py       # Memory management service\n│   ├── 🌐 websocket/                # WebSocket handling layer\n│   │   ├── connection_manager.py    # Connection pool management\n│   │   └── handlers.py              # Message processing\n│   ├── 🛠️ utils/                    # Utility packages\n│   │   ├── error_handler.py         # Error handling and retry logic\n│   │   └── health_check.py          # Health monitoring service\n│   └── 📄 templates/                # Template files\n│       └── twiml.xml                # TwiML configuration\n├── 🚀 run.py                        # Application startup script\n├── ⚡ quick_start.py                # Quick setup script\n├── 🧪 test_server.py                # Testing utilities\n└── 📋 requirements.txt              # Python dependencies\n```\n\n## 🛠️ Installation \u0026 Setup\n\n### Prerequisites\n\n- Python 3.8+\n- OpenAI API key with Realtime API access\n- Twilio account with phone number\n- ngrok or similar tunneling service for local development\n\n### 1. Install Dependencies\n\n```bash\nuv sync\nor \npip install -r requirements.txt\n```\n\n### 2. Environment Configuration\n\nCreate a `.env` file in the project root:\n\n```bash\n# OpenAI API Key (Required)\nOPENAI_API_KEY=your-openai-api-key-here\n\n# Public URL for Twilio callbacks (Required)\nPUBLIC_URL=https://your-domain.ngrok.io\n\n# Server port (Optional, default: 8081)\nPORT=8081\n\n# Log level (Optional, default: INFO)\nLOG_LEVEL=INFO\n```\n\n**Important Notes**:\n- `OPENAI_API_KEY`: Obtain from [OpenAI Platform](https://platform.openai.com/api-keys)\n- `PUBLIC_URL`: Use ngrok or similar service to expose your local server\n\n### 3. Start the Server\n\nChoose one of the following methods:\n\n```bash\n# Method 1: Using the startup script (Recommended)\npython run.py\n\n# Method 2: Using uvicorn directly\nuvicorn app.main:app --host 0.0.0.0 --port 8081 --reload\n\n```\n\n### 4. Expose Service to Internet\n\n```bash\n# Install ngrok if not already installed\n# Visit https://ngrok.com/ to download\n\n# Start ngrok tunnel\nngrok http 8081\n\n# Copy the HTTPS URL to your .env file as PUBLIC_URL\n```\n\n### 5. Configure Twilio\n\n1. Log into [Twilio Console](https://console.twilio.com/)\n2. Purchase or configure a phone number\n3. Set Webhook URL to: `https://your-domain.ngrok.io/twiml`\n4. Set HTTP method to `POST`\n\n## 📡 API Endpoints\n\n### HTTP Endpoints\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/` | Server health status |\n| `GET` | `/public-url` | Get configured public URL |\n| `GET\\|POST` | `/twiml` | TwiML response endpoint (used by Twilio) |\n| `GET` | `/tools` | List available function tools |\n| `GET` | `/docs` | Interactive API documentation |\n| `GET` | `/health` | Detailed health check information |\n\n### WebSocket Endpoints\n\n| Endpoint | Purpose |\n|----------|---------|\n| `/ws/call` | Twilio audio stream connection |\n| `/ws/logs` | Frontend monitoring and logging |\n\n## 🔧 Core Components\n\n### Session Manager\n- **Three-way Connection Management**: Coordinates Twilio ↔ OpenAI ↔ Frontend connections\n- **Real-time Audio Forwarding**: Streams audio between services with minimal latency\n- **State Management**: Maintains session state across all connections\n- **Message Routing**: Intelligent message distribution and processing\n\n### OpenAI Client\n- **Realtime API Integration**: WebSocket connection to OpenAI's Realtime API\n- **Audio Stream Processing**: Handles audio encoding/decoding (g711_ulaw format)\n- **Function Call Support**: Executes custom functions and returns results\n- **Auto-Reconnection**: Intelligent reconnection with exponential backoff\n\n### WebSocket Connection Manager\n- **Multi-type Connection Pools**: Manages different connection types efficiently\n- **Connection Lifecycle Control**: Automatic connection cleanup and health monitoring\n- **Message Broadcasting**: Supports both unicast and multicast messaging\n- **Health Monitoring**: Continuous connection health assessment\n\n### Function Handler System\n- **Dynamic Registration**: Plugin-style function registration system\n- **Schema Validation**: Type-safe parameter validation using Pydantic\n- **Async Execution**: Non-blocking function execution with proper error handling\n- **Extensible Design**: Easy to add new functions and capabilities\n\n## 🔄 Auto-Reconnection Features\n\nThis implementation includes robust auto-reconnection capabilities not present in the original TypeScript version:\n\n### Reconnection Strategy\n- **Exponential Backoff**: Gradually increasing delays to avoid overwhelming servers\n- **Configurable Parameters**: Customizable retry counts, delays, and timeouts\n- **Intelligent Error Classification**: Different handling for different error types\n- **State Recovery**: Automatic session configuration recovery after reconnection\n\n### Reconnection Configuration\n```python\n# Configure reconnection parameters\nopenai_client.configure_reconnect(\n    auto_reconnect=True,      # Enable automatic reconnection\n    max_attempts=10,          # Maximum retry attempts\n    initial_delay=2.0,        # Initial delay in seconds\n    max_delay=60.0           # Maximum delay cap\n)\n```\n\n### Connection Status Monitoring\nReal-time status updates are sent to the frontend:\n```json\n{\n  \"type\": \"connection_status\",\n  \"status\": \"openai_connected\",\n  \"message\": \"OpenAI connection established\"\n}\n```\n\n## 🧪 Testing \u0026 Development\n\n### Run Tests\n```bash\n# Comprehensive server testing\npython test_server.py\n\n```\n\n### Development Tools\n- **Interactive API Docs**: Visit `/docs` for Swagger UI\n- **Health Monitoring**: Visit `/health` for system status\n- **Real-time Logs**: Connect to `/ws/logs` for live monitoring\n- **Function Testing**: Use `/tools` endpoint to verify available functions\n\n### Testing a Call\n1. Ensure server is running and accessible via ngrok\n2. Call your configured Twilio phone number\n3. After hearing \"Connected\", start speaking\n4. Try asking about weather to test function calling\n5. Monitor logs via WebSocket or console output\n\n## 🔌 Extending Functionality\n\n### Adding Custom Functions\n\nAdd new functions in `app/services/function_handlers.py`:\n\n```python\n# Register your function\nself.register_function(\n    name=\"your_custom_function\",\n    description=\"Description of what your function does\",\n    parameters={\n        \"type\": \"object\",\n        \"properties\": {\n            \"param1\": {\"type\": \"string\", \"description\": \"Parameter description\"}\n        },\n        \"required\": [\"param1\"]\n    },\n    handler=self._your_custom_handler\n)\n\n# Implement the handler\nasync def _your_custom_handler(self, args: Dict[str, Any]) -\u003e str:\n    # Your custom logic here\n    result = {\"status\": \"success\", \"data\": \"your_result\"}\n    return json.dumps(result)\n```\n\n### Customizing OpenAI Configuration\n\nModify the `OpenAISessionConfig` in `app/models/schemas.py`:\n\n```python\nclass OpenAISessionConfig(BaseModel):\n    modalities: List[str] = [\"text\", \"audio\"]\n    turn_detection: Dict[str, str] = {\"type\": \"server_vad\"}\n    voice: str = \"ash\"  # Options: ash, ballad, coral, sage, verse\n    temperature: float = 0.8\n    max_response_output_tokens: int = 4096\n```\n\n## 🚀 Performance Features\n\n### High Concurrency\n- **Async I/O Processing**: Non-blocking operations throughout\n- **WebSocket Connection Pooling**: Efficient connection management\n- **Event-driven Architecture**: Reactive programming patterns\n- **Stream Processing**: Real-time audio stream handling\n\n### Memory Optimization\n- **Automatic Connection Cleanup**: Prevents memory leaks\n- **Streaming Audio Processing**: Minimal memory footprint\n- **Garbage Collection Optimization**: Efficient resource management\n- **Session Cleanup Service**: Periodic cleanup of expired sessions\n\n### Network Optimization\n- **WebSocket Long Connections**: Persistent, low-latency connections\n- **Intelligent Reconnection**: Minimizes connection overhead\n- **Connection Pool Management**: Efficient resource utilization\n- **Compression Support**: Reduced bandwidth usage\n\n## 🛡️ Production Considerations\n\n### Security Features\n- **Input Validation**: Comprehensive Pydantic data validation\n- **Error Information Filtering**: Prevents sensitive data leakage\n- **WebSocket Authentication**: Support for connection authentication\n- **CORS Configuration**: Configurable cross-origin resource sharing\n\n### Monitoring \u0026 Observability\n- **Health Check Endpoints**: Comprehensive system health reporting\n- **Structured Logging**: JSON-formatted logs for easy parsing\n- **Performance Metrics**: Built-in performance monitoring\n- **Error Tracking**: Detailed error collection and analysis\n\n### Deployment Options\n- **Docker Support**: Container-ready configuration\n- **Cloud Platform Ready**: Compatible with major cloud providers\n- **Horizontal Scaling**: Stateless design supports load balancing\n- **Environment Configuration**: Flexible configuration management\n\n## 🔧 Configuration Reference\n\n### Environment Variables\n\n| Variable | Required | Default | Description |\n|----------|----------|---------|-------------|\n| `OPENAI_API_KEY` | Yes | - | OpenAI API key for Realtime API access |\n| `PUBLIC_URL` | Yes | - | Public URL for Twilio webhook callbacks |\n| `PORT` | No | 8081 | Server port number |\n| `LOG_LEVEL` | No | INFO | Logging level (DEBUG, INFO, WARNING, ERROR) |\n\n### Advanced Configuration\n\nFor production deployments, consider these additional configurations:\n\n```bash\n# Production environment\nENVIRONMENT=production\nLOG_FORMAT=json\nCORS_ORIGINS=[\"https://yourdomain.com\"]\nMAX_CONNECTIONS=100\nHEALTH_CHECK_INTERVAL=30\n```\n\n## 🐛 Troubleshooting\n\n### Common Issues\n\n#### 1. OpenAI Connection Failures\n- Verify `OPENAI_API_KEY` is correct and has Realtime API access\n- Check network connectivity and firewall settings\n- Ensure sufficient OpenAI account credits\n- Monitor logs for specific error messages\n\n#### 2. Twilio Connection Issues\n- Verify `PUBLIC_URL` is accessible from the internet\n- Ensure ngrok or tunneling service is running\n- Check Twilio webhook configuration\n- Verify TwiML endpoint responds correctly\n\n#### 3. Audio Quality Problems\n- Check network latency and stability\n- Verify audio format configuration (g711_ulaw)\n- Monitor OpenAI Realtime API status\n- Ensure proper WebSocket connection handling\n\n#### 4. Function Call Failures\n- Verify function registration in `function_handlers.py`\n- Check function parameter validation\n- Monitor function execution logs\n- Ensure proper JSON response formatting\n\n### Debug Mode\n\nEnable debug logging for detailed troubleshooting:\n\n```bash\nexport LOG_LEVEL=DEBUG\npython run.py\n```\n\n## 📊 Performance Metrics\n\n### System Capabilities\n- **Concurrent Connections**: Supports 100+ simultaneous WebSocket connections\n- **Audio Latency**: \u003c200ms end-to-end audio processing\n- **Memory Usage**: \u003c50MB base memory footprint\n- **CPU Efficiency**: Async processing minimizes CPU usage\n- **Network Throughput**: Optimized for real-time audio streaming\n\n### Comparison with Original TypeScript Version\n\n| Feature | TypeScript Version | FastAPI Version |\n|---------|-------------------|-----------------|\n| Auto-Reconnection | ❌ Not available | ✅ Full support |\n| Health Monitoring | ❌ Basic | ✅ Comprehensive |\n| Error Recovery | ❌ Limited | ✅ Advanced |\n| Memory Management | ❌ Manual | ✅ Automatic |\n| Type Safety | ✅ TypeScript | ✅ Pydantic |\n| API Documentation | ❌ Manual | ✅ Auto-generated |\n| Testing Tools | ❌ Limited | ✅ Comprehensive |\n| Production Ready | ⚠️ Basic | ✅ Enterprise-grade |\n\n## 📚 Additional Resources\n\n### Documentation\n- [OpenAI Realtime API Documentation](https://platform.openai.com/docs/guides/realtime)\n- [Twilio WebSocket Documentation](https://www.twilio.com/docs/voice/twiml/stream)\n- [FastAPI Documentation](https://fastapi.tiangolo.com/)\n\n### Community \u0026 Support\n- Report issues on the project repository\n- Contribute improvements via pull requests\n- Join discussions in the community forums\n\n## 📄 License\n\nThis project is a Python FastAPI reimplementation of the original TypeScript demo, maintaining the same open-source license terms.\n\n---\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F51bitquant%2Fopenai_voice_agent_twilio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F51bitquant%2Fopenai_voice_agent_twilio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F51bitquant%2Fopenai_voice_agent_twilio/lists"}