Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/51bitquant/openai_voice_agent_twilio

Twilio Streaming Voice Assistant using OpenAI Realtime Agent
https://github.com/51bitquant/openai_voice_agent_twilio

agent fastapi openai openai-realtime realtime realtime-api twilio twilio-voice

Last synced: 10 months ago
JSON representation

Twilio Streaming Voice Assistant using OpenAI Realtime Agent

Awesome Lists containing this project

README 

          
# Twilio OpenAI Realtime API Demo - Python FastAPI Implementation



A comprehensive FastAPI reimplementation of the Twilio + OpenAI Realtime voice calling system, featuring enhanced reliability, modularity, and production-ready capabilities.

If you’d like to explore the TypeScript/JavaScript version, check it out here: https://github.com/openai/openai-realtime-twilio-demo

## 🚀 Features



- 🎙️ **Real-time Voice Calls**: Audio streaming through Twilio WebSocket connections

- 🤖 **OpenAI Integration**: Voice recognition and generation using OpenAI Realtime API

- 🔧 **Function Calling**: Support for OpenAI custom function execution (weather queries, etc.)

- 📱 **Web Monitoring**: Real-time call logs and status monitoring interface

- 🏗️ **Modular Architecture**: Clean separation of concerns with layered design

- 🔄 **Auto-Reconnection**: Intelligent reconnection with exponential backoff strategy

- 🛡️ **Enhanced Error Handling**: Comprehensive exception handling and recovery mechanisms

- 📊 **Health Monitoring**: Built-in system resource monitoring and health checks

- 🧹 **Memory Management**: Automatic session cleanup and connection management



## 🏗️ Architecture Overview



![img.png](img.png)



This FastAPI implementation features a modern, production-ready architecture:



### Design Patterns

- **Layered Architecture**: Clear separation between presentation, business, and data layers

- **Dependency Injection**: Global service lifecycle management

- **Factory Pattern**: Dynamic function registration and execution

- **Observer Pattern**: WebSocket connection management and event distribution

- **Singleton Pattern**: Global service managers



### Project Structure



```

openai_voice_agent_twilio/

├── 📱 app/                          # Main application package

│   ├── 📄 main.py                   # FastAPI application entry point

│   ├── ⚙️ config.py                 # Configuration management

│   ├── 📊 models/                   # Data model layer

│   │   └── schemas.py               # Pydantic data models

│   ├── 🔧 services/                 # Business logic layer

│   │   ├── session_manager.py       # Core session management

│   │   ├── openai_client.py         # OpenAI WebSocket client

│   │   ├── function_handlers.py     # Function execution handlers

│   │   └── session_cleanup.py       # Memory management service

│   ├── 🌐 websocket/                # WebSocket handling layer

│   │   ├── connection_manager.py    # Connection pool management

│   │   └── handlers.py              # Message processing

│   ├── 🛠️ utils/                    # Utility packages

│   │   ├── error_handler.py         # Error handling and retry logic

│   │   └── health_check.py          # Health monitoring service

│   └── 📄 templates/                # Template files

│       └── twiml.xml                # TwiML configuration

├── 🚀 run.py                        # Application startup script

├── ⚡ quick_start.py                # Quick setup script

├── 🧪 test_server.py                # Testing utilities

└── 📋 requirements.txt              # Python dependencies

```



## 🛠️ Installation & Setup



### Prerequisites



- Python 3.8+

- OpenAI API key with Realtime API access

- Twilio account with phone number

- ngrok or similar tunneling service for local development



### 1. Install Dependencies



```bash

uv sync

or 

pip install -r requirements.txt

```



### 2. Environment Configuration



Create a `.env` file in the project root:



```bash

# OpenAI API Key (Required)

OPENAI_API_KEY=your-openai-api-key-here



# Public URL for Twilio callbacks (Required)

PUBLIC_URL=https://your-domain.ngrok.io



# Server port (Optional, default: 8081)

PORT=8081



# Log level (Optional, default: INFO)

LOG_LEVEL=INFO

```



**Important Notes**:

- `OPENAI_API_KEY`: Obtain from [OpenAI Platform](https://platform.openai.com/api-keys)

- `PUBLIC_URL`: Use ngrok or similar service to expose your local server



### 3. Start the Server



Choose one of the following methods:



```bash

# Method 1: Using the startup script (Recommended)

python run.py



# Method 2: Using uvicorn directly

uvicorn app.main:app --host 0.0.0.0 --port 8081 --reload



```



### 4. Expose Service to Internet



```bash

# Install ngrok if not already installed

# Visit https://ngrok.com/ to download



# Start ngrok tunnel

ngrok http 8081



# Copy the HTTPS URL to your .env file as PUBLIC_URL

```



### 5. Configure Twilio



1. Log into [Twilio Console](https://console.twilio.com/)

2. Purchase or configure a phone number

3. Set Webhook URL to: `https://your-domain.ngrok.io/twiml`

4. Set HTTP method to `POST`



## 📡 API Endpoints



### HTTP Endpoints



| Method | Endpoint | Description |

|--------|----------|-------------|

| `GET` | `/` | Server health status |

| `GET` | `/public-url` | Get configured public URL |

| `GET\|POST` | `/twiml` | TwiML response endpoint (used by Twilio) |

| `GET` | `/tools` | List available function tools |

| `GET` | `/docs` | Interactive API documentation |

| `GET` | `/health` | Detailed health check information |



### WebSocket Endpoints



| Endpoint | Purpose |

|----------|---------|

| `/ws/call` | Twilio audio stream connection |

| `/ws/logs` | Frontend monitoring and logging |



## 🔧 Core Components



### Session Manager

- **Three-way Connection Management**: Coordinates Twilio ↔ OpenAI ↔ Frontend connections

- **Real-time Audio Forwarding**: Streams audio between services with minimal latency

- **State Management**: Maintains session state across all connections

- **Message Routing**: Intelligent message distribution and processing



### OpenAI Client

- **Realtime API Integration**: WebSocket connection to OpenAI's Realtime API

- **Audio Stream Processing**: Handles audio encoding/decoding (g711_ulaw format)

- **Function Call Support**: Executes custom functions and returns results

- **Auto-Reconnection**: Intelligent reconnection with exponential backoff



### WebSocket Connection Manager

- **Multi-type Connection Pools**: Manages different connection types efficiently

- **Connection Lifecycle Control**: Automatic connection cleanup and health monitoring

- **Message Broadcasting**: Supports both unicast and multicast messaging

- **Health Monitoring**: Continuous connection health assessment



### Function Handler System

- **Dynamic Registration**: Plugin-style function registration system

- **Schema Validation**: Type-safe parameter validation using Pydantic

- **Async Execution**: Non-blocking function execution with proper error handling

- **Extensible Design**: Easy to add new functions and capabilities



## 🔄 Auto-Reconnection Features



This implementation includes robust auto-reconnection capabilities not present in the original TypeScript version:



### Reconnection Strategy

- **Exponential Backoff**: Gradually increasing delays to avoid overwhelming servers

- **Configurable Parameters**: Customizable retry counts, delays, and timeouts

- **Intelligent Error Classification**: Different handling for different error types

- **State Recovery**: Automatic session configuration recovery after reconnection



### Reconnection Configuration

```python

# Configure reconnection parameters

openai_client.configure_reconnect(

    auto_reconnect=True,      # Enable automatic reconnection

    max_attempts=10,          # Maximum retry attempts

    initial_delay=2.0,        # Initial delay in seconds

    max_delay=60.0           # Maximum delay cap

)

```



### Connection Status Monitoring

Real-time status updates are sent to the frontend:

```json

{

  "type": "connection_status",

  "status": "openai_connected",

  "message": "OpenAI connection established"

}

```



## 🧪 Testing & Development



### Run Tests

```bash

# Comprehensive server testing

python test_server.py



```



### Development Tools

- **Interactive API Docs**: Visit `/docs` for Swagger UI

- **Health Monitoring**: Visit `/health` for system status

- **Real-time Logs**: Connect to `/ws/logs` for live monitoring

- **Function Testing**: Use `/tools` endpoint to verify available functions



### Testing a Call

1. Ensure server is running and accessible via ngrok

2. Call your configured Twilio phone number

3. After hearing "Connected", start speaking

4. Try asking about weather to test function calling

5. Monitor logs via WebSocket or console output



## 🔌 Extending Functionality



### Adding Custom Functions



Add new functions in `app/services/function_handlers.py`:



```python

# Register your function

self.register_function(

    name="your_custom_function",

    description="Description of what your function does",

    parameters={

        "type": "object",

        "properties": {

            "param1": {"type": "string", "description": "Parameter description"}

        },

        "required": ["param1"]

    },

    handler=self._your_custom_handler

)



# Implement the handler

async def _your_custom_handler(self, args: Dict[str, Any]) -> str:

    # Your custom logic here

    result = {"status": "success", "data": "your_result"}

    return json.dumps(result)

```



### Customizing OpenAI Configuration



Modify the `OpenAISessionConfig` in `app/models/schemas.py`:



```python

class OpenAISessionConfig(BaseModel):

    modalities: List[str] = ["text", "audio"]

    turn_detection: Dict[str, str] = {"type": "server_vad"}

    voice: str = "ash"  # Options: ash, ballad, coral, sage, verse

    temperature: float = 0.8

    max_response_output_tokens: int = 4096

```



## 🚀 Performance Features



### High Concurrency

- **Async I/O Processing**: Non-blocking operations throughout

- **WebSocket Connection Pooling**: Efficient connection management

- **Event-driven Architecture**: Reactive programming patterns

- **Stream Processing**: Real-time audio stream handling



### Memory Optimization

- **Automatic Connection Cleanup**: Prevents memory leaks

- **Streaming Audio Processing**: Minimal memory footprint

- **Garbage Collection Optimization**: Efficient resource management

- **Session Cleanup Service**: Periodic cleanup of expired sessions



### Network Optimization

- **WebSocket Long Connections**: Persistent, low-latency connections

- **Intelligent Reconnection**: Minimizes connection overhead

- **Connection Pool Management**: Efficient resource utilization

- **Compression Support**: Reduced bandwidth usage



## 🛡️ Production Considerations



### Security Features

- **Input Validation**: Comprehensive Pydantic data validation

- **Error Information Filtering**: Prevents sensitive data leakage

- **WebSocket Authentication**: Support for connection authentication

- **CORS Configuration**: Configurable cross-origin resource sharing



### Monitoring & Observability

- **Health Check Endpoints**: Comprehensive system health reporting

- **Structured Logging**: JSON-formatted logs for easy parsing

- **Performance Metrics**: Built-in performance monitoring

- **Error Tracking**: Detailed error collection and analysis



### Deployment Options

- **Docker Support**: Container-ready configuration

- **Cloud Platform Ready**: Compatible with major cloud providers

- **Horizontal Scaling**: Stateless design supports load balancing

- **Environment Configuration**: Flexible configuration management



## 🔧 Configuration Reference



### Environment Variables



| Variable | Required | Default | Description |

|----------|----------|---------|-------------|

| `OPENAI_API_KEY` | Yes | - | OpenAI API key for Realtime API access |

| `PUBLIC_URL` | Yes | - | Public URL for Twilio webhook callbacks |

| `PORT` | No | 8081 | Server port number |

| `LOG_LEVEL` | No | INFO | Logging level (DEBUG, INFO, WARNING, ERROR) |



### Advanced Configuration



For production deployments, consider these additional configurations:



```bash

# Production environment

ENVIRONMENT=production

LOG_FORMAT=json

CORS_ORIGINS=["https://yourdomain.com"]

MAX_CONNECTIONS=100

HEALTH_CHECK_INTERVAL=30

```



## 🐛 Troubleshooting



### Common Issues



#### 1. OpenAI Connection Failures

- Verify `OPENAI_API_KEY` is correct and has Realtime API access

- Check network connectivity and firewall settings

- Ensure sufficient OpenAI account credits

- Monitor logs for specific error messages



#### 2. Twilio Connection Issues

- Verify `PUBLIC_URL` is accessible from the internet

- Ensure ngrok or tunneling service is running

- Check Twilio webhook configuration

- Verify TwiML endpoint responds correctly



#### 3. Audio Quality Problems

- Check network latency and stability

- Verify audio format configuration (g711_ulaw)

- Monitor OpenAI Realtime API status

- Ensure proper WebSocket connection handling



#### 4. Function Call Failures

- Verify function registration in `function_handlers.py`

- Check function parameter validation

- Monitor function execution logs

- Ensure proper JSON response formatting



### Debug Mode



Enable debug logging for detailed troubleshooting:



```bash

export LOG_LEVEL=DEBUG

python run.py

```



## 📊 Performance Metrics



### System Capabilities

- **Concurrent Connections**: Supports 100+ simultaneous WebSocket connections

- **Audio Latency**: <200ms end-to-end audio processing

- **Memory Usage**: <50MB base memory footprint

- **CPU Efficiency**: Async processing minimizes CPU usage

- **Network Throughput**: Optimized for real-time audio streaming



### Comparison with Original TypeScript Version



| Feature | TypeScript Version | FastAPI Version |

|---------|-------------------|-----------------|

| Auto-Reconnection | ❌ Not available | ✅ Full support |

| Health Monitoring | ❌ Basic | ✅ Comprehensive |

| Error Recovery | ❌ Limited | ✅ Advanced |

| Memory Management | ❌ Manual | ✅ Automatic |

| Type Safety | ✅ TypeScript | ✅ Pydantic |

| API Documentation | ❌ Manual | ✅ Auto-generated |

| Testing Tools | ❌ Limited | ✅ Comprehensive |

| Production Ready | ⚠️ Basic | ✅ Enterprise-grade |



## 📚 Additional Resources



### Documentation

- [OpenAI Realtime API Documentation](https://platform.openai.com/docs/guides/realtime)

- [Twilio WebSocket Documentation](https://www.twilio.com/docs/voice/twiml/stream)

- [FastAPI Documentation](https://fastapi.tiangolo.com/)



### Community & Support

- Report issues on the project repository

- Contribute improvements via pull requests

- Join discussions in the community forums



## 📄 License



This project is a Python FastAPI reimplementation of the original TypeScript demo, maintaining the same open-source license terms.



---