{"id":24482103,"url":"https://github.com/mostafa-drz/openai-realtime-webrtc","last_synced_at":"2026-02-21T03:01:47.226Z","repository":{"id":273462620,"uuid":"919796603","full_name":"mostafa-drz/openai-realtime-webrtc","owner":"mostafa-drz","description":"A sample demo showcasing how to integrate WebRTC with the OpenAI Realtime API for real-time audio and text interactions. Built with Next.js and TypeScript, this project serves as a simple starting point for developers exploring OpenAI’s Realtime capabilities.","archived":false,"fork":false,"pushed_at":"2025-07-07T15:20:23.000Z","size":657,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-07T15:45:46.296Z","etag":null,"topics":["generative-ai","nextjs","openai-api","openai-api-chatbot","webrtc"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mostafa-drz.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-21T03:06:20.000Z","updated_at":"2025-07-07T15:19:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"6274bbdd-4441-45e9-bc53-f05a88d4c569","html_url":"https://github.com/mostafa-drz/openai-realtime-webrtc","commit_stats":null,"previous_names":["mostafa-drz/openai-realtime-webrtc"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/mostafa-drz/openai-realtime-webrtc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa-drz%2Fopenai-realtime-webrtc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa-drz%2Fopenai-realtime-webrtc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa-drz%2Fopenai-realtime-webrtc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa-drz%2Fopenai-realtime-webrtc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mostafa-drz","download_url":"https://codeload.github.com/mostafa-drz/openai-realtime-webrtc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa-drz%2Fopenai-realtime-webrtc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29672253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T00:11:43.526Z","status":"online","status_checked_at":"2026-02-21T02:00:07.432Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generative-ai","nextjs","openai-api","openai-api-chatbot","webrtc"],"created_at":"2025-01-21T12:12:26.654Z","updated_at":"2026-02-21T03:01:47.219Z","avatar_url":"https://github.com/mostafa-drz.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## OpenAI Realtime WebRTC Integration (Next.js)\n\nThis project provides a reusable, minimal boilerplate to integrate the OpenAI Realtime API with WebRTC using TypeScript and Next.js. It handles session creation, audio + data channel transport, and exposes a clean abstraction for frontend developers.\n\n### 🔧 Features\n\n- Ephemeral token authentication (client-secret)\n- WebRTC connection management (audio + data)\n- Voice input and output stream support\n- Custom `RealtimeClient` class with high-level abstraction\n- `RealtimeClient` class with comprehensive event system\n- **Comprehensive Event System** - Raw event access with high-level APIs\n- **NEW: Speaker-Specific Transcript Callbacks** - Separate handling for user vs assistant transcripts\n- **NEW: Chat-like Interface** - Live streaming and final transcript display\n- **NEW: Enhanced Demo App** - Full-featured UI showcasing all capabilities with real-time transcript visualization\n- **Environment Configuration** - Fully configurable via environment variables\n- Future-ready structure for publishing as an npm package\n\n### 🚀 Quick Start\n\n\u003e **💡 Don't have Node.js installed?** Use Docker instead:\n\u003e\n\u003e ```bash\n\u003e git clone https://github.com/mostafa-drz/openai-realtime-webrtc.git\n\u003e cd openai-realtime-webrtc\n\u003e cp env.example .env.local\n\u003e # Edit .env.local with your OpenAI API key\n\u003e docker-compose up --build\n\u003e ```\n\u003e\n\u003e **Or with Docker directly:**\n\u003e\n\u003e ```bash\n\u003e docker build -t app .\n\u003e docker run -p 3000:3000 --env-file .env.local app\n\u003e ```\n\n#### **Prerequisites**\n\n- Node.js 18+ and npm\n- OpenAI API key with Realtime API access\n- Modern browser with WebRTC support\n\n#### **Installation**\n\n```bash\n# Clone the repository\ngit clone https://github.com/mostafa-drz/openai-realtime-webrtc.git\ncd openai-realtime-webrtc\n\n# Install dependencies\nnpm install\n\n# Set up environment variables\ncp env.example .env.local\n```\n\n#### **Environment Configuration**\n\nAdd your OpenAI configuration to `.env.local`:\n\n```env\n# OpenAI API Configuration\nOPENAI_API_KEY=your_openai_api_key_here\nOPENAI_API_BASE_URL=https://api.openai.com/v1\n\n# Realtime API Endpoints\nOPENAI_REALTIME_SESSION_URL=https://api.openai.com/v1/realtime/sessions\nNEXT_PUBLIC_OPENAI_REALTIME_WEBRTC_URL=https://api.openai.com/v1/realtime\n\n# Model Configuration\nNEXT_PUBLIC_OPENAI_MODEL=gpt-4o-realtime-preview-2024-12-17\n```\n\n**Environment Variables Explained:**\n\n- **`OPENAI_API_KEY`** - Your OpenAI API key for session creation\n- **`OPENAI_API_BASE_URL`** - Base URL for all OpenAI API calls (optional, defaults to production)\n- **`OPENAI_REALTIME_SESSION_URL`** - Session creation endpoint (server-side)\n- **`NEXT_PUBLIC_OPENAI_REALTIME_WEBRTC_URL`** - WebRTC connection endpoint (client-side)\n- **`NEXT_PUBLIC_OPENAI_MODEL`** - Default model for the demo application\n\n**Note:** Variables prefixed with `NEXT_PUBLIC_` are exposed to the client-side code.\n\n#### **Running the Demo**\n\n```bash\n# Start the development server\nnpm run dev\n\n# Open http://localhost:3000\n```\n\n### 📸 Demo Preview\n\n![OpenAI Realtime WebRTC Demo](/public/screenshots/main.png)\n\n**[🎥 Watch Demo Video](https://youtu.be/6egwG8aExas)** - See the demo in action with real-time voice conversations and speaker-specific transcript callbacks.\n\n#### **Docker Development (Recommended for Quick Start)**\n\n```bash\n# Using Docker Compose (with hot reloading)\ndocker-compose up --build\n\n# Or using Docker directly\ndocker build -t app .\ndocker run -p 3000:3000 --env-file .env.local app\n\n# Open http://localhost:3000\n```\n\n**Docker Benefits:**\n\n- ✅ No Node.js installation required\n- ✅ Hot reloading with code changes\n- ✅ Consistent environment across machines\n- ✅ Quick setup for experimentation\n\n### 🧠 Architecture\n\n#### **OpenAI Realtime API Flow**\n\nThe OpenAI Realtime API operates in two distinct phases:\n\n**Phase 1: Session Creation**\n\n```\nClient → Server Action → OpenAI API\nPOST /api/realtime/sessions\n{\n  \"model\": \"gpt-4o-realtime-preview-2024-12-17\",\n  \"voice\": \"echo\",\n  \"temperature\": 0.8,\n  \"instructions\": \"You are a helpful assistant\"\n}\n```\n\n**Response:**\n\n```json\n{\n  \"id\": \"session_abc123\",\n  \"client_secret\": {\n    \"value\": \"rt_xyz789...\",\n    \"expires_at\": 1234567890\n  },\n  \"model\": \"gpt-4o-realtime-preview-2024-12-17\",\n  \"voice\": \"echo\"\n}\n```\n\n**Phase 2: WebRTC Connection**\n\n```\nClient → OpenAI WebRTC Endpoint\nPOST /api/realtime?model=gpt-4o-realtime-preview-2024-12-17\nHeaders: {\n  \"Authorization\": \"Bearer rt_xyz789...\",\n  \"Content-Type\": \"application/sdp\"\n}\nBody: SDP Offer\n```\n\n**Response:**\n\n```\nSDP Answer (text/plain)\n```\n\n#### **Event-Based Communication Architecture**\n\nOnce connected, the system uses a comprehensive event system for real-time communication:\n\n**Client Events** (Browser → OpenAI):\n\n- `session.update` - Update session configuration\n- `input_audio_buffer.append/commit/clear` - Audio buffer management\n- `conversation.item.create` - Add conversation messages\n- `response.create/cancel` - Request/cancel AI responses\n- `output_audio_buffer.clear` - Clear output audio\n\n**Server Events** (OpenAI → Browser):\n\n- `session.created/updated` - Session state changes\n- `input_audio_buffer.speech_started/stopped` - Voice activity detection\n- `conversation.item.input_audio_transcription.delta/completed` - User speech-to-text results\n- `response.audio_transcript.delta/done` - Assistant speech-to-text results\n- `response.content_part.added` - AI response streaming\n- `output_audio_buffer.started/stopped` - Audio playback control\n- `error` - Error notifications\n\n#### **Our Implementation Architecture**\n\n```\n┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐\n│   React Demo    │    │  Server Actions  │    │  OpenAI API     │\n│                 │    │                  │    │                 │\n│ ┌─────────────┐ │    │ ┌──────────────┐ │    │ ┌─────────────┐ │\n│ │Realtime     │ │    │ │createSession │ │    │ │Session      │ │\n│ │Client       │ │    │ │Server Action │ │    │ │Creation     │ │\n│ └─────────────┘ │    │ └──────────────┘ │    │ └─────────────┘ │\n│         │       │    │         │        │    │         │       │\n│         ▼       │    │         ▼        │    │         ▼       │\n│ ┌─────────────┐ │    │ ┌──────────────┐ │    │ ┌─────────────┐ │\n│ │WebRTC       │ │    │ │Server Module │ │    │ │WebRTC       │ │\n│ │Connection   │ │    │ │(API Wrapper) │ │    │ │Connection   │ │\n│ └─────────────┘ │    │ └──────────────┘ │    │ └─────────────┘ │\n└─────────────────┘    └──────────────────┘    └─────────────────┘\n```\n\n#### **Abstraction Layer Design**\n\nOur abstraction layer provides multiple levels of API access:\n\n**High-Level APIs** (Recommended for most use cases):\n\n```typescript\n// Simple conversation flow\nconst client = new RealtimeClient(config);\nawait client.sendTextMessage('Hello, how are you?');\nawait client.requestResponse();\nconst items = client.getConversationItems();\nconst responding = client.isResponding();\n```\n\n**Event Callbacks** (For reactive UI updates):\n\n```typescript\nconst client = new RealtimeClient({\n  onConversationItemCreated: (item) =\u003e console.log('New message:', item),\n  onResponseCreated: (response) =\u003e console.log('AI started responding'),\n  onSpeechStarted: () =\u003e console.log('User started speaking'),\n  onSpeechStopped: () =\u003e console.log('User stopped speaking'),\n\n  // NEW: Speaker-specific transcript callbacks\n  onUserTranscriptDelta: (text) =\u003e console.log('User speaking:', text),\n  onUserTranscriptDone: (text) =\u003e console.log('User finished:', text),\n  onAssistantTranscriptDelta: (text) =\u003e\n    console.log('Assistant speaking:', text),\n  onAssistantTranscriptDone: (text) =\u003e console.log('Assistant finished:', text),\n  onTranscriptionError: (error) =\u003e\n    console.error('Transcription failed:', error),\n});\n```\n\n**Raw Event Access** (For advanced use cases):\n\n```typescript\nconst client = new RealtimeClient({\n  onRawEvent: (event) =\u003e {\n    // Handle any server event with full type safety\n    switch (event.type) {\n      case 'response.content_part.added':\n        // Custom handling\n        break;\n    }\n  },\n});\n```\n\n#### **Key Components**\n\n**1. Server Actions (`src/lib/actions.ts`)**\n\n- Next.js 15 server actions for session creation\n- Calls OpenAI API with session configuration\n- Returns client secret for WebRTC authentication\n- Uses `OPENAI_REALTIME_SESSION_URL` environment variable\n\n**2. RealtimeClient (`src/lib/openai-realtime/client/RealtimeClient.ts`)**\n\n- Manages WebRTC peer connection\n- Handles SDP offer/answer exchange\n- Manages data channel for events\n- Processes audio streams\n- **High-level methods**: `sendTextMessage()`, `requestResponse()`, `cancelResponse()`\n- **State management**: Conversation tracking, response status\n- **Event abstraction**: Converts raw events to callbacks\n\n**3. Type System (`src/lib/openai-realtime/types/`)**\n\n- Complete TypeScript definitions\n- Session configuration types\n- Event type definitions (`client-events.ts`, `server-events.ts`)\n- API response types\n\n#### **Data Flow**\n\n```\n1. User clicks \"Start New Session\"\n   ↓\n2. Server Action creates session via OpenAI API\n   ↓\n3. Client secret returned to frontend\n   ↓\n4. RealtimeClient establishes WebRTC connection\n   ↓\n5. SDP handshake with OpenAI WebRTC endpoint\n   ↓\n6. Data channel opens for event communication\n   ↓\n7. Audio streams established (input/output)\n   ↓\n8. Real-time conversation begins\n   ↓\n9. Events flow through abstraction layer\n   ↓\n10. UI updates reactively via callbacks\n```\n\n### 🗣️ Turn Detection: Server VAD vs Semantic VAD\n\nThe OpenAI Realtime API supports two advanced turn detection modes for voice interaction:\n\n#### **Server VAD (Voice Activity Detection)**\n\n- Detects speech start/end based on audio volume.\n- Responds at the end of user speech.\n- **Fast**, but may cut off if you pause.\n\n#### **Semantic VAD**\n\n- Uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking.\n- Dynamically sets a timeout based on this probability.\n- For example, if user audio trails off with \"uhhm\", the model will score a low probability of turn end and wait longer for the user to continue speaking.\n- **More natural conversations**, but may have higher latency.\n\n\u003e **From OpenAI docs:**\n\u003e\n\u003e \"Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with 'uhhm', the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.\"\n\n#### **Manual Mode**\n\n- You can also set turn detection to `null` to disable automatic detection. In this case, the client must manually trigger model response (e.g., push-to-talk or commit audio buffer).\n\n#### **Demo Session Types**\n\n- **Live Chat (VAD):** Uses Server VAD for fast, automatic turn detection.\n- **Natural Chat (Semantic):** Uses Semantic VAD for more natural, conversational flow.\n- **Manual Chat/Transcription:** Uses manual commit/clear for full control.\n\nSee the navigation bar in the demo for hands-on examples of each mode.\n\n### 📦 Usage\n\n#### **Basic Setup**\n\n```typescript\nimport { RealtimeClient } from '@/lib/openai-realtime/client/RealtimeClient';\n\nfunction MyComponent() {\n  const client = new RealtimeClient({\n    clientSecret: 'your-client-secret',\n    realtimeUrl: process.env.NEXT_PUBLIC_OPENAI_REALTIME_WEBRTC_URL,\n    onMessageToken: (token) =\u003e console.log('AI:', token),\n\n    // NEW: Speaker-specific transcript handling\n    onUserTranscriptDelta: (text) =\u003e console.log('You said:', text),\n    onAssistantTranscriptDelta: (text) =\u003e console.log('AI said:', text),\n    onTranscriptionError: (error) =\u003e console.error('Transcription error:', error),\n  });\n\n  const startConversation = async () =\u003e {\n    await client.connect();\n    await client.sendTextMessage(\"Hello!\");\n    await client.requestResponse();\n  };\n\n  return (\n    \u003cdiv\u003e\n      \u003cbutton onClick={startConversation}\u003e\n        Start Conversation\n      \u003c/button\u003e\n    \u003c/div\u003e\n  );\n}\n```\n\n#### **Enhanced Transcript Features**\n\nThe latest version includes speaker-specific transcript callbacks for precise control over user and assistant speech:\n\n```typescript\nconst client = new RealtimeClient({\n  clientSecret: 'your-secret',\n  realtimeUrl: process.env.NEXT_PUBLIC_OPENAI_REALTIME_WEBRTC_URL,\n\n  // User transcript callbacks\n  onUserTranscriptDelta: (text) =\u003e {\n    // Live user speech as they speak\n    console.log('User speaking:', text);\n    updateUserSubtitle(text);\n  },\n  onUserTranscriptDone: (text) =\u003e {\n    // Final user transcript when they finish speaking\n    console.log('User finished:', text);\n    saveUserTranscript(text);\n  },\n\n  // Assistant transcript callbacks\n  onAssistantTranscriptDelta: (text) =\u003e {\n    // Live assistant speech as AI responds\n    console.log('Assistant speaking:', text);\n    updateAssistantSubtitle(text);\n  },\n  onAssistantTranscriptDone: (text) =\u003e {\n    // Final assistant transcript when AI finishes\n    console.log('Assistant finished:', text);\n    saveAssistantTranscript(text);\n  },\n\n  // Error handling\n  onTranscriptionError: (error) =\u003e {\n    console.error('Transcription failed:', error.message);\n    showTranscriptionError(error.message);\n  },\n});\n```\n\n**Chat-like Interface Features:**\n\n- **Live Streaming**: Real-time transcript display as users speak\n- **Final Transcripts**: Completed messages moved to conversation history\n- **Speaker Distinction**: Clear visual separation between user and assistant\n- **Error Handling**: Inline error display with recovery options\n- **Auto-scroll**: Automatic scrolling to keep latest messages visible\n- **Session Types**: Works for both regular chat and transcription-only sessions\n\n#### **Advanced Usage with Event Callbacks**\n\n```typescript\nconst client = new RealtimeClient({\n  clientSecret: 'your-secret',\n  realtimeUrl: process.env.NEXT_PUBLIC_OPENAI_REALTIME_WEBRTC_URL,\n\n  // High-level callbacks\n  onConversationItemCreated: (item) =\u003e {\n    console.log('New conversation item:', item);\n  },\n  onResponseCreated: (response) =\u003e {\n    console.log('AI started responding:', response.id);\n  },\n  onResponseDone: (response) =\u003e {\n    console.log('AI finished responding:', response.id);\n  },\n\n  // Speech detection\n  onSpeechStarted: () =\u003e {\n    console.log('User started speaking');\n  },\n  onSpeechStopped: () =\u003e {\n    console.log('User stopped speaking');\n  },\n\n  // Raw event access (for advanced use cases)\n  onRawEvent: (event) =\u003e {\n    console.log('Raw event:', event.type, event);\n  },\n});\n\n// Connect and start conversation\nawait client.connect();\nawait client.sendTextMessage('Hello!');\nawait client.requestResponse();\n```\n\n#### **State Management (Consumer Responsibility)**\n\nThe RealtimeClient no longer manages conversation state internally. Consumers are responsible for managing their own state:\n\n```typescript\n// Example: Managing conversation state in your application\nconst [conversationItems, setConversationItems] = useState([]);\nconst [isResponding, setIsResponding] = useState(false);\nconst [isSpeaking, setIsSpeaking] = useState(false);\nconst [hasAudioBuffer, setHasAudioBuffer] = useState(false);\n\nconst client = new RealtimeClient({\n  // ... config\n  onResponseCreated: () =\u003e setIsResponding(true),\n  onResponseDone: () =\u003e setIsResponding(false),\n  onSpeechStarted: () =\u003e setIsSpeaking(true),\n  onSpeechStopped: () =\u003e setIsSpeaking(false),\n  onConversationItemCreated: (item) =\u003e {\n    setConversationItems((prev) =\u003e [...prev, item]);\n  },\n});\n```\n\n#### **Migration Guide**\n\nIf you're upgrading from a previous version that had built-in state management:\n\n**Before (Old API):**\n\n```typescript\n// These methods no longer exist\nif (client.isResponding()) { ... }\nif (client.isSpeaking()) { ... }\nif (client.hasAudioBuffer()) { ... }\nconst items = client.getConversationItems(); // Never existed but implied\n```\n\n**After (New API):**\n\n```typescript\n// Manage state in your application\nconst [isResponding, setIsResponding] = useState(false);\nconst [isSpeaking, setIsSpeaking] = useState(false);\nconst [hasAudioBuffer, setHasAudioBuffer] = useState(false);\nconst [conversationItems, setConversationItems] = useState([]);\n\n// Use callbacks to update state\nconst client = new RealtimeClient({\n  // ... config\n  onResponseCreated: () =\u003e setIsResponding(true),\n  onResponseDone: () =\u003e setIsResponding(false),\n  onSpeechStarted: () =\u003e setIsSpeaking(true),\n  onSpeechStopped: () =\u003e setIsSpeaking(false),\n  onConversationItemCreated: (item) =\u003e {\n    setConversationItems((prev) =\u003e [...prev, item]);\n  },\n});\n```\n\n#### **Enhanced Audio and Conversation Management**\n\n```typescript\nconst client = new RealtimeClient({\n  clientSecret: 'your-secret',\n  realtimeUrl: process.env.NEXT_PUBLIC_OPENAI_REALTIME_WEBRTC_URL,\n});\n\n// Audio buffer management\nawait client.appendAudioData(base64AudioData);\nawait client.commitAudioBuffer();\nawait client.clearAudioBuffer();\n\n// Conversation management\nawait client.retrieveConversationItem('item_123');\nawait client.truncateConversationItem(5000); // Truncate at 5 seconds\nawait client.deleteConversationItem();\n\n// Enhanced response management\nawait client.cancelSpecificResponse('response_456', 'User interrupted');\n\n// Check states\nif (client.isSpeaking()) {\n  console.log('User is currently speaking');\n}\nif (client.hasAudioBuffer()) {\n  console.log('Audio buffer has data');\n}\n```\n\n#### **API Reference**\n\n| Method                                 | Description                                          | Parameters                                    | Use Case                                                                      |\n| -------------------------------------- | ---------------------------------------------------- | --------------------------------------------- | ----------------------------------------------------------------------------- |\n| `constructor(config)`                  | Creates a new RealtimeClient instance                | `config: RealtimeClientConfig`                | Initialize the client with authentication and event handlers                  |\n| `connect()`                            | Establishes WebRTC connection to OpenAI Realtime API | None                                          | Start a new session and connect to the API                                    |\n| `updateSession(config)`                | Sends session update event to server                 | `config: Partial\u003cSessionConfig\u003e`              | Update session parameters during active session                               |\n| `updateTranscriptionSession(config)`   | Sends transcription session update event             | `config: Partial\u003cTranscriptionSessionConfig\u003e` | Update transcription parameters during active session                         |\n| `disconnect()`                         | Closes WebRTC connection and cleans up resources     | None                                          | Properly end session and free system resources                                |\n| `isConnected()`                        | Returns connection status                            | None                                          | Check if client is connected before making API calls                          |\n| `getSessionId()`                       | Returns current session ID                           | None                                          | Get session identifier for logging or debugging                               |\n| `getSessionType()`                     | Returns session type (regular/transcription)         | None                                          | Determine session capabilities and behavior                                   |\n| `getConnectionState()`                 | Returns current connection state                     | None                                          | Get detailed connection status for UI feedback                                |\n| `sendTextMessage(text, role?)`         | Sends text message to conversation                   | `text: string`, `role?: MessageRole`          | Add text messages in regular sessions                                         |\n| `requestResponse(options?)`            | Requests AI response from conversation               | `options?: Partial\u003cResponseConfig\u003e`           | Trigger AI response after adding messages                                     |\n| `cancelResponse(reason?)`              | Cancels current AI response                          | `reason?: string`                             | Stop ongoing AI response generation                                           |\n| `commitAudioBuffer()`                  | Commits current audio buffer to conversation         | None                                          | **Manual audio control**: Push-to-talk, walkie-talkie, manual transcription   |\n| `clearAudioBuffer()`                   | Clears current audio buffer without committing       | None                                          | **Manual audio control**: Cancel recording, re-record, error recovery         |\n| `clearOutputAudioBuffer()`             | Clears output audio buffer                           | None                                          | Stop currently playing AI audio output                                        |\n| `appendAudioData(audioBase64)`         | Appends audio data to buffer                         | `audioBase64: string`                         | **Manual audio control**: Build up audio buffer before committing             |\n| `retrieveConversationItem(itemId)`     | Retrieves specific conversation item                 | `itemId: string`                              | Load messages from history, implement search                                  |\n| `truncateConversationItem(audioEndMs)` | Truncates assistant message audio at timestamp       | `audioEndMs: number`                          | **User interruption**: Truncate unplayed AI audio, sync playback with context |\n| `deleteConversationItem()`             | Deletes any item from conversation history           | None                                          | Remove unwanted messages, clean up history, privacy control                   |\n\n#### **Transcript Callback Interface**\n\nThe latest version provides speaker-specific transcript callbacks for precise control:\n\n| Callback                     | Description                         | Parameters     | Use Case                                 |\n| ---------------------------- | ----------------------------------- | -------------- | ---------------------------------------- |\n| `onUserTranscriptDelta`      | Live user speech transcription      | `text: string` | Show real-time user speech as they speak |\n| `onUserTranscriptDone`       | Final user transcript               | `text: string` | Save completed user message to history   |\n| `onAssistantTranscriptDelta` | Live assistant speech transcription | `text: string` | Show real-time AI speech as it responds  |\n| `onAssistantTranscriptDone`  | Final assistant transcript          | `text: string` | Save completed AI response to history    |\n| `onTranscriptionError`       | Transcription error handling        | `error: Error` | Handle transcription failures gracefully |\n\n**Note:** Methods marked with **Manual audio control** are essential for push-to-talk, walkie-talkie, or manual transcription interfaces where you want to control when audio is sent to the AI, as opposed to automatic streaming used in the current demo.\n\n**State Management:** The client no longer provides state checking methods like `isResponding()`, `isSpeaking()`, or `hasAudioBuffer()`. Consumers must manage these states using the provided event callbacks.\n\n### 🎯 Demo Application\n\n**Try the live demo!** This project includes a comprehensive demo application that showcases:\n\n- **Real-time Voice Conversations**: Natural voice interaction with AI\n- **Chat-like Interface**: Live streaming and final transcript display\n- **Speaker-Specific Transcripts**: Separate handling for user vs assistant speech\n- **Session Configuration**: Voice selection, temperature, speed controls\n- **Event Logging**: Comprehensive debugging and monitoring\n- **Responsive UI**: Modern, accessible interface with auto-scroll\n- **Error Handling**: Inline transcription error display\n- **Developer Tools**: API examples and integration patterns\n\n**Key Demo Features:**\n\n- **Live Transcript Streaming**: See your speech transcribed in real-time as you speak\n- **Final Transcript History**: Completed messages saved to conversation history\n- **Assistant Response Visualization**: Watch AI responses stream in real-time\n- **Error Recovery**: Graceful handling of transcription failures\n- **Session Type Support**: Both regular chat and transcription-only modes\n- **Event Timeline**: Complete event logging for debugging\n\n**Quick Start:**\n\n```bash\nnpm install\nnpm run dev\n# Open http://localhost:3000\n```\n\nSee [DEMO_README.md](./DEMO_README.md) for detailed demo documentation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmostafa-drz%2Fopenai-realtime-webrtc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmostafa-drz%2Fopenai-realtime-webrtc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmostafa-drz%2Fopenai-realtime-webrtc/lists"}