Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/transitive-bullshit/openai-realtime-api
TypeScript client for OpenAI's realtime voice API.
https://github.com/transitive-bullshit/openai-realtime-api
openai realtime voice
Last synced: 3 months ago
JSON representation
TypeScript client for OpenAI's realtime voice API.
- Host: GitHub
- URL: https://github.com/transitive-bullshit/openai-realtime-api
- Owner: transitive-bullshit
- License: mit
- Created: 2024-10-15T04:50:21.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2024-10-21T21:15:47.000Z (3 months ago)
- Last Synced: 2024-10-22T08:06:53.488Z (3 months ago)
- Topics: openai, realtime, voice
- Language: TypeScript
- Homepage:
- Size: 1.14 MB
- Stars: 249
- Watchers: 2
- Forks: 11
- Open Issues: 1
-
Metadata Files:
- Readme: readme.md
- Funding: .github/funding.yml
- License: license
Awesome Lists containing this project
- awesome - transitive-bullshit/openai-realtime-api - TypeScript client for OpenAI's realtime voice API. (TypeScript)
README
# openai-realtime-api
> TypeScript client for OpenAI's realtime voice API.
- [Features](#features)
- [Install](#install)
- [Usage](#usage)
- [Server Usage](#server-usage)
- [Browser Usage](#browser-usage)
- [Relay Server](#relay-server)
- [Examples](#examples)
- [Node.js Basic](#nodejs-basic)
- [Node.js Audio](#nodejs-audio)
- [Node.js Conversation](#nodejs-conversation)
- [OpenAI Realtime Console](#openai-realtime-console)
- [TODO](#todo)
- [License](#license)
## Features
- **Strongly typed** TS fork of [openai/openai-realtime-api-beta](https://github.com/openai/openai-realtime-api-beta)
- [All events](./src/events.ts) and handlers are 100% typed
- **Drop-in replacement for OpenAI's JS version**
- Fixes many small bugs and inconsistencies
- ([#3](https://github.com/openai/openai-realtime-api-beta/issues/3), [#11](https://github.com/openai/openai-realtime-api-beta/pull/11), [#12](https://github.com/openai/openai-realtime-api-beta/pull/12), [#14](https://github.com/openai/openai-realtime-api-beta/issues/14), [#17](https://github.com/openai/openai-realtime-api-beta/pull/17), [#29](https://github.com/openai/openai-realtime-api-beta/pull/29), [#34](https://github.com/openai/openai-realtime-api-beta/pull/34), [#35](https://github.com/openai/openai-realtime-api-beta/pull/35), [#37](https://github.com/openai/openai-realtime-api-beta/pull/37), [#43](https://github.com/openai/openai-realtime-api-beta/pull/43), [#44](https://github.com/openai/openai-realtime-api-beta/pull/44), and likely others)
- Published to NPM
- Supports Node.js, browser, deno, bun, CF workers, etc
- Includes Node.js CLI examples for easy local testing
- Includes a simple relay server
- Includes the [OpenAI Realtime Console demo](#openai-realtime-console) using this package 🔥
## Install
```sh
npm install openai-realtime-api
```
This package is [ESM-only](https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c). It requires `Node.js >= 18`, a browser environment, or an equivalent JS runtime (Deno, Bun, CF workers, etc).
## Usage
> [!IMPORTANT]
> All usage and events are 100% compatible with the [OpenAI JS version](https://github.com/openai/openai-realtime-api-beta). The main difference aside from bug fixes is that **all events are fully-typed**.
```ts
import { RealtimeClient } from 'openai-realtime-api'
// Create a new client; all params are optional; apiKey defaults to the
// `OPENAI_API_KEY` environment variable (when using Node.js).
const client = new RealtimeClient({
sessionConfig: {
instructions: 'You are a great, upbeat friend.',
voice: 'alloy'
}
})
// Can change session config ahead of connecting.
client.updateSession({
turn_detection: null,
input_audio_transcription: { model: 'whisper-1' }
})
// Example of custom event handling
client.on('conversation.updated', (event) => {
// All events are fully-typed based on the event name.
// In this case, `event` will have the type `RealtimeCustomEvents.ConversationUpdatedEvent`
const { item, delta } = event
// Access the full list of conversation items.
const items = client.conversation.getItems()
})
// Connect to the Realtime API.
await client.connect()
// Send a text message and trigger a response generation.
client.sendUserMessageContent([{ type: 'input_text', text: 'How are you?' }])
// Wait for a completed response from the model.
// (`event` will be of type `RealtimeServerEvents.ResponseDoneEvent`)
const event = await client.realtime.waitForNext('response.done')
```
See [examples](#examples) for more complete demos.
See also the official [OpenAI Realtime API Guide](https://platform.openai.com/docs/guides/realtime) and [API Reference](https://platform.openai.com/docs/api-reference/realtime).
For more info on usage, tools, and custom events, see [OpenAI's readme](https://github.com/openai/openai-realtime-api-beta). Note that this package is 100% compatible with OpenAI's beta package in terms of both official and unofficial events. The only difference is that all events are typed.
### Server Usage
`RealtimeClient` takes in an optional `apiKey` which defaults to `process.env.OPENAI_API_KEY`.
### Browser Usage
`RealtimeClient` takes in an optional `url` which can be pointed at a relay server.
```ts
import { RealtimeClient } from 'openai-realtime-api'
// Create a browser client which points to a relay server.
const client = new RealtimeClient({ url: RELAY_SERVER_URL })
```
Alternatively, you can use `apiKey` with `RealtimeClient` in the browser, but you also have to pass `dangerouslyAllowAPIKeyInBrowser: true`.
```ts
import { RealtimeClient } from 'openai-realtime-api'
// Create a browser client which connects directly to the OpenAI realtime API
// with an unsafe, client-side API key.
const client = new RealtimeClient({
apiKey: process.env.OPENAI_API_KEY,
dangerouslyAllowAPIKeyInBrowser: true
})
```
> [!CAUTION]
> We strongly recommend against including your API key in any client (mobile or browser). It can be useful for local testing, but for production, you should be using a relay server.
### Relay Server
```ts
import { RealtimeClient } from 'openai-realtime-api'
import { RealtimeRelay } from 'openai-realtime-api/node'
// Setting `relay: true` disables tool calls and directly modifying the session,
// since that will be the responsibility of the upstream client.
const client = new RealtimeClient({ relay: true })
const relay = new RealtimeRelay({ client })
relay.listen(8081)
```
Note that `RealtimeRelay` uses a different import path because it contains Node.js-specific code.
A full example is included in [examples/node/relay-server.ts](./examples/node/relay-server.ts).
## Examples
To run the included examples (requires `Node.js >= 18`):
1. Clone this repo
2. Run `pnpm install`
3. Setup `.env` with your `OPENAI_API_KEY`
You can set `debug: true` in the `RealtimeClient` constructor of these examples to print out the full event log.
### Node.js Basic
Simple Node.js demo using the `RealtimeClient` which sends a text message and waits for a complete response.
- [examples/node/basic.ts](./examples/node/basic.ts)
- Run `npx tsx examples/node/basic.ts`
### Node.js Audio
Simple Node.js demo using the `RealtimeClient` which sends a short audio message and waits for a complete response.
- [examples/node/audio.ts](./examples/node/audio.ts)
- Run `npx tsx examples/node/audio.ts`
### Node.js Conversation
Simple Node.js demo using the `RealtimeClient` with a microphone and speaker to simulate a full, back & forth conversation from the terminal.
- [examples/node/convo.ts](./examples/node/convo.ts)
- This demo uses the [mic](https://github.com/ashishbajaj99/mic) and [speaker](https://github.com/TooTallNate/node-speaker) npm packages
- `mic` requires [sox](https://sourceforge.net/projects/sox/); on macOS, you can run `brew install sox`
- `npx tsx examples/node/convo.ts`
### OpenAI Realtime Console
This example has been imported from https://github.com/openai/openai-realtime-console ([at commit 6ea4dba](https://github.com/openai/openai-realtime-console/tree/6ea4dba795fee868c60ea9e8e7eba7469974b3e9)). The only change has been to replace `@openai/realtime-api-beta` with `openai-realtime-api` and to fix a few types.
To run the realtime console example:
```sh
pnpm install
cd examples/openai-realtime-console
pnpm start
```
## TODO
- add an example using tools
- add an example next.js app
- improve readme docs
## License
MIT © [Travis Fischer](https://x.com/transitive_bs)
If you found this project interesting, [consider following me on Twitter](https://x.com/transitive_bs).