{"id":22820516,"url":"https://github.com/codenamev/live-research","last_synced_at":"2026-02-27T06:03:00.425Z","repository":{"id":264993351,"uuid":"886684907","full_name":"codenamev/live-research","owner":"codenamev","description":"A real-time research assistant","archived":false,"fork":false,"pushed_at":"2025-02-25T14:44:36.000Z","size":1953,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-11T14:40:52.812Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codenamev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-11T12:23:40.000Z","updated_at":"2024-11-11T13:02:09.000Z","dependencies_parsed_at":"2025-05-05T20:08:56.288Z","dependency_job_id":"cd6bd4a5-1f28-4009-bcfc-92802b744bd1","html_url":"https://github.com/codenamev/live-research","commit_stats":null,"previous_names":["codenamev/live-research"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/codenamev/live-research","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codenamev%2Flive-research","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codenamev%2Flive-research/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codenamev%2Flive-research/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codenamev%2Flive-research/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codenamev","download_url":"https://codeload.github.com/codenamev/live-research/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codenamev%2Flive-research/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29887007,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-27T05:38:26.446Z","status":"ssl_error","status_checked_at":"2026-02-27T05:38:25.235Z","response_time":57,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-12T15:24:54.113Z","updated_at":"2026-02-27T06:03:00.390Z","avatar_url":"https://github.com/codenamev.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenAI Realtime Console\n\nThe OpenAI Realtime Console is intended as an inspector and interactive API reference\nfor the OpenAI Realtime API. It comes packaged with two utility libraries,\n[openai/openai-realtime-api-beta](https://github.com/openai/openai-realtime-api-beta)\nthat acts as a **Reference Client** (for browser and Node.js) and\n[`/src/lib/wavtools`](./src/lib/wavtools) which allows for simple audio\nmanagement in the browser.\n\n\u003cimg src=\"/readme/realtime-console-demo.png\" width=\"800\" /\u003e\n\n# Starting the console\n\nThis is a React project created using `create-react-app` that is bundled via Webpack.\nInstall it by extracting the contents of this package and using;\n\n```shell\n$ npm i\n```\n\nStart your server with:\n\n```shell\n$ npm start\n```\n\nIt should be available via `localhost:3000`.\n\n# Table of contents\n\n1. [Using the console](#using-the-console)\n   1. [Using a relay server](#using-a-relay-server)\n1. [Realtime API reference client](#realtime-api-reference-client)\n   1. [Sending streaming audio](#sending-streaming-audio)\n   1. [Adding and using tools](#adding-and-using-tools)\n   1. [Interrupting the model](#interrupting-the-model)\n   1. [Reference client events](#reference-client-events)\n1. [Wavtools](#wavtools)\n   1. [WavRecorder quickstart](#wavrecorder-quickstart)\n   1. [WavStreamPlayer quickstart](#wavstreamplayer-quickstart)\n1. [Acknowledgements and contact](#acknowledgements-and-contact)\n\n# Using the console\n\nThe console requires an OpenAI API key (**user key** or **project key**) that has access to the\nRealtime API. You'll be prompted on startup to enter it. It will be saved via `localStorage` and can be\nchanged at any time from the UI.\n\nTo start a session you'll need to **connect**. This will require microphone access.\nYou can then choose between **manual** (Push-to-talk) and **vad** (Voice Activity Detection)\nconversation modes, and switch between them at any time.\n\nThere are two functions enabled;\n\n- `get_weather`: Ask for the weather anywhere and the model will do its best to pinpoint the\n  location, show it on a map, and get the weather for that location. Note that it doesn't\n  have location access, and coordinates are \"guessed\" from the model's training data so\n  accuracy might not be perfect.\n- `set_memory`: You can ask the model to remember information for you, and it will store it in\n  a JSON blob on the left.\n\nYou can freely interrupt the model at any time in push-to-talk or VAD mode.\n\n## Using a relay server\n\nIf you would like to build a more robust implementation and play around with the reference\nclient using your own server, we have included a Node.js [Relay Server](/relay-server/index.js).\n\n```shell\n$ npm run relay\n```\n\nIt will start automatically on `localhost:8081`.\n\n**You will need to create a `.env` file** with the following configuration:\n\n```conf\nOPENAI_API_KEY=YOUR_API_KEY\nREACT_APP_LOCAL_RELAY_SERVER_URL=http://localhost:8081\n```\n\nYou will need to restart both your React app and relay server for the `.env.` changes\nto take effect. The local server URL is loaded via [`ConsolePage.tsx`](/src/pages/ConsolePage.tsx).\nTo stop using the relay server at any time, simply delete the environment\nvariable or set it to empty string.\n\n```javascript\n/**\n * Running a local relay server will allow you to hide your API key\n * and run custom logic on the server\n *\n * Set the local relay server address to:\n * REACT_APP_LOCAL_RELAY_SERVER_URL=http://localhost:8081\n *\n * This will also require you to set OPENAI_API_KEY= in a `.env` file\n * You can run it with `npm run relay`, in parallel with `npm start`\n */\nconst LOCAL_RELAY_SERVER_URL: string =\n  process.env.REACT_APP_LOCAL_RELAY_SERVER_URL || '';\n```\n\nThis server is **only a simple message relay**, but it can be extended to:\n\n- Hide API credentials if you would like to ship an app to play with online\n- Handle certain calls you would like to keep secret (e.g. `instructions`) on\n  the server directly\n- Restrict what types of events the client can receive and send\n\nYou will have to implement these features yourself.\n\n# Realtime API reference client\n\nThe latest reference client and documentation are available on GitHub at\n[openai/openai-realtime-api-beta](https://github.com/openai/openai-realtime-api-beta).\n\nYou can use this client yourself in any React (front-end) or Node.js project.\nFor full documentation, refer to the GitHub repository, but you can use the\nguide here as a primer to get started.\n\n```javascript\nimport { RealtimeClient } from '/src/lib/realtime-api-beta/index.js';\n\nconst client = new RealtimeClient({ apiKey: process.env.OPENAI_API_KEY });\n\n// Can set parameters ahead of connecting\nclient.updateSession({ instructions: 'You are a great, upbeat friend.' });\nclient.updateSession({ voice: 'alloy' });\nclient.updateSession({ turn_detection: 'server_vad' });\nclient.updateSession({ input_audio_transcription: { model: 'whisper-1' } });\n\n// Set up event handling\nclient.on('conversation.updated', ({ item, delta }) =\u003e {\n  const items = client.conversation.getItems(); // can use this to render all items\n  /* includes all changes to conversations, delta may be populated */\n});\n\n// Connect to Realtime API\nawait client.connect();\n\n// Send an item and triggers a generation\nclient.sendUserMessageContent([{ type: 'text', text: `How are you?` }]);\n```\n\n## Sending streaming audio\n\nTo send streaming audio, use the `.appendInputAudio()` method. If you're in `turn_detection: 'disabled'` mode,\nthen you need to use `.generate()` to tell the model to respond.\n\n```javascript\n// Send user audio, must be Int16Array or ArrayBuffer\n// Default audio format is pcm16 with sample rate of 24,000 Hz\n// This populates 1s of noise in 0.1s chunks\nfor (let i = 0; i \u003c 10; i++) {\n  const data = new Int16Array(2400);\n  for (let n = 0; n \u003c 2400; n++) {\n    const value = Math.floor((Math.random() * 2 - 1) * 0x8000);\n    data[n] = value;\n  }\n  client.appendInputAudio(data);\n}\n// Pending audio is committed and model is asked to generate\nclient.createResponse();\n```\n\n## Adding and using tools\n\nWorking with tools is easy. Just call `.addTool()` and set a callback as the second parameter.\nThe callback will be executed with the parameters for the tool, and the result will be automatically\nsent back to the model.\n\n```javascript\n// We can add tools as well, with callbacks specified\nclient.addTool(\n  {\n    name: 'get_weather',\n    description:\n      'Retrieves the weather for a given lat, lng coordinate pair. Specify a label for the location.',\n    parameters: {\n      type: 'object',\n      properties: {\n        lat: {\n          type: 'number',\n          description: 'Latitude',\n        },\n        lng: {\n          type: 'number',\n          description: 'Longitude',\n        },\n        location: {\n          type: 'string',\n          description: 'Name of the location',\n        },\n      },\n      required: ['lat', 'lng', 'location'],\n    },\n  },\n  async ({ lat, lng, location }) =\u003e {\n    const result = await fetch(\n      `https://api.open-meteo.com/v1/forecast?latitude=${lat}\u0026longitude=${lng}\u0026current=temperature_2m,wind_speed_10m`\n    );\n    const json = await result.json();\n    return json;\n  }\n);\n```\n\n## Interrupting the model\n\nYou may want to manually interrupt the model, especially in `turn_detection: 'disabled'` mode.\nTo do this, we can use:\n\n```javascript\n// id is the id of the item currently being generated\n// sampleCount is the number of audio samples that have been heard by the listener\nclient.cancelResponse(id, sampleCount);\n```\n\nThis method will cause the model to immediately cease generation, but also truncate the\nitem being played by removing all audio after `sampleCount` and clearing the text\nresponse. By using this method you can interrupt the model and prevent it from \"remembering\"\nanything it has generated that is ahead of where the user's state is.\n\n## Reference client events\n\nThere are five main client events for application control flow in `RealtimeClient`.\nNote that this is only an overview of using the client, the full Realtime API\nevent specification is considerably larger, if you need more control check out the GitHub repository:\n[openai/openai-realtime-api-beta](https://github.com/openai/openai-realtime-api-beta).\n\n```javascript\n// errors like connection failures\nclient.on('error', (event) =\u003e {\n  // do thing\n});\n\n// in VAD mode, the user starts speaking\n// we can use this to stop audio playback of a previous response if necessary\nclient.on('conversation.interrupted', () =\u003e {\n  /* do something */\n});\n\n// includes all changes to conversations\n// delta may be populated\nclient.on('conversation.updated', ({ item, delta }) =\u003e {\n  // get all items, e.g. if you need to update a chat window\n  const items = client.conversation.getItems();\n  switch (item.type) {\n    case 'message':\n      // system, user, or assistant message (item.role)\n      break;\n    case 'function_call':\n      // always a function call from the model\n      break;\n    case 'function_call_output':\n      // always a response from the user / application\n      break;\n  }\n  if (delta) {\n    // Only one of the following will be populated for any given event\n    // delta.audio = Int16Array, audio added\n    // delta.transcript = string, transcript added\n    // delta.arguments = string, function arguments added\n  }\n});\n\n// only triggered after item added to conversation\nclient.on('conversation.item.appended', ({ item }) =\u003e {\n  /* item status can be 'in_progress' or 'completed' */\n});\n\n// only triggered after item completed in conversation\n// will always be triggered after conversation.item.appended\nclient.on('conversation.item.completed', ({ item }) =\u003e {\n  /* item status will always be 'completed' */\n});\n```\n\n# Wavtools\n\nWavtools contains easy management of PCM16 audio streams in the browser, both\nrecording and playing.\n\n## WavRecorder Quickstart\n\n```javascript\nimport { WavRecorder } from '/src/lib/wavtools/index.js';\n\nconst wavRecorder = new WavRecorder({ sampleRate: 24000 });\nwavRecorder.getStatus(); // \"ended\"\n\n// request permissions, connect microphone\nawait wavRecorder.begin();\nwavRecorder.getStatus(); // \"paused\"\n\n// Start recording\n// This callback will be triggered in chunks of 8192 samples by default\n// { mono, raw } are Int16Array (PCM16) mono \u0026 full channel data\nawait wavRecorder.record((data) =\u003e {\n  const { mono, raw } = data;\n});\nwavRecorder.getStatus(); // \"recording\"\n\n// Stop recording\nawait wavRecorder.pause();\nwavRecorder.getStatus(); // \"paused\"\n\n// outputs \"audio/wav\" audio file\nconst audio = await wavRecorder.save();\n\n// clears current audio buffer and starts recording\nawait wavRecorder.clear();\nawait wavRecorder.record();\n\n// get data for visualization\nconst frequencyData = wavRecorder.getFrequencies();\n\n// Stop recording, disconnects microphone, output file\nawait wavRecorder.pause();\nconst finalAudio = await wavRecorder.end();\n\n// Listen for device change; e.g. if somebody disconnects a microphone\n// deviceList is array of MediaDeviceInfo[] + `default` property\nwavRecorder.listenForDeviceChange((deviceList) =\u003e {});\n```\n\n## WavStreamPlayer Quickstart\n\n```javascript\nimport { WavStreamPlayer } from '/src/lib/wavtools/index.js';\n\nconst wavStreamPlayer = new WavStreamPlayer({ sampleRate: 24000 });\n\n// Connect to audio output\nawait wavStreamPlayer.connect();\n\n// Create 1s of empty PCM16 audio\nconst audio = new Int16Array(24000);\n// Queue 3s of audio, will start playing immediately\nwavStreamPlayer.add16BitPCM(audio, 'my-track');\nwavStreamPlayer.add16BitPCM(audio, 'my-track');\nwavStreamPlayer.add16BitPCM(audio, 'my-track');\n\n// get data for visualization\nconst frequencyData = wavStreamPlayer.getFrequencies();\n\n// Interrupt the audio (halt playback) at any time\n// To restart, need to call .add16BitPCM() again\nconst trackOffset = await wavStreamPlayer.interrupt();\ntrackOffset.trackId; // \"my-track\"\ntrackOffset.offset; // sample number\ntrackOffset.currentTime; // time in track\n```\n\n# Acknowledgements and contact\n\nThanks for checking out the Realtime Console. We hope you have fun with the Realtime API.\nSpecial thanks to the whole Realtime API team for making this possible. Please feel free\nto reach out, ask questions, or give feedback by creating an issue on the repository.\nYou can also reach out and let us know what you think directly!\n\n- OpenAI Developers / [@OpenAIDevs](https://x.com/OpenAIDevs)\n- Jordan Sitkin / API / [@dustmason](https://x.com/dustmason)\n- Mark Hudnall / API / [@landakram](https://x.com/landakram)\n- Peter Bakkum / API / [@pbbakkum](https://x.com/pbbakkum)\n- Atty Eleti / API / [@athyuttamre](https://x.com/athyuttamre)\n- Jason Clark / API / [@onebitToo](https://x.com/onebitToo)\n- Karolis Kosas / Design / [@karoliskosas](https://x.com/karoliskosas)\n- Keith Horwood / API + DX / [@keithwhor](https://x.com/keithwhor)\n- Romain Huet / DX / [@romainhuet](https://x.com/romainhuet)\n- Katia Gil Guzman / DX / [@kagigz](https://x.com/kagigz)\n- Ilan Bigio / DX / [@ilanbigio](https://x.com/ilanbigio)\n- Kevin Whinnery / DX / [@kevinwhinnery](https://x.com/kevinwhinnery)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodenamev%2Flive-research","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodenamev%2Flive-research","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodenamev%2Flive-research/lists"}