{"id":15321526,"url":"https://github.com/zsais/selfservicekiosk-audio-streaming","last_synced_at":"2025-10-09T08:31:27.047Z","repository":{"id":38827231,"uuid":"494181500","full_name":"zsais/selfservicekiosk-audio-streaming","owner":"zsais","description":"A best practice for streaming audio from a browser microphone to Dialogflow or Google Cloud STT by using websockets.","archived":false,"fork":true,"pushed_at":"2022-06-03T17:55:13.000Z","size":2437,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-27T05:33:54.907Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"dialogflow/selfservicekiosk-audio-streaming","license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zsais.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-19T18:10:43.000Z","updated_at":"2022-06-02T23:11:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/zsais/selfservicekiosk-audio-streaming","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zsais/selfservicekiosk-audio-streaming","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zsais%2Fselfservicekiosk-audio-streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zsais%2Fselfservicekiosk-audio-streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zsais%2Fselfservicekiosk-audio-streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zsais%2Fselfservicekiosk-audio-streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zsais","download_url":"https://codeload.github.com/zsais/selfservicekiosk-audio-streaming/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zsais%2Fselfservicekiosk-audio-streaming/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001064,"owners_count":26082991,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T09:11:19.652Z","updated_at":"2025-10-09T08:31:26.539Z","avatar_url":"https://github.com/zsais.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n# Google Cloud / Dialogflow - Self Service Kiosk Demo\n\n[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.svg)](https://console.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2Fdialogflow%2Fselfservicekiosk-audio-streaming\u0026cloudshell_tutorial=TUTORIAL.md)\n\nA best practice for streaming audio from a browser microphone to Dialogflow or Google Cloud STT by using websockets.\n\nAirport SelfService Kiosk demo, to demonstrate how microphone streaming to GCP works, from a web application.\n\nIt makes use of the following GCP resources:\n\n* Dialogflow \u0026 Knowledge Bases\n* Speech to Text\n* Text to Speech\n* Translate API\n* (optionally) App Engine Flex\n\nIn this demo, you can start recording your voice, it will display answers on a screen and synthesize the speech.\n\n![alt text](https://github.com/dialogflow/selfservicekiosk-audio-streaming/blob/master/docs/architecture.png \"Architecture\")\n\n![alt text](https://github.com/dialogflow/selfservicekiosk-audio-streaming/blob/master/docs/screen.png \"Screenshot\")\n\n\n# Live demo\n\nA working demo can be found here: [http://selfservicedesk.appspot.com/](http://selfservicedesk.appspot.com/)\n\n\n# Blog posts\n\nI wrote very extensive blog articles on how to setup your streaming project. Want to exactly learn how this code works? Have a start here:\n\nBlog 1: [Introduction to the GCP conversational AI components, and integrating your own voice AI in a web app.](https://medium.com/@ladysign/building-your-own-conversational-voice-ai-with-dialogflow-speech-to-text-in-web-apps-part-i-b92770bd8b47)  \nBlog 2: [Building a client-side web application which streams audio from a browser microphone to a server.](https://medium.com/@ladysign/building-a-client-side-web-app-which-streams-audio-from-a-browser-microphone-to-a-server-part-ii-df20ddb47d4e)  \nBlog 3: [Building a web server which receives a browser microphone stream and uses Dialogflow or the Speech to Text API for retrieving text results.](https://medium.com/google-cloud/building-a-web-server-which-receives-a-browser-microphone-stream-and-uses-dialogflow-or-the-speech-62b47499fc71)  \nBlog 4: [Getting Audio Data from Text (Text to Speech) and play it in your browser.](https://medium.com/@ladysign/getting-audio-data-from-text-text-to-speech-and-play-it-in-your-browser-part-iv-cd2d6ea71c6a)\n\n# Slides \u0026 Video\n\nThere's a [presentation](https://speakerdeck.com/savelee/implementing-a-custom-ai-voice-assistant-by-streaming-webrtc-to-dialogflow-and-cloud-speech) and a [video](https://youtu.be/6JD8WC1LV7g) that accompanies the tutorial.\n\n[![Slidedeck AudioStreaming](https://github.com/dialogflow/selfservicekiosk-audio-streaming/blob/master/docs/slidedeck.png)](https://speakerdeck.com/savelee/implementing-a-custom-ai-voice-assistant-by-streaming-webrtc-to-dialogflow-and-cloud-speech)\n\n# Setup Local Environment\n\n## Get a Node.js environment\n\n1. `apt-get install nodejs -y`\n\n1. `apt-get npm`\n\n## Get an Angular environment\n\n1. `sudo npm install -g @angular/cli`\n\n## Clone Repo\n\n1. `git clone https://github.com/dialogflow/selfservicekiosk-audio-streaming.git selfservicekiosk`\n\n2. Set the PROJECT_ID variable: export PROJECT_ID=[gcp-project-id]\n\n3. Set the project: `gcloud config set project $PROJECT_ID`\n\n4. Download the service account key.\n\n5. Assign the key to environment var: **GOOGLE_APPLICATION_CREDENTIALS**\n\n LINUX/MAC\n `export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account.json`\n WIN\n `set GOOGLE_APPLICATION_CREDENTIALS=c:\\path\\to\\service_account.json`\n\n6. Login: `gcloud auth login`\n\n7. Open **server/env.txt**, change the environment variables and rename the file to **server/.env**\n\n8. Enable APIs:\n\n ```\n  gcloud services enable \\\n  appengineflex.googleapis.com \\\n  containerregistry.googleapis.com \\\n  cloudbuild.googleapis.com \\\n  cloudtrace.googleapis.com \\\n  dialogflow.googleapis.com \\\n  logging.googleapis.com \\\n  monitoring.googleapis.com \\\n  sourcerepo.googleapis.com \\\n  speech.googleapis.com \\\n  mediatranslation.googleapis.com \\\n  texttospeech.googleapis.com \\\n  translate.googleapis.com\n```\n\n9. Build the client-side Angular app:\n    \n    ```\n    cd client \u0026\u0026 sudo npm install\n    npm run-script build\n    ```\n\n10. Start the server Typescript app, which is exposed on port 8080:\n\n    ```\n    cd ../server \u0026\u0026 sudo npm install\n    npm run-script watch\n    ```\n\n3. Browse to http://localhost:8080\n\n## Setup Dialogflow\n\n1. Create a Dialogflow agent at: http://console.dialogflow.com\n\n1. Zip the contents of the *dialogflow* folder, from this repo.\n\n1. Click **settings** \u003e **Import**, and upload the Dialogflow agent zip, you just created.\n\n1. *Caution: Knowledge connector settings are not currently included when exporting, importing, or restoring agents.*\n\n    Make sure you have enabled *Beta* features in settings.\n\n    1. Select *Knowledge* from the left menu.\n    1. Create a Knowledge Base: **Airports**\n    1. Add the following Knowledge Base **FAQs**, as **text/html** documents:\n\n    * https://www.panynj.gov/port-authority/en/help-center/faq/airports-faq-help-center.html\n    * https://www.schiphol.nl/en/before-you-take-off/\n    * https://www.flysfo.com/faqs\n\n    1. As a response it requires the following custom payload:\n\n    ```\n    {\n    \"knowledgebase\": true,\n    \"QUESTION\": \"$Knowledge.Question[1]\",\n    \"ANSWER\": \"$Knowledge.Answer[1]\"\n    }\n    ```\n\n    1. And to make the Text to Speech version of the answer working add the following Text SSML response:\n\n    ```\n    $Knowledge.Answer[1]\n    ```\n\n# Deploy with App Engine Flex\n\nThis demo makes heavy use of websockets and\nthe microphone `getUserMedia()` HTML5 API requires\nto run over HTTPS. Therefore, I deploy this demo\nwith a custom runtime, so I can include my own **Dockerfile**.\n\n1. Edit the **app.yaml** to tweak the environment variables.\nSet the correct Project ID.\n\n1. Deploy with: `gcloud app deploy`\n\n1. Browse: `gcloud app browse`\n\n\n# Examples\n\nThe selfservice kiosk is a full end to end application.\nTo showcase smaller examples, I've created 6 small demos. Here's how you can get these running:\n\n1. Install the required libraries, run the following command from the *examples* folder:\n\n    `npm install`\n\n2. Start the simpleserver node app:\n\n   `npm --EXAMPLE=1 --PORT=8080 --PROJECT_ID=[your-gcp-project-id] run start`\n\nTo switch to the various examples, edit the EXAMPLE variable to one of these:\n\n* Example **1**: Dialogflow Speech Intent Detection\n* Example **2**: Dialogflow Speech Detection through streaming\n* Example **3**: Dialogflow Speech Intent Detection with Text to Speech output\n* Example **4**: Speech to Text Transcribe Recognize Call\n* Example **5**: Speech to Text Transcribe Streaming Recognize\n* Example **6**: Text to Speech in a browser\n\n3. Browse to http://localhost:8080. Open the inspector, to preview the\nDialogflow results object.\n\nThe code required for these examples can be found in **simpleserver.js** for the different Dialogflow \u0026 STT calls. - **example1.html** - **example5.html** will show the client-side implementations.\n\n# License\n\nApache 2.0\n\nThis is not an official Google product.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzsais%2Fselfservicekiosk-audio-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzsais%2Fselfservicekiosk-audio-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzsais%2Fselfservicekiosk-audio-streaming/lists"}