{"id":13575867,"url":"https://github.com/google/live-transcribe-speech-engine","last_synced_at":"2025-05-16T06:00:27.734Z","repository":{"id":39352105,"uuid":"201338678","full_name":"google/live-transcribe-speech-engine","owner":"google","description":"Live Transcribe is an Android application that provides real-time captioning for people who are deaf or hard of hearing. This repository contains the Android client libraries for communicating with Google's Cloud Speech API that are used in Live Transcribe.","archived":false,"fork":false,"pushed_at":"2022-07-20T03:44:43.000Z","size":6978,"stargazers_count":1456,"open_issues_count":11,"forks_count":218,"subscribers_count":52,"default_branch":"master","last_synced_at":"2025-04-19T22:27:27.738Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-08T21:15:19.000Z","updated_at":"2025-04-12T06:52:55.000Z","dependencies_parsed_at":"2022-08-01T08:19:24.522Z","dependency_job_id":null,"html_url":"https://github.com/google/live-transcribe-speech-engine","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Flive-transcribe-speech-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Flive-transcribe-speech-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Flive-transcribe-speech-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Flive-transcribe-speech-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google","download_url":"https://codeload.github.com/google/live-transcribe-speech-engine/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478160,"owners_count":22077675,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:01:04.908Z","updated_at":"2025-05-16T06:00:25.052Z","avatar_url":"https://github.com/google.png","language":"Java","funding_links":[],"categories":["Java","人工智能"],"sub_categories":[],"readme":"# The Live Transcribe Speech Engine\n\n(This is not an official Google product!)\n\n[Live Transcribe](https://www.android.com/accessibility/live-transcribe/) is an\nAndroid application that provides real-time captioning for people who are deaf \nor hard of hearing. This repository contains the Android client libraries for\ncommunicating with Google's Cloud Speech API that are used in Live Transcribe.\n\nThe automatic speech recognition (ASR) module has the following features:\n\n- Infinite streaming\n- Support for [70+ languages](https://cloud.google.com/speech-to-text/docs/languages)\n- Robust to brief network loss (which occurs often when traveling and switching\n between network/wifi). Text is not lost, only delayed.\n- Robust to extended network loss. Will reconnect again even if network has been\n out for hours. Of course, no speech recognition can be delivered without a\n connection.\n- Robust to server errors\n- Opus, AMR-WB, FLAC encoding can be easily enabled and configured.\n- Contains a text formatting library for visualizing ASR confidence, speaker\n ID, and more\n- Extensible to offline models\n- Built-in support for speech detectors, which can be used to stop ASR during\n extended silences to save money and data (Note that speech detector\n implementation is not provided)\n- Built-in support for speaker identification, which can be used to label or\n color text according to speaker number (Note that speaker identification\n implementation is not provided)\n\nThe libraries provided are nearly identical to those running in the production\napplication Live Transcribe. They have been extensively field tested and unit\ntested. However, the tests themselves are not open sourced at this time.\n\nContact lt-speech-engine-maintainers@google.com with questions/issues.\n\nTo try this library out using our sample Android application, follow the\ninstructions below. These instructions assume that the host operating \nsystem is an Ubuntu-like flavor of Linux. If other operating systems can be \nsupported with reasonable, additional efforts we may do so on a case-by-case \nbasis, however we unfortunately cannot claim that this library will be maintained \nacross all operating systems and build environments. Sorry for any inconvenience.\nSee note below about testing this on other environments.\n\nWe have also provided APKs so that you can try out the library without building\nany code.\n\nWhether you're using our code or our sample APKs, an API key is required. See the\n[documentation on API keys](https://cloud.google.com/docs/authentication/api-keys)\nto learn more. In the sample APK, you can copy/paste your API key into the pop-up\ndialog.\n\n\n```\nRequirements: CMake, Gradle, Android SDK/NDK\n(1) Export the path of your Android SDK. The NDK is assumed to be located at\n   $ANDROID_SDK_PATH/ndk-bundle [Android NDK](https://developer.android.com/ndk/guides)\n   export ANDROID_SDK_PATH=\"/wherever/your/sdk/is\"\n(2) Build the APKs (location is app/build/outputs/apk/)\n   ./build_all.sh\n```\n\n# What this library does (and important tricks for getting good results):\n\nThe purpose of this library is to simulate an infinite connection to Google's\nCloud Speech API. The API itself currently does not support infinite streaming\nand forces a timeout after 5 minutes of streaming. It turns out that there are a\nlot of subtleties involved in covering up this timeout, and we believe that not\neveryone should have to figure this out on their own. The main logic for\nmanaging streaming requests is the RepeatingRecognitionSession class. This class\nmaintains a single bidirectional streaming request, called a session, and takes\nmeasures to avoid hitting the timeout. Namely, we use the following strategies:\n- Try to close sessions during pauses once the server-side timeout time is\napproaching (this is done using the is_final field from the returned speech\nresults as they are an estimate of when a pause has happened).\n- Close sessions that have been silent for a very long time prior to hitting the\ntimeout. If someone were to start talking, it is better that that be towards the\nbeginning of the session.\n- Store a buffer of audio in between sessions and send it once the new session\nstarts. The device may jump from one network to another, or from network to\nWiFi. Even on a steady connection, it can take more than the typical hundred or\nso milliseconds to open a new session after the previous one has closed. This\nbuffer is crucial for making sure all audio gets to the server.\n\n## Model selection\nIn most cases, the [best model](https://cloud.google.com/speech-to-text/docs/transcription-model)\nto use for general purpose recognition is the default model. For languages where\nit is available, the \"video\" model has much better performance.\n\n```\n// As seen in CloudSpeechSession.java\nRecognitionConfig.Builder.newBuilder()\n    .setModel(\"video\")\n    ...  // Other options.\n    .build()\n```\n\nAt the time of writing this, the video model exists only for the \"en-US\" locale\nand is offered at a different price point.\n\nNote that [server performance](https://issuetracker.google.com/issues/137672586)\ncan have a serious impact on result quality.\n\n## Codec notes\nThere are other concerns that arise when trying to stream infinitely. In some\ncountries, data is expensive and sending uncompressed PCM formatted audio is\nsimply not practical. At 16kHz, streaming uncompressed 16-bit PCM data requires\n256 kilobits per second of data, ignoring the comparatively small overhead of\nheader/auxiliary data. On low-bandwidth connections, this is too high of a data\nrate to reliably use the cloud for recognition. It becomes necessary to use a\ncodec. Of the codecs supported by the speech APIs, we experimented with FLAC,\nAMR-WB, and Opus (in an Ogg container). For the former two, we leverage the\nAndroid framework's encoder. FLAC is a lossless codec (unlike most audio codecs)\nand will get you roughly a factor of 2 in data compression. It introduces a few\nhundred milliseconds of latency, but is quite acceptable in most cases. AMR-WB offers\na much more appealing compression ratio, but in relatively noisy conditions\nperforms very badly for speech recognition. We do not recommend using AMR-WB for\nspeech recognition under any circumstances.\n\nFinally, the Opus codec delivers quite impressive results for speech\nrecognition. Unfortunately, the Android framework does not ship with an Opus\nencoder, so we included a native implementation in our library.\nAt rates at least as low as 24 kilobits per second (a compression\nratio of nearly 11), recognition quality does not seem to be impacted at all. We\nknow that with captions, accuracy is critical, so in Live Transcribe, we\nconfigured our Opus codec to use a more conservative 32 kbps with variable\nbitrate (VBR) enabled. This is sufficient for minimizing data streaming costs\n(note that music streaming services use bitrates many times higher than this).\nHowever, there is still a bit of latency associated with Opus compression. There\nis one more fairly technical detail that we use to minimize latency in our Opus\nstream. For every block of audio that is pushed into the Ogg/Opus stream, we\nflush the Ogg stream rather than let the ogg library decide on its own when to\npush out the next block of data. This causes a slight increase in bitrate, but\na significant reduction in latency. For the curious, deep in our encoder is a\n\"low_latency_mode\" flag. As a user of this library nothing need be done to\nenable that. Just request the following settings for your\nCloudSpeechSessionParams:\n\n```\nCloudSpeechSessionParams.newBuilder()\n        .setEncoderParams(CloudSpeechSessionParams.EncoderParams.newBuilder()\n            .setEnableEncoder(true)\n            .setAllowVbr(true)\n            .setCodec(CodecAndBitrate.OGG_OPUS_BITRATE_32KBPS))\n        .build()\n```\n\n## Running on different environments\nWe expect that our code will build easily on Ubuntu, but on other systems, you \nmight meet challenges building the c++ Opus library. Fortunately you can \nlearn a lot about this library without using Opus at all. Here are some \ninstructions for how to do so. They aren't extremely elegant, so please bear\nin mind that we provide this info with the goal of getting you unblocked on\na system that we do not claim support for.\n\nIf you change the line [here](https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/MainActivity.java#L180) to request uncompressed audio, you will no longer have a dependency on Opus. \n```\nCloudSpeechSessionParams.newBuilder()\n        .setEncoderParams(CloudSpeechSessionParams.EncoderParams.newBuilder()\n            .setEnableEncoder(false)\n        .build()\n```\n\nComment out the [the static import of the Opus lib](https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/StreamingAudioEncoder.java#L541). \n\nYour code no longer depends on Opus, but that doesn't mean the gradle files won't look for those deps.\nGo through the gradle files and comment out any references to the c++ libs. \n\nFor development and local testing, Opus vs. uncompressed audio doesn't matter \nmuch. It's nearly enough to decide whether these libraries suit your needs, so you \ncan postpone figuring out Opus for when you decide whether to put your application\ninto production.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Flive-transcribe-speech-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle%2Flive-transcribe-speech-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Flive-transcribe-speech-engine/lists"}