{"id":21679209,"url":"https://github.com/3wille/bbb-kaldi-connector","last_synced_at":"2026-05-10T15:32:16.247Z","repository":{"id":80988440,"uuid":"287934102","full_name":"3wille/bbb-kaldi-connector","owner":"3wille","description":"Connect BigBlueButton conferences to Kaldi automatic-speech-recognition","archived":false,"fork":false,"pushed_at":"2020-10-28T20:30:17.000Z","size":211,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-20T10:21:34.239Z","etag":null,"topics":["bigbluebutton","kaldi-asr","nlp"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/3wille.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-16T11:53:59.000Z","updated_at":"2022-12-13T19:51:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"856caf2e-19e7-4c96-b6df-47932c777ca9","html_url":"https://github.com/3wille/bbb-kaldi-connector","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/3wille/bbb-kaldi-connector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3wille%2Fbbb-kaldi-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3wille%2Fbbb-kaldi-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3wille%2Fbbb-kaldi-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3wille%2Fbbb-kaldi-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/3wille","download_url":"https://codeload.github.com/3wille/bbb-kaldi-connector/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3wille%2Fbbb-kaldi-connector/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32861420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"ssl_error","status_checked_at":"2026-05-10T13:40:02.145Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigbluebutton","kaldi-asr","nlp"],"created_at":"2024-11-25T14:49:42.051Z","updated_at":"2026-05-10T15:32:16.226Z","avatar_url":"https://github.com/3wille.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BBB Kaldi Connector\n\nThis repository is the result of my master programm project at the [LT Group](https://github.com/uhh-lt) at the Univeristy of Hamburg.\nThe main goal of this project was to integrate a speech recognition pipeline into the [BigBlueButton Conference System](https://github.com/bigbluebutton/bigbluebutton).\nIt further allows to display recognized speech as closed captions in BBB's HTML5 client.\n\n## Architecture\n\n![Architecture](./docs/architecture.svg)\n\nThis project consists of three components that interact with some of the BBB components.\nMost notably, Redis PubSub is used both for listening for newly created meetings as well as communication between this project's components.\n\n### Audio Relay\n![Audio Relay](./docs/audio_relay.svg)\n\nFor each new meeting, the Relay initializes a VoIP call to Freeswitch over SIP.\nThis results in an RTP connection that transfers mono OPUS 48kHz frames.\nThe frames are decoded into 16bit PCM samples that are published over pubsub in a meeting specific channel.\n\n### Kaldi Model Server\n\n![Kaldi Model Server](./docs/kms.svg)\n\nThis is a project by [UHH LT group](https://github.com/uhh-lt/kaldi-model-server).\nThe kaldi-start is a program that listens on redis for new meeting and then starts KMS with meeting specific channels for audio input and text output.\n\n### MongoDB Connector\n\n![MongoDB Connector](./docs/mongodb.svg)\n\nThis component listens on PubSub for utterance messages by KMS, extracts the utterance and the corresponding meeting ID.\nThe BBB HTML5 client is built with Meteor.js which uses a MongoDB.\nIn the MongoDB, there is a document for each meeting and locale combination.\nWe update the respective document with the just received utterance and meteor.js can recognize the change and update its client through websockets.\n\n#### BBB-HTML5 (Meteor.js)\nTo properly display captions inserted into the MongoDB a slightly modified version of html5 client needs to be used.\nSee the following two commits for reference:\n* https://github.com/3wille/bigbluebutton/commit/6b066f4bbf53dfc1b6844a9d7a8134bd512f436c\n* https://github.com/3wille/bigbluebutton/commit/2322054d922159ba9ef9a30ad269a709d3e3f320\n\n## Setup \u0026 Usage\n\nAll three components need to be compiled with a Golang compiler that supports module (probably \u003e=1.13).\nIn my setup I have the relay and mongo-connector running on the same server as BBB and Kaldi running on a different server.\nCommunication with KMS happens through Redis, so make sure that all have access to the same redis server.\n\nCompile the binaries with:\n\n```\ngo build audio_relay.go\ngo build kaldi-starter.go\ngo build kaldi_mongo_connector/kaldi_mongo_connector.go\n```\nIf you are compiling for a server without glibc, consider usign ``CGO_ENABLED=0``.\n\nIf you have Golang installed on your server, you can also run the programs directly, e.g.:\n```\ngo run kaldi-starter.go\n```\nOtherwise run the compiled binaries.\n\n### Running Programs \u0026 Options\n\n#### Relay\n```\naudio_relay bbb_secret_path_token sentry_url\n```\n* ``sentry_url``: I use sentry for error tracking, you can make an account, but removing the code should be easy as well\n* ``bbb_secret_path_token`` is a path suffix used to circumvent BBB authorization. Append the following to ``/etc/bigbluebutton/nginx/sip.nginx``:\n\n```\nlocation /ws_BBB_SECRET_PATH_TOKEN {\n        proxy_pass https://ip_address_as_above:7443;\n        proxy_http_version 1.1;\n        proxy_set_header Upgrade $http_upgrade;\n        proxy_set_header Connection \"Upgrade\";\n        proxy_read_timeout 6h;\n        proxy_send_timeout 6h;\n        client_body_timeout 6h;\n        send_timeout 6h;\n\n\terror_log /var/log/nginx/bbb_websocket.log info;\n}\n```\n#### MongoDB Connector\n```\nkaldi_mongo_connector sentry_url\n```\n* ``sentry_url``: I use sentry for error tracking, you can make an account, but removing the code should be easy as well\n#### Kaldi Starter\n```\nkaldi-starter\n```\n\nTo work, the starter of course needs a running version of [Kaldi Model Server](https://github.com/uhh-lt/kaldi-model-server).\nThat in turn, needs a running version of PyKaldi and Kaldi, please see the respective documentations on how to install those.\nKMS also needs a model to work with, the respective settings yaml file can be set in https://github.com/3wille/bbb-kaldi-connector/blob/master/kaldi-starter.go#L38\n\n## Results\n\nWhen everything is working fine, you should be able to see Redis PubSub channels of this pattern ``asr_text_INTERNAL_MEETING_ID`` and on each of those channels messages with partial and complete utterances:\n\n```\n{\n    \"handle\": \"partialUtterance\",\n    \"utterance\": \"die Universität Hamburg ist mit über vierzig Tausend Studierenden die größte\n\tUniversität in der Freien und Hansestadt Hamburg die größte Forschungs und Ausbildungseinrichtung\nin Norddeutschland und eine der zehn größten Hochschulen in\",\n    \"key\": \"mic0-utt329-part52\",\n    \"speaker\": \"speaker0\",\n    \"time\": 1751.2375078201294\n}\n```\n```\n{\n    \"handle\": \"completeUtterance\",\n    \"utterance\": \"die Universität Hamburg ist mit über vierzig Tausend Studierenden die größte Universität in der Freien und Hansestadt Hamburg die größte Forschungs und Ausbildungseinrichtung in Norddeutschland und eine der zehn größten Hochschulen in Deutschland\",\n    \"confidences\": [0.9995377659797668, 1.0, 0.867841362953186, 0.8684811592102051, 0.9593820571899414, 1.0, 1.0, 0.9992160797119141, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.9999002814292908, 1.0, 0.9989173412322998, 1.0, 1.0, 1.0, 0.997087836265564, 0.9883859157562256, 1.0, 1.0, 0.9992555379867554, 1.0, 1.0, 0.973842442035675, 0.9845209121704102, 1.0, 1.0, 1.0, 1.0, 0.9999350905418396, 1.0],\n    \"key\": \"mic0-utt443-part67\",\n    \"speaker\": \"speaker0\",\n    \"time\": 2371.5270574092865\n}\n```\n(text source: https://de.wikipedia.org/wiki/Universit%C3%A4t_Hamburg)\n\nIf you don't see this message, you can check on ``asr_audio_INTERNAL_MEETING_ID`` to see whether the problem is with the relay or kaldi.\n\nWhen the MongoDB Connector is running, you should be able to see the CC button in the bottom-left corner of the client's main pane.\nOnce activated, the closed captions should show like\n![MongoDB Connector](./docs/captions.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3wille%2Fbbb-kaldi-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F3wille%2Fbbb-kaldi-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3wille%2Fbbb-kaldi-connector/lists"}