https://github.com/didi/meetdot
https://github.com/didi/meetdot
Last synced: 7 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/didi/meetdot
- Owner: didi
- License: apache-2.0
- Created: 2021-09-08T10:52:51.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2021-11-04T10:19:22.000Z (about 4 years ago)
- Last Synced: 2024-06-20T14:33:10.383Z (over 1 year ago)
- Language: Python
- Size: 1.4 MB
- Stars: 11
- Watchers: 5
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# MeetDot: Speech Translation for Video Calling
MeetDot is a videoconferencing system with live translation captions overlaid onscreen.
The system aims to facilitate conversation between people who speak different languages,
reducing communication barriers between multilingual participants.
Currently, our system supports speech and captions in four languages (English, Chinese, Spanish, Portuguese)
and combines automatic speech recognition (ASR) and machine translation (MT) in a cascade.
- [Key Features](#key-features)
- [Compatibility](#compatibility)
- [Setup](#setup)
- [Create Python Environment](#create-python-environment)
- [Set Up Backend](#set-up-backend)
- [Create the Credentials](#create-the-credentials)
- [Configuration](#configuration)
- [Run the Backend](#run-the-backend)
- [Set Up Frontend](#set-up-frontend)
- [Testing](#testing)
- [Committing](#committing)
- [Evaluation](#evaluation)
- [Security Notes](#security-notes)
- [Deployment](#deployment)
- [License](#license)
## Key Features
- Smooth scrolling captions to reduce flicker and therefore cognitive load
- Modular architecture to integrate different ASR and MT services
- Integrated evaluation suite to optimize metrics such as accuracy, latency and erasure
- Word-guessing game for an extrinsic measure of end-to-end performance
## Compatibility
- conda 64-bit with Python3.7+
- Tested on Chrome and Firefox
## Setup
### Create Python Environment
Create a conda environment that is Python 3.7 or 3.8 (newer versions will probably also work)
1. `conda create -n streaming_translation python=3.7.9`
2. `conda activate streaming_translation`
### Set Up Backend
1. `cd backend`
2. `pip install -r requirements.txt`, from your conda environment.
3. `pip install -e .`
4. Under the repository's root directory, execute `python env_setup.py --copy-keys`. Sets environment variables neccessary to run APIs and copies credentials. It also installs git commit hooks. If not running on our shared server, see `python env_setup.py --help`
### Create Credentials
In MeetDot, we provide ASR/MT integrations with Google Cloud so it can be run without access to our servers or internal APIs.
If not running on our shared server, do the following:
1. Go to Google Cloud and get a custom google api key [link](https://cloud.google.com/docs/authentication/getting-started) and copy it into `backend/resources/credentials.json`
2. We use API from [daily.co](https://www.daily.co) to optimize real-time video call quality. Create an account with daily.co, and add your URL and key to `.env`.
### Configuration
In MeetDot, we uses dotenv in the frontend and backend.
Variables can be set by editing the `.env` file or on the command line `BACKEND_PORT=5888 python src/app.py`.
See `.env.default` for a list of possible keys.
### Run the Backend
1. `python src/app.py` (run with the `--debug` flag to restart on code changes, useful for development)
### Set Up Frontend
1. `cd frontend`
2. `yarn install`
3. `yarn run serve --port=8080` (You can use `--mode=production` to serve a minified bundle)
4. Open localhost:8080 in browser.
## Testing
1. `cd backend/`
2. `pytest`
## Committing
We use [pre-commit](https://pre-commit.com/) for managing git hooks.
Run: `pre-commit install` to set it up.
This should run the formatter (Prettier for javascript) and syntax checks whenever you commit.
It should be set up to update itself and library installation, but see `.pre-commit-config.yaml` for more information.
## Evaluation
1. [Download test data](./backend/src/services/evaluation/)
2. `cd backend/`
3. `python src/services/evaluation/evaluate.py`
## Security Notes
In the first time running MeetDot system, the browser needs microphone permission to run the demo. Modern browsers prevent unsecure websites from accessing the microphone.
Local environments (`localhost`) are considered secure. For remote deployment, you will need to do one of the following:
1. Serve the website over HTTPS. Set paths to your key and cert files in `backend/resources/keys.json`. The filenames are defaults generated by LetsEncrypt.
2. Use a self-signed certificate and have your users click through the browser warnings. (Not recommended)
3. Have users enable experimental flags to bypass browser security setttings. (Also not recommended). In Chrome, this is `chrome://flags#unsafely-treat-insecure-origin-as-secure`. In Firefox, this is `media.devices.insecure.enabled` in `about:config`.
## Deployment
Our deployment script copies files into $DEPLOY_ROOT (e.g. `/srv/example.com`, set in gitlab variables).
Static frontend files are served from $DEPLOY_ROOT/dist with nginx.
Backend files are saved in $DEPLOY_ROOT/backend and served with a bare flask-socketio server.
These processes (nginx, the backend) are managed by systemd scripts.
## License
[Apache License 2.0](LICENSE)