{"id":17095758,"url":"https://github.com/bramvanroy/mateo-demo","last_synced_at":"2026-06-01T03:30:14.911Z","repository":{"id":137973142,"uuid":"607072353","full_name":"BramVanroy/mateo-demo","owner":"BramVanroy","description":"MAchine Translation Evaluation Online (MATEO)","archived":false,"fork":false,"pushed_at":"2024-03-15T08:27:36.000Z","size":303,"stargazers_count":19,"open_issues_count":6,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-21T10:03:43.198Z","etag":null,"topics":["bertscore","bleu","bleurt","chrf","clarin","comet","machine-translation","machine-translation-evaluation","machine-translation-metrics","streamlit","ter"],"latest_commit_sha":null,"homepage":"https://mateo.ivdnt.org/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BramVanroy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-02-27T08:54:53.000Z","updated_at":"2025-02-19T02:17:23.000Z","dependencies_parsed_at":"2024-03-15T09:46:38.645Z","dependency_job_id":null,"html_url":"https://github.com/BramVanroy/mateo-demo","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BramVanroy%2Fmateo-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BramVanroy%2Fmateo-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BramVanroy%2Fmateo-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BramVanroy%2Fmateo-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BramVanroy","download_url":"https://codeload.github.com/BramVanroy/mateo-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240170070,"owners_count":19759140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bertscore","bleu","bleurt","chrf","clarin","comet","machine-translation","machine-translation-evaluation","machine-translation-metrics","streamlit","ter"],"created_at":"2024-10-14T14:43:43.180Z","updated_at":"2026-06-01T03:30:14.851Z","avatar_url":"https://github.com/BramVanroy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MAchine Translation Evaluation Online (MATEO)\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://huggingface.co/spaces/BramVanroy/mateo-demo\" target=\"_blank\"\u003e\u003cimg alt=\"HF Spaces shield\" src=\"https://img.shields.io/badge/%F0%9F%A4%97-%20HF%20Spaces-orange?style=flat\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.gnu.org/licenses/gpl-3.0\" target=\"_blank\"\u003e\u003cimg alt=\"License shield\" src=\"https://img.shields.io/badge/License-GPLv3-blue.svg?style=flat\"\u003e\u003c/a\u003e\n  \u003cimg alt=\"Code style black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg?style=flat\"\u003e\n  \u003cimg alt=\"Built with Streamlit\" src=\"https://img.shields.io/static/v1?style=for-the-badge\u0026message=Streamlit\u0026color=FF4B4B\u0026logo=Streamlit\u0026logoColor=FFFFFF\u0026label\u0026style=flat\"\u003e\n\u003c/p\u003e\n\nWe present MAchine Translation Evaluation Online (MATEO), a project that aims to facilitate machine translation (MT)\nevaluation by means of an easy-to-use interface that can evaluate given machine translations with a battery of\nautomatic metrics. It caters to both experienced and novice users who are working with MT, such as MT system builders,\nand also researchers from Social Sciences and Humanities, and teachers and students of (machine) translation.\n\nMATEO can be accessed on [this website](https://mateo.ivdnt.org/), \nhosted by the [CLARIN](https://www.clarin.eu/) B center at [Instituut voor de Nederlandse Taal](https://ivdnt.org/).\nIt is also available on Hugging Face [Spaces](https://huggingface.co/spaces/BramVanroy/mateo-demo).\n\nIf you use the MATEO interface for your work, please cite our project paper!\n\nVanroy, B., Tezcan, A., \u0026 Macken, L. (2023). [MATEO: MAchine Translation Evaluation Online](https://aclanthology.org/2023.eamt-1.52/). In M. Nurminen, J. Brenner, M. Koponen, S. Latomaa, M. Mikhailov, F. Schierl, … H. Moniz (Eds.), _Proceedings of the 24th Annual Conference of the European Association for Machine Translation_ (pp. 499–500). Tampere, Finland: European Association for Machine Translation (EAMT).\n\n```bibtex\n@inproceedings{vanroy-etal-2023-mateo,\n    title = \"{MATEO}: {MA}chine {T}ranslation {E}valuation {O}nline\",\n    author = \"Vanroy, Bram  and\n      Tezcan, Arda  and\n      Macken, Lieve\",\n    booktitle = \"Proceedings of the 24th Annual Conference of the European Association for Machine Translation\",\n    month = jun,\n    year = \"2023\",\n    address = \"Tampere, Finland\",\n    publisher = \"European Association for Machine Translation\",\n    url = \"https://aclanthology.org/2023.eamt-1.52\",\n    pages = \"499--500\",\n}\n```\n\n## Self-hosting\n\nThe MATEO [website](https://lt3.ugent.be/mateo/) is provided for free as a hosted application. That means that you, or\nanyone else, can use it. The implication is that it is possible that the service will be slow depending on the usage of\nthe system. As such, specific attention was paid to making it easy for you to set up your own instance that you can use!\n\n### Duplicating a Hugging Face Spaces\n\nMATEO is also [running](https://huggingface.co/spaces/BramVanroy/mateo-demo) on the free platform of 🤗 Hugging Face in a\nso-called 'Space'. If you have an account (free) on that platform, you can easily duplicate the running MATEO instance\nto your own profile. That means that you can create a private duplication of the MATEO interface **just for you** and\nfree of charge! You can simply click [this link](https://huggingface.co/spaces/BramVanroy/mateo-demo?duplicate=true)\nor, if that does not work, follow these steps:\n\n1. Go to the [Space](https://huggingface.co/spaces/BramVanroy/mateo-demo);\n2. in the top right (below your profile picture) you should click on the three vertical dots;\n3. choose 'Duplicate space', _et\u0026nbsp;voilà!_, a new space should now be running on your own profile\n\n### Install locally with Python\n\nYou can clone and install the library on your own device (laptop, computer, server). I recommend to run this in a new \nvirtual environment. It requires `python \u003e= 3.10`.\n\nRun the following commands:\n\n```shell\ngit clone https://github.com/BramVanroy/mateo-demo.git\ncd mateo-demo\npython -m pip install .\ncd src/mateo_st\nstreamlit run 01_🎈_MATEO.py\n```\n\nThe streamlit server will then start on your own computer. You can access the website via a local address,\n[http://localhost:8501](http://localhost:8501) by default. \n\nConfiguration options specific to Streamlit can be found\n[here](https://docs.streamlit.io/library/advanced-features/configuration). They are more related to server-side configurations\nthat you typically do not need when you are running this directly through Python. But you may need them when you are\nusing Docker, e.g. setting the `--server.port` that streamlit is running on (see [Docker](#running-with-docker)).\n\nA number of command-line arguments are available to change the interface to your needs.\n\n```shell\n--use_cuda             whether to use CUDA for translation task (CUDA for metrics not supported) (default: False)                                                                                                                                      \n--demo_mode           when demo mode is enabled, only a limited range of neural check-points are available. So all metrics are available but not all of the checkpoints. (default: False)\n```\n\nThese can be passed to the Streamlit launcher by adding a `--` after the streamlit command and streamlit-specific\noptions, followed by any of the options above.\n\nFor instance, if you want to run streamlit specifically on port 1234 and you want to use the demo mode, you can modify\nyour command to look like this:\n\n```shell\nstreamlit run 01_🎈_MATEO.py --server.port 1234 -- --demo_mode\n```\n\nNote the separating `--` in the middle so that streamlit can distinguish between streamlit's own options and the MATEO\nconfiguration parameters.\n\n\n### Running with Docker\n\nIf you have docker installed, it is very easy to get a MATEO instance running.\n\nThe following Dockerfiles are available in the [`docker`](docker) directory. They are a little bit different depending\non the specific needs.\n\n- [`hf-spaces`](docker/hf-spaces/Dockerfile): specific configuration for Hugging Face spaces but without env options\n- [`default`](docker/default/Dockerfile): a more intricate Dockerfile that accepts environment variables to be used\nthat are specific to the server, demo functionality, and CUDA. These Docker environment variables are available.\n\n  - PORT: server port to expose and to run the streamlit server on (default: 7860)\n  - SERVER: server address to run on (default: 'localhost')\n  - BASE: base path (default: '')\n  - DEMO_MODE: set to `true` to disable some options for neural metrics and to limit the max. upload size to 1MB \n  per file (default: '')\n\nAs an example, to build and run the repository on port 5034 with CUDA disabled and demo mode enabled, you can run the\nfollowing commands which will automatically use the most recent `cpu` Dockerfile from Github.\n\n```shell\ndocker build -t mateo https://raw.githubusercontent.com/BramVanroy/mateo-demo/main/docker/cpu/Dockerfile\ndocker run --rm -d --name mateo-demo -p 5034:5034 --env PORT=5034 --env DEMO_MODE=true mateo\n```\n\nNote how the opened ports in Docker's `-p` must correspond with the env variable `PORT`!\n\nMATEO is now running on port 5034 and available on the local address [http://localhost:5034/](http://localhost:5034/).\n\nAs mentioned before, you can modify the Dockerfiles as you wish. Most notably you may want to change the `streamlit`\nlauncher command itself. Therefore you could use the [streamlit options]([here](https://docs.streamlit.io/library/advanced-features/configuration))\nalongside custom options for MATEO specifically, which were mentioned in the [previous section](#install-locally-with-python).\n\n## Tests\n\nThe tests are run using `pytest` and `playwright`. To ensure that the right dependencies are installed, you can run\n\n```shell\npython -m pip install -e .[dev]\n```\n\nThen, install the appropriate chromium version for playwright. You can do this by running the following command.\n\n```shell\nplaywright install --with-deps chromium\n```\n\nNow you can run the tests by running the following command in the root directory of the project.\n\n```shell\npython -m pytest\n```\n\n## Notes\n\n### Using CUDA\n\nUsing CUDA for the metrics is currently not supported. However, it is possible to use CUDA for the translation task.\nThis can be done by setting the `--use_cuda` flag when running the Streamlit server. This will enable the use of CUDA\nfor the translation task, but not for the metrics. The reason for this is the memory consumption since streamlit \ncreates a separate instance for each user, the GPU may run OOM quickly and moving on/off devices is not feasible. \n\nI have not found a solution for this yet. A queueing system would solve the issue with a separate backend and dedicated \nworkers, but that defeats the purpose of having a simple, easy-to-use interface. It would also lead to the requirement\nof strong data for longer, which many users may not want to, considering that I've received many questions whether I \nsave their data on disk (I don't - the current approach processes everything in memory).\n\n## Acknowledgements\n\nThis project was kickstarted by a Sponsorship project from the\n[European Association for Machine Translation](https://eamt.org/), and\na substantial follow-up grant by the support of [CLARIN.eu](https://www.clarin.eu/).\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://eamt.org/\" target=\"_blank\"\u003e\u003cimg alt=\"EAMT logo\" src=\"src/mateo_st/img/eamt.png\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.clarin.eu/\" target=\"_blank\"\u003e\u003cimg alt=\"CLARIN logo\" src=\"src/mateo_st/img/clarin.png\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbramvanroy%2Fmateo-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbramvanroy%2Fmateo-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbramvanroy%2Fmateo-demo/lists"}