{"id":25476493,"url":"https://github.com/jonsafari/multiway-corpus","last_synced_at":"2026-02-27T04:32:04.236Z","repository":{"id":90546756,"uuid":"79048983","full_name":"jonsafari/multiway-corpus","owner":"jonsafari","description":"Build an n-way multilingual corpus","archived":false,"fork":false,"pushed_at":"2017-01-30T14:45:11.000Z","size":70,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-28T06:07:29.903Z","etag":null,"topics":["corpus-data","machine-translation","mt","multilingual","multiway-corpus","zero-shot"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonsafari.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-01-15T17:52:30.000Z","updated_at":"2020-06-09T11:29:25.000Z","dependencies_parsed_at":"2023-07-18T11:00:56.329Z","dependency_job_id":null,"html_url":"https://github.com/jonsafari/multiway-corpus","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jonsafari/multiway-corpus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonsafari%2Fmultiway-corpus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonsafari%2Fmultiway-corpus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonsafari%2Fmultiway-corpus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonsafari%2Fmultiway-corpus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonsafari","download_url":"https://codeload.github.com/jonsafari/multiway-corpus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonsafari%2Fmultiway-corpus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29884709,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-26T23:51:21.483Z","status":"online","status_checked_at":"2026-02-27T02:00:06.759Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["corpus-data","machine-translation","mt","multilingual","multiway-corpus","zero-shot"],"created_at":"2025-02-18T12:57:24.284Z","updated_at":"2026-02-27T04:32:04.231Z","avatar_url":"https://github.com/jonsafari.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multiway Corpus\n\nThis builds an *n*-way multilingual corpus, from the data in the awesome [Tatoeba](http://tatoeba.org) dataset.\nThis allows you to do pivot-free [zero-shot](https://arxiv.org/abs/1611.04558) machine translation, as well as have unusual language combinations.\n\nUsage is:\n\n    python3 intersect_tatoeba.py Spanish jpn English\n\nThe arguments are the languages that you want to intersect, either the [ISO 639-3](data/lang_codes_iso-639-3.tsv) names (eg. English) or codes (eg. `eng`).\nThe output in this example will be `corpus.jpn`, `corpus.spa`, and `corpus.eng` .\n\nFirst download two files into this directory, as these are constantly being updated upstream:\n\n```bash\nwget -c http://downloads.tatoeba.org/exports/sentences.tar.bz2  \u0026\u0026  tar jxvf sentences.tar.bz2\nwget -c http://downloads.tatoeba.org/exports/links.tar.bz2      \u0026\u0026  tar jxvf links.tar.bz2\n```\n\nThen run the script.  Enjoy!\n\nHere are some languages in the upstream dataset:\n\n| Language | ISO 639-3 Code | Sentences |\n| --- | --- | --- |\n| English | eng | 641421 |\n| Esperanto | epo | 511221 |\n| Turkish | tur | 503109 |\n| Russian | rus | 479397 |\n| Italian | ita | 474880 |\n| German | deu | 366934 |\n| French | fra | 315677|\n| Spanish | spa | 265058 |\n| Portuguese | por | 231807 |\n| Hungarian | hun | 191328 |\n| Japanese | jpn | 184296 |\n| Hebrew | heb | 153655 |\n| Berber | ber | 104842 |\n| (Hundreds more languages) | | |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonsafari%2Fmultiway-corpus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonsafari%2Fmultiway-corpus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonsafari%2Fmultiway-corpus/lists"}