{"id":21478960,"url":"https://github.com/mjpost/wmt15","last_synced_at":"2025-03-17T08:27:35.012Z","repository":{"id":35333885,"uuid":"39595976","full_name":"mjpost/wmt15","owner":"mjpost","description":"Data and code for computing the results of the WMT15 manual evaluation","archived":false,"fork":false,"pushed_at":"2016-03-29T16:43:20.000Z","size":12163,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-06T02:27:21.090Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mjpost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-23T22:07:53.000Z","updated_at":"2022-09-18T19:21:43.000Z","dependencies_parsed_at":"2022-08-31T20:10:31.126Z","dependency_job_id":null,"html_url":"https://github.com/mjpost/wmt15","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjpost%2Fwmt15","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjpost%2Fwmt15/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjpost%2Fwmt15/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjpost%2Fwmt15/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mjpost","download_url":"https://codeload.github.com/mjpost/wmt15/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243998690,"owners_count":20381201,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T11:20:18.475Z","updated_at":"2025-03-17T08:27:34.983Z","avatar_url":"https://github.com/mjpost.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to produce the WMT human rankings\n\n0. If you *really* want to start from the beginning, download the raw XML dump of the \n   ranking tasks from [Appraise](http://appraise.cf/admin/wmt15/hit/) and save it\n   as data/wmt15.xml.gz. Then run\n\n        cd data\n        python xml2csv.py wmt15.xml.gz\n\n   This will generate wmt15.XXX.csv files, one for each language pair, and anonymize the\n   judges.\n\n   Note that WMT15 had judges compare unique outputs, instead of system outputs (which\n   was done prior to WMT15). Rankings were then assigned to all systems that had a unique\n   output. For this reason, the WMT CSV file has a slightly different format: each line\n   is a pairwise (2-way) comparison instead of a 5-way one. The last field, rankingID, groups\n   together the judgments that were in an individual ranking task.\n\n1. Now, to compute the rankings. Install wmt-trueskill:\n\n        git clone https://github.com/mjpost/wmt-trueskill\n\n2. Install the trueskill code\n\n        cd wmt-trueskill/src\n        git clone https://github.com/sublee/trueskill\n\n3. Compute the rankings\n\n        for type in ew ts; do \n          for lang in $(ls data/wmt15*csv | cut -d. -f2); do \n            qsub scripts/run-1000.sh $lang $type\n          done\n        done\n\n   If you're not on a cluster, it's\n\n        for type in ew ts; do \n          for lang in $(ls data/wmt15*csv | cut -d. -f2); do \n            for num in $(seq 1 1000); do\n              SGE_TASK_ID=$num ./scripts/run-1000.sh $lang $type\n            done\n          done\n        done\n\n4. Compute the clusters\n\n        for lang in $(ls data/wmt15*csv | cut -d. -f2); do \n          ./wmt-trueskill/eval/cluster.py -by-rank results/$lang/ts/*/*.json \u003e results/wmt15.$lang.txt; \n        done\n\n5. That's it! Please direct questions to Matt Post \u003cpost@cs.jhu.edu\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmjpost%2Fwmt15","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmjpost%2Fwmt15","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmjpost%2Fwmt15/lists"}