{"id":21478953,"url":"https://github.com/mjpost/contrapro","last_synced_at":"2026-07-17T15:33:35.900Z","repository":{"id":169885327,"uuid":"439752277","full_name":"mjpost/ContraPro","owner":"mjpost","description":null,"archived":false,"fork":false,"pushed_at":"2023-05-26T12:55:47.000Z","size":1433,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-29T19:48:28.387Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mjpost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-12-19T01:42:32.000Z","updated_at":"2023-05-26T21:23:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"a263d660-7c5f-4dca-a5cf-8ce44f43c0df","html_url":"https://github.com/mjpost/ContraPro","commit_stats":null,"previous_names":["mjpost/contrapro"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mjpost/ContraPro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjpost%2FContraPro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjpost%2FContraPro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjpost%2FContraPro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjpost%2FContraPro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mjpost","download_url":"https://codeload.github.com/mjpost/ContraPro/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mjpost%2FContraPro/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35587132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-17T02:00:06.162Z","response_time":116,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T11:20:15.850Z","updated_at":"2026-07-17T15:33:35.895Z","avatar_url":"https://github.com/mjpost.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"ContraPro\n---------\n\n`ContraPro` is a large-scale test set meant to\n- evaluate a specific discourse phenomenon: pronoun translation - automatically\n- promote contrastive evaluation of machine translation systems\n\nThe test set allows for a targeted evaluation of **English--German** pronoun translation, with a _contrastive_ set of translations.\n\nPlease note: this repository does not contain any OpenSubtitles data. Instead, it includes code that automatically downloads resources.\n\nContrastive Evaluation\n----------------------\n\nContrastive evaluation means to use a trained translation model to produce _scores_. Crucially, it does not involve any translation;\nthe translations are already given. Any translation system evaluated with this method must be able to provide model scores (negative log probabilities) for existing translations.\n\nThe input for scoring is a sentence pair, and the output is a single number. For instance:\n\n    (\"Say, if you get near a song, play it.\", \"Wenn Ihnen ein Song über den Weg läuft, spielen Sie ihn einfach.\") -\u003e 0.1975\n\nThe key idea of contrastive evaluation is to compare this score (`0.1975` in the example) to the score obtained with another pair of sentences,\nwhere the translation is _corrupted_ in a certain way. In our case, we replace correct pronouns with wrong ones, as in:\n\n    \"Wenn Ihnen ein Song über den Weg läuft, spielen Sie es einfach.\"\n\nAnd if a translation model gives lower scores to those _contrastive_ pairs, as in, for example:\n\n    (\"Say, if you get near a song, play it.\", \"Wenn Ihnen ein Song über den Weg läuft, spielen Sie es einfach.\") -\u003e 0.0043\n\nWe refer to this as a \"correct decision\" by the model. If this happens consistently, we conclude that the model can\ndiscriminate between good and bad translations.\n\n\nUsage Instructions\n------------------\n\nDownload ContraPro, for instance by cloning:\n\n    git clone https://github.com/ZurichNLP/ContraPro\n    cd ContraPro\n\nDownload Opensubtitles2018 and extract documents, preferably by just running this predefined script:\n\n    ./setup_opensubs.sh\n\nExtract raw text (plus context) for the ContraPro test set. Note that you can choose the number of context sentences according to what your translation system supports: a sentence-level system does not see any context, a context-aware system might observe 1 to n sentences as context.\n\n    perl conversion_scripts/json2text_and_context.pl --source en --target de --dir \\\n    [/path/to/OpenSubtitles_with_document_splitting, e.g. \"documents\"] --json contrapro.json --context 1\n    \nOr use the Python script, which generates n-n source-target pairs, tab-delimited, on STDOUT (e.g., for use with `marian-scorer`):\n\n    # context + sentence pair max 250 tokens with SPM\n    python3 conversion_scripts/json2text.py --dir /path/to/opensubtitles/dir -m 250 --spm /path/to/spm model\n\n    # context 1 sentence\n    python3 conversion_scripts/json2text.py --dir /path/to/opensubtitles/dir -m 1 --count-sents\n\nThe previous step will produce 4 files:\n\n- `contrapro.text.{en,de}`: Source and target sentences, one sentence per line.\n- `contrapro.context.{en,de}`: Source and target contexts, one sentence per line. If a sentence in `contrapro.text.{en,de}` has no context (e.g. because it is the first sentence in a document), this corresponds to an _empty line_ in `contrapro.context.{en,de}`.\n\nApply the preprocessing necessary for your system, and score each line in `contrapro.text.de` with your translation system (conditioned on the source in `contrapro.text.en`, and the context in `contrapro.context.{en,de}` - it is your responsibility to pass these in the appropriate format to your system).\n\nUse the scores produced in the previous step (one per line) to evaluate your system. By default, lower scores are interpreted as better. If your system produces scores where higher is better, add the argument `--maximize`\n\n    python evaluate.py --reference contrapro.json --scores [/path/to/your/scores]\n\n\nPublication\n-----------\n\nIf you use ContraPro, please cite the following paper:\n\nMathias Müller; Annette Rios; Elena Voita; Rico Sennrich (2018). A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation. In WMT 2018. Brussels, Belgium. http://www.statmt.org/wmt18/pdf/WMT007.pdf\n\n```\n@inproceedings{mueller2018,\n  address = \"Brussels, Belgium\",\n  author = \"M{\\\"u}ller, Mathias and Rios, Annette and Voita, Elena and Sennrich, Rico\",\n  booktitle = \"{WMT 2018}\",\n  publisher = \"Association for Computational Linguistics\",\n  title = \"{A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation}\",\n  year = \"2018\"\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmjpost%2Fcontrapro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmjpost%2Fcontrapro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmjpost%2Fcontrapro/lists"}