{"id":19287816,"url":"https://github.com/deeppavlov/diff_beam_search","last_synced_at":"2025-10-13T02:43:06.908Z","repository":{"id":83450940,"uuid":"126012133","full_name":"deeppavlov/diff_beam_search","owner":"deeppavlov","description":null,"archived":false,"fork":false,"pushed_at":"2018-09-26T10:49:16.000Z","size":605,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-10-13T02:43:06.210Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deeppavlov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"support/res.png","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-03-20T12:12:20.000Z","updated_at":"2023-02-06T09:50:44.000Z","dependencies_parsed_at":"2023-07-18T16:31:40.714Z","dependency_job_id":null,"html_url":"https://github.com/deeppavlov/diff_beam_search","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/deeppavlov/diff_beam_search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fdiff_beam_search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fdiff_beam_search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fdiff_beam_search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fdiff_beam_search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deeppavlov","download_url":"https://codeload.github.com/deeppavlov/diff_beam_search/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2Fdiff_beam_search/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279014017,"owners_count":26085346,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T22:07:18.509Z","updated_at":"2025-10-13T02:43:06.903Z","avatar_url":"https://github.com/deeppavlov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Code for paper: XXX\n\n![alt text](https://raw.githubusercontent.com/deepmipt-VladZhukov/pytorch_nmt/master/support/res.png)\n\n## File Structure\n\n* `nmt.py`: main file\n* `vocab.py`: script used to generate `.bin` vocabulary file from parallel corpus\n* `util.py`: script containing helper functions\n## Usage\nSee ```EXAMPLE_SETUP.md```\n* Pytorch **0.4** is used.\n\n* Add expected_bleu module to PYTHONPATH\n\n``` export PYTHONPATH=\"${PYTHONPATH}:/\u003cPATH\u003e/\u003cTO\u003e/\u003crepo(bleu_lower_bound)\u003e/expected_bleu\" ```\nIf you just want to include this loss into your MT - see expected_bleu directory. If you want to reproduce read on.\n* (See NOTE below) Run the script (borrowed from [Harvard NLP repo](https://github.com/harvardnlp/BSO/tree/master/data_prep/MT)) to download and preprocess IWSLT'14 dataset:\n```shell\n$ cd preprocessing\n$ source prepareData.sh\n```\nNOTE: this script requires Lua and luaTorch. As an alternative, you can download all necessary files(data directory) from [this repo](https://github.com/pcyin/pytorch_nmt/tree/master/data) or via this [link](https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/pcyin/pytorch_nmt/tree/master/data)\n\n* Generate Vocabulary Files\n\n```\npython vocab.py\n```\nExample:\n```\npython vocab.py --train_src data/train.de-en.de --train_tgt data/train.de-en.en --output data/vocab.bin\n```\n\n* Vanilla Maximum Likelihood Training\n\n```\n. scripts/run_mle.sh\n```\n* BLEU LB train\n\n```\n. scripts/run_custom_train\n```\n\n* REINFORCE train\n\n```\n. scripts/run_custom_train3\n```\n\n#### Training (one experiment)\n```\nbash scripts/run_mle.sh \u003cgpu_id\u003e\nbash scripts/run_custom_train.sh models/model_name \u003cgpu_id\u003e\n\nbash scripts/test_mle.sh \u003cpath to model\u003e \u003cgpu_id\u003e \u003cmode_name\u003e\n```\n```\u003cmode_name\u003e``` equals to test or train (depends on what dataset you want to evaluate).\n\nSee result in logs directory.\n\nFor multi experiments testing see scripts:\n```run_experiments.sh``` - script which will run run_all.sh script for all gpus you’ll write in (see file content).\n\n```run_all.sh``` - script for training (uncomment/comment different phases of training)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fdiff_beam_search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeppavlov%2Fdiff_beam_search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fdiff_beam_search/lists"}