{"id":13563728,"url":"https://github.com/facebookresearch/Mask-Predict","last_synced_at":"2025-04-03T20:31:36.296Z","repository":{"id":41081462,"uuid":"205230497","full_name":"facebookresearch/Mask-Predict","owner":"facebookresearch","description":"A masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. ","archived":true,"fork":false,"pushed_at":"2021-09-17T22:07:33.000Z","size":2472,"stargazers_count":241,"open_issues_count":13,"forks_count":38,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-12-17T01:37:40.771Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookresearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-29T18:53:08.000Z","updated_at":"2024-11-12T00:32:56.000Z","dependencies_parsed_at":"2022-08-31T05:03:26.419Z","dependency_job_id":null,"html_url":"https://github.com/facebookresearch/Mask-Predict","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2FMask-Predict","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2FMask-Predict/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2FMask-Predict/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2FMask-Predict/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookresearch","download_url":"https://codeload.github.com/facebookresearch/Mask-Predict/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247074716,"owners_count":20879305,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:22.668Z","updated_at":"2025-04-03T20:31:35.396Z","avatar_url":"https://github.com/facebookresearch.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Mask-Predict\n\n\n### Download model \nDescription | Dataset | Model\n---|---|---\nMASK-PREDICT | [WMT14 English-German] | [download (.tar.bz2)](http://dl.fbaipublicfiles.com/fairseq/models/maskPredict_en_de.tar.gz)\nMASK-PREDICT | [WMT14 German-English] | [download (.tar.bz2)](http://dl.fbaipublicfiles.com/fairseq/models/maskPredict_de_en.tar.gz)\nMASK-PREDICT | [WMT16 English-Romanian] | [download (.tar.bz2)](http://dl.fbaipublicfiles.com/fairseq/models/maskPredict_en_ro.tar.gz)\nMASK-PREDICT | [WMT16 Romanian-English] | [download (.tar.bz2)](http://dl.fbaipublicfiles.com/fairseq/models/maskPredict_ro_en.tar.gz)\nMASK-PREDICT | [WMT17 English-Chinese] | [download (.tar.bz2)](http://dl.fbaipublicfiles.com/fairseq/models/maskPredict_en_zh.tar.gz)\nMASK-PREDICT | [WMT17 Chinese-English] | [download (.tar.bz2)](http://dl.fbaipublicfiles.com/fairseq/models/maskPredict_zh_en.tar.gz)\n\n### Preprocess\n\ntext=PATH_YOUR_DATA\n\noutput_dir=PATH_YOUR_OUTPUT\n\nsrc=source_language\n\ntgt=target_language\n\nmodel_path=PATH_TO_MASKPREDICT_MODEL_DIR\n\npython preprocess.py --source-lang ${src} --target-lang ${tgt} --trainpref $text/train --validpref $text/valid --testpref $text/test  --destdir ${output_dir}/data-bin  --workers 60  --srcdict ${model_path}/maskPredict_${src}_${tgt}/dict.${src}.txt --tgtdict ${model_path}/maskPredict_${src}_${tgt}/dict.${tgt}.txt\n\n### Train\n\n\nmodel_dir=PLACE_TO_SAVE_YOUR_MODEL\n\npython train.py ${output_dir}/data-bin --arch bert_transformer_seq2seq --share-all-embeddings --criterion label_smoothed_length_cross_entropy --label-smoothing 0.1 --lr 5e-4 --warmup-init-lr 1e-7 --min-lr 1e-9 --lr-scheduler inverse_sqrt --warmup-updates 10000 --optimizer adam --adam-betas '(0.9, 0.999)' --adam-eps 1e-6 --task translation_self --max-tokens 8192 --weight-decay 0.01 --dropout 0.3 --encoder-layers 6 --encoder-embed-dim 512 --decoder-layers 6 --decoder-embed-dim 512  --fp16 --max-source-positions 10000 --max-target-positions 10000 --max-update 300000 --seed 0 --save-dir ${model_dir}\n\n### Evaluation\n\n\npython generate_cmlm.py ${output_dir}/data-bin  --path ${model_dir}/checkpoint_best_average.pt  --task translation_self --remove-bpe --max-sentences 20 --decoding-iterations 10  --decoding-strategy mask_predict\n\n# License\nMASK-PREDICT is CC-BY-NC 4.0.\nThe license applies to the pre-trained models as well.\n\n# Citation\n\nPlease cite as:\n\n```bibtex\n@inproceedings{ghazvininejad2019MaskPredict,\n  title = {Mask-Predict: Parallel Decoding of Conditional Masked Language Models},\n  author = {Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer},\n  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},\n  year = {2019},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2FMask-Predict","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookresearch%2FMask-Predict","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2FMask-Predict/lists"}