{"id":18887149,"url":"https://github.com/thunlp-mt/trice","last_synced_at":"2025-04-14T22:33:48.466Z","repository":{"id":86099763,"uuid":"370619452","full_name":"THUNLP-MT/TRICE","owner":"THUNLP-MT","description":"Code for our paper \"Transfer Learning for Sequence Generation: from Single-source to Multi-source\" in ACL 2021.","archived":false,"fork":false,"pushed_at":"2021-06-01T14:39:39.000Z","size":75,"stargazers_count":11,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-28T10:54:18.142Z","etag":null,"topics":["automatic-post-editing","finetuning","machine-translation","multi-source-translation","natural-language-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-05-25T08:26:08.000Z","updated_at":"2023-12-28T06:02:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"e08eeba7-0dd8-44e2-a9c2-bb110cb05422","html_url":"https://github.com/THUNLP-MT/TRICE","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FTRICE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FTRICE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FTRICE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FTRICE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/TRICE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248973046,"owners_count":21191903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automatic-post-editing","finetuning","machine-translation","multi-source-translation","natural-language-processing"],"created_at":"2024-11-08T07:34:46.684Z","updated_at":"2025-04-14T22:33:43.453Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TRICE: a task-agnostic transferring framework for multi-source sequence generation\n\nThis is the source code of our work \u003ca href=\"https://arxiv.org/abs/2105.14809\"\u003eTransfer Learning for Sequence Generation: from Single-source to Multi-source (ACL 2021)\u003c/a\u003e. \n\nWe propose TRICE, a task-agnostic Transferring fRamework for multI-sourCe sEquence generation, \nfor transferring pretrained models to multi-source sequence generation tasks \n(e.g., automatic post-editing, multi-source translation, and multi-document summarization).\nTRICE achieves new state-of-the-art results on the WMT17 APE task and the multi-source translation task using the WMT14 test set.\nWelcome to take a quick glance at our [blog](https://thumtblog.github.io/2021/05/27/TRICE/). \n\nThe implementation is on top of the open-source NMT toolkit [THUMT](https://github.com/thumt/THUMT). \n\n```bibtex\n@misc{huang2021transfer,\n      title={Transfer Learning for Sequence Generation: from Single-source to Multi-source}, \n      author={Xuancheng Huang and Jingfang Xu and Maosong Sun and Yang Liu},\n      year={2021},\n      eprint={2105.14809},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n\n## Contents\n* [Prerequisites](#Prerequisites)\n* [Pretrained\u0026#32;model](#Pretrained-model)\n* [Finetuning](#Finetuning)\n* [Inference](#Inference)\n* [Contact](#Contact)\n\n## Prerequisites\n* Python \u003e= 3.6\n* tensorflow-cpu \u003e= 2.0\n* torch \u003e= 1.7\n* transformers \u003e= 3.4\n* sentencepiece \u003e= 0.1\n\n## Pretrained\u0026#32;model\nWe adopt [mbart-large-cc25](https://huggingface.co/facebook/mbart-large-cc25) in our experiments. \nOther sequence-to-sequence pretrained models can also be used with only a few modifications.\n\nIf your GPUs do not have enough memories, you can prune the original large vocabulary (25k) to a small vocabulary (e.g., 3k) with little performance loss.\n\n## Finetuning\n\n### Single-source finetuning\n```powershell\nPYTHONPATH=${path_to_TRICE} \\\npython ${path_to_TRICE}/thumt/bin/trainer.py \\\n    --input ${train_src1} ${train_src2} ${train_trg} \\\n    --vocabulary ${vocab_joint} ${vocab_joint} \\\n    --validation ${dev_src1} ${dev_src2} \\\n    --references ${dev_ref} \\\n    --model transformer --half --hparam_set big \\\n    --output single_finetuned \\\n    --parameters \\\nfixed_batch_size=false,batch_size=820,train_steps=120000,update_cycle=5,device_list=[0,1,2,3],\\\nkeep_checkpoint_max=2,save_checkpoint_steps=2000,\\\neval_steps=2001,decode_alpha=1.0,decode_batch_size=16,keep_top_checkpoint_max=1,\\\nattention_dropout=0.1,relu_dropout=0.1,residual_dropout=0.1,learning_rate=5e-05,warmup_steps=4000,initial_learning_rate=5e-8,\\\nseparate_encode=false,separate_cross_att=false,segment_embedding=false,\\\ninput_type=\"single_random\",adapter_type=\"None\",num_fine_encoder_layers=0,normalization=\"after\",\\\nsrc_lang_tok=\"en_XX\",hyp_lang_tok=\"de_DE\",tgt_lang_tok=\"de_DE\",mbart_model_code=\"facebook/mbart-large-cc25\",\\\nspm_path=\"sentence.bpe.model\",pad=\"\u003cpad\u003e\",bos=\"\u003cs\u003e\",eos=\"\u003c/s\u003e\",unk=\"\u003cunk\u003e\"\n```\n\n### Multi-source finetuning\n```powershell\nPYTHONPATH=${path_to_TRICE} \\\npython ${path_to_TRICE}/thumt/bin/trainer.py \\\n    --input ${train_src1} ${train_src2} ${train_tgt} \\\n    --vocabulary ${vocab_joint} ${vocab_joint} \\\n    --validation ${dev_src1} ${dev_src2} \\\n    --references ${dev_ref} \\\n    --model transformer --half --hparam_set big \\\n    --checkpoint single_finetuned/eval/model-best.pt \\\n    --output multi_finetuned \\\n    --parameters \\\nfixed_batch_size=false,batch_size=820,train_steps=120000,update_cycle=5,device_list=[0,1,2,3],\\\nkeep_checkpoint_max=2,save_checkpoint_steps=2000,\\\neval_steps=2001,decode_alpha=1.0,decode_batch_size=16,keep_top_checkpoint_max=1,\\\nattention_dropout=0.1,relu_dropout=0.1,residual_dropout=0.1,learning_rate=5e-05,warmup_steps=4000,initial_learning_rate=5e-8,special_learning_rate=5e-04,special_var_name=\"adapter\",\\\nseparate_encode=false,separate_cross_att=true,segment_embedding=true,\\\ninput_type=\"\",adapter_type=\"Cross-attn\",num_fine_encoder_layers=1,normalization=\"after\",\\\nsrc_lang_tok=\"en_XX\",hyp_lang_tok=\"de_DE\",tgt_lang_tok=\"de_DE\",mbart_model_code=\"facebook/mbart-large-cc25\",\\\nspm_path=\"sentence.bpe.model\",pad=\"\u003cpad\u003e\",bos=\"\u003cs\u003e\",eos=\"\u003c/s\u003e\",unk=\"\u003cunk\u003e\"\n```\n\n### Arguments to be explained\n\n** `special_learning_rate`: if a variable's name contains `special_var_name`, the learning rate of it will be `special_learning_rate`. We give the fine encoder a larger learning rate.  \n** `separate_encode`: whether to encode multiple sources separately before the fine encoder.  \n** `separate_cross_att`: whether to use separated cross-attention described in our paper.  \n** `segment_embedding`: whether to use sinusoidal segment embedding described in our paper.  \n** `input_type`: `\"single_random\"` for single-source finetuning , `\"\"` for multi-source finetuning.  \n** `adapter_type`: `\"None\"` for no fine encoder,  `\"Cross-attn\"` for fine encoder with cross-attention.  \n** ``num_fine_encoder_layers``: number of fine encoder layers.  \n** ``src_lang_tok``: language token for the first source sentence. Please refer to [here](https://huggingface.co/facebook/mbart-large-cc25) for language tokens for all 25 languages.  \n** ``hyp_lang_tok``: language token for the second source sentence.  \n** ``tgt_lang_tok``: language token for the target sentence.  \n** ``mbart_model_code``: model code for [transformers](https://huggingface.co/models).  \n** ``spm_path``: sentence piece model (can download from [here](https://huggingface.co/facebook/mbart-large-cc25/tree/main)).\n\nExplanations for other arguments could be found in the user manual of [THUMT](https://github.com/thumt/THUMT).\n\n## Inference\n```\nPYTHONPATH=${path_to_TRICE} \\\npython ${path_to_TRICE}/thumt/bin/translator.py \\\n  --input ${test_src1} ${test_src2} --output ${test_tgt} \\\n  --vocabulary ${vocab_joint} ${vocab_joint} \\\n  --checkpoints multi_finetuned/eval/model-best.pt \\\n  --model transformer --half \\\n  --parameters device_list=[0,1,2,3],decode_alpha=1.0,decode_batch_size=32\n# recover sentence piece tokenization\n...\n# calculate BLEU\n...\n```\n\n## Contact\n\nIf you have questions, suggestions and bug reports, please email [xchuang17@163.com](xchuang17@163.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Ftrice","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Ftrice","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Ftrice/lists"}