{"id":18887126,"url":"https://github.com/thunlp-mt/ktnmt","last_synced_at":"2025-10-03T16:17:29.638Z","repository":{"id":244947012,"uuid":"679206630","full_name":"THUNLP-MT/ktnmt","owner":"THUNLP-MT","description":null,"archived":false,"fork":false,"pushed_at":"2023-08-16T10:35:15.000Z","size":8209,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-28T10:54:14.242Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-16T10:28:18.000Z","updated_at":"2024-12-04T07:20:56.000Z","dependencies_parsed_at":"2024-06-18T14:53:34.269Z","dependency_job_id":null,"html_url":"https://github.com/THUNLP-MT/ktnmt","commit_stats":null,"previous_names":["thunlp-mt/ktnmt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2Fktnmt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2Fktnmt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2Fktnmt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2Fktnmt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/ktnmt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248973005,"owners_count":21191897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T07:34:23.471Z","updated_at":"2025-10-03T16:17:29.537Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KTNMT\n\n## Introduction\nSource code for the ACL 2023 main conference long paper \"Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation\" (Outstanding Paper Award)\n\nIn this work, we propose a knowledge transfer method that can efficiently adapt original MNMT models to diverse incremental language pairs.\n\n---\n## Get Started\n(Core) Data Preprocessing.\n\nStandard BPE Procedure: following https://github.com/google/sentencepiece with 64k merged BPE tokens.\n\nAfter obtaining the original vocabulary and the incremental vocabulary, you must get an incremental vocabulary of external models.\n\nModel Training:\n\nThis system has been tested in the following environment.\n\nPython version == 3.7\n\nPytorch version == 1.8.0\n\nFairseq version == 0.12.0 (pip install fairseq)\n\nNote that it only influences the training procedure of the original and incremental model. You can choose your favorite deep learning library for model training.\n\n---\n## Incremental Learning\nWe build the incremental learning procedure for Multilingual Neural Machine Translation as follows:\n\n1. Get original multilingual translation models (or train a multilingual translation model by yourself).\n\n2. Preprocessing incremental data\n- Data Clean (optional, if needed)\n- Get Vocabulary (optional, if needed, follow standard BPE procedure)\n\n3. Incremental Training.\n    We provide all runing scripts in the folder ''src''. \n    Here is an example:\n    ```bash\n    echo \"Task format: \"\n    export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\n    export SEED=222\n    export CHECKPOINT_PATH='' # save new model\n    export CL_CHECKPOINT_PATH='' # old model\n    export DATA_PATH='' # incremental data bin\n    export USER_PATH='mkt/' # user dir\n    export ARCH=transformer_adapter\n    export TASK=translation_multi_adapter\n    export ADAPTER_MODULE=serial # serial / parallel / parallel_gate / parallel_norm / parallel_dp\n    export KRAIL_MODULE='' # krail / krail_ffn / krail_map / krail_map_last / krail_map_avg / krail_adaptive / krail_adaptive_final\n    export LANG_DICT='misc/cl_adapter/lang_dicts_uk.txt' # new lang dict\n    export KRAIL_MODEL='mPLM/418M_last_checkpoint.pt' # external model path\n    export NEW_DICT_PATH='misc/cl_adapter/model_dict.m2m100.vex.txt' # combination model dict, we can get this vocab by scripts\n    export KRAIL_DICT_PATH='misc/m2m100/model_dict.128k.txt' # external model dict\n    export lang_pairs=en-uk\n    export CLIP_NORM=0.0\n    export OPTIMIZER=adam\n    export ADAM_EPS=1e-9\n    export LR=5e-4\n    export LR_SCHEDULER=inverse_sqrt\n    export WARMUP=4000\n    export DROPOUT=0.2\n    export ATT_DROPOUT=0.2\n    export WEIGHT_DECAY=0.0001\n    export CRITERION=label_smoothed_cross_entropy\n    export LABEL_SMOOTHING=0.1\n    export MAX_TOKENS=1024\n    export UPDATE_FREQ=8\n    export SAVE_INTERVAL_UPDATES=2500\n    export KEEP_INTERVAL_UPDATES=1\n    export MAX_UPDATE=500000\n\n    fairseq-train $DATA_PATH \\\n    --user-dir $USER_PATH \\\n    --share-all-embeddings \\\n    --encoder-normalize-before --decoder-normalize-before \\\n    --encoder-embed-dim 1024 --encoder-ffn-embed-dim 8192 --encoder-attention-heads 16 \\\n    --decoder-embed-dim 1024 --decoder-ffn-embed-dim 8192 --decoder-attention-heads 16 \\\n    --encoder-layers 24 --decoder-layers 24 \\\n    --left-pad-source False --left-pad-target False \\\n    --arch $ARCH \\\n    --task $TASK \\\n    --sampling-method temperature \\\n    --sampling-temperature 5 \\\n    --lang-tok-style multilingual \\\n    --lang-dict $LANG_DICT \\\n    --lang-pairs $lang_pairs \\\n    --encoder-langtok src \\\n    --decoder-langtok \\\n    --clip-norm $CLIP_NORM \\\n    --optimizer $OPTIMIZER \\\n    --adam-betas '(0.9, 0.98)' \\\n    --adam-eps $ADAM_EPS \\\n    --lr $LR \\\n    --lr-scheduler $LR_SCHEDULER \\\n    --warmup-updates $WARMUP \\\n    --dropout $DROPOUT \\\n    --attention-dropout $ATT_DROPOUT \\\n    --weight-decay $WEIGHT_DECAY \\\n    --criterion $CRITERION \\\n    --label-smoothing $LABEL_SMOOTHING \\\n    --max-tokens $MAX_TOKENS \\\n    --save-dir $CHECKPOINT_PATH/checkpoints/ \\\n    --update-freq $UPDATE_FREQ \\\n    --save-interval-updates $SAVE_INTERVAL_UPDATES \\\n    --keep-interval-updates $KEEP_INTERVAL_UPDATES \\\n    --max-update $MAX_UPDATE \\\n    --no-epoch-checkpoints \\\n    --seed $SEED --log-format simple --log-interval 300 \\\n    --patience 10 \\\n    --tensorboard-logdir $CHECKPOINT_PATH/logs/ \\\n    --no-progress-bar \\\n    --ddp-backend no_c10d \\\n    --finetune-from-model $CL_CHECKPOINT_PATH \\\n    --freeze-all \\\n    --adapter-inner-dims 4096 \\\n    --adapter-module $ADAPTER_MODULE \\\n    --krail-module $KRAIL_MODULE \\\n    --vocab-adapter \\\n    --base-embed-dim 32211 \\\n    --krail-model $KRAIL_MODEL \\\n    --new-dict-path $NEW_DICT_PATH \\\n    --krail-dict-path $KRAIL_DICT_PATH\n    ```\n---\n## Inference \u0026 Evaluation\nPlease refer to src/inference.sh \u0026 run_sh/evaluate.sh\n\n---\n## Citation\n```\n@inproceedings{huang-etal-2023-knowledge,\n    title = \"Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation\",\n    author = \"Huang, Kaiyu  and\n      Li, Peng  and\n      Ma, Jin  and\n      Yao, Ting  and\n      Liu, Yang\",\n    booktitle = \"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)\",\n    month = jul,\n    year = \"2023\",\n    address = \"Toronto, Canada\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2023.acl-long.852\",\n    doi = \"10.18653/v1/2023.acl-long.852\",\n    pages = \"15286--15304\",\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fktnmt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Fktnmt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fktnmt/lists"}