{"id":20467402,"url":"https://github.com/zhaocq-nlp/njunmt-tf","last_synced_at":"2025-10-16T20:06:40.357Z","repository":{"id":201455506,"uuid":"115672915","full_name":"zhaocq-nlp/NJUNMT-tf","owner":"zhaocq-nlp","description":"An open-source neural machine translation system developed by Natural Language Processing Group, Nanjing University.","archived":false,"fork":false,"pushed_at":"2018-09-06T07:51:52.000Z","size":855,"stargazers_count":103,"open_issues_count":1,"forks_count":41,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-13T09:12:13.166Z","etag":null,"topics":["attention","neural-machine-translation","njunmt-tf","nmt","tensor2tensor","tensorflow","transformer","translation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhaocq-nlp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-12-29T01:17:43.000Z","updated_at":"2025-02-23T11:55:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"8f92573f-0789-48f0-85e7-1c3cacda63b5","html_url":"https://github.com/zhaocq-nlp/NJUNMT-tf","commit_stats":null,"previous_names":["zhaocq-nlp/njunmt-tf"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zhaocq-nlp/NJUNMT-tf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhaocq-nlp%2FNJUNMT-tf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhaocq-nlp%2FNJUNMT-tf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhaocq-nlp%2FNJUNMT-tf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhaocq-nlp%2FNJUNMT-tf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhaocq-nlp","download_url":"https://codeload.github.com/zhaocq-nlp/NJUNMT-tf/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhaocq-nlp%2FNJUNMT-tf/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262748798,"owners_count":23358273,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","neural-machine-translation","njunmt-tf","nmt","tensor2tensor","tensorflow","transformer","translation"],"created_at":"2024-11-15T13:28:25.607Z","updated_at":"2025-10-16T20:06:35.310Z","avatar_url":"https://github.com/zhaocq-nlp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NJUNMT-tf\n\nNJUNMT-tf is a general purpose sequence modeling tool in TensorFlow while neural machine translation is the main target task.\n\n\n## Key features\n\n**NJUNMT-tf builds NMT models almost from scratch without any high-level TensorFlow APIs which often hide details of many network components and lead to obscure code structure that is difficult to understand and manipulate. NJUNMT-tf only depends on basic TensorFlow modules, like array_ops, math_ops and nn_ops. Each operation in the code is under control.** \u003c/br\u003e\n\nNJUNMT-tf focuses on modularity and extensibility using standard TensorFlow modules and practices to support advanced modeling capability:\n\n- arbitrarily complex encoder architectures, e.g. Bidirectional RNN encoder, Unidirectional RNN encoder and self-attention.\n- arbitrarily complex decoder architectures, e.g. Conditional GRU/LSTM decoder, attention decoder and self-attention.\n- hybrid encoder-decoder models, e.g. self-attention encoder and RNN decoder or vice versa.\n\nand all of the above can be used simultaneously to train novel and complex architectures.\n\nThe code also supports:\n\n- model ensemble.\n- learning rate decaying according to loss on evaluation data.\n- model validation on evaluation data with BLEU  score and early stop strategy.\n- monitoring with [TensorBoard](https://www.tensorflow.org/get_started/summaries_and_tensorboard).\n- capability for [BPE](https://github.com/rsennrich/subword-nmt)\n\n\n## Requirements\n\n- `tensorflow` (`\u003e=1.6`)\n- `pyyaml`\n\n\n## Quickstart\n\nHere is a minimal workflow to get you started in using NJUNMT-tf. This example uses a toy Chinese-English dataset for machine translation with a toy setting.\n\n1\\. Build the word vocabularies:\n\n``` bash\npython -m bin.generate_vocab testdata/toy.zh --max_vocab_size 100  \u003e testdata/vocab.zh\npython -m bin.generate_vocab testdata/toy.en0 --max_vocab_size 100  \u003e testdata/vocab.en\n```\n\n2\\. Train with preset sequence-to-sequence parameters:\n``` bash\nexport CUDA_VISIBLE_DEVICES=\npython -m bin.train --model_dir test_model \\\n    --config_paths \"\n        ./njunmt/example_configs/toy_seq2seq.yml,\n        ./njunmt/example_configs/toy_training_options.yml,\n        ./default_configs/default_optimizer.yml\"\n```\n\n3\\. Translate a test file with the latest checkpoint:\n``` bash\nexport CUDA_VISIBLE_DEVICES=\npython -m bin.infer --model_dir test_models \\\n  --infer \"\n    beam_size: 4\n    source_words_vocabulary: testdata/vocab.zh\n    target_words_vocabulary: testdata/vocab.en\" \\\n  --infer_data \"\n    - features_file: testdata/toy.zh\n      labels_file: testdata/toy.en\n      output_file: toy.trans\n      output_attention: false\"\n```\n\n**Note:** do not expect any good translation results with this toy example. Consider training on [larger parallel datasets](http://www.statmt.org/wmt16/translation-task.html) instead.\n\n## Configuration\n\nAs you can see, there are two ways to manipulate hyperparameters of the process:\n\n- tf FLAGS\n- yaml-style config file\n\nFor example, there is a config file specifying the datasets for training procedure.\n```\n# datasets.yml\ndata:\n  train_features_file: testdata/toy.zh\n  train_labels_file: testdata/toy.en0\n  eval_features_file: testdata/toy.zh\n  eval_labels_file: testdata/toy.en\n  source_words_vocabulary: testdata/vocab.zh\n  target_words_vocabulary: testdata/vocab.en\n```\n\nYou can either use the command:\n``` bash\npython -m bin.train --config_paths \"datasets.yml\" ...\n```\nor\n``` bash\npython -m bin.train --data \"\n    train_features_file: testdata/toy.zh\n    train_labels_file: testdata/toy.en0\n    eval_features_file: testdata/toy.zh\n    eval_labels_file: testdata/toy.en\n    source_words_vocabulary: testdata/vocab.zh\n    target_words_vocabulary: testdata/vocab.en\" ...\n```\nThey are of the same effect.\n\nThe available FLAGS (or the top levels of yaml configs) for bin.train are as follows:\n- **config_paths**: the paths for config files\n- **model_dir**: the directory for saving checkpoints\n- **problem_name**: The top name scope, \"seq2seq\" by default\n- **train**: training options, e.g. batch size, maximum length\n- **data**: training data, evaluation data, vocabulary and (optional) BPE codes\n- **hooks**: a list of training hooks (not provided, in the current version)\n- **metrics**: a list of evaluation metrics on evaluation data\n- **model**: the class name of the model\n- **model_params**: parameters for the model\n- **optimizer_params**: parameters for optimizer\n\nThe available FLAGS (or the top levels of yaml configs) for bin.infer are as follows:\n- **config_paths**: the paths for config files\n- **model_dir**: the checkpoint directory or directories separated by commas for model ensemble\n- **infer**: inference options, e.g. beam size, length penalty rate\n- **infer_data**: a list of data file to be translated\n- **weight_scheme**: the weight scheme for model ensemble (only \"average\" available now)\n\n**Note that:**\n- each FLAG should be a string of yaml-style\n- the hyperparameters provided by FLAGS will overwrite those presented in config files\n- illegal parameters will interrupt the program, so see [sample.yml](https://github.com/zhaocq-nlp/NJUNMT-tf/blob/master/njunmt/example_configs/sample.yml) of more detailed discription for each parameter.\n\n\n## Benchmarks\n\nThe RNN benchmarks are performed on 1 GTX 1080Ti GPU with predefined configurations:\n\n- `default_configs/adam_loss_decay.yml`\n- `default_configs/default_metrics.yml`\n- `default_configs/default_training_options.yml`\n- `default_configs/seq2seq_cgru.yml`\n\nThe Transformer benchmarks are performed on 1 GTX 1080Ti GPU with predefined configurations:\n\n- `default_configs/transformer_base.yml`\n- `default_configs/transformer_training_options.yml`\n\nNote that in Transformer model, we set `batch_tokens_size=2500` with `update_cycle=10` to realize pseudo parallel training.\n\n\nThe beam sizes for RNN and Transformer are 10 and 4 respectively.\n\nThe datasets are preprocessed using [fetch_wmt2017_ende.sh](https://github.com/zhaocq-nlp/MT-data-processing/blob/master/fetch_wmt2017_ende.sh) and [fetch_wmt2018_zhen.sh](https://github.com/zhaocq-nlp/MT-data-processing/blob/master/fetch_wmt2018_zhen.sh) referring to [Edinburgh’s Report](http://statmt.org/wmt17/pdf/WMT39.pdf).\n\n\nThe BLEU scores are evaluated by the wrapper script [run_mteval.sh](https://github.com/zhaocq-nlp/NJUNMT-tf/blob/master/njunmt/tools/mteval/run_mteval.sh). For EN-ZH experiments, the BLEU scores are evaluated at character-level while others are evaluated at word-level.\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth rowspan=\"2\"\u003eDataset\u003c/th\u003e\n    \u003cth rowspan=\"2\"\u003e Model\u003c/th\u003e\n    \u003cth colspan=\"2\"\u003e BLEU \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003enewstest2016(dev)\u003c/td\u003e\n    \u003ctd\u003enewstest2017\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"2\"\u003eWMT17 EN-DE\u003c/td\u003e\n    \u003ctd\u003eRNN\u003c/td\u003e\n    \u003ctd\u003e29.6\u003c/td\u003e\n    \u003ctd\u003e23.6\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eTransformer\u003c/td\u003e\n    \u003ctd\u003e33.5\u003c/td\u003e\n    \u003ctd\u003e27.0\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"2\"\u003eWMT17 DE-EN\u003c/td\u003e\n    \u003ctd\u003eRNN\u003c/td\u003e\n    \u003ctd\u003e34.0\u003c/td\u003e\n    \u003ctd\u003e29.6\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eTransformer\u003c/td\u003e\n    \u003ctd\u003e37.6\u003c/td\u003e\n    \u003ctd\u003e33.1\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth rowspan=\"2\"\u003eDataset\u003c/th\u003e\n    \u003cth rowspan=\"2\"\u003e Model\u003c/th\u003e\n    \u003cth colspan=\"2\"\u003e BLEU \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003enewsdev2017(dev)\u003c/td\u003e\n    \u003ctd\u003enewstest2017\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"2\"\u003eWMT17 ZH-EN\u003c/td\u003e\n    \u003ctd\u003eRNN\u003c/td\u003e\n    \u003ctd\u003e19.7\u003c/td\u003e\n    \u003ctd\u003e21.2\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eTransformer\u003c/td\u003e\n    \u003ctd\u003e22.7\u003c/td\u003e\n    \u003ctd\u003e25.0\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"2\"\u003eWMT17 EN-ZH\u003c/td\u003e\n    \u003ctd\u003eRNN\u003c/td\u003e\n    \u003ctd\u003e30.0\u003c/td\u003e\n    \u003ctd\u003e30.2\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eTransformer\u003c/td\u003e\n    \u003ctd\u003e34.9\u003c/td\u003e\n    \u003ctd\u003e35.0\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n## TODO\n\nThe following features remain unimplemented:\n\n- multi-gpu training\n- schedule sampling\n- minimum risk training\n\n\n## Acknowledgments\n\nThe implementation is inspired by the following:\n- *[Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473)*\n- [dl4mt-tutorial](https://github.com/nyu-dl/dl4mt-tutorial)\n- [OpenNMT-tf](https://github.com/OpenNMT/OpenNMT-tf)\n- [Google's seq2seq](https://github.com/google/seq2seq) \u003c/br\u003e\n*[Massive Exploration of Neural Machine Translation Architectures](https://arxiv.org/abs/1703.03906)*\n- [THUMT](https://github.com/thumt/THUMT)\n- [Google's tensor2tensor](https://github.com/tensorflow/tensor2tensor) \u003c/br\u003e\n*[Attention is All You Need](https://arxiv.org/abs/1706.03762)*\n- *[Stronger Baselines for Trustable Results in Neural Machine Translation](http://www.aclweb.org/anthology/W17-3203.pdf)*\n\n## Contact\n\nAny comments or suggestions are welcome.\n\nPlease email [zhaocq.nlp@gmail.com](mailto:zhaocq.nlp@gmail.com).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhaocq-nlp%2Fnjunmt-tf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhaocq-nlp%2Fnjunmt-tf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhaocq-nlp%2Fnjunmt-tf/lists"}