{"id":13564174,"url":"https://github.com/google-research/multilingual-t5","last_synced_at":"2025-04-03T21:30:28.355Z","repository":{"id":38322658,"uuid":"306422385","full_name":"google-research/multilingual-t5","owner":"google-research","description":null,"archived":true,"fork":false,"pushed_at":"2022-12-15T16:32:35.000Z","size":84,"stargazers_count":1250,"open_issues_count":19,"forks_count":128,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-11-04T17:47:07.454Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-22T18:12:24.000Z","updated_at":"2024-11-04T10:54:45.000Z","dependencies_parsed_at":"2023-01-29T03:45:15.612Z","dependency_job_id":null,"html_url":"https://github.com/google-research/multilingual-t5","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fmultilingual-t5","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fmultilingual-t5/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fmultilingual-t5/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fmultilingual-t5/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/multilingual-t5/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247082804,"owners_count":20880722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:27.552Z","updated_at":"2025-04-03T21:30:28.040Z","avatar_url":"https://github.com/google-research.png","language":"Python","funding_links":[],"categories":["Python","A01_文本生成_文本对话","**Tools, Libraries, Models**","3. Pretrained models","Pre-trained resources","Urdu NLP Tools, Libraries and Models"],"sub_categories":["其他_文本生成_文本对话","Transformers, BERT","Sequence-to-sequence models","Multilingual Transformer models","Language Models"],"readme":"# mT5: Multilingual T5\n\nMultilingual T5 (mT5) is a massively multilingual pretrained text-to-text\ntransformer model, trained following a similar recipe as\n[T5](https://github.com/google-research/text-to-text-transfer-transformer).\nThis repo can be used to reproduce the experiments in the [mT5 paper][paper].\n\n## Table of Contents\n\n* [Languages covered](#languages-covered)\n* [Results](#results)\n* [Usage](#usage)\n  * [Training](#training)\n  * [Fine-Tuning](#fine-tuning)\n* [Released Model Checkpoints](#released-model-checkpoints)\n* [How to Cite](#how-to-cite)\n\n## Languages covered\n\nmT5 is pretrained on the [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual_nights_stay) corpus, covering 101 languages:\n\nAfrikaans, Albanian, Amharic, Arabic, Armenian, Azerbaijani, Basque,\nBelarusian, Bengali, Bulgarian, Burmese, Catalan, Cebuano, Chichewa, Chinese,\nCorsican, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino,\nFinnish, French, Galician, Georgian, German, Greek, Gujarati, Haitian Creole,\nHausa, Hawaiian, Hebrew, Hindi, Hmong, Hungarian, Icelandic, Igbo, Indonesian,\nIrish, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Kurdish,\nKyrgyz, Lao, Latin, Latvian, Lithuanian, Luxembourgish, Macedonian, Malagasy,\nMalay, Malayalam, Maltese, Maori, Marathi, Mongolian, Nepali, Norwegian,\nPashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Samoan,\nScottish Gaelic, Serbian, Shona, Sindhi, Sinhala, Slovak, Slovenian, Somali,\nSotho, Spanish, Sundanese, Swahili, Swedish, Tajik, Tamil, Telugu, Thai,\nTurkish, Ukrainian, Urdu, Uzbek, Vietnamese, Welsh, West Frisian, Xhosa,\nYiddish, Yoruba, Zulu.\n\n## Results\n\nmT5 achieves state-of-the-art performance on many cross-lingual NLP tasks, as\nof November 2020. For example, on\n[XTREME](https://github.com/google-research/xtreme) zero-shot classification,\nstructured prediction and QA tasks (showing F1 scores):\n\n| Model | XNLI | PAWS-X | WikiAnn-NER | XQuAD | MLQA | TyDiQA-GoldP |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| mBERT | 65.4 | 81.9 | 62.2 | 64.5 | 61.4 | 59.7 |\n| XLM | 69.1 | 80.9 | 61.2 | 59.8 | 48.5 | 43.6 |\n| InfoXLM | 81.4 | - | - | - | 73.6 | - |\n| X-STILTs | 80.4 | 87.7 | 64.7 | 77.2 | 72.3 | 76.0 |\n| XLM-R | 79.2 | 86.4 | 65.4 | 76.6 | 71.6 | 65.1 |\n| VECO | 79.9 | 88.7 | 65.7 | 77.3 | 71.7 | 67.6 |\n| RemBERT | 80.8 | 87.5 | **70.1** | 79.6 | 73.1 | 77.0 |\n| mT5-Small | 67.5 | 82.4 | 50.5 | 58.1 | 54.6 | 36.4 |\n| mT5-Base | 75.4 | 86.4 | 55.7 | 67.0 | 64.6 | 59.1 |\n| mT5-Large | 81.1 | 88.9 | 58.5 | 77.8 | 71.2 | 68.4 |\n| mT5-XL | 82.9 | 89.6 | 65.5 | 79.5 | 73.5 | 77.8 |\n| mT5-XXL | **85.0** | **90.0** | 69.2 | **82.5** | **76.0** | **82.0** |\n\n## Usage\n\n### Training\n\nTo run this code, you need to install the [t5\nlibrary](https://pypi.org/project/t5/). General instructions for training,\nfine-tuning, evaluation, and exporting models for inference can be found in the\n[t5\nrepo](https://github.com/google-research/text-to-text-transfer-transformer). In\norder to use the additional mT5 tasks provided in this library with the\n`t5_mesh_transformer` command, run from this directory and add the flag\n`--module_import=\"multilingual_t5.tasks\"`. There is also support for [mT5 in\nHuggingFace](https://huggingface.co/transformers/model_doc/mt5.html); see\ninstructions in the T5 repo\n[here](https://github.com/google-research/text-to-text-transfer-transformer#t5models).\n\nTo train an `mT5-Large` model on the\n[mc4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual_nights_stay)\ntask from scratch as described in the paper:\n\n```\nexport PROJECT=yourproject\nexport ZONE=yourzone\nexport BUCKET=yourbucket\nexport TPU=yourtpu\n\nctpu up --name=$TPU --project=$PROJECT --zone=$ZONE --tpu-size=v3-256 --tpu-only --noconf\n\nTASK=mc4\nMODEL_DIR=\"${BUCKET}${TASK}\"\n\npython -m t5.models.mesh_transformer_main \\\n  --tpu=\"${TPU}\" \\\n  --gcp_project=\"${PROJECT}\" \\\n  --tpu_zone=\"${ZONE}\" \\\n  --model_dir=\"${MODEL_DIR}\" \\\n  --gin_file=\"models/t5.1.1.large.gin\" \\\n  --gin_param=\"MIXTURE_NAME = '${TASK}'\" \\\n  --gin_param=\"utils.run.sequence_length = {'inputs': 1024, 'targets': 256}\" \\\n  --gin_param=\"utils.run.batch_size = ('tokens_per_batch', 1048576)\" \\\n  --gin_param=\"utils.run.learning_rate_schedule=@learning_rate_schedules.rsqrt_no_ramp_down\" \\\n  --gin_param=\"run.train_steps = 1000000\" \\\n  --gin_param=\"utils.tpu_mesh_shape.model_parallelism = 1\" \\\n  --gin_param=\"utils.tpu_mesh_shape.tpu_topology = 'v3-256'\" \\\n  --eval_mode=\"perplexity_eval\" \\\n  --eval_gin_param=\"mesh_eval_dataset_fn.num_eval_examples = 10000\" \\\n  --t5_tfds_data_dir=\"${BUCKET}/t5-tfds\" \\\n  --module_import=\"multilingual_t5.tasks\"\n```\n\n### Fine-Tuning\n\nThe example below shows how to finetune the `mT5-Large` model on the XNLI\nzeroshot task. See `finetune_mt5_tasks.sh` for hyperparameter settings for\nother tasks.\n\n```\nexport PROJECT=yourproject\nexport ZONE=yourzone\nexport BUCKET=yourbucket\nexport TPU=yourtpu\n\nctpu up --name=$TPU --project=$PROJECT --zone=$ZONE --tpu-size=v3-256 --tpu-only --noconf\n\nTASK=mt5_xnli_zeroshot\nSEQUENCE_LENGTH_GIN=xnli\nPRETRAINED_DIR=gs://t5-data/pretrained_models/mt5/large\nPRETRAINED_STEPS=1000000\nFINETUNE_STEPS=20000\nMODEL_DIR=\"${BUCKET}${TASK}\"\n\n# Run fine-tuning\npython -m t5.models.mesh_transformer_main \\\n  --tpu=\"${TPU}\" \\\n  --gcp_project=\"${PROJECT}\" \\\n  --tpu_zone=\"${ZONE}\" \\\n  --model_dir=\"${MODEL_DIR}\" \\\n  --gin_file=\"${PRETRAINED_DIR}/operative_config.gin\" \\\n  --gin_file=\"sequence_lengths/${SEQUENCE_LENGTH_GIN}.gin\" \\\n  --gin_param=\"utils.tpu_mesh_shape.tpu_topology = 'v3-256'\" \\\n  --gin_param=\"MIXTURE_NAME = '${TASK}'\" \\\n  --gin_param=\"utils.run.train_steps=$((PRETRAINED_STEPS+FINETUNE_STEPS))\" \\\n  --gin_param=\"utils.run.init_checkpoint='${PRETRAINED_DIR}/model.ckpt-${PRETRAINED_STEPS}'\" \\\n  --t5_tfds_data_dir=\"${BUCKET}/t5-tfds\" \\\n  --module_import=\"multilingual_t5.tasks\" \\\n  --gin_param=\"utils.run.batch_size = ('tokens_per_batch', 1048576)\" \\\n  --gin_location_prefix=\"multilingual_t5/gin/\"\n```\n\nThe remaining experiments are shown in the [tasks.py](multilingual_t5/tasks.py) file.\n\n## Released Model Checkpoints\n\nWe have released the following checkpoints for pre-trained models described in our [paper][paper]:\n\n* **mT5-Small** (300 million parameters): [gs://t5-data/pretrained_models/mt5/small](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/mt5/small/)\n* **mT5-Base** (580 million parameters): [gs://t5-data/pretrained_models/mt5/base](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/mt5/base/)\n* **mT5-Large** (1.2 billion parameters): [gs://t5-data/pretrained_models/mt5/large](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/mt5/large/)\n* **mT5-XL** (3.7 billion parameters): [gs://t5-data/pretrained_models/mt5/xl](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/mt5/xl/)\n* **mT5-XXL** (13 billion parameters): [gs://t5-data/pretrained_models/mt5/xxl](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/mt5/xxl/)\n\n# How to Cite\n\nIf you extend or use this work, please cite the [paper][paper] where it was\nintroduced:\n\n```\n@inproceedings{xue-etal-2021-mt5,\n    title = \"m{T}5: A Massively Multilingual Pre-trained Text-to-Text Transformer\",\n    author = \"Xue, Linting  and\n      Constant, Noah  and\n      Roberts, Adam  and\n      Kale, Mihir  and\n      Al-Rfou, Rami  and\n      Siddhant, Aditya  and\n      Barua, Aditya  and\n      Raffel, Colin\",\n    booktitle = \"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies\",\n    month = jun,\n    year = \"2021\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.naacl-main.41\",\n    doi = \"10.18653/v1/2021.naacl-main.41\",\n    pages = \"483--498\"\n}\n```\n\n[paper]: https://aclanthology.org/2021.naacl-main.41/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fmultilingual-t5","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fmultilingual-t5","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fmultilingual-t5/lists"}