{"id":13493743,"url":"https://github.com/ai-forever/ru-gpts","last_synced_at":"2025-05-15T14:04:37.898Z","repository":{"id":37573529,"uuid":"303541321","full_name":"ai-forever/ru-gpts","owner":"ai-forever","description":"Russian GPT3 models.","archived":false,"fork":false,"pushed_at":"2022-12-12T08:42:02.000Z","size":392,"stargazers_count":2091,"open_issues_count":12,"forks_count":442,"subscribers_count":86,"default_branch":"master","last_synced_at":"2025-04-07T18:09:13.811Z","etag":null,"topics":["deep-learning","gpt3","language-model","russian","russian-language","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ai-forever.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-13T00:00:15.000Z","updated_at":"2025-03-30T10:44:53.000Z","dependencies_parsed_at":"2022-07-12T16:24:20.538Z","dependency_job_id":null,"html_url":"https://github.com/ai-forever/ru-gpts","commit_stats":null,"previous_names":["sberbank-ai/ru-gpts"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2Fru-gpts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2Fru-gpts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2Fru-gpts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2Fru-gpts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ai-forever","download_url":"https://codeload.github.com/ai-forever/ru-gpts/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254355334,"owners_count":22057354,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","gpt3","language-model","russian","russian-language","transformers"],"created_at":"2024-07-31T19:01:18.379Z","updated_at":"2025-05-15T14:04:37.877Z","avatar_url":"https://github.com/ai-forever.png","language":"Python","funding_links":[],"categories":["Python","Искусственный интеллект и машинное обучение","Stale"],"sub_categories":["ML \u0026 Training"],"readme":"#### Russian GPT-3 models\r\n# ruGPT3XL, ruGPT3Large, ruGPT3Medium, ruGPT3Small and ruGPT2Large\r\nThis repository contains bunch of autoregressive transformer language models trained on a huge dataset of russian language.\r\n\r\n * Russian GPT-3 models (ruGPT3XL, ruGPT3Large, ruGPT3Medium, ruGPT3Small) trained with 2048 sequence length with sparse and dense attention blocks. We also provide Russian GPT-2 large model (ruGPT2Large) trained with 1024 sequence length.\r\n\r\n * Try Model Generation In Colab! ruGPT-3 XL: [![Try Model Generation In Colab!](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ai-forever/ru-gpts/blob/master/examples/ruGPT3XL_generation.ipynb) or ruGPT-3 smaller models: [![Try Model Generation In Colab!](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ai-forever/ru-gpts/blob/master/examples/Generate_text_with_RuGPTs_HF.ipynb)\r\n \r\n * Usage examples are described in detail [here](examples/). See how fine-tuning works: [![Try Model Generation In Colab!](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ai-forever/ru-gpts/blob/master/examples/ruGPT3XL_finetune_example.ipynb)\r\n\r\n## Table of contents\r\n* [ruGPT3XL](#ruGPT3XL)\r\n  * Setup\r\n  * Usage\r\n  * Finetune\r\n  * [Pretraining details ruGPT3XL](#Pretraining-details-ruGPT3XL)\r\n* [ruGPT3Large, ruGPT3Medium, ruGPT3Small, ruGPT2Large](ruGPT3Large,-ruGPT3Medium,-ruGPT3Small,-ruGPT2Large)\r\n  * Setup\r\n  * Usage\r\n  * Pretraining details\r\n    * [Pretraining details ruGPT3Large](#Pretraining-details-ruGPT3Large)\r\n    * [Pretraining details ruGPT3Medium](#Pretraining-details-ruGPT3Medium)\r\n    * [Pretraining details ruGPT3Small](#Pretraining-details-ruGPT3Small)\r\n    * [Pretraining details ruGPT2Large](#Pretraining-details-ruGPT2Large)\r\n* [Papers mentioning ruGPT3](#Papers-mentioning-ruGPT3)\r\n* [OpenSource Solutions with ruGPT3](#OpenSource-Solutions-with-ruGPT3)\r\n\r\n# ruGPT3XL\r\n## Setup\r\nFor colab we recommend use the following installation instructions:\r\n\r\n```%%bash\r\nexport LD_LIBRARY_PATH=/usr/lib/\r\napt-get install clang-9 llvm-9 llvm-9-dev llvm-9-tools\r\ngit clone https://github.com/qywu/apex\r\ncd apex\r\npip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ./\r\npip install triton\r\nDS_BUILD_CPU_ADAM=1 DS_BUILD_SPARSE_ATTN=1 pip install deepspeed\r\npip install transformers\r\npip install huggingface_hub\r\npip install timm==0.3.2\r\ngit clone  https://github.com/sberbank-ai/ru-gpts\r\ncp ru-gpts/src_utils/trainer_pt_utils.py /usr/local/lib/python3.8/dist-packages/transformers/trainer_pt_utils.py\r\ncp ru-gpts/src_utils/_amp_state.py /usr/local/lib/python3.8/dist-packages/apex/amp/_amp_state.py\r\n```\r\n\r\nAfter installation env please restart colab. For checking is all ok, run the following commands:\r\n\r\n```\r\n!ds_report\r\n# Output:\r\n...\r\nsparse_attn ............ [YES] ...... [OKAY]\r\n...\r\nimport deepspeed.ops.sparse_attention.sparse_attn_op\r\n```\r\n\r\n## Usage\r\nHere is a simple example of usage. For more see this [example](examples/ruGPT3XL_generation.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ai-forever/ru-gpts/blob/master/examples/ruGPT3XL_generation.ipynb).\r\n\r\n```python\r\nimport sys\r\nfrom src.xl_wrapper import RuGPT3XL\r\nimport os\r\n\r\n# If run to from content root.\r\nsys.path.append(\"ru-gpts/\")\r\nos.environ[\"USE_DEEPSPEED\"] = \"1\"\r\n# We can change address and port\r\nos.environ[\"MASTER_ADDR\"] = \"127.0.0.1\"\r\nos.environ[\"MASTER_PORT\"] = \"5000\"\r\ngpt = RuGPT3XL.from_pretrained(\"sberbank-ai/rugpt3xl\", seq_len=512)\r\ngpt.generate(\r\n    \"Кто был президентом США в 2020? \",\r\n    max_length=50,\r\n    no_repeat_ngram_size=3,\r\n    repetition_penalty=2.,\r\n)\r\n```\r\n## Finetuning\r\nExample of finetune, load finetuned model and generate is [here](examples/ruGPT3XL_finetune_example.ipynb).\r\n\r\nOur example of finetuning script [here](scripts/deepspeed_gpt3_xl_finetune.sh)\r\n\r\n## Pretraining details ruGPT3XL\r\nModel was trained with 512 sequence length using [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron](https://github.com/NVIDIA/Megatron-LM) code by [Devices](https://sberdevices.ru/) team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.  \r\n*Note! Model has sparse attention blocks.*\r\n\r\nTotal training time was around 10 days on 256 GPUs.  \r\nFinal perplexity on test set is `12.05`.\r\n\r\n🤗HuggingFace model card [link](https://HuggingFace.co/ai-forever/rugpt3xl).\r\n\r\n# ruGPT3Large, ruGPT3Medium, ruGPT3Small, ruGPT2Large\r\n## Setup\r\nFor using ruGPT3Large, ruGPT3Medium, ruGPT3Small, ruGPT2Large just install 🤗HuggingFace transformers.\r\n\r\n```bash\r\npip install transformers==4.24.0\r\n```\r\n\r\n## Usage\r\nHere we can obtain examples of [finetuning](examples/RuGPT3FinetuneHF.ipynb) or [generation](examples/Generate_text_with_RuGPTs_HF.ipynb).\r\n\r\nAlso this examples is adapted for google colab:\r\n* finetuning: [![finetuning](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ai-forever/ru-gpts/blob/master/examples/RuGPT3FinetuneHF.ipynb).\r\n* generation: [![generation](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ai-forever/ru-gpts/blob/master/examples/Generate_text_with_RuGPTs_HF.ipynb)\r\n\r\n```python\r\nfrom transformers import GPT2LMHeadModel, GPT2Tokenizer\r\n\r\n\r\nmodel_name_or_path = \"sberbank-ai/rugpt3large_based_on_gpt2\"\r\ntokenizer = GPT2Tokenizer.from_pretrained(model_name_or_path)\r\nmodel = GPT2LMHeadModel.from_pretrained(model_name_or_path).cuda()\r\ntext = \"Александр Сергеевич Пушкин родился в \"\r\ninput_ids = tokenizer.encode(text, return_tensors=\"pt\").cuda()\r\nout = model.generate(input_ids.cuda())\r\ngenerated_text = list(map(tokenizer.decode, out))[0]\r\nprint(generated_text)\r\n# Output should be like this:\r\n# Александр Сергеевич Пушкин родился в \\n1799 году. Его отец был крепостным крестьянином, а мать – крепостной крестьянкой. Детство и юность Пушкина прошли в деревне Михайловское под Петербургом. В 1820-х годах семья переехала\r\n```\r\n\r\n### Pretraining details\r\nAll pretraining was done on Nvidia Tesla V100-SXM3 32 Gb GPUs on a [Christofari Cluster](https://en.wikipedia.org/wiki/Christofari). Following are the details of pretraining for each model.\r\n\r\n#### Pretraining details ruGPT3Large\r\nModel was trained with sequence length 1024 using transformers lib by [Devices](https://sberdevices.ru/) team on 80B tokens for 3 epochs. After that model was finetuned 1 epoch with sequence length 2048. \r\n\r\nTotal training time was around 14 days on 128 GPUs for 1024 context and few days on 16 GPUs for 2048 context.  \r\nFinal perplexity on test set is `13.6`.\r\n\r\nYou can obtain this model by using transformers with model name `sberbank-ai/rugpt3large_based_on_gpt2`.\r\n\r\n🤗HuggingFace model card [link](https://HuggingFace.co/sberbank-ai/rugpt3large_based_on_gpt2)\r\n\r\nOur pretraining script [here](scripts/deepspeed_gpt3_large.sh)\r\n\r\n#### Pretraining details ruGPT3Medium\r\nModel was trained with sequence length 1024 using transformers lib by [Devices](https://sberdevices.ru/) team on 80B tokens for 3 epoch. After that model was finetuned on 2048 context.\r\n\r\nTotal training time was around 16 days on 64 GPUs.  \r\nFinal perplexity on test set is `17.4`.\r\n\r\nYou can obtain this model by using transformers with model name `sberbank-ai/rugpt3medium_based_on_gpt2`. \r\n\r\n🤗HuggingFace model card [link](https://HuggingFace.co/sberbank-ai/rugpt3medium_based_on_gpt2)\r\n\r\nOur pretraining script [here](scripts/deepspeed_gpt3_medium.sh)\r\n\r\n#### Pretraining details ruGPT3Small\r\nModel was trained with sequence length 1024 using transformers by [Devices](https://sberdevices.ru/) team on 80B tokens around 3 epoch. After that model was finetuned on 2048 context.\r\n\r\nTotal training time took around one week on 32 GPUs.\r\n\r\nYou can obtain this model by using transformers with model name `sberbank-ai/rugpt3small_based_on_gpt2`. \r\n\r\n🤗HuggingFace model card [link](https://HuggingFace.co/sberbank-ai/rugpt3small_based_on_gpt2)\r\n\r\nOur pretraining script [here](scripts/deepspeed_gpt3_small.sh)\r\n\r\n#### Pretraining details ruGPT2Large\r\nModel was trained with sequence length 1024 using transformers by [Devices](https://sberdevices.ru/) team on 170Gb data on 64 GPUs 3 weeks.\r\n\r\nYou can obtain this model by using transformers with model name `sberbank-ai/rugpt2large`.\r\n\r\n🤗HuggingFace model card [link](https://HuggingFace.co/sberbank-ai/rugpt2large)\r\n\r\n## OpenSource Solutions with ruGPT3\r\n\r\n* ruCLIP [Github](https://github.com/ai-forever/ru-clip)\r\n* Simplification with ruGPT-3 XL [Github](https://github.com/Alenush/rugpt3simplification_rsse )\r\n* Word normalization (RuNormAS shared task) [Github](https://github.com/RussianNLP/RuNormAS-solution)\r\n* AI CopyWriter [Github](https://github.com/dilyararimovna/text_expansion)\r\n* ЕГЭ Generation [Github](https://github.com/orzhan/rugpt3-question-generation )\r\n* NeuroZhirinovsky [Github](https://github.com/GraphGrailAi/ruGPT3-ZhirV)\r\n* PseudoKant [Github](https://github.com/AsakoKabe/pseudo-kant )\r\n* DostoevskyDoesntWriteIt [Github](https://github.com/K7chyp/DostoevskyDoesntWriteIt)\r\n\r\n\r\n## Papers mentioning ruGPT3\r\nAccording to google scholar [search](https://scholar.google.com/scholar?hl=ru\u0026as_sdt=0%2C5\u0026q=rugpt3\u0026btnG=) - feel free to add links to this list\r\n\r\n### Text Simplification\r\n```\r\n@article{shatilovsentence,\r\n  title={Sentence simplification with ruGPT3},\r\n  author={Shatilov, AA and Rey, AI},\r\n  url={http://www.dialog-21.ru/media/5281/shatilovaaplusreyai142.pdf}\r\n}\r\n\r\n@article{fenogenovatext,\r\n  title={Text Simplification with Autoregressive Models},\r\n  author={Fenogenova, Alena},\r\n  url={http://www.dialog-21.ru/media/5250/fenogenovaa141.pdf}}\r\n  ```\r\n\r\n### Text Detoxification\r\n```\r\n@article{dementieva2021methods,\r\n  title={Methods for Detoxification of Texts for the Russian Language},\r\n  author={Dementieva, Daryna and Moskovskiy, Daniil and Logacheva, Varvara and Dale, David and Kozlova, Olga and Semenov, Nikita and Panchenko, Alexander},\r\n  journal={arXiv preprint arXiv:2105.09052},\r\n  year={2021},\r\n  url={https://arxiv.org/abs/2105.09052}\r\n}\r\n```\r\n\r\n### Paraphrasing and Data Augmentation\r\n```\r\n@inproceedings{fenogenova2021russian,\r\n  title={Russian Paraphrasers: Paraphrase with Transformers},\r\n  author={Fenogenova, Alena},\r\n  booktitle={Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing},\r\n  pages={11--19},\r\n  year={2021},\r\n  url={https://www.aclweb.org/anthology/2021.bsnlp-1.2.pdf}\r\n}\r\n``` \r\n\r\n### Model Evaluation\r\n```\r\n@article{malykh2021morocco,\r\n  title={MOROCCO: Model Resource Comparison Framework},\r\n  author={Malykh, Valentin and Kukushkin, Alexander and Artemova, Ekaterina and Mikhailov, Vladislav and Tikhonova, Maria and Shavrina, Tatiana},\r\n  journal={arXiv preprint arXiv:2104.14314},\r\n  year={2021},\r\n  url={https://arxiv.org/abs/2104.14314}}\r\n  ``` \r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-forever%2Fru-gpts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai-forever%2Fru-gpts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-forever%2Fru-gpts/lists"}