{"id":17989532,"url":"https://github.com/evilfreelancer/rugpt-3.5-training","last_synced_at":"2025-03-25T23:30:47.086Z","repository":{"id":199122112,"uuid":"701726949","full_name":"EvilFreelancer/ruGPT-3.5-training","owner":"EvilFreelancer","description":null,"archived":false,"fork":false,"pushed_at":"2023-10-09T20:06:44.000Z","size":60,"stargazers_count":16,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-20T22:09:52.945Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EvilFreelancer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-07T11:42:58.000Z","updated_at":"2025-03-03T13:10:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"45bc2409-c963-4b8b-9732-bb52b3a8b8f5","html_url":"https://github.com/EvilFreelancer/ruGPT-3.5-training","commit_stats":null,"previous_names":["evilfreelancer/rugpt-3.5-training"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2FruGPT-3.5-training","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2FruGPT-3.5-training/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2FruGPT-3.5-training/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2FruGPT-3.5-training/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EvilFreelancer","download_url":"https://codeload.github.com/EvilFreelancer/ruGPT-3.5-training/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245561156,"owners_count":20635683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T19:14:53.429Z","updated_at":"2025-03-25T23:30:44.217Z","avatar_url":"https://github.com/EvilFreelancer.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Training ruGPT-3.5 13B with LoRA\n\nThis repository provides a curated collection of scripts and a Jupyter Notebook designed for training a custom\n`ruGPT-3.5-13B` model in the `load_in_8bit` mode utilizing datasets and some scripts\nfrom [Saiga-2 (rulm)](https://github.com/IlyaGusev/rulm).\n\nThe training process outlined here leverages Peft/LoRA technology. The resources provided are designed to facilitate a\nsmooth training experience, seamless merging of LoRA weights with the original model, and a straightforward conversion\nof the model into the GGML format.\n\n\u003e Note: While the training settings for this model mirror those used\n\u003e in [GigaSaiga](https://huggingface.co/IlyaGusev/gigasaiga_lora), but my model is enriched with\n\u003e additional dataset.\n\nThe primary objective of this repository is to reproduce the success achieved by GigaSaiga and to provide a detailed,\nstep-by-step documentation of the training procedure. This initiative aims to empower and support the Russian-speaking\nML community by making the process of training the ruGPT-3.5-13B model more accessible and understandable.\n\nFor your convenience, pretrained models are readily available at the following locations:\n\n- https://huggingface.co/evilfreelancer/ruGPT-3.5-13B-lora\n- https://huggingface.co/evilfreelancer/ruGPT-3.5-13B-ggml\n\nBy following the instructions and using the scripts provided in this repository, users can efficiently train their\nversions of the ruGPT-3.5-13B model with the flexibility to incorporate additional datasets as necessary.\n\n## Acknowledgments\n\nFirst of all I would like to extend our sincere gratitude to the following authors and contributors:\n\n- The [Sber AI](https://ai.sber.ru/) Team, the brains behind the original `ruGPT-3.5-13B` model. Their groundbreaking\n  work\n  and continuous efforts in advancing AI and machine learning technologies have laid a solid foundation for this project\n  and many others in the AI community.\n\n- [IlyaGusev](https://github.com/IlyaGusev) and the [rulm](https://github.com/IlyaGusev/rulm) project team for their\n  invaluable resources and datasets from Saiga-2/GigaSaiga, which have been fundamental in the training process of this\n  custom ruGPT-3.5-13B model.\n\n- [graysonwhite](https://github.com/graysonwhite) and the [gglm](https://github.com/graysonwhite/gglm) project team. I'm\n  particularly thankful for their comprehensive documentation on the ggml project, which has been indispensable in\n  guiding me through the correct procedures for model transformation.\n\n- [iashchak](https://huggingface.co/iashchak) for\n  his [ruGPT-3.5-13B-ggml](https://huggingface.co/iashchak/ruGPT-3.5-13B-ggml) repository on HuggingFace. His\n  contributions and shared expertise with [llm-rs-python](https://github.com/LLukas22/llm-rs-python) have been crucial\n  in the successful creation of this project.\n\nThis project has been significantly enriched and made possible through the cumulative efforts and shared knowledge of\nthese incredible individuals and teams. I deeply appreciate their contributions and are immensely thankful for their\nopenness to sharing resources with the broader community.\n\nFor anyone looking to understand, extend, or build upon my work, I strongly recommend referring to and acknowledging\nthese original authors and contributors, as their work represents the cornerstone of this project and many others\nin the field.\n\n## Global requirements\n\nBefore embarking on the training process, ensure your system meets the following requirements:\n\n* ~100 GB of system RAM\n* ~200 GB on HDD/SSD\n* Nvidia GPU with at least 20 GB VRAM (eg. RTX 3090 or 4090)\n* CUDA 12.2\n\n## Install libraries manually\n\nRequirements:\n\n* Python 3.10\n* Python VirtualEnv\n\nClone the repo with all submodules:\n\n```shell\ngit clone --recurse-submodules https://github.com/EvilFreelancer/ruGPT-3.5-training.git\n```\n\nInstantiate a virtual environment:\n\n```shell\npython -m venv venv\n```\n\nSwitch to a virtual environment:\n\n```shell\nsource venv/bin/activate\n```\n\nDownload Python packages:\n\n```shell\npip install -r requirements.txt\n```\n\n## Install libraries in Docker\n\nRequirements:\n\n* Docker\n* Docker Compose\n* Nvidia Docker Runtime\n\nSolution based\non [nvidia/cuda:12.2.0-devel-ubuntu22.04](https://hub.docker.com/r/nvidia/cuda/tags?page=1\u0026name=12.2.0-devel-ubuntu22.04)\nimage.\n\nClone the repo with all submodules:\n\n```shell\ngit clone --recurse-submodules https://github.com/EvilFreelancer/ruGPT-3.5-training.git\n```\n\nCopy compose config from dist (and change settings if you need):\n\n```shell\ncp docker-compose.dist.yml docker-compose.yml\n```\n\nBuild an image:\n\n```shell\ndocker-compose build\n```\n\nStart container:\n\n```shell\ndocker-compose build\n```\n\nAttach to container's shell:\n\n```shell\ndocker-compose exec app bash\n```\n\n## Training with LoRA\n\nThe entire process is broken down into four main steps, each corresponding to a\nscript in the project’s root directory. Below is a step-by-step guide.\n\n### Step 1: Dataset Download and Merging\n\n```shell\npython3 1_dataset.py\n```\n\nThe datasets utilized for training this model are consistent with those used\nfor [Saiga-2 (rulm)](https://github.com/IlyaGusev/rulm).\n\nHere's the comprehensive list:\n\n- [ru_turbo_alpaca](https://huggingface.co/datasets/IlyaGusev/ru_turbo_alpaca)\n- [ru_turbo_alpaca_evol_instruct](https://huggingface.co/datasets/IlyaGusev/ru_turbo_alpaca_evol_instruct)\n- [ru_turbo_saiga](https://huggingface.co/datasets/IlyaGusev/ru_turbo_saiga)\n- [ru_sharegpt_cleaned](https://huggingface.co/datasets/IlyaGusev/ru_sharegpt_cleaned)\n- [oasst1_ru_main_branch](https://huggingface.co/datasets/IlyaGusev/oasst1_ru_main_branch)\n- [gpt_roleplay_realm](https://huggingface.co/datasets/IlyaGusev/gpt_roleplay_realm)\n- [ru_instruct_gpt4](https://huggingface.co/datasets/lksy/ru_instruct_gpt4)\n\nTo download and merge all datasets from this list you need to execute:\n\nThe resultant datasets `train_full.jsonl` and `val_full.jsonl` are generated in chat format.\n\n### Step 2: Model Training LoRA\n\n```shell\npython3 2_train.py\n```\n\nThe sequence of operations performed by this script is as follows:\n\n1. **Download Original Model**: The script initiates by downloading the original `ruGPT-3.5-13B` model from HuggingFace.\n   The downloaded files are stored in the `ruGPT-3.5-13B` folder.\n\n2. **Configuration Modification**: After the download is complete, the script copies and modifies the configuration\n   files located in `ruGPT-3.5-13B` folder. The altered configurations, which are necessary to enable training, are then\n   placed in the `output` folder.\n\n3. **Training Initialization**: The script subsequently instantiates the `src.train` Python module from the `rulm`\n   project. This operation occurs within the `rulm/self_instruct` subdirectory.\n\n4. **Output Files**: Upon the completion of the above steps, `adapter_model.bin` and `adapter_config.json` are generated\n   and saved in the `output` folder.\n\nEach of the generated files plays a crucial role in the subsequent steps of the model training and application process.\n\n### Step 3: Model Merging\n\n```shell\npython3 3_merge.py\n```\n\nThis script performs the following tasks:\n\n- **Weights Merging**: It uses a modified version of the [convert_to_native.py] script. The script seamlessly merges the\n  LoRA adapter weights with the weights of the base `ruGPT-3.5-13B` model. This merging process is crucial for enhancing\n  the model’s performance with the learned adaptations from the LoRA training.\n\n- **Saving Merged Model**: After the merging process is complete, the script saves the resultant model with the filename\n  `pytorch_model.bin` in the `output` directory of project.\n\nEnsure you have sufficient storage space available in the `output` directory as the merged model file can be quite\nlarge.\n\n### Step 4: GGML Conversion\n\n```shell\npython3 4_ggml.py\n```\n\nThis step involves two main tasks:\n\n1. **Conversion to GGML-Compatible Format**: The script starts by converting the `pytorch_model.bin` file into a format\n   that is compatible with GGML. This converted format serves as an intermediate step that prepares the model for\n   subsequent quantization processes.\n\n2. **Quantization**: Following the initial conversion, the script performs quantization on the model. The quantization\n   process generates various quantized versions of the model, specifically: q4_0, q4_1, q5_0, q5_1, and q8_0. Each\n   quantized version is optimized for different levels of precision and performance requirements.\n\n3. **Library Utilization**: This entire process utilizes the [llm-rs-python](https://github.com/LLukas22/llm-rs-python)\n   library. Ensure that this library is installed and accessible, as it plays a pivotal role in the GGML conversion and\n   quantization processes.\n\n4. **Saving GGML Models**: Upon completion of the conversion and quantization steps, the script saves the resultant GGML\n   models in the `output_ggml` directory within your project’s root.\n\nEnsure you have adequate storage space available in the `output_ggml` directory, as the GGML models, especially the\nquantized versions, may occupy significant space.\n\n## Testing Scripts\n\nThe root directory contains four additional scripts for testing each intermediate step:\n\n- **test_gigasaiga.py**: Demonstrates the functionality of the original GigaSaiga as implemented by the authors of the\n  rulm project.\n- **test_lora.py**: Tests the on-the-fly merging of the LoRA adapter with adapter_model.bin from the output directory.\n- **test_merged.py**: Shows the functionality of the original ruGPT-3.5 model after LoRA weights merging.\n- **test_ggml.py**: Tests the GGML versions of the model to ensure proper functioning.\n\n## Support and Contribution\n\nFeel free to open issues or pull requests if you have suggestions or encounter issues.\nContributions to improve or expand this project are always welcome!\n\n## Links\n\n- https://huggingface.co/ai-forever/ruGPT-3.5-13B\n- https://github.com/IlyaGusev/rulm\n- https://github.com/ggerganov/ggml\n- https://github.com/LLukas22/llm-rs-python\n- https://huggingface.co/iashchak/ruGPT-3.5-13B-ggml\n- https://huggingface.co/iashchak/ruGPT-3.5-13B-ggml/discussions/3\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilfreelancer%2Frugpt-3.5-training","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevilfreelancer%2Frugpt-3.5-training","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilfreelancer%2Frugpt-3.5-training/lists"}