{"id":17989528,"url":"https://github.com/evilfreelancer/saiga-custom","last_synced_at":"2025-11-03T17:00:08.695Z","repository":{"id":207471536,"uuid":"719305686","full_name":"EvilFreelancer/saiga-custom","owner":"EvilFreelancer","description":"Bunch of notebooks for pre-training custom Saiga-like LLM","archived":false,"fork":false,"pushed_at":"2024-02-09T07:03:41.000Z","size":1611,"stargazers_count":13,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-06T14:21:23.393Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EvilFreelancer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":["https://pay.cloudtips.ru/p/937f48ac","https://boosty.to/evilfreelancer"]}},"created_at":"2023-11-15T22:15:04.000Z","updated_at":"2024-05-14T02:04:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"47960856-61dd-4ef5-b01c-c244204ed34f","html_url":"https://github.com/EvilFreelancer/saiga-custom","commit_stats":null,"previous_names":["evilfreelancer/saiga-custom"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EvilFreelancer/saiga-custom","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2Fsaiga-custom","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2Fsaiga-custom/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2Fsaiga-custom/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2Fsaiga-custom/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EvilFreelancer","download_url":"https://codeload.github.com/EvilFreelancer/saiga-custom/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2Fsaiga-custom/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":282494848,"owners_count":26678624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-03T02:00:05.676Z","response_time":108,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T19:14:52.976Z","updated_at":"2025-11-03T17:00:08.269Z","avatar_url":"https://github.com/EvilFreelancer.png","language":"Jupyter Notebook","funding_links":["https://pay.cloudtips.ru/p/937f48ac","https://boosty.to/evilfreelancer"],"categories":[],"sub_categories":[],"readme":"# Saiga-Custom Project\n\nWelcome to the `saiga-custom` project, a comprehensive collection of Jupyter notebooks specifically designed for\ntraining large language models on datasets from the [Saiga (rulm)](https://github.com/IlyaGusev/rulm) project. This\nrepository is an essential resource for anyone looking to leverage the advanced capabilities of the Saiga datasets for\nlanguage model training.\n\nOur notebooks are crafted to provide intuitive, step-by-step guidance for training state-of-the-art LoRA adapters for\ndifferent language models, ensuring that even those new to the field can successfully navigate the complexities of\nlanguage model training.\n\n## Repository Contents\n\n### Jupyter Notebooks\n\n* [yarn_mistral_7b_128k.ipynb](./yarn_mistral_7b_128k.ipynb) - this notebook contains a script for training the\n  [NousResearch/Yarn-Mistral-7b-128k](https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k) model. This model, an\n  advancement over the base [Mistral 7B v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1), incorporates\n  the [Flash Attention 2](https://github.com/Dao-AILab/flash-attention) algorithm, enabling it to handle a\n  context size of up to 128k tokens. The notebook provides a detailed and user-friendly guide for training the LoRA\n  adapter specifically for the Yarn-Mistral-7b-128k model. It meticulously outlines the necessary steps and parameters\n  required to optimize performance and achieve the best possible results with this enhanced model.\n* [rugpt35_13b.ipynb](./rugpt35_13b.ipynb) - This notebook focuses on training\n  the [ruGPT-3.5-13B](https://huggingface.co/ai-forever/ruGPT-3.5-13B) model, a powerful\n  language model specifically tailored for understanding and generating Russian text. It guides users through creating a\n  LoRA layer for model adaptation and subsequently performing a conversion to the GGML format for optimized deployment.\n* [llama2_7b_yakovlev.ipynb](./llama2_7b_yakovlev.ipynb) - This notebook provides a detailed guide for training a\n  Russian language model based on the [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)\n  model. The model is trained to imitate a historical figure\n  named [Ivan Yakovlevich Yakovlev](https://en.wikipedia.org/wiki/Ivan_Yakovlev).\n* [pavelgpt_7b_128k.ipynb](./pavelgpt_7b_128k.ipynb) - This notebook provides a detailed guide for training a Russian\n  language model based on\n  the [NousResearch/Yarn-Mistral-7b-128k](https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k) model. It is able to\n  generate text in Russian, answer questions, solve simple logical puzzles and simple math calculations. It is optimized\n  for INSTRUCT mode and it works better if you give it system prompt and only one instruction (without history at all).\n\n### Scripts\n\n* [test_lora.py](./test_lora.py) - this script features a console-based chat interface and a Conversation class that\n  maintains a message history. It is specifically adapted to function seamlessly with the Mistral model. The script\n  demonstrates a practical application of the model, showcasing its conversational abilities and providing a template\n  for further custom implementations.\n* [test_gguf.py](./test_gguf.py) - this script features a console-based chat interface and a Conversation class that\n  maintains a message history. If is adapted to work with the GGML format of models.\n\n## Pretrained models\n\n* [evilfreelancer/saiga_mistral_7b_128k_lora](https://huggingface.co/evilfreelancer/saiga_mistral_7b_128k_lora)\n* [evilfreelancer/ruGPT-3.5-13B-lora](https://huggingface.co/evilfreelancer/ruGPT-3.5-13B-lora)\n* [evilfreelancer/ruGPT-3.5-13B-ggml](https://huggingface.co/evilfreelancer/ruGPT-3.5-13B-ggml)\n* [evilfreelancer/llama2_7b_gguf_yakovlev](https://huggingface.co/evilfreelancer/llama2_7b_gguf_yakovlev)\n* [evilfreelancer/PavelGPT-7B-128K-v0.1-LoRA](https://huggingface.co/evilfreelancer/PavelGPT-7B-128K-v0.1-LoRA)\n\n## Dependencies\n\nThe notebooks and scripts in this repository depend on specific libraries and frameworks. Ensure you have the latest\nversions of these dependencies installed:\n\n* Python 3.11\n* Jupyter Lab\n* PyTorch \u003e= 2.1\n* transformers \u003e= 4.30\n* flash-attn \u003e= 2.3\n* joblib \u003e= 1.1\n\nTo install all dependencies just execute following command:\n\n```shell\npip install -r requirements.txt\n```\n\n## Contribution\n\nContributions to the saiga-custom project are welcome. If you have suggestions for improvement or have developed\nadditional tools or scripts that could benefit the community, please feel free to submit a pull request.\n\n## License\n\nThis project is licensed under the MIT License.\n\n## Acknowledgements\n\nSpecial thanks to the [Saiga (rulm)](https://github.com/IlyaGusev/rulm) project and all the contributors who have made\nthis work possible. For more information about the Saiga project, please visit their GitHub repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilfreelancer%2Fsaiga-custom","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevilfreelancer%2Fsaiga-custom","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilfreelancer%2Fsaiga-custom/lists"}