{"id":17989508,"url":"https://github.com/evilfreelancer/yalm-100b-quantization","last_synced_at":"2025-11-03T17:01:34.904Z","repository":{"id":236662473,"uuid":"659495750","full_name":"EvilFreelancer/yalm-100b-quantization","owner":"EvilFreelancer","description":"This experimental project aims to adapt the YaLM-100B language model to run on consumer-grade GPUs, making advanced AI capabilities more accessible to individual users.","archived":false,"fork":false,"pushed_at":"2023-06-28T01:22:43.000Z","size":12,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-09T15:15:36.748Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EvilFreelancer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-28T01:05:52.000Z","updated_at":"2024-10-31T13:24:07.000Z","dependencies_parsed_at":"2024-04-28T05:53:30.045Z","dependency_job_id":"e9068c3e-0683-4dc1-ac4f-a337d69593f1","html_url":"https://github.com/EvilFreelancer/yalm-100b-quantization","commit_stats":null,"previous_names":["evilfreelancer/yalm-100b-quantization"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2Fyalm-100b-quantization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2Fyalm-100b-quantization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2Fyalm-100b-quantization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvilFreelancer%2Fyalm-100b-quantization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EvilFreelancer","download_url":"https://codeload.github.com/EvilFreelancer/yalm-100b-quantization/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247117749,"owners_count":20886439,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T19:14:50.548Z","updated_at":"2025-11-03T17:01:34.822Z","avatar_url":"https://github.com/EvilFreelancer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# YaLM-100B Quantization Experiment\n\nThis project aims to enable the execution of the YaLM-100B Language Model on consumer-grade hardware by implementing a\nquantization and prune procedures. It utilizes the Megatron-LM framework (forked and modified by Yandex), Nvidia Apex\nand includes additional scripts for model quantization and pruning, in order to optimize model size and inference speed.\n\nMain goal is to optimize the model size and making it feasible to run this large-scale model on `home` computers (not\njust on a cluster of industrial cards) with powerful GPUs, such as a single RTX 3090/4090 video card.\n\nHowever, it's essential to note that the model's large number of hidden layers (10240) pose a significant challenge. The\nMegatron-LM framework reserves a substantial amount of memory (approximately 200GB VRAM) due to these layers, which\nprevents successful model execution even after quantization and pruning.\n\nThe most likely solution to this challenge involves implementing an algorithm that uses the GPT-2 transformer in\ncombination with `torch.load` to load each weight file separately. Each weight file would then be used in a synchronous\nsequential run of a seed text, with the results displayed to the user upon completion. Another promising directions\ninvolves optimizing the Megatron-LM framework to enable it to run not only on GPUs but also on CPUs, or altering the\nlogic of VRAM reservation to match the actual memory footprint of the weights.\n\n## Table of Contents\n\n* [Whats inside](#Whats-inside)\n* [Installation](#Installation)\n    * [Clone repository](#Clone-repository)\n    * [Downloading checkpoint](#Downloading-checkpoint)\n    * [Docker](#Docker)\n    * [Local setup](#Local-setup)\n* [How to use](#How-to-use)\n    * [Quantization only](#Quantization-only)\n    * [Quantization and Pruning](#Quantization-and-Pruning)\n* [Roadmap](#Roadmap)\n* [Links](#Links)\n\n## Whats inside\n\n* `quantize.py` - script for quantization weights to 8/4/2 bits\n* `quantize_prune.py` - script for quantization weights to 8/4/2 bits and pruning model\n* `predict.py` - script for model inference\n* `interactive.sh` - script for interactive model inference, it will call `predict.py` script with required arguments\n* `pr.py` - experimental script for utilizing `torch.load` to load each weight file separately\n\n## Installation\n\n### Clone repository\n\nClone the repository using the following command:\n\n```shell\ngit clone --recurse-submodules\n``` \n\nIt will clone the repository and all submodules (Megatron-LM, YaLM-100B and Apex).\n\n### Downloading checkpoint\n\nDownload model weights and vocabulary.\n\n```shell\nbash YaLM-100B/download/download.sh\n```\n\nBy default, weights will be downloaded to `./YaLM-100B/download/yalm100b_checkpoint/weights/`, and vocabulary will be\ndownloaded to `./YaLM-100B/download/yalm100b_checkpoint/vocab/`, as another option, you can clone our HF repo and pull\nthe checkpoint.\n\n### Docker\n\nRequirements:\n\n* CUDA 11.7\n* Docker\n* Docker Compose\n* Nvidia Container Toolkit\n* Nvidia Runtime Docker\n\nTo set up the project using Docker, follow the steps below:\n\n1. Make sure Docker and Docker Compose are installed on your system. You can download them from the official Docker\n   website.\n2. Navigate to the project root directory.\n3. Build the Docker image:\n\n    ```shell\n    cp docker-compose.dist.yml docker-compose.yml \n    docker-compose build\n    ```\n\n4. Run the Docker container:\n\n   ```shell\n   docker-compose up -d\n   ```\n\n5. Log into the Docker container:\n\n   ```shell\n   docker-compose exec app bash\n   ```\n\n### Local setup\n\nOr you can set up the project locally.\n\nRequirements:\n\n* Python 3.8 (or 3.9)\n* PyTorch 1.13.1 (megatron-lm from YaLM-100B repo doesn't work with newer versions, because of `six` package)\n* CUDA 11.7 (because of Nvidia repo, there is only PyTorch 1.13.1+cu117 is available for download)\n\nTo set up the project locally, follow the steps below:\n\n1. Prepare environment\n\n   ```shell\n   python3.8 -m venv venv\n   source venv/bin/activate\n   ```\n\n2. Install all required dependencies\n\n   ```shell\n   pip install --upgrade pip\n   pip install packaging==23.0 torch==1.13.1+cu117 -f https://download.pytorch.org/whl/torch_stable.html\n   pip install sentencepiece==0.1.99 deepspeed==0.9.5 six==1.16.0\n   pip install --no-cache-dir -r requirements.txt\n   ```\n\n3. Then install Nvidia Apex\n\n   ```shell\n   git clone https://github.com/NVIDIA/apex.git\n   pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ./apex\n   ```\n\n## How to use\n\n### Quantization only\n\nThe script requires three parameters:\n\n* `input_dir`: The directory containing the original checkpoints.\n* `output_dir`: The directory where the quantized and pruned checkpoints will be saved.\n* `bits`: The bit depth for quantization. Possible choices are 8, 4, and 2.\n\n```shell\npython quantize.py ./YaLM-100B/download/yalm100b_checkpoint/ ./data 8\n```\n\n### Quantization and Pruning\n\nThe script requires four parameters:\n\n* `input_dir`: The directory containing the original checkpoints.\n* `output_dir`: The directory where the quantized and pruned checkpoints will be saved.\n* `bits`: The bit depth for quantization. Possible choices are 8, 4, and 2.\n* `prune_ratio`: The pruning ratio.\n\n```shell\npython quantize_prune.py ./YaLM-100B/download/yalm100b_checkpoint/ ./data 8 0.1\n```\n\n### Model inference\n\n```shell\nsh interactive.sh\n```\n\nIt will run `predict.py` script with required arguments and start interactive inference.\n\n## Roadmap\n\nThe project is currently at an intermediate stage. Here is an overview of the development progress:\n\n* [x] Model quantization script\n* [x] Model pruning script\n* [x] Script for standard model execution using Megatron-LM\n* [ ] Script for quantized model execution using PyTorch only\n    * [ ] Implementation for loading model layers sequentially rather than all at once\n    * [ ] Parsing results and displaying them to the user\n* [ ] Tunes of Megatron-LM framework\n    * [ ] Enable it to run on CPUs\n    * [ ] Reduce VRAM reservation\n\nThe final points in the roadmap present significant areas for future development. Sequential loading of model layers\ncould provide a solution to the current memory limitation issue, allowing for the execution of larger models on hardware\nwith less available VRAM. Moreover, enabling execution without reliance on the Megatron-LM framework may provide\nadditional flexibility for model deployment and execution.\n\nContributions to the project to help achieve these roadmap goals are welcomed and appreciated.\n\n## Links\n\n* https://github.com/yandex/YaLM-100B\n* https://github.com/NVIDIA/Megatron-LM\n* https://github.com/NVIDIA/apex\n* https://huggingface.co/yandex/yalm-100b\n* https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6\n* https://www.reddit.com/r/MachineLearning/comments/vivji3/p_yandex_open_sources_100b_large_language_model/\n* https://www.reddit.com/r/MachineLearning/comments/vpn0r1/d_has_anyone_got_yalm100b_to_run/\n* https://xailient.com/blog/4-popular-model-compression-techniques-explained/","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilfreelancer%2Fyalm-100b-quantization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevilfreelancer%2Fyalm-100b-quantization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilfreelancer%2Fyalm-100b-quantization/lists"}