{"id":26517469,"url":"https://github.com/dhakalnirajan/llama-bitnet","last_synced_at":"2025-03-21T08:27:21.236Z","repository":{"id":230570682,"uuid":"779593082","full_name":"dhakalnirajan/LLaMA-BitNet","owner":"dhakalnirajan","description":"LLaMA-BitNet is a repository dedicated to empowering users to train their own BitNet models built upon LLaMA 2 model, inspired by the groundbreaking paper 'The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits'.","archived":false,"fork":false,"pushed_at":"2024-03-31T06:05:54.000Z","size":12,"stargazers_count":8,"open_issues_count":3,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-18T21:59:08.280Z","etag":null,"topics":["large-language-models","llama","llama2","llm","llms","meta","microsoft"],"latest_commit_sha":null,"homepage":"https://arxiv.org/pdf/2402.17764","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dhakalnirajan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-30T08:44:54.000Z","updated_at":"2024-08-03T04:13:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"ff11ef13-7378-4c9c-a6ff-625a461e07b2","html_url":"https://github.com/dhakalnirajan/LLaMA-BitNet","commit_stats":{"total_commits":9,"total_committers":2,"mean_commits":4.5,"dds":"0.11111111111111116","last_synced_commit":"6114fee4520e333c4c3cbb1ba867fd41dcc81725"},"previous_names":["dhakalnirajan/llama-bitnet"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dhakalnirajan%2FLLaMA-BitNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dhakalnirajan%2FLLaMA-BitNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dhakalnirajan%2FLLaMA-BitNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dhakalnirajan%2FLLaMA-BitNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dhakalnirajan","download_url":"https://codeload.github.com/dhakalnirajan/LLaMA-BitNet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244762860,"owners_count":20506360,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["large-language-models","llama","llama2","llm","llms","meta","microsoft"],"created_at":"2025-03-21T08:27:20.771Z","updated_at":"2025-03-21T08:27:21.228Z","avatar_url":"https://github.com/dhakalnirajan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cspan style=\"color:#fa362f\"\u003eWelcome to LLaMA-BitNet\u003c/span\u003e\n\n\u003cspan style=\"color:#ffa042\"\u003eWelcome to the LLaMA-BitNet repository, where you can dive into the fascinating world of BitNet models. Our repository is your gateway to training your very own BitNet model, as highlighted in the groundbreaking paper [\u003cspan style=\"color:#52d1ff;text-decoration: underline;\"\u003eThe Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/abs/2402.17764\u003c/span\u003e). Built upon the cutting-edge [\u003cspan style=\"color:#52d1ff;text-decoration: underline;\"\u003eLLaMA 2](https://llama.meta.com\u003c/span\u003e) architecture, this project allows you to unleash the potential of a model wielding approximately 78 million parameters, trained on a staggering corpus of around 1.5 billion tokens.\u003c/span\u003e\n\n\u003cbr\u003e\n\n\u003e Note: You need to have access to LLaMA model if you wish to run code without modifications. To get access to LLaMA family of models, you need to go to [https://llama.meta.com/llama-downloads/](https://llama.meta.com/llama-downloads/) and provide credentials which you use in Hugging Face. After that, you will receive mail to either download weights directly to your device or to use LLaMA through API.\n\n\u003cbr\u003e\n\n[![Python](https://img.shields.io/badge/Python-3776AB?style=for-the-badge\u0026logo=python\u0026logoColor=white)](https://www.python.org/)\n![PyTorch](https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?style=for-the-badge\u0026logo=PyTorch\u0026logoColor=white)\n[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-transformers%20%7C%20datasets%20%7C%20Models-blueviolet?style=for-the-badge\u0026logo=huggingface\u0026labelColor=ee4c2c\u0026color=ff9d00)](https://huggingface.co/)\n![GitHub License](https://img.shields.io/github/license/dhakalnirajan/LLaMA-BitNet?style=for-the-badge\u0026logo=github\u0026logoColor=white\u0026label=License\u0026labelColor=purple\u0026color=orange)\n[![Repository Stars](https://img.shields.io/github/stars/dhakalnirajan/LLaMA-BitNet)](https://github.com/dhakalnirajan/LLaMA-BitNet/stargazers)\n![GitHub forks](https://img.shields.io/github/forks/dhakalnirajan/LLaMA-BitNet)\n[![Dataset on HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-sm.svg)](https://huggingface.co/datasets)\n[![Follow me on HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/follow-me-on-HF-sm.svg)](https://huggingface.co/nirajandhakal)[![X (formerly Twitter) URL](https://img.shields.io/twitter/url?url=https%3A%2F%2Ftwitter.com%2Fnirajandhakal_7\u0026style=for-the-badge\u0026logo=X\u0026logoColor=blue\u0026label=Follow%20%40nirajandhakal_7%20\u0026labelColor=black\u0026color=black\u0026link=https%3A%2F%2Ftwitter.com%2Fnirajandhakal_7)](https://twitter.com/nirajandhakal_7)\n\u003cimg src=\"https://chunte-hfba.static.hf.space/images/Outlined%20Huggies/Greeting%20Huggy%20left.png\" alt=\"HuggingFace Logo\" height=80 width=80\u003e\n\n## \u003cspan style=\"color:#ff52fc\"\u003eEasy Installation\u003c/span\u003e\n\nGetting started with LLaMA-BitNet is a breeze! Follow these simple steps to install all the necessary modules:\n\n```shell\npip install -r requirements.txt\n```\n\n## \u003cspan style=\"color:#5e6eff\"\u003eIntuitive File Structure\u003c/span\u003e\n\nOur repository boasts a clear and intuitive file structure designed for effortless navigation and customization:\n\n```\nLLaMA-BitNet                    (root folder)\n|\n│   ├── inference.py            (Run inference with the trained BitNet model)\n│   ├── LICENSE                 (MIT License)\n│   ├── README.md\n│   ├── requirements.txt        (List of required modules for installation)\n│   ├── train.py                (Run the training process)\n│   └── utils.py                (Contains utility functions)\n```\n\n## \u003cspan style=\"color:#00ff00\"\u003eEmpowering Training Data\u003c/span\u003e\n\nHarness the power of a 15% subset of the \u003cspan style=\"color:#ff006a\"\u003e`OpenWebText2`\u003c/span\u003e dataset meticulously prepared for training. This subset, tokenized with a context length of 256 for seamless testing, offers unparalleled versatility. However, our code also facilitates manual tokenization, allowing you to train on datasets of your choice effortlessly.\n\n## \u003cspan style=\"color:#ff4592\"\u003eStreamlined Dependencies\u003c/span\u003e\n\nWe've curated a set of essential dependencies listed in the \u003cspan style=\"color:#ff006a\"\u003e`requirements.txt`\u003c/span\u003e file, ensuring a seamless installation process:\n\n```text\ntransformers\ndatasets\ntorch\nwandb\nhuggingface_hub\n```\n\n## \u003cspan style=\"color:#ff45f9\"\u003eUnleash the Full Potential of BitNet\u003c/span\u003e\n\nOur BitNet architecture is engineered for excellence, drawing inspiration from the meticulous design laid out in the training details manuscript, [\u003cspan style=\"color:#52d1ff;text-decoration: underline;\"\u003eThe-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf](https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf)\u003c/span\u003e. By seamlessly integrating BitLinear and leveraging HuggingFace's \u003cspan style=\"color:#ffff00; background-color:#112211;\"\u003e`LlamaForCasualLM`\u003c/span\u003e, we empower you to unlock the true power of BitNet.\n\nExplore, train, and revolutionize with LLaMA-BitNet!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdhakalnirajan%2Fllama-bitnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdhakalnirajan%2Fllama-bitnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdhakalnirajan%2Fllama-bitnet/lists"}