https://github.com/dhakalnirajan/llama-bitnet
LLaMA-BitNet is a repository dedicated to empowering users to train their own BitNet models built upon LLaMA 2 model, inspired by the groundbreaking paper 'The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits'.
https://github.com/dhakalnirajan/llama-bitnet
large-language-models llama llama2 llm llms meta microsoft
Last synced: 2 months ago
JSON representation
LLaMA-BitNet is a repository dedicated to empowering users to train their own BitNet models built upon LLaMA 2 model, inspired by the groundbreaking paper 'The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits'.
- Host: GitHub
- URL: https://github.com/dhakalnirajan/llama-bitnet
- Owner: dhakalnirajan
- License: mit
- Created: 2024-03-30T08:44:54.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-31T06:05:54.000Z (about 1 year ago)
- Last Synced: 2024-10-18T21:59:08.280Z (7 months ago)
- Topics: large-language-models, llama, llama2, llm, llms, meta, microsoft
- Language: Python
- Homepage: https://arxiv.org/pdf/2402.17764
- Size: 11.7 KB
- Stars: 8
- Watchers: 3
- Forks: 1
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Welcome to LLaMA-BitNet
Welcome to the LLaMA-BitNet repository, where you can dive into the fascinating world of BitNet models. Our repository is your gateway to training your very own BitNet model, as highlighted in the groundbreaking paper [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/abs/2402.17764). Built upon the cutting-edge [LLaMA 2](https://llama.meta.com) architecture, this project allows you to unleash the potential of a model wielding approximately 78 million parameters, trained on a staggering corpus of around 1.5 billion tokens.
> Note: You need to have access to LLaMA model if you wish to run code without modifications. To get access to LLaMA family of models, you need to go to [https://llama.meta.com/llama-downloads/](https://llama.meta.com/llama-downloads/) and provide credentials which you use in Hugging Face. After that, you will receive mail to either download weights directly to your device or to use LLaMA through API.
[](https://www.python.org/)
[](https://huggingface.co/)
[](https://github.com/dhakalnirajan/LLaMA-BitNet/stargazers)
[](https://huggingface.co/datasets)
[](https://huggingface.co/nirajandhakal)[](https://twitter.com/nirajandhakal_7)
## Easy Installation
Getting started with LLaMA-BitNet is a breeze! Follow these simple steps to install all the necessary modules:
```shell
pip install -r requirements.txt
```
## Intuitive File Structure
Our repository boasts a clear and intuitive file structure designed for effortless navigation and customization:
```
LLaMA-BitNet (root folder)
|
│ ├── inference.py (Run inference with the trained BitNet model)
│ ├── LICENSE (MIT License)
│ ├── README.md
│ ├── requirements.txt (List of required modules for installation)
│ ├── train.py (Run the training process)
│ └── utils.py (Contains utility functions)
```
## Empowering Training Data
Harness the power of a 15% subset of the `OpenWebText2` dataset meticulously prepared for training. This subset, tokenized with a context length of 256 for seamless testing, offers unparalleled versatility. However, our code also facilitates manual tokenization, allowing you to train on datasets of your choice effortlessly.
## Streamlined Dependencies
We've curated a set of essential dependencies listed in the `requirements.txt` file, ensuring a seamless installation process:
```text
transformers
datasets
torch
wandb
huggingface_hub
```
## Unleash the Full Potential of BitNet
Our BitNet architecture is engineered for excellence, drawing inspiration from the meticulous design laid out in the training details manuscript, [The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf](https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf). By seamlessly integrating BitLinear and leveraging HuggingFace's `LlamaForCasualLM`, we empower you to unlock the true power of BitNet.
Explore, train, and revolutionize with LLaMA-BitNet!