Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/nouamanetazi/bloomz.cpp
C++ implementation for BLOOM
https://github.com/nouamanetazi/bloomz.cpp
bloom cpp multilingual
Last synced: about 2 months ago
JSON representation
C++ implementation for BLOOM
- Host: GitHub
- URL: https://github.com/nouamanetazi/bloomz.cpp
- Owner: NouamaneTazi
- License: mit
- Created: 2023-03-12T18:40:53.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-05-13T10:45:38.000Z (over 1 year ago)
- Last Synced: 2024-04-23T00:12:53.743Z (5 months ago)
- Topics: bloom, cpp, multilingual
- Language: C
- Homepage:
- Size: 2.09 MB
- Stars: 807
- Watchers: 15
- Forks: 63
- Open Issues: 19
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-ChatGPT-repositories - bloomz.cpp - C++ implementation for BLOOM (Reimplementations)
README
# bloomz.cpp
Inference of HuggingFace's [BLOOM-like](https://huggingface.co/docs/transformers/model_doc/bloom) models in pure C/C++.
The repo was built on top of the amazing [llama.cpp](https://github.com/ggerganov/llama.cpp) repo by [@ggerganov](https://github.com/ggerganov), to support [BLOOM](https://huggingface.co/docs/transformers/model_doc/bloom) models. It supports all models that can be loaded using `BloomForCausalLM.from_pretrained()`.
## Demo
## Usage
First, you need to clone the repo and build it:
```bash
git clone https://github.com/NouamaneTazi/bloomz.cpp
cd bloomz.cpp
make
```
### Convert weights
Then, you must convert the model weights to the ggml format. Any BLOOM model can be converted.
Some weights hosted on the Hub are already converted. You can find the list [here](https://huggingface.co/models?other=bloom&other=ggml).
Otherwise, the quickest way to convert weights is to use this [converter tool](https://huggingface.co/spaces/Wauplin/bloomz.cpp-converter).
It is a Space hosted on the Huggingface Hub that converts and quantizes weights for you and upload them to the repository of your choice.
If you prefer, you can manually convert the weights on your machine:
```bash
# install required libraries
python3 -m pip install torch numpy transformers accelerate
# download and convert the 7B1 model to ggml FP16 format
python3 convert-hf-to-ggml.py bigscience/bloomz-7b1 ./models
# Note: you can add --use-f32 to convert to FP32 instead of FP16
```
Optionally, you can quantize the model to 4-bits.
```bash
./quantize ./models/ggml-model-bloomz-7b1-f16.bin ./models/ggml-model-bloomz-7b1-f16-q4_0.bin 2
```
### Run inference
Finally, you can run the inference.
```bash
./main -m ./models/ggml-model-bloomz-7b1-f16-q4_0.bin -t 8 -n 128
```
Your output should look like this:
```bash
make && ./main -m models/ggml-model-bloomz-7b1-f16-q4_0.bin -p 'Translate "Hi, how are you?" in French:' -t 8 -n 256
I llama.cpp build info:
I UNAME_S: Darwin
I UNAME_P: arm
I UNAME_M: arm64
I CFLAGS: -I. -O3 -DNDEBUG -std=c11 -fPIC -pthread -DGGML_USE_ACCELERATE
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread
I LDFLAGS: -framework Accelerate
I CC: Apple clang version 13.1.6 (clang-1316.0.21.2.5)
I CXX: Apple clang version 13.1.6 (clang-1316.0.21.2.5)
make: Nothing to be done for `default'.
main: seed = 1678899845
llama_model_load: loading model from 'models/ggml-model-bloomz-7b1-f16-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 250880
llama_model_load: n_ctx = 512
llama_model_load: n_embd = 4096
llama_model_load: n_mult = 1
llama_model_load: n_head = 32
llama_model_load: n_layer = 30
llama_model_load: f16 = 2
llama_model_load: n_ff = 16384
llama_model_load: n_parts = 1
llama_model_load: ggml ctx size = 5312.64 MB
llama_model_load: memory_size = 480.00 MB, n_mem = 15360
llama_model_load: loading model part 1/1 from 'models/ggml-model-bloomz-7b1-f16-q4_0.bin'
llama_model_load: ............................................. done
llama_model_load: model size = 4831.16 MB / num tensors = 366
main: prompt: 'Translate "Hi, how are you?" in French:'
main: number of tokens in prompt = 11
153772 -> 'Translate'
17959 -> ' "H'
76 -> 'i'
98257 -> ', '
20263 -> 'how'
1306 -> ' are'
1152 -> ' you'
2040 -> '?'
5 -> '"'
361 -> ' in'
196427 -> ' French:'
sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000
Translate "Hi, how are you?" in French: Bonjour, comment ça va? [end of text]
main: mem per token = 24017564 bytes
main: load time = 3092.29 ms
main: sample time = 2.40 ms
main: predict time = 1003.04 ms / 59.00 ms per token
main: total time = 5307.23 ms
```
## Advanced usage
Here's a list of the available options:
```bash
usage: ./main [options]
options:
-h, --help show this help message and exit
-s SEED, --seed SEED RNG seed (default: -1)
-t N, --threads N number of threads to use during computation (default: 4)
-p PROMPT, --prompt PROMPT
prompt to start generation with (default: random)
-n N, --n_predict N number of tokens to predict (default: 128)
--top_k N top-k sampling (default: 40)
--top_p N top-p sampling (default: 0.9)
--repeat_last_n N last n tokens to consider for penalize (default: 64)
--repeat_penalty N penalize repeat sequence of tokens (default: 1.3)
--temp N temperature (default: 0.8)
-b N, --batch_size N batch size for prompt processing (default: 8)
-m FNAME, --model FNAME
model path (default: models/ggml-model-bloomz-7b1-f16-q4_0.bin)
```
## Memory usage
| Model | Disk | Mem |
| --- | --- | --- |
| `bloomz-7b1-f16-q4_0` | 4.7 GB | 5.3 GB |
## iOS App
The repo includes a proof-of-concept iOS app in the `Bloomer` directory. You need to provide the converted model weights, placing a file called `ggml-model-bloomz-560m-f16.bin` inside that folder. This is what it looks like on an iPhone:
