https://github.com/johnclaw/chatllm.rs
rust api wrapper for llm-inference chatllm.cpp
https://github.com/johnclaw/chatllm.rs
api-wrapper bindings chatbot chatllm cpu-inference gemma ggml inference llama llm llm-inference llms mistral quantization qwen rust
Last synced: 3 months ago
JSON representation
rust api wrapper for llm-inference chatllm.cpp
- Host: GitHub
- URL: https://github.com/johnclaw/chatllm.rs
- Owner: JohnClaw
- Created: 2024-11-27T16:14:29.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-11-27T16:29:20.000Z (7 months ago)
- Last Synced: 2025-01-27T18:13:35.236Z (5 months ago)
- Topics: api-wrapper, bindings, chatbot, chatllm, cpu-inference, gemma, ggml, inference, llama, llm, llm-inference, llms, mistral, quantization, qwen, rust
- Language: Rust
- Homepage:
- Size: 784 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# chatllm.rs
rust api wrapper for llm-inference chatllm.cpp
All credits go to original repo: https://github.com/foldl/chatllm.cpp and Qwen 2.5 32b Coder Instruct which made 99% of work. I only guided it with prompts.
To compile your project into an executable, open a terminal and navigate to your project directory.
Then run the following command:
cargo build --release
When exe will be ready launch it like this: main.exe -m qwen2.5-1.5b.bin
Links for quantatized models:
QWen-2.5 1.5B - https://modelscope.cn/api/v1/models/judd2024/chatllm_quantized_qwen2.5/repo?Revision=master&FilePath=qwen2.5-1.5b.bin
Gemma-2 2B - https://modelscope.cn/api/v1/models/judd2024/chatllm_quantized_gemma2_2b/repo?Revision=master&FilePath=gemma2-2b.bin
If you need more quantatized models use this python model downloader: https://github.com/foldl/chatllm.cpp/blob/master/scripts/model_downloader.py
You can convert custom safetensors model to inner chatllm.cpp format by using this script: https://github.com/foldl/chatllm.cpp/blob/master/convert.py
Converting tutorial: https://github.com/foldl/chatllm.cpp?tab=readme-ov-file#quantize-model
List of supported llm architecture types suitable for conversion: https://github.com/foldl/chatllm.cpp/blob/master/docs/models.md