https://github.com/tracel-ai/burn-lm
Democratizing large model inference and training on any device.
https://github.com/tracel-ai/burn-lm
burn large-model-inference open-webui openai-api rust tui
Last synced: 9 months ago
JSON representation
Democratizing large model inference and training on any device.
- Host: GitHub
- URL: https://github.com/tracel-ai/burn-lm
- Owner: tracel-ai
- Created: 2024-12-03T21:07:17.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-09-11T16:48:36.000Z (9 months ago)
- Last Synced: 2025-09-11T18:46:36.667Z (9 months ago)
- Topics: burn, large-model-inference, open-webui, openai-api, rust, tui
- Language: Rust
- Homepage: https://burn.dev
- Size: 910 KB
- Stars: 143
- Watchers: 6
- Forks: 11
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Burn LM
[](https://discord.gg/uPEBbYYDB6)
---
**Burn-LM aims at democratizing large model inference and training on any device.**
## Quick Start
Launch a Burn LM shell with:
```sh
git clone https://github.com/tracel-ai/burn-lm.git
cd burn-lm
cargo burn-lm
```
Type `help` to get a list of commands.
## Available Models
The list of models is very small at the moment since we're focused on performance optimization.
Still, we're accepting high quality contributions to port open-source models to Burn-LM.
Here's the current list of supported models:
| Model | Size |
| --------- | ------ |
| Llama 3 | 8B |
| Llama 3.1 | 8B |
| Llama 3.2 | 1B, 3B |
| TinyLlama | 1.1B |
### Adding a New Model
Models can be easily integrated with Burn LM by implementing the `InferenceServer`
trait to create a pluggable server that can be added to the Burn LM registry.
To bootstrap a new model server you can use the dedicated command `new`:
```sh
cargo burn-lm new "my-model"
```
This will create a new crate named `burn-lm-inference-my-model` and automatically
register it in `burn-lm-registry`.
The bootstraped server is a model-less server that just repeat the prompt it is
given. You can also get inspiration from the other crate with the crate `burn-lm-llama`.