https://github.com/lenml/llm-boot
Running local LLMs is as simple as breathing.
https://github.com/lenml/llm-boot
Last synced: over 1 year ago
JSON representation
Running local LLMs is as simple as breathing.
- Host: GitHub
- URL: https://github.com/lenml/llm-boot
- Owner: lenML
- License: agpl-3.0
- Created: 2024-12-07T20:56:39.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-21T15:34:49.000Z (over 1 year ago)
- Last Synced: 2025-02-21T16:30:55.858Z (over 1 year ago)
- Language: TypeScript
- Size: 73.2 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# llm-boot
Running local LLMs is as simple as breathing.
# Roadmap
- [ ] openai compatible api
- [x] Basic features
- [x] `/v1/completions`
- [x] `/v1/chat/completions`
- [x] `/v1/embedding`
- [x] `/v1/models`
- [ ] Advanced features
- [ ] logprobs
- [ ] fake multi-model (base on whisper/florence onnx model)
- [ ] prompt cache
- [ ] vram manager: auto offload / vram limit / cpu limit
- [ ] Packaged to bin file
- [ ] evaluator
- [ ] Special API provided for model evaluation
# Running
## from executable release
WIP
## from source
```
pnpm install
pnpm dev
```
# Configure
运行程序需要在执行目录下创建一个配置文件 `boot.config.json`
```json
{
"server": {
"port": 4567
},
"no_docs": false,
"model_dirs": ["~/llm_models","~/ggufs"]
}
```
# LICENSE
AGPL-3.0 license