{"id":49523292,"url":"https://github.com/loopyd/easyllama","last_synced_at":"2026-07-04T01:09:11.315Z","repository":{"id":353470526,"uuid":"1219550538","full_name":"loopyd/easyllama","owner":"loopyd","description":"Minimal Docker + CUDA llama.cpp runner with safe public defaults","archived":false,"fork":false,"pushed_at":"2026-05-17T01:21:04.000Z","size":171,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-17T03:30:46.652Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/loopyd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-24T01:45:48.000Z","updated_at":"2026-05-17T01:21:07.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/loopyd/easyllama","commit_stats":null,"previous_names":["loopyd/easyllama"],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/loopyd/easyllama","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loopyd%2Feasyllama","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loopyd%2Feasyllama/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loopyd%2Feasyllama/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loopyd%2Feasyllama/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/loopyd","download_url":"https://codeload.github.com/loopyd/easyllama/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loopyd%2Feasyllama/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35106615,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-02T01:04:13.525Z","updated_at":"2026-07-04T01:09:11.310Z","avatar_url":"https://github.com/loopyd.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# easy llama(cpp)\n\nRun local `llama.cpp`-style backends behind one `llama-swap` endpoint at `http://127.0.0.1:8080`.\n\nProject goal: one host command surface, one public port, one shared model cache, multiple backend modes.\n\n## Contents\n\n- [easy llama(cpp)](#easy-llamacpp)\n  - [Contents](#contents)\n  - [At glance](#at-glance)\n  - [Modes](#modes)\n  - [System requirements](#system-requirements)\n  - [Install](#install)\n  - [Quick start](#quick-start)\n    - [1. Create credentials](#1-create-credentials)\n    - [2. Copy mode config templates](#2-copy-mode-config-templates)\n    - [3. Build, start, warm](#3-build-start-warm)\n    - [4. Verify runtime](#4-verify-runtime)\n  - [Common commands](#common-commands)\n  - [File map](#file-map)\n  - [Environment overrides](#environment-overrides)\n  - [Reader path](#reader-path)\n  - [Troubleshooting](#troubleshooting)\n  - [Contributing](#contributing)\n  - [License](#license)\n\n## At glance\n\nWhy reader cares:\n\n- One entrypoint: `./run.sh`\n- One API base URL: `http://127.0.0.1:8080`\n- One shared Hugging Face cache: `models/`\n- One shared mmproj asset directory: `mmproj/`\n- Stable model IDs exposed through `/v1/models`\n- Per-model `concurrencyLimit: 4` in llama-swap configs to cap parallel requests\n- `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` set in runtime Docker stage for oversubscribed VRAM (RTX 5090)\n- Lazy downloads by default; use warmup for predictable first-request latency\n\n## Modes\n\nPick mode by backend behavior, not by install flow. Setup path stays same.\n\n- Mode-specific defaults live in the tracked templates under `config/`.\n\n| Mode | Best for | `qwen3-chat` backend | Default chat weights | Extra API surface |\n| --- | --- | --- | --- | --- |\n| `basic` | Plain llama.cpp path | `llama-server-basic` | `unsloth/Qwen3.6-27B-GGUF:Q4_K_M` | none |\n| `turboquant` | Turboquant KV-cache experiments | `llama-server-turboquant` | `HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive:Q5_K_P` | none |\n| `mtp` | MTP experiments (separate llama.cpp fork/config) | `llama-server-basic` (runtime mode) | `havenoammo/Qwen3.6-27B-MTP-UD-GGUF:Qwen3.6-27B-MTP-UD-Q6_K_XL.gguf` | none |\n| `spiritbuun` | buun-llama-cpp DFlash experiments | `easyllama server spiritbuun` | `unsloth/Qwen3.6-27B-GGUF:Q5_K_M` + `Ardenzard/Qwen3.6-27B-DFlash-GGUF:Qwen3.6-27B-DFlash-Q5_K_M.gguf` | none |\n| `lucebox` | Luce dflash/pflash experiments | `easyllama server lucebox` | `unsloth/Qwen3.6-27B-GGUF:Q4_K_M` + `KingsonHO/Qwen3.6-27B-DFlash:model.safetensors` | `POST /v1/messages` |\n\n## System requirements\n\n- Linux with Bash `4.1+`\n- Docker daemon running\n- `docker buildx`\n- NVIDIA drivers and working `nvidia-smi`\n- NVIDIA container runtime in Docker\n- Python `3.11+`\n- `curl`\n- `jq`\n\n## Install\n\nMinimal host setup for running `./run.sh` from checkout:\n\n```bash\npython3 -m venv ./.venv\n.venv/bin/activate\npython -m pip install -r ./requirements.txt\n```\n\nEditable development install:\n\n```bash\npython3 -m venv ./.venv\n.venv/bin/activate\npython -m pip install -e . -r ./requirements.txt -r ./requirements-dev.txt\n```\n\n## Quick start\n\nFastest path from fresh checkout to working local endpoint.\n\n### 1. Create credentials\n\n```bash\ncp auth.json.example auth.json\n```\n\nSet:\n\n- `hf_token` for private or rate-limited Hugging Face pulls\n- `api_key` for `Authorization: Bearer ...` protection on `/v1/*` routes\n\n### 2. Copy mode config templates\n\n```bash\ncp config/config.basic.yml.example config/config.basic.yml\ncp config/config.turboquant.yml.example config/config.turboquant.yml\ncp config/config.spiritbuun.yml.example config/config.spiritbuun.yml\ncp config/config.mtp.yml.example config/config.mtp.yml\ncp config/config.lucebox.yml.example config/config.lucebox.yml\n```\n\nEdit configs as needed. For more config detail, see `llama-swap` docs:\n[llama-swap configuration docs](https://github.com/mostlygeek/llama-swap/blob/main/docs/configuration.md)\n\nIf `config/config.\u003cmode\u003e.yml` does not exist, `run.sh` falls back to the matching example file in `config/`.\n\n### 3. Build, start, warm\n\n```bash\n./run.sh --mode \u003cmode\u003e build\n./run.sh --mode \u003cmode\u003e start\n./run.sh --mode \u003cmode\u003e warmup\n```\n\nPass model IDs to warm only subset:\n\n```bash\n./run.sh --mode \u003cmode\u003e warmup qwen3-chat qmd-rerank\n```\n\nWith no model arguments, warmup hits every model exposed by `/v1/models`.\n\n### 4. Verify runtime\n\n```bash\nAPI_KEY=\"$(jq -r '.api_key // empty' auth.json)\"\nAUTH=()\nif [[ -n \"${API_KEY}\" ]]; then\n  AUTH=(-H \"Authorization: Bearer ${API_KEY}\")\nfi\n\n./run.sh status\ncurl -sS http://127.0.0.1:8080/health\ncurl -sS \"${AUTH[@]}\" http://127.0.0.1:8080/v1/models | jq -r '.data[].id'\n```\n\n## Common commands\n\nMost-used host commands through `./run.sh`.\n\n| Command | Action |\n| --- | --- |\n| `./run.sh build` | Build default `basic` image |\n| `./run.sh --mode \u003cmode\u003e build` | Build selected mode image |\n| `./run.sh start` | Start default `basic` container |\n| `./run.sh --mode \u003cmode\u003e start` | Start selected mode |\n| `./run.sh warmup [model...]` | Preload one or more models through `llama-swap` |\n| `./run.sh restart` | Restart selected mode container |\n| `./run.sh stop` | Stop and remove runtime container |\n| `./run.sh logs` | Follow runtime logs |\n| `./run.sh status` | Show runtime status and built images |\n| `./run.sh clean` | Remove current mode image and container |\n| `./run.sh clean --all-images` | Remove all mode images and runtime container |\n| `./run.sh serve` | Run `llama-swap` inside container |\n| `./run.sh server ...` | Run mode-specific upstream server directly |\n| `./run.sh help` | Show CLI help |\n\n## File map\n\n| Path | Purpose |\n| --- | --- |\n| `run.sh` | Host and container entrypoint |\n| `auth.json` | Local Hugging Face token and optional API key |\n| `auth.json.example` | Credential template |\n| `config/config.basic.yml` | Editable config for `basic` |\n| `config/config.turboquant.yml` | Editable config for `turboquant` |\n| `config/config.spiritbuun.yml` | Editable config for `spiritbuun` |\n| `config/config.mtp.yml` | Editable config for `mtp` |\n| `config/config.lucebox.yml` | Editable config for `lucebox` |\n| `config/config.basic.yml.example` | Tracked `basic` template |\n| `config/config.turboquant.yml.example` | Tracked `turboquant` template |\n| `config/config.spiritbuun.yml.example` | Tracked `spiritbuun` template |\n| `config/config.mtp.yml.example` | Tracked `mtp` template |\n| `config/config.lucebox.yml.example` | Tracked `lucebox` template |\n| `models/` | Shared Hugging Face cache |\n| `mmproj/` | Shared mmproj assets |\n| `chat_template/` | Mounted chat templates |\n| `easyllama/` | Python package: runtime, CLI, Docker orchestration, launchers |\n| `API.md` | API reference and request examples |\n| `CHANGELOG.md` | Release history |\n\n## Environment overrides\n\n| Variable | Purpose |\n| --- | --- |\n| `LLAMACPP_MODE` | Select `basic`, `turboquant`, `mtp`, `spiritbuun`, or `lucebox` |\n| `LLAMACPP_LLAMA_CPP_REPO` | Git repo URL for the llama.cpp fork used by the active mode **(unified)** |\n| `LLAMACPP_LLAMA_CPP_REF` | Git branch/tag/commit for the llama.cpp fork **(unified)** |\n| `LLAMACPP_LUCEBOX_HUB_REPO` | Git repo URL for Luce dflash hub (lucebox mode only) |\n| `LLAMACPP_LUCEBOX_HUB_REF` | Git branch/tag/commit for Luce dflash hub |\n| `LLAMACPP_LS_CONFIG_FILE` | Use explicit config file instead of mode lookup |\n| `LLAMACPP_HOST_PORT` | Change published host port |\n| `LLAMACPP_AUTH_FILE` | Use different auth JSON file |\n| `HF_TOKEN` or `LLAMACPP_HF_TOKEN` | Override Hugging Face token |\n| `API_KEY` or `LLAMACPP_API_KEY` | Override local API key |\n| `LLAMACPP_MMPROJ_FILE` | Use local mmproj path, `mmproj/...`, or URL |\n| `LLAMACPP_HF_MMPROJ` | Use HF mmproj asset as `owner/repo/file.gguf` |\n| `LLAMACPP_CMAKE_CUDA_ARCHITECTURES` | Override auto-detected CUDA arch build values |\n\nNotes:\n\n- **Unified repo vars**: All llama.cpp variants (basic, turboquant, mtp, spiritbuun) now share one `LLAMACPP_LLAMA_CPP_REPO`/`LLAMACPP_LLAMA_CPP_REF` pair. The old mode-specific env vars (`LLAMACPP_MTP_LLAMA_CPP_REPO`, `LLAMACPP_TURBOQUANT_LLAMA_CPP_REPO`, `LLAMACPP_SPIRITBUUN_LLAMA_CPP_REPO`, etc.) are removed. Set the unified pair to override the repo defaults in `pyproject.toml`.\n- Lucebox requires two repos: the main llama.cpp fork plus `LLAMACPP_LUCEBOX_HUB_REPO`/`LLAMACPP_LUCEBOX_HUB_REF` for the dflash hub.\n- If `auth.json` contains `api_key`, `/v1/*` routes require `Authorization: Bearer \u003capi_key\u003e`.\n- `LLAMACPP_LS_CONFIG_FILE` overrides mode-based config selection.\n\n## Reader path\n\nNew reader:\n\n1. Read this file for setup and mode selection.\n2. Read [API.md](API.md) for endpoint usage.\n3. Read [CHANGELOG.md](CHANGELOG.md) for release history and upgrade context.\n\n## Troubleshooting\n\nFast map from symptom to likely fix.\n\n| Problem | Likely cause | What to do |\n| --- | --- | --- |\n| `docker buildx` build fails fast | Buildx missing or not bootstrapped | Install Buildx, then run `docker buildx inspect --bootstrap` |\n| First request is slow | Model download or first load happening lazily | Run `./run.sh warmup ...` first |\n| `POST /v1/messages` fails | Not running `lucebox` mode | Restart with `./run.sh --mode lucebox start` |\n| `/v1/models` returns `401` | API key enabled | Send `Authorization: Bearer \u003capi_key\u003e` |\n| Config edit does nothing | Wrong mode file edited or `LLAMACPP_LS_CONFIG_FILE` set | Check active mode and config path |\n| Python change seems ignored | Running image stale | Rebuild affected mode, then restart |\n| Port `8080` busy | Another process owns host port | Start with `LLAMACPP_HOST_PORT=8090 ./run.sh start` |\n| Private HF downloads fail | No usable HF token | Set `hf_token` in `auth.json` or export `HF_TOKEN` |\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md).\n\n## License\n\nSee [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floopyd%2Feasyllama","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Floopyd%2Feasyllama","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floopyd%2Feasyllama/lists"}