{"id":50436991,"url":"https://github.com/lukaspustina/4lm","last_synced_at":"2026-05-31T17:31:16.505Z","repository":{"id":358529693,"uuid":"1221776134","full_name":"lukaspustina/4lm","owner":"lukaspustina","description":"Local LLM control plane for Apple Silicon — one command, your hardware, your data.","archived":false,"fork":false,"pushed_at":"2026-05-17T19:48:46.000Z","size":1206,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-17T21:40:34.147Z","etag":null,"topics":["apple-silicon","cli","inference","llm","local-llm","macos","mlx","ollama","open-webui","openai-compatible","qwen3","self-hosted"],"latest_commit_sha":null,"homepage":"https://lukaspustina.github.io/4lm/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lukaspustina.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-26T17:05:16.000Z","updated_at":"2026-05-17T19:48:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lukaspustina/4lm","commit_stats":null,"previous_names":["lukaspustina/4lm"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/lukaspustina/4lm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukaspustina%2F4lm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukaspustina%2F4lm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukaspustina%2F4lm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukaspustina%2F4lm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lukaspustina","download_url":"https://codeload.github.com/lukaspustina/4lm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukaspustina%2F4lm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33742185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","cli","inference","llm","local-llm","macos","mlx","ollama","open-webui","openai-compatible","qwen3","self-hosted"],"created_at":"2026-05-31T17:31:15.719Z","updated_at":"2026-05-31T17:31:16.498Z","avatar_url":"https://github.com/lukaspustina.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 4lm — Local LLM control plane for Apple Silicon\n\n[![CI](https://github.com/lukaspustina/4lm/actions/workflows/ci.yml/badge.svg)](https://github.com/lukaspustina/4lm/actions/workflows/ci.yml)\n[![Release](https://img.shields.io/github/v/release/lukaspustina/4lm?sort=semver)](https://github.com/lukaspustina/4lm/releases)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow)](LICENSE)\n[![Platform: macOS · Apple Silicon](https://img.shields.io/badge/platform-macOS%20%C2%B7%20Apple%20Silicon-blue)](#quickstart)\n\n**A local LLM stack you don't have to babysit.**\n\nYour Mac. Or a Mac in your closet. Either way, one command.\nYour hardware, your data, no metered bill.\n\nRunning multiple local LLMs on Apple Silicon means juggling ~140 GB\nof weights, wired-memory limits, profile YAMLs, and Open WebUI config\ndrift. You end up administering the stack instead of using it.\n\n`4lm` is the CLI that refuses to let that happen.\n\n## Two shapes\n\nOne installer, one CLI, two ways to run it.\n\n### Workstation — your Mac IS the LLM\n\n```sh\n./install.sh           # omlx + Open WebUI + opencode TUI\n4lm start\nopen http://localhost:3000   # register your admin account here, first\n4lm opencode                 # daily driver\n```\n\nOpen WebUI on `http://localhost:3000` with RAG, web search, code\ninterpreter, and memory wired in by default. `opencode` in your\nterminal, pointed at the local `/v1`. Your laptop is the assistant.\n\n\u003e **Register your WebUI admin account before doing anything else.** New\n\u003e accounts default to the `pending` role with no privileges, and\n\u003e registration is locked after the first user. If you skip this and\n\u003e later run `4lm expose lan --confirm` on a network you don't fully\n\u003e control, you can lock yourself out.\n\n### Appliance — a Mac in your closet serves the LAN\n\n```sh\n./install.sh --backend-only   # skips Open WebUI + opencode\n4lm start\n4lm expose lan --confirm\n```\n\nHeadless OpenAI-compatible `/v1/*` API on the LAN. Other machines run\ntheir own clients (`opencode`, Open WebUI, Continue.dev — anything that\nspeaks `/v1`) pointed at `http://\u003chost\u003e:8000`. The Mac Studio in the\ncloset does the inference; the Air on the couch does the typing.\n\n\u003e **The backend has no auth.** Anyone who can reach `:8000` on your LAN\n\u003e can call `/v1/*`. Use this on a network you trust, or front it with\n\u003e Tailscale (or another VPN that does authenticate).\n\n## What it refuses to do\n\n- **Never auto-starts after reboot.** A 70 GB working set should not\n  sneak onto wired memory before you've made coffee. Plists live in\n  `~/.4lm/launchd/` — launchd never finds them unless `4lm start`\n  says so. Opt in with `4lm autostart enable` if you want it.\n- **Never binds to LAN without `--confirm`.** No env-var bypass, no\n  config typo, no *\"I thought it was already local.\"* `4lm expose lan`\n  is a deliberate two-step. (Tailscale is still the better answer.)\n- **Never silently breaks profile switches.** `4lm profile set \u003cname\u003e`\n  validates the YAML → swaps the active symlink → polls `/v1/models`\n  for 30 s → on timeout, restores the previous symlink and re-polls.\n  Bad YAML never kills the stack.\n- **Never invalidates your knowledge base across profiles.** Every\n  omlx profile serves the embedder as `qwen3-embedding` and the\n  reranker as `qwen3-reranker`. Switch from `default` (65 GB) to\n  `lean` (40 GB) to `max-100gb` (92 GB) — the same RAG index keeps\n  working. Switch profiles like you switch branches.\n- **Never lets you OOM silently.** `install.sh` enforces\n  `iogpu.wired_limit_mb=98304` via sudoers + sysctl. `4lm doctor`\n  smoke-tests inference. `4lm diag` shows what's actually running\n  when the fans spin up.\n\n## What it actually does\n\n- **One CLI**, three backends behind one OpenAI-compatible `/v1`\n  seam — `omlx` (default; vLLM-style MLX, paged KV cache, multi-model\n  EnginePool), `mlx_lm` (upstream MLX, single-model), `ollama`\n  (GGUF/llama.cpp). Clients don't know or care which is running.\n- **Atomic profile switching with bounded rollback** (above).\n- **Same-name reissue = live config reload.** Edit a profile YAML\n  and re-run `4lm profile set \u003csame-name\u003e` — re-renders\n  `model_settings.json`, re-stages model symlinks, kickstarts the\n  backend. No full stop/start.\n- **Open WebUI, preconfigured for daily use.** DuckDuckGo web search,\n  Pyodide code interpreter, personal memory, follow-up + autocomplete\n  suggestions, file-upload RAG. Embeddings + reranker served by the\n  same omlx backend → no second service, no cloud calls, fully offline.\n  (Most of these settings are PersistentConfig: copied into `webui.db`\n  on first init only — after that the admin UI is source of truth.\n  Toggles in the admin panel survive restarts; env-var changes do not.)\n- **Idempotent install / upgrade / uninstall.** Every step is a\n  no-op on re-run: sudoers, sysctl, newsyslog, pipx, opencode config.\n  Re-running `--backend-only` over a full install does not strip the\n  WebUI; re-running the full installer over `--backend-only` upgrades\n  cleanly.\n- **Visibility commands.** `4lm doctor` (prereqs + smoke-test\n  inference), `4lm diag` (live clients, in-flight requests, top CPU\n  consumers), `4lm outdated` / `4lm upgrade` (PyPI + Homebrew + HF).\n\n## Quickstart\n\n```sh\nmake bootstrap   # Brewfile + Brewfile-tui (skipped if BACKEND_ONLY=1)\n                 # core: shellcheck, shfmt, bats-core, jq, python@3.12,\n                 # pipx, llmfit, ollama; tui extra: opencode\nmake install     # ~/.4lm/, sudoers, sysctl, pipx-installed deps,\n                 # log rotation, opencode config\nmake models      # ~140 GB from HuggingFace (idempotent; same target updates)\n4lm start        # bootstrap launchd agents\n4lm opencode     # daily driver (alias: 4lm code)\n```\n\n\u003e **64 GB Macs: switch to the `lean` profile first** —\n\u003e `4lm profile set lean` before `make models`. The `default` profile\n\u003e wants 96 GB+ steady; `lean` fits in 40 GB and downloads ~80 GB instead\n\u003e of ~140 GB. `4lm doctor` will warn you if the active profile doesn't\n\u003e fit your hardware.\n\nAfter a reboot: `4lm start`. There's no autostart and that's a feature.\n\nBackend-only variant:\n\n```sh\nmake bootstrap BACKEND_ONLY=1   # skips opencode brew formula\n./install.sh --backend-only     # skips Open WebUI + opencode + their plists/configs\n4lm start \u0026\u0026 4lm expose lan --confirm\n```\n\nSee [`docs/setup.md`](docs/setup.md) for the operator runbook,\nincluding the `OPENAI_API_BASE_URL` wiring for consumer hosts on\nthe LAN.\n\n## Profile lineup\n\nSix profiles. The three Qwen3-stack tiers share an 8B embedder so\nknowledge bases stay valid across switches.\n\n| Profile         | Backend | Coder                   | Chat            | Embed | Rerank | Vision | Steady | Fits on |\n|-----------------|---------|-------------------------|-----------------|-------|--------|--------|--------|---------|\n| `lean`          | omlx    | Qwen3-Coder-30B-A3B     | Qwen3.6-35B-A3B | 8B    | 0.6B   | —      | ~40 GB | 64 GB+  |\n| `default`       | omlx    | Qwen3-Coder-Next (80B)  | Qwen3.6-35B-A3B | 8B    | 0.6B   | VL-8B  | ~65 GB | 96 GB+  |\n| `max-100gb`     | omlx    | Qwen3-Coder-Next (80B)  | Qwen3-Next-80B  | 8B    | 4B     | VL-8B  | ~92 GB | 128 GB  |\n| `mlx-coding`    | omlx    | Qwen3-Coder-Next (80B)  | —               | —     | —      | —      | ~42 GB | 64 GB+  |\n| `mlx-knowledge` | omlx    | —                       | Qwen3.6-35B-A3B | 8B    | 0.6B   | —      | ~23 GB | 36 GB+  |\n| `ollama`        | ollama  | qwen3-coder-next:q4_K_M | —               | —     | —      | —      | ~22 GB | 36 GB+  |\n\n**Memory math for `default` on a 128 GB Mac.** Qwen3-Coder-Next 80B\n(~42 GB 4-bit) + Qwen3.6-35B-A3B (~12 GB) + Qwen3-Embedding-8B\n(~5 GB) + Qwen3-Reranker-0.6B (~0.4 GB) + Qwen3-VL-8B (~5 GB) ≈\n65 GB steady. Both 80B-class models are MoE → ~3B active params each\n→ KV cache and batched decoding fit comfortably in the remaining\n~33 GB of the wired-memory budget.\n\nThe everyday ladder is `lean` → `default` → `max-100gb`. `mlx-coding`\nstrips everything except the 80B coder so long agentic sessions get\nmaximum KV-cache headroom. `mlx-knowledge` is the text-only vault-\nsynthesis tier. `ollama` is the GGUF smoke test — switch to it\noccasionally to confirm Ollama still works, then switch back.\n\nEach profile YAML carries an extensive header comment documenting\nslot-by-slot rationale, memory math, when-to-use, and\nassumptions-to-validate.\n\n## Architecture\n\n```\n                       4lm  (single control command)\n                              │ bootstrap / bootout / kickstart\n                              ▼\n        ┌──────────────────────────────────┐    ┌──────────────────────────────────┐\n        │ com.4lm.backend                  │    │ com.4lm.webui                    │\n        │   omlx | mlx_lm | ollama         │    │   open-webui serve               │\n        │   :8000 (OpenAI API)             │←───│   :3000 (Web UI)                 │\n        └──────────────────────────────────┘    └──────────────────────────────────┘\n                  ▲                                ▲\n                  │ HTTP                           │ HTTP (browser)\n            ┌─────┴────┐                     ┌─────┴────┐\n            │ opencode │                     │  Safari  │\n            │   TUI    │                     │  Chrome  │\n            └──────────┘                     └──────────┘\n```\n\nThe backend is the source of truth. Open WebUI is a stateless\nfrontend proxying to it. `opencode` talks directly to `:8000/v1`.\nNone of them know or care about each other → the OpenAI-compatible\nAPI is the seam.\n\nIn `--backend-only` mode the WebUI block is absent on disk; `bin/4lm`\nprobes for the plist and treats the layer as not installed. Consumer\nhosts on the LAN run their own clients against the backend's\n`:8000/v1`.\n\n## Common operations\n\n```sh\n# Lifecycle\n4lm start | stop | restart [backend|webui|all]\n4lm                                # status (alias for `4lm status`; --json for parseable)\n4lm logs [backend|webui]           # tail -F\n\n# Profiles\n4lm profile list\n4lm profile set \u003cname\u003e             # atomic, validated, rollback on failure\n4lm profile show [\u003cname\u003e]          # YAML of active (or named) profile\n4lm profile validate [--all]\n\n# Models\nmake models                        # download/update everything in config/profiles/\n4lm model list                     # what's loaded vs cached\n4lm model recommend [\u003cuse-case\u003e]   # top picks via llmfit + localmaxxing benchmarks\n\n# Updates\n4lm outdated                       # check PyPI / Homebrew / HF\n4lm upgrade [brew|models|python]   # apply pending updates\n\n# Diagnostics\n4lm doctor                         # prereqs + smoke-test inference\n4lm diag                           # live clients, in-flight inference, top CPU\n\n# Network exposure (refuses without --confirm)\n4lm expose lan --confirm\n4lm expose local --confirm\n\n# Autostart at login (off by default)\n4lm autostart enable [backend|webui|all]\n4lm autostart disable [backend|webui|all]\n4lm autostart status\n\n# Removal\n4lm uninstall                      # bootout, remove ~/.local/bin/4lm; keep ~/.4lm/\nmake uninstall                     # full: bootout, ~/.4lm/, sudoers, newsyslog, pipx packages\n```\n\nEvery command has `--help`.\n\n## File layout\n\n```\n~/.4lm/\n├── bin/                   control + wrappers (called by launchd)\n├── launchd/               com.4lm.{backend,webui}.plist\n├── config/\n│   ├── active-profile     symlink → profiles/\u003cactive\u003e.yaml\n│   ├── previous-profile   plain text, used for rollback\n│   ├── network.yaml       bind mode + ports (single config channel)\n│   ├── webui_secret_key   mode 0600, generated on first lan-mode start\n│   └── profiles/          lean / default / max-100gb / mlx-coding /\n│                          mlx-knowledge / ollama YAMLs\n├── logs/                  backend.log, webui.log (merged stdout+stderr)\n└── openwebui-data/        Open WebUI db, settings, RAG index\n\n~/.local/bin/4lm                   symlink to ~/.4lm/bin/4lm\n~/.config/opencode/opencode.jsonc  seeded by install.sh from the repo template\n/etc/sudoers.d/4lm-stack           NOPASSWD for the backend's sysctl call\n/etc/newsyslog.d/4lm.conf          log rotation (10 MB, 7 generations, gzipped)\n```\n\n## What this isn't\n\n- Not a multi-user server. Single-user, single-host, single login session.\n- Not a Docker or pip-into-system-Python project. PEP 668 is honoured\n  via pipx; Python 3.12 is pinned for compatibility with the MLX ecosystem.\n- Not auto-starting. After reboot you run `4lm start`. On purpose.\n- Not a cloud-fallback router. If the local stack fails, fall back by\n  hand; 4lm does not silently route to Anthropic, OpenAI, or anyone else.\n\n## Where this is going\n\nToday: best Apple Silicon models, fastest local inference, a\nClaude-Desktop-shaped Open WebUI in front of it. Next: tool calling\n+ MCP so local models can *use* Open WebUI's tools (web search, RAG,\ncode interpreter, memory) instead of hallucinating them — and reach\ntoward Claude-Desktop feature parity. The phased roadmap lives in\n[`specs/sdd/webui-tools-and-mcp.md`](specs/sdd/webui-tools-and-mcp.md).\nFoundation is shipped at v0.6; agentic phases are draft.\n\n## Development\n\n```sh\nmake bootstrap    # one-time: Brewfile + Brewfile-tui + pipx ensurepath\nmake lint         # shellcheck + shfmt -d\nmake fmt          # shfmt -w\nmake test         # bats + pytest\nmake check        # everything; CI runs this on macos-latest\nmake ci           # mirror the CI matrix locally (default + backend-only legs)\n```\n\nCI on every PR (`.github/workflows/ci.yml`):\n`shellcheck`, `shfmt -d`, `bash -n`, `plutil -lint`, `xmllint --noout`,\nprofile YAML validation, bats suite — both install modes — driven by\n`make check`.\n\n## Conventions\n\n- Bash scripts: `set -euo pipefail`, `shellcheck` clean, `shfmt -i 2 -ci`.\n- Plists carry `__HOME__` placeholders, substituted by `install.sh`.\n- Profiles validated by `validate_profile()` in `bin/4lm` (regex name\n  whitelist, schema check, parser-enum match against backend defaults).\n- Conventional-commit prefixes (`feat:`, `fix:`, `refactor:`, `chore:`,\n  `docs:`, `test:`); subject ≤ 72 chars.\n- No `Co-Authored-By: Claude` lines.\n- Formatting-only changes ship in their own commits.\n\n## Model history (short)\n\n4lm migrated through `mlx-openai-server` → `mlx_lm` → `omlx` as Apple\nSilicon MLX tooling matured. The initial model set (GLM-4.7-Flash +\nQwen3.6-35B-A3B) gave way to Qwen3-Coder + Qwen3.6-27B in v0.3 after\nthe Qwen3 thinking-loop bug forced a custom Jinja workaround. v0.6\nconsolidates onto the full Qwen3 stack (coder + chat + embed + rerank\n+ vision) on a single omlx process. Per-version detail and the\nthinking-mode template story live in [`CHANGELOG.md`](CHANGELOG.md).\n\n## Credits\n\n4lm is glue around several upstream projects that do the actual heavy\nlifting. Go give them stars:\n\n- [**omlx**](https://github.com/jundot/omlx) — vLLM-style MLX inference\n  server with paged KV cache, continuous batching, and multi-model\n  EnginePool. Primary backend.\n- [**mlx_lm**](https://github.com/ml-explore/mlx-lm) — Apple's reference\n  MLX language-model library. Alternative single-model backend.\n- [**Ollama**](https://github.com/ollama/ollama) — llama.cpp + Metal\n  GGUF serving. Smoke-test backend.\n- [**Open WebUI**](https://github.com/open-webui/open-webui) — the\n  frontend. Without their PersistentConfig surface this stack would be\n  half the experience.\n- [**opencode**](https://github.com/sst/opencode) — the TUI client.\n- [**Qwen team @ Alibaba**](https://github.com/QwenLM) — the model\n  family carrying the default profile (Qwen3-Coder, Qwen3.6,\n  Qwen3-Embedding, Qwen3-Reranker, Qwen3-VL). Apache-2.0 licensed.\n- [**llmfit**](https://github.com/lukaspustina/llmfit) +\n  [**localmaxxing.com**](https://localmaxxing.com) — hardware-fit\n  scoring and community benchmarks used by `4lm model recommend`.\n\n## License\n\n[MIT](LICENSE).\n\n## Documentation\n\n- [`docs/setup.md`](docs/setup.md) — operator runbook (sudoers, troubleshooting, model pulls, LAN client wiring)\n- [`docs/profile-schema.md`](docs/profile-schema.md) — YAML key reference for all backends\n- [`docs/autostart.md`](docs/autostart.md) — opt-in login autostart mechanics\n- [`specs/sdd/webui-tools-and-mcp.md`](specs/sdd/webui-tools-and-mcp.md) — active SDD for tool calling + MCP\n- [`specs/done/sdd/4lm-rework-2026-05-09.md`](specs/done/sdd/4lm-rework-2026-05-09.md) — archived design doc this repo implements\n- [`CONTRIBUTING.md`](CONTRIBUTING.md) — dev setup, commit style, PR checklist\n- [`SECURITY.md`](SECURITY.md) — threat model + vulnerability reporting\n- [`CLAUDE.md`](CLAUDE.md) — orientation for AI assistants working in this repo\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukaspustina%2F4lm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukaspustina%2F4lm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukaspustina%2F4lm/lists"}