{"id":50752431,"url":"https://github.com/hvasconcelos/libmlxforge","last_synced_at":"2026-06-11T02:05:36.770Z","repository":{"id":361948334,"uuid":"1256544369","full_name":"hvasconcelos/libmlxforge","owner":"hvasconcelos","description":"Embeddable, batched MLX LLM engine for Apple Silicon — one C ABI, bound from Node, Swift \u0026 Rust. Continuous batching, streaming, JSON-constrained structured output, and embeddings.","archived":false,"fork":false,"pushed_at":"2026-06-09T22:04:43.000Z","size":3538,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-09T22:12:02.197Z","etag":null,"topics":["apple-silicon","cpp","inference","llama","llm","metal","mlx","openai-api"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hvasconcelos.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-01T22:01:12.000Z","updated_at":"2026-06-09T20:51:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hvasconcelos/libmlxforge","commit_stats":null,"previous_names":["hvasconcelos/mlxforge","hvasconcelos/libmlxforge"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/hvasconcelos/libmlxforge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Flibmlxforge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Flibmlxforge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Flibmlxforge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Flibmlxforge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hvasconcelos","download_url":"https://codeload.github.com/hvasconcelos/libmlxforge/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Flibmlxforge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34178825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","cpp","inference","llama","llm","metal","mlx","openai-api"],"created_at":"2026-06-11T02:05:36.038Z","updated_at":"2026-06-11T02:05:36.765Z","avatar_url":"https://github.com/hvasconcelos.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# libmlxforge\n\n**An embeddable, batched MLX LLM engine you bind from any language.**\n\nA from-scratch local inference engine in **C++ on Apple MLX** with **continuous\nbatching**, exposed through a stable C ABI so Node (first), then Swift and Rust, can\nembed the *same batched engine* in-process — not just array ops, and not one stream at a\ntime.\n\nmlxforge loads raw safetensors / GGUF weights, runs a numerically-correct transformer\nforward pass on the Metal GPU, and serves many concurrent requests through a vLLM-style\nsingle-worker / three-queue scheduler. Every numerically-sensitive phase is validated\nagainst an `mlx-lm` golden reference, because the failure mode here is **silent garbage,\nnot a crash** — the guarantee an embedded engine has to make.\n\n\u003e **The library is the product.** The OpenAI-compatible HTTP server and the CLI in this\n\u003e repo are **QA harnesses** that exercise and prove the engine's stability — they are\n\u003e dev/QA tools, not the deliverable. See [`doc/applications.md`](./doc/applications.md).\n\n## Why mlxforge (vs the rest of the MLX ecosystem)\n\nApple Silicon already has plenty of ways to run a local model — but each existing option\nis either an *array framework* (you build the engine yourself), a *single-stream* LLM\nlib, or a *Python server* you can't embed. None is a complete, batched engine bindable\nfrom another language. That's the gap mlxforge fills:\n\n| Option | What it is | In-process | MLX-native | Batched / concurrent |\n| --- | --- | :---: | :---: | :---: |\n| `mlx-c` / `mlx-rs` / `mlx-rust` | array-framework bindings | ✅ | ✅ | build it yourself |\n| **`node-llama-cpp`** | Node LLM lib (llama.cpp/GGUF) | ✅ | ❌ | ⚠️ weak |\n| **`node-mlx`** | Node LLM lib (MLX) | ✅ | ✅ | ❌ single-stream |\n| Apple **`MLXLLM`** (mlx-swift) | Swift LLM lib | ✅ | ✅ | ❌ single-stream |\n| **`vllm-mlx`** / **`omlx`** | Python MLX servers | ❌ (HTTP) | ✅ | ✅ |\n| Ollama (`+ ollama` npm) | Go sidecar (MLX since 0.19) | ❌ (HTTP) | ✅ | ✅ |\n| **mlxforge** | **batched engine + C ABI** | ✅ | ✅ | ✅ |\n\nmlxforge is the only row that is **in-process + MLX-native + batched** — and the only\nbatched MLX engine you can bind from Node/Swift/Rust instead of running as a separate\nPython/Go process. See [`doc/embedding.md`](./doc/embedding.md) for the full thesis and\nthe C-ABI / Node quickstart.\n\n## Features\n\n- **Embeddable batched engine** — `runtime/engine` is an HTTP-free, in-process engine\n  (one GPU worker + scheduler + batched KV cache) wrapped by a stable `extern \"C\"` ABI\n  (`src/capi/mlxforge.h`, *in progress*) and bound from other languages. This is the\n  product.\n- **Numerically correct** — forward-pass logits and greedy tokens match `mlx-lm`\n  (golden-reference `.npy` fixtures gate every step).\n- **Continuous batching** — one GPU worker thread owns all MLX state and is the only\n  caller of `eval`/`async_eval`; exactly **one `async_eval` per decode step** over the\n  whole batch, with batch-size bucketing.\n- **KV cache** — single-sequence and batched (`BatchKVCache`), left-padded, grown in\n  256-token blocks, with `filter` (eviction) / `merge` (admission).\n- **Sampling as graph ops** — greedy, temperature, top-k, top-p (no host readback).\n- **Embeddings** — `engine.embed` runs the decoder to its final hidden states, pools\n  (mean or last-token) and L2-normalizes. **Qwen3-Embedding** is first-class. Exposed\n  through the C ABI, the bindings, the CLI `embed` command, and `POST /v1/embeddings`.\n- **C++ tokenizer** — from-scratch byte-level BPE and SentencePiece-BPE over HF\n  `tokenizer.json` (no Rust), with chat templates and UTF-8-safe streaming detokenization.\n- **OpenAI server harness** — drives the engine over HTTP (`/v1/chat/completions`,\n  `/v1/completions`, `/v1/embeddings`, `/v1/models`, `/health`), streaming + tool calling.\n  Built only when `MLXFORGE_BUILD_SERVER=ON`; the released library ships without it.\n- **Optional 4-bit quantization** — `quantized_matmul` (group_size 64), ~0.65 GiB\n  resident vs ~2.3 GiB fp16.\n- **Configurable logging** (spdlog) — level / file / format via `MLXFORGE_LOG_*`.\n\n## Supported models\n\nThe forward pass (the `DecoderModel` base in `model/`) is family-shared and runs\nLlama-3.2, Qwen3 (dense / MoE), and Qwen3.5 hybrid models from safetensors (fp16 /\n4-bit) or a single-file GGUF, plus **Qwen3-VL** vision-language (image → text). The\nchat template and special-token handling are selected from `config.json` — loading\nis org-agnostic.\n\n| Family | Example repo | Precision |\n| --- | --- | --- |\n| Llama-3.2 | `mlx-community/Llama-3.2-1B-Instruct-bf16` / `-4bit` | fp16 / 4-bit |\n| Llama-3.2 (GGUF) | `bartowski/Llama-3.2-1B-Instruct-GGUF` | Q4_0/Q4_1/Q8_0, Q4_K/Q5_K/Q6_K |\n| Qwen3 (dense) | `mlx-community/Qwen3-0.6B-bf16` / `Qwen3-4B-4bit` | fp16 / 4-bit |\n| Qwen3 (GGUF) | `Qwen/Qwen3-0.6B-GGUF` | Q4_0/Q4_1/Q8_0, Q4_K/Q5_K/Q6_K |\n| Qwen3 (MoE) | `mlx-community/Qwen3-30B-A3B-4bit` | 4-bit / fp16 |\n| Qwen3.5 (hybrid) | `mlx-community/Qwen3.5-0.8B-4bit` | 4-bit (text tower) |\n| Qwen3-VL (vision) | `mlx-community/Qwen3-VL-4B-Instruct-4bit` | 4-bit (image → text) |\n\nSee [`doc/supported-models.md`](./doc/supported-models.md) for the full compatibility\nmatrix, the per-family deltas, and how to add a new family.\n\n## Requirements\n\n- Apple Silicon (the MLX Metal backend) + the Xcode **Metal Toolchain**\n  (`xcodebuild -downloadComponent MetalToolchain`).\n- CMake ≥ 3.24, a C++17 compiler (Apple clang).\n- System **libcurl** (in the macOS SDK; found via `find_package(CURL)`) — for\n  HuggingFace downloads.\n- (Optional, for regenerating golden fixtures) Python 3.12 + `mlx-lm`.\n\nAll C++ dependencies (MLX, cpp-httplib, nlohmann/json, doctest, spdlog) are fetched and\npinned by CMake (`cmake/Dependencies.cmake`). The tokenizer is our own, so there is no\nRust/`cargo` requirement.\n\n## Build\n\n```sh\ncmake -S . -B build\ncmake --build build --parallel\n```\n\nThe first build compiles MLX's Metal kernels, so it takes a few minutes. Outputs:\n\n- `build/libmlxforge.dylib` — **the product**: the engine behind the C ABI\n  (`src/capi/mlxforge.h`); link this to embed mlxforge (see\n  [`doc/embedding.md`](./doc/embedding.md)).\n- `build/mlxforge` — the OpenAI HTTP server *harness*.\n- `build/mlxforge-cli` — CLI *harness* (weight dump + single-stream generation).\n- `build/tests/mlxforge_tests` — the doctest suite.\n\nThe harnesses are on by default for development; the released library is built with them\noff (a lean dylib, no httplib/curl). For the build-option matrix see\n[`doc/applications.md`](./doc/applications.md).\n\n## Quickstart\n\nThe server and CLI accept a model **spec** — a HuggingFace repo id (downloaded and\ncached on first use) or a local model directory — passed directly, llama.cpp-style:\n\n```sh\n# CLI: stream generated text from a chat prompt (downloads the model if needed)\n./build/mlxforge-cli generate mlx-community/Llama-3.2-1B-Instruct-4bit \\\n  \"What is the capital of France?\" 64\n\n# Server: speak the OpenAI API so existing clients can hammer the scheduler\n./build/mlxforge -m mlx-community/Llama-3.2-1B-Instruct-4bit --port 8080\n```\n\n```python\nfrom openai import OpenAI\nc = OpenAI(base_url=\"http://127.0.0.1:8080/v1\", api_key=\"x\")\nr = c.chat.completions.create(model=\"mlxforge\",\n    messages=[{\"role\": \"user\", \"content\": \"What is the capital of France?\"}],\n    max_tokens=32)\nprint(r.choices[0].message.content)            # \"The capital of France is Paris.\"\n```\n\nHow specs are resolved (HF cache reuse, download cache, gated repos) is documented in\n[`doc/supported-models.md`](./doc/supported-models.md); the harness flags, endpoints,\nstreaming, tool calling, and `embed` are in [`doc/applications.md`](./doc/applications.md).\n\n## Documentation\n\nThe [`doc/`](./doc) folder is the design reference:\n\n- [`doc/embedding.md`](./doc/embedding.md) — **start here**: the library / cross-language\n  batched-engine thesis, the competitive landscape, and the C-ABI + Node quickstart.\n- [`doc/architecture.md`](./doc/architecture.md) — engine architecture, the\n  single-GPU-thread model, request lifecycle, continuous batching, and the module map.\n- [`doc/llm-architecture.md`](./doc/llm-architecture.md) — the transformer forward pass\n  (embedding, RMSNorm, RoPE, GQA attention, SwiGLU, KV cache, sampling, quantization).\n- [`doc/supported-models.md`](./doc/supported-models.md) — model families, getting the\n  weights, spec resolution, and adding a new family.\n- [`doc/tokenizer.md`](./doc/tokenizer.md) — the from-scratch byte-level BPE / SPM\n  tokenizers and their byte-exact validation against mlx-lm.\n- [`doc/applications.md`](./doc/applications.md) — the server and CLI **harnesses**,\n  the build options, the OpenAI API surface, and logging.\n- [`doc/distribution.md`](./doc/distribution.md) — release artifacts and how Homebrew,\n  SwiftPM, npm, and the Rust crate consume them.\n- [`doc/abi-stability.md`](./doc/abi-stability.md) — the C-ABI versioning policy and the\n  automated symbol/leak guard.\n- [`doc/conformance.md`](./doc/conformance.md) — validate a build against the `mlx-lm`\n  golden reference.\n- [`doc/contributing.md`](./doc/contributing.md) — build/test workflow, conventions, the\n  golden-reference discipline, and the hard-won numerical gotchas.\n\n## License\n\nmlxforge is released under the [MIT License](./LICENSE).\n\nModel weights are **not** covered by this license and remain subject to their own terms\n(e.g. the Llama Community License for the Llama-3.2 weights).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhvasconcelos%2Flibmlxforge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhvasconcelos%2Flibmlxforge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhvasconcelos%2Flibmlxforge/lists"}