{"id":50489181,"url":"https://github.com/hvasconcelos/mlxforge","last_synced_at":"2026-06-02T01:02:42.788Z","repository":{"id":361948334,"uuid":"1256544369","full_name":"hvasconcelos/mlxforge","owner":"hvasconcelos","description":"From-scratch LLaMA inference engine in C++ on Apple MLX — OpenAI-compatible HTTP API with continuous batching, Apple Silicon (Metal) only.","archived":false,"fork":false,"pushed_at":"2026-06-01T22:53:25.000Z","size":674,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-02T00:30:39.819Z","etag":null,"topics":["apple-silicon","cpp","inference","llama","llm","metal","mlx","openai-api"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hvasconcelos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-01T22:01:12.000Z","updated_at":"2026-06-01T22:53:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hvasconcelos/mlxforge","commit_stats":null,"previous_names":["hvasconcelos/mlxforge"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/hvasconcelos/mlxforge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Fmlxforge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Fmlxforge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Fmlxforge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Fmlxforge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hvasconcelos","download_url":"https://codeload.github.com/hvasconcelos/mlxforge/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hvasconcelos%2Fmlxforge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33800676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-01T02:00:06.963Z","response_time":115,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","cpp","inference","llama","llm","metal","mlx","openai-api"],"created_at":"2026-06-02T01:02:37.941Z","updated_at":"2026-06-02T01:02:42.779Z","avatar_url":"https://github.com/hvasconcelos.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mlxforge\n\nA from-scratch LLaMA inference engine in **C++ on Apple MLX**, served behind an\n**OpenAI-compatible HTTP API** with **continuous batching**.\n\nmlxforge loads raw safetensors weights, runs a numerically-correct transformer\nforward pass on the Metal GPU, and serves concurrent users through a vLLM-style\nsingle-worker / three-queue scheduler. Every numerically-sensitive phase is\nvalidated against an `mlx-lm` golden reference, because the failure mode here is\n**silent garbage, not a crash**.\n\nPrimary model: `mlx-community/Llama-3.2-1B-Instruct` (fp16 by default; optional\n4-bit). 16 layers, hidden 2048, 32 query / 8 KV heads (GQA), head_dim 64,\nRMSNorm, RoPE (llama3 scaling), SwiGLU, tied embeddings.\n\nThe forward pass is architecture-shared across the LLaMA family, so\n**Mistral-7B-Instruct-v0.3** is also supported (RMSNorm + SwiGLU + plain RoPE +\nGQA + separate LM head, no attention bias, no sliding window). See\n[Supported models](#supported-models).\n\n## Features\n\n- **Numerically correct** — forward-pass logits and greedy tokens match `mlx-lm`\n  (golden-reference `.npy` fixtures gate every step).\n- **KV cache** — single-sequence and batched (`BatchKVCache`), left-padded,\n  grown in 256-token blocks, with `filter` (eviction) / `merge` (admission).\n- **Continuous batching** — one GPU worker thread owns all MLX state and is the\n  only caller of `eval`/`async_eval`; exactly **one `async_eval` per decode\n  step** over the whole batch; batch-size bucketing so the graph shape recurs.\n- **Sampling as graph ops** — greedy, temperature, top-k, top-p (no host\n  readback of logits).\n- **C++ tokenizer** — HF `tokenizer.json` via `tokenizers-cpp`, the chat template\n  (Llama-3.2 or Mistral, selected from `config.json`'s `model_type`), and\n  UTF-8-safe incremental detokenization.\n- **OpenAI server** (cpp-httplib) — `/v1/chat/completions`, `/v1/completions`,\n  `/v1/models`, `/health`; non-streaming and SSE streaming; cancellation on\n  client disconnect; per-request metrics; OpenAI-shaped errors (400/429/503).\n- **Optional 4-bit quantization** — `quantized_matmul` (group_size 64), ~0.65\n  GiB resident vs ~2.3 GiB fp16.\n\n## Requirements\n\n- Apple Silicon (the MLX Metal backend) + the Xcode **Metal Toolchain**\n  (`xcodebuild -downloadComponent MetalToolchain`).\n- CMake ≥ 3.24, a C++17 compiler (Apple clang).\n- `cargo` / Rust — `tokenizers-cpp` builds the Rust HF `tokenizers` crate.\n- (Optional, for regenerating golden fixtures) Python 3.12 + `mlx-lm`.\n\nAll C++ dependencies (MLX, cpp-httplib, nlohmann/json, doctest, tokenizers-cpp)\nare fetched and pinned by CMake — see `cmake/Dependencies.cmake`.\n\n## Build\n\n```sh\ncmake -S . -B build\ncmake --build build --parallel\n```\n\nThe first build compiles MLX's Metal kernels and the Rust tokenizer crate, so it\ntakes a few minutes. Outputs:\n\n- `build/mlxforge` — the OpenAI HTTP server\n- `build/mlxforge-cli` — CLI (weight dump + single-stream generation)\n- `build/tests/mlxforge_tests` — the doctest suite\n\n## Get the model\n\n```sh\n# fp16 (full precision)\nhuggingface-cli download mlx-community/Llama-3.2-1B-Instruct-bf16\n# or 4-bit\nhuggingface-cli download mlx-community/Llama-3.2-1B-Instruct-4bit\n```\n\n`MODEL_DIR` below is the resolved snapshot directory under\n`~/.cache/huggingface/hub/.../snapshots/\u003crev\u003e` (or any local dir containing\n`config.json`, `tokenizer.json`, and `model.safetensors`).\n\n## Supported models\n\n| Family | Example repo | Chat format |\n| --- | --- | --- |\n| Llama-3.2 | `mlx-community/Llama-3.2-1B-Instruct-bf16` / `-4bit` | `\u003c|start_header_id|\u003e…` |\n| Mistral | `mlx-community/Mistral-7B-Instruct-v0.3-4bit` | `\u003cs\u003e[INST] … [/INST]` |\n\nThe transformer (`model/llama`) is shared — adding the Mistral family needed no\nforward-pass changes, only tokenizer/chat-format work. The chat template is\nselected automatically from `config.json`'s `model_type`, and BOS / special-token\nhandling is driven by `config.json` + `tokenizer.json` (no hard-coded ids), so\npointing the server or CLI at a Mistral snapshot just works:\n\n```sh\nhuggingface-cli download mlx-community/Mistral-7B-Instruct-v0.3-4bit\n./build/mlxforge-cli --model \"$MISTRAL_DIR\" \"What is the capital of France?\"\n```\n\nEach model is gated against its own `mlx-lm` golden reference. Regenerate the\nfixtures (rarely needed) with:\n\n```sh\nreference/.venv/bin/python reference/dump_ref.py --model llama     # -\u003e reference/fixtures/\nreference/.venv/bin/python reference/dump_ref.py --model mistral   # -\u003e reference/fixtures_mistral/\n```\n\n**Limitations.** Mistral's `[INST]` template has no system role, so a leading\nsystem message is folded into the first user turn. Sliding-window attention\n(Mistral v0.1) and tool/function-calling tokens are not implemented; v0.2/v0.3\ndisable the sliding window and so run as plain causal attention.\n\n## Run the server\n\n```sh\n./build/mlxforge \"$MODEL_DIR\" --port 8080 --max-ctx 8192 --max-waiting 256\n```\n\nThen use the official `openai` client:\n\n```python\nfrom openai import OpenAI\nc = OpenAI(base_url=\"http://127.0.0.1:8080/v1\", api_key=\"x\")\n\n# non-streaming\nr = c.chat.completions.create(model=\"mlxforge\",\n    messages=[{\"role\": \"user\", \"content\": \"What is the capital of France?\"}],\n    max_tokens=32)\nprint(r.choices[0].message.content)            # \"The capital of France is Paris.\"\n\n# streaming\nfor ev in c.chat.completions.create(model=\"mlxforge\",\n        messages=[{\"role\": \"user\", \"content\": \"Tell me a joke.\"}],\n        max_tokens=64, stream=True):\n    print(ev.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\nConfig knobs are also read from the environment (`MLXFORGE_HOST`, `MLXFORGE_PORT`,\n`MLXFORGE_MAX_CTX`, `MLXFORGE_MAX_WAITING`, `MLXFORGE_KV_BUDGET`). `SIGINT`/`SIGTERM`\ntrigger a graceful shutdown that drains in-flight requests.\n\n## Run the CLI\n\n```sh\n# stream generated text from a chat prompt\n./build/mlxforge-cli generate \"$MODEL_DIR\" \"What is the capital of France?\" 64\n\n# generate from a pre-tokenized .npy prompt (ids printed/streamed)\n./build/mlxforge-cli generate \"$MODEL_DIR\" reference/fixtures/prompt_0_ids.npy 20\n\n# inspect weights: key -\u003e shape -\u003e dtype, assert fp16, report peak memory\n./build/mlxforge-cli dump-weights \"$MODEL_DIR\"\n```\n\n## Tests\n\n```sh\nctest --test-dir build --output-on-failure\n```\n\nTests come in two tiers:\n\n- **Unit tests** — pure logic, no GPU/weights (config parsing, sanitize,\n  KV-cache bookkeeping, sampler math, request/response (de)serialization, SSE\n  framing). Always run.\n- **Golden-reference / integration tests** — numerically-sensitive stages\n  validated against the `mlx-lm` reference dump (tensor-closeness / exact tokens)\n  and the continuous-batching scheduler. These run only if the model is present\n  locally; otherwise they self-skip (CMake globs the HF cache for the snapshot\n  dir).\n\nTo (re)generate the golden fixtures (committed under `reference/fixtures/`):\n\n```sh\npython3.12 -m venv reference/.venv\nreference/.venv/bin/pip install mlx-lm numpy\nreference/.venv/bin/python reference/dump_ref.py\n```\n\n## Architecture\n\n```\nHTTP request ─▶ server/http_server (cpp-httplib)\n                │  parse OpenAI JSON, apply chat template, tokenize\n                ▼\n            scheduler/  ── waiting queue (mutex + cv) ──▶ runtime/worker\n                                                          (the ONE GPU thread)\n                                                          │ owns weights,\n                                                          │ BatchKVCache, sampler\n                                                          ▼\n   admit (prefill ─▶ merge) ─▶ decode step (1 async_eval/batch) ─▶ evict (filter)\n                                                          │\n   each row's tokens ─▶ per-request bounded TokenQueue ─▶ SSE / blocking response\n```\n\nSource layout (`src/`):\n\n| Module | Responsibility |\n|---|---|\n| `core/config` | parse `config.json` into `ModelConfig` (incl. rope_scaling, quantization) |\n| `core/weights` | load safetensors (single/sharded), sanitize keys, fp16-cast |\n| `model/llama` | the transformer: embedding, RMSNorm, RoPE, GQA SDPA, SwiGLU, LM head; fp16 + quantized paths; single-stream and batched forward |\n| `cache/kv_cache` | single-sequence KV cache |\n| `cache/batch_kv_cache` | batched, left-padded KV cache: `update_and_fetch`, `filter`, `merge`, `pad_dummies` |\n| `cache/kv_budget` | KV memory projection / admission gate |\n| `sample/sampler` | greedy / temperature / top-k / top-p as MLX graph ops |\n| `scheduler/request` | `Request` + bounded `TokenQueue` |\n| `scheduler/scheduler` | the waiting queue + handoff |\n| `runtime/worker` | the single GPU worker: admit / decode / evict loop |\n| `runtime/batching` | prefill pass + batch-size bucketing |\n| `runtime/single_stream` | the CLI's greedy generation loop |\n| `tokenizer/tokenizer` | `tokenizers-cpp` wrapper, chat template, streaming detokenizer |\n| `server/openai` | OpenAI request parse + response serialize (pure) |\n| `server/http_server` | routes, blocking + SSE handlers, error shapes |\n| `server/config` | CLI/env server configuration |\n\nSee `SPECIFICATION.md` for the full design and `STORIES.md` for the 25-story\nimplementation breakdown.\n\n## License\n\nResearch/educational. Model weights are subject to their own licenses (Llama\nCommunity License for the underlying Llama-3.2 weights).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhvasconcelos%2Fmlxforge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhvasconcelos%2Fmlxforge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhvasconcelos%2Fmlxforge/lists"}