{"id":47351817,"url":"https://github.com/jjang-ai/vmlx","last_synced_at":"2026-06-18T01:01:16.991Z","repository":{"id":344784462,"uuid":"1160596966","full_name":"jjang-ai/vmlx","owner":"jjang-ai","description":"vMLX - JANGTQ Uber Compressed MLX Models - L2 Disk Cache (survives restart) + L1 Paged (super fast ttft) + Hybrid SSM Scheduler  + Cont Batching + etc!","archived":false,"fork":false,"pushed_at":"2026-06-16T00:25:44.000Z","size":90273,"stargazers_count":671,"open_issues_count":36,"forks_count":70,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-06-16T02:15:30.307Z","etag":null,"topics":["anthropic-api","kvcache-compression","kvcache-optimization","kvcache-reuse","llm","lmstudio","macbook","mcp-server","mlx","mlxllm","mlxstudio","omlx","omlx-alternative","openai-api","openclaw","openclaw-agent","persistent-memory","prefix-cache","vmlx"],"latest_commit_sha":null,"homepage":"https://vmlx.net","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jjang-ai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-18T06:11:55.000Z","updated_at":"2026-06-16T00:48:20.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jjang-ai/vmlx","commit_stats":null,"previous_names":["jjang-ai/vmlx"],"tags_count":124,"template":false,"template_full_name":null,"purl":"pkg:github/jjang-ai/vmlx","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjang-ai%2Fvmlx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjang-ai%2Fvmlx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjang-ai%2Fvmlx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjang-ai%2Fvmlx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jjang-ai","download_url":"https://codeload.github.com/jjang-ai/vmlx/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjang-ai%2Fvmlx/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34471639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic-api","kvcache-compression","kvcache-optimization","kvcache-reuse","llm","lmstudio","macbook","mcp-server","mlx","mlxllm","mlxstudio","omlx","omlx-alternative","openai-api","openclaw","openclaw-agent","persistent-memory","prefix-cache","vmlx"],"created_at":"2026-03-18T00:17:28.370Z","updated_at":"2026-06-18T01:01:16.941Z","avatar_url":"https://github.com/jjang-ai.png","language":"Python","funding_links":["https://ko-fi.com/jangml"],"categories":["Python","Rising projects"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/logo-wide-dark.png\"\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/logo-wide-light.png\"\u003e\n    \u003cimg alt=\"vMLX\" src=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/logo-wide-light.png\" width=\"400\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003eMLX Inference Server for Apple Silicon\u003c/h3\u003e\n\n\u003cp align=\"center\"\u003e\n  Self-hosted inference server for LLMs, VLMs, and image generation on Apple Silicon.\u003cbr\u003e\n  OpenAI + Anthropic + Ollama compatible HTTP API. Self-hosted; no third-party API keys required.\u003cbr\u003e\n  Native MTP artifact detection and family-specific cache policy gates keep speculative/cache settings explicit and model-safe.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cem\u003eLooking for a native Swift macOS app or Swift inference engine? See \u003ca href=\"https://osaurus.ai\"\u003eosaurus.ai\u003c/a\u003e.\u003c/em\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/vmlx/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/vmlx?color=%234B8BBE\u0026label=PyPI\u0026logo=python\u0026logoColor=white\" alt=\"PyPI\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/jjang-ai/vmlx/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-Apache_2.0-green?logo=apache\" alt=\"License\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/jjang-ai/vmlx\"\u003e\u003cimg src=\"https://img.shields.io/github/stars/jjang-ai/vmlx?style=social\" alt=\"Stars\" /\u003e\u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Apple_Silicon-M1%2FM2%2FM3%2FM4-black?logo=apple\" alt=\"Apple Silicon\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Python-3.10+-3776AB?logo=python\u0026logoColor=white\" alt=\"Python\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Electron-28-47848F?logo=electron\u0026logoColor=white\" alt=\"Electron\" /\u003e\n  \u003ca href=\"https://ko-fi.com/jangml\"\u003e\u003cimg src=\"https://img.shields.io/badge/Support-Ko--fi-FF5E5B?logo=ko-fi\u0026logoColor=white\" alt=\"Ko-fi\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#quickstart\"\u003eQuickstart\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#model-support\"\u003eModels\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#features\"\u003eFeatures\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#image-generation--editing\"\u003eImage Gen\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#api-reference\"\u003eAPI\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#desktop-app\"\u003eDesktop App\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#advanced-quantization\"\u003eJANG\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#cli-commands\"\u003eCLI\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#configuration\"\u003eConfig\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e \u0026bull;\n  \u003ca href=\"#한국어-korean\"\u003e한국어\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n\u003e **JANG 2-bit destroys MLX 4-bit on [MiniMax M2.5](https://huggingface.co/JANGQ-AI/MiniMax-M2.5-JANG_2L):**\n\u003e\n\u003e | Quantization | MMLU (200q) | Size |\n\u003e |---|---|---|\n\u003e | **JANG\\_2L (2-bit)** | **74%** | **89 GB** |\n\u003e | MLX 4-bit | 26.5% | 120 GB |\n\u003e | MLX 3-bit | 24.5% | 93 GB |\n\u003e | MLX 2-bit | 25% | 68 GB |\n\u003e\n\u003e Adaptive mixed-precision keeps critical layers at higher precision. Scores at [jangq.ai](https://jangq.ai). Models at [JANGQ-AI](https://huggingface.co/JANGQ-AI).\n\n\u003ctable align=\"center\"\u003e\n\u003ctr\u003e\n\u003ctd align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/chat-tab.png\" width=\"500\" alt=\"Chat interface\" /\u003e\u003c/td\u003e\n\u003ctd align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/agentic-chat.png\" width=\"500\" alt=\"Agentic coding chat\" /\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"center\"\u003e\u003cem\u003eChat with any MLX model -- thinking mode, streaming, and syntax highlighting\u003c/em\u003e\u003c/td\u003e\n\u003ctd align=\"center\"\u003e\u003cem\u003eAgentic chat with full coding capabilities -- tool use and structured output\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n---\n\n## Quickstart\n\n### Install from PyPI\n\nPublished on [PyPI as `vmlx`](https://pypi.org/project/vmlx/) -- install and run in one command:\n\n```bash\n# Recommended: uv (fast, no venv hassle)\nbrew install uv\nuv tool install vmlx\nvmlx serve mlx-community/Qwen3-8B-4bit\n\n# Or: pipx (isolates from system Python)\nbrew install pipx\npipx install vmlx\nvmlx serve mlx-community/Qwen3-8B-4bit\n\n# Or: pip in a virtual environment\npython3 -m venv ~/.vmlx-env \u0026\u0026 source ~/.vmlx-env/bin/activate\npip install vmlx\nvmlx serve mlx-community/Qwen3-8B-4bit\n```\n\n\u003e **Note:** On macOS 14+, bare `pip install` fails with \"externally-managed-environment\". Use `uv`, `pipx`, or a venv.\n\nThe vMLX inference server is now running at `http://0.0.0.0:8000` with an OpenAI + Anthropic compatible API. Works with any model from [mlx-community](https://huggingface.co/mlx-community) -- thousands of models ready to go.\n\n### Or download the desktop app\n\nGet [MLX Studio](https://github.com/jjang-ai/mlxstudio/releases/latest) -- a native macOS app with chat UI, model management, image generation, and developer tools. No terminal required. Just download the DMG and drag to Applications.\n\n### Use with OpenAI SDK\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"not-needed\")\nresponse = client.chat.completions.create(\n    model=\"local\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n    stream=True,\n)\nfor chunk in response:\n    print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\n### Use with Anthropic SDK\n\n```python\nimport anthropic\n\nclient = anthropic.Anthropic(base_url=\"http://localhost:8000/v1\", api_key=\"not-needed\")\nmessage = client.messages.create(\n    model=\"local\",\n    max_tokens=1024,\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n)\nprint(message.content[0].text)\n```\n\n### Use with curl\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"local\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"stream\": true\n  }'\n```\n\n---\n\n## Model Support\n\nvMLX runs any MLX model. Point it at a HuggingFace repo or local path and go.\n\n| Type | Models |\n|------|--------|\n| **Text LLMs** | Qwen 2/2.5/3/3.5/3.6, Llama 3/3.1/3.2/3.3/4, Mistral/Mixtral, **Mistral-Medium-3.5** (ministral3), Mistral-Small-4, Gemma 3/4, Phi-4, DeepSeek V2/V3/V4, GLM-4/5, MiniMax M2.5/M2.7, Nemotron, **Laguna** (poolside), **ZAYA** (CCA + MoE), Kimi K2.5/K2.6, StepFun, and any mlx-lm model |\n| **Vision LLMs** | Qwen-VL, Qwen3.5-VL / Qwen3.6-VL, Pixtral, InternVL, LLaVA, Gemma 3n / 4-VL, Mistral-Medium-3.5 (PIXTRAL) |\n| **Multimodal Omni** | **Nemotron-3-Nano-Omni** (text + image + audio + video) — Parakeet audio encoder + RADIO ViT vision tower; routed via OmniMultimodalDispatcher across `/v1/chat/completions`, `/v1/messages`, `/v1/responses`, `/api/chat` |\n| **MoE Models** | Qwen 3.5/3.6 MoE (A3B/A10B), Mixtral, DeepSeek V2/V3/V4, MiniMax M2.5/M2.7, Llama 4, Laguna (256 routed experts top-8) |\n| **Hybrid SSM** | Nemotron-H, Jamba, GatedDeltaNet (Mamba + Attention), Qwen3.5-A3B hybrid, Granite MoE Hybrid, LFM2 |\n| **Image Gen** | Flux Schnell/Dev, Z-Image Turbo (via mflux) |\n| **Image Edit** | Qwen Image Edit (via mflux) |\n| **Embeddings** | Any mlx-lm compatible embedding model |\n| **Reranking** | Cross-encoder reranking models |\n| **Audio** | Kokoro TTS, Whisper STT (via mlx-audio) |\n\n---\n\n## Features\n\n### Inference Engine\n\n| Feature | Description |\n|---------|-------------|\n| **Continuous Batching** | Handle multiple concurrent requests efficiently |\n| **Prefix Cache** | Reuse KV states for repeated prompts -- makes follow-up messages instant |\n| **Paged KV Cache** | Block-based caching with content-addressable deduplication |\n| **KV Cache Quantization** | Compress cached states to q4/q8 for 2-4x memory savings |\n| **Disk Cache (L2)** | Persist prompt caches to SSD -- survives server restarts |\n| **Block Disk Cache** | Per-block persistent cache paired with paged KV cache |\n| **Speculative Decoding** | Small draft model proposes tokens for 20-90% speedup |\n| **Prompt Lookup Decoding** | No draft model needed — reuses n-gram matches from the prompt/context. Best for structured or repetitive output (code, JSON, schemas). Enable with `--enable-pld`. |\n| **JIT Compilation** | `mx.compile` Metal kernel fusion (experimental) |\n| **Hybrid SSM Support** | Mamba/GatedDeltaNet layers handled correctly alongside attention |\n| **Distributed Compute** | Pipeline parallelism across multiple Macs via Thunderbolt 5 / Ethernet / WiFi |\n\n### Distributed Inference (Multi-Mac)\n\nRun models too large for a single Mac across 2+ machines. Each Mac loads a subset of transformer layers and they communicate hidden states over the network.\n\n```bash\n# On worker Macs:\npip install vmlx\nvmlx-worker --secret mysecret\n\n# On coordinator Mac (runs the server):\nvmlx serve JANGQ-AI/Qwen3.5-Coder-Rerank-397B-A27B-JANG_2L --distributed --cluster-secret mysecret\n```\n\n| Feature | Description |\n|---------|-------------|\n| **Pipeline Parallelism** | Split layers across nodes -- hidden state (~8KB/step) flows sequentially |\n| **Auto-Discovery** | Bonjour mDNS, UDP broadcast, HTTP probes, Tailscale, cached peers, manual IP |\n| **Capability-Scored Election** | Most powerful Mac becomes coordinator automatically |\n| **Any Network Works** | TB5 (120 Gbps), 10GbE, 1GbE, WiFi, Tailscale -- PP is not bandwidth-bound |\n| **JANG Support** | Each worker loads its layer range from JANG safetensors (mmap) |\n| **Live Node List** | Desktop app shows discovered nodes, link type, latency, layer assignments |\n| **Cluster API** | `/v1/cluster/status`, `/v1/cluster/nodes`, `/v1/cluster/scan` REST endpoints |\n\n### 5-Layer Cache Architecture\n\n```\nRequest -\u003e Tokens\n    |\nL1: Memory-Aware Prefix Cache (or Paged Cache)\n    | miss\nL2: Disk Cache (or Block Disk Store)\n    | miss\nInference -\u003e float16 KV states\n    |\nKV Quantization -\u003e q4/q8 for storage\n    |\nStore back into L1 + L2\n```\n\n### Tool Calling\n\nAuto-detected parsers for every major model family:\n\n`qwen` - `llama` - `mistral` - `hermes` - `deepseek` - `glm47` - `minimax` - `nemotron` - `granite` - `functionary` - `xlam` - `kimi` - `step3p5`\n\n### Reasoning / Thinking Mode\n\nAuto-detected reasoning parsers that extract `\u003cthink\u003e` blocks:\n\n`qwen3` (Qwen3, QwQ, StepFun) - `minimax_m2` (MiniMax M2/M2.5/M2.7) - `deepseek_r1` (DeepSeek R1, Gemma 3, GLM, Phi-4) - `openai_gptoss` (GLM Flash, GPT-OSS)\n\n### Audio\n\n| Feature | Description |\n|---------|-------------|\n| **Text-to-Speech** | Kokoro TTS via mlx-audio -- multiple voices, streaming output |\n| **Speech-to-Text** | Whisper STT via mlx-audio -- transcription and translation |\n\n---\n\n## Image Generation \u0026 Editing\n\nGenerate and edit images locally with Flux models via [mflux](https://github.com/filipstrand/mflux).\n\n```bash\npip install vmlx[image]\n\n# Image generation\nvmlx serve schnell                    # or dev, z-image-turbo\nvmlx serve ~/.mlxstudio/models/image/flux1-schnell-4bit\n\n# Image editing\nvmlx serve qwen-image-edit            # instruction-based editing\n```\n\n### Generation API\n\n```bash\ncurl http://localhost:8000/v1/images/generations \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"schnell\",\n    \"prompt\": \"A cat astronaut floating in space with Earth in the background\",\n    \"size\": \"1024x1024\",\n    \"n\": 1\n  }'\n```\n\n```python\n# Python (OpenAI SDK)\nresponse = client.images.generate(\n    model=\"schnell\",\n    prompt=\"A cat astronaut floating in space\",\n    size=\"1024x1024\",\n    n=1,\n)\n```\n\n### Editing API\n\n```bash\n# Edit an image with a text prompt (Flux Kontext / Qwen Image Edit)\ncurl http://localhost:8000/v1/images/edits \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"flux-kontext\",\n    \"prompt\": \"Change the background to a sunset\",\n    \"image\": \"\u003cbase64-encoded-image\u003e\",\n    \"size\": \"1024x1024\",\n    \"strength\": 0.8\n  }'\n```\n\n```python\n# Python\nimport base64\nwith open(\"source.png\", \"rb\") as f:\n    image_b64 = base64.b64encode(f.read()).decode()\n\nresponse = requests.post(\"http://localhost:8000/v1/images/edits\", json={\n    \"model\": \"flux-kontext\",\n    \"prompt\": \"Make the sky purple\",\n    \"image\": image_b64,\n    \"size\": \"1024x1024\",\n    \"strength\": 0.8,\n})\n```\n\n### Supported Image Models\n\n**Generation Models:**\n\n| Model | Steps | Speed | Memory |\n|-------|-------|-------|--------|\n| **Flux Schnell** | 4 | Fastest | ~6-24 GB |\n| **Z-Image Turbo** | 4 | Fast | ~6-24 GB |\n| **Flux Dev** | 20 | Slow | ~6-24 GB |\n\n**Editing Models:**\n\n| Model | Steps | Type | Memory |\n|-------|-------|------|--------|\n| **Qwen Image Edit** | 28 | Instruction-based editing | ~54 GB |\n\n---\n\n## API Reference\n\n### API Gateway\n\nThe desktop app runs an **API Gateway** on a single port (default `8080`) that routes requests to all loaded models by name. Run multiple models simultaneously and access them all through one URL.\n\n```bash\n# All models accessible through the gateway\ncurl http://localhost:8080/v1/chat/completions \\\n  -d '{\"model\": \"Qwen3.5-122B\", \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}]}'\n\n# Works with Ollama CLI too\nOLLAMA_HOST=http://localhost:8080 ollama run Qwen3.5-122B\n```\n\nThe gateway supports **OpenAI**, **Anthropic**, and **Ollama** wire formats. Configure the port in the API tab.\n\n### Endpoints\n\n**OpenAI / Anthropic**\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `POST` | `/v1/chat/completions` | OpenAI Chat Completions API (streaming + non-streaming) |\n| `POST` | `/v1/messages` | Anthropic Messages API |\n| `POST` | `/v1/responses` | OpenAI Responses API |\n| `POST` | `/v1/completions` | Text completions |\n| `POST` | `/v1/images/generations` | Image generation |\n| `POST` | `/v1/images/edits` | Image editing (Qwen Image Edit) |\n| `POST` | `/v1/embeddings` | Text embeddings |\n| `POST` | `/v1/rerank` | Document reranking |\n| `POST` | `/v1/audio/transcriptions` | Speech-to-text (Whisper) |\n| `POST` | `/v1/audio/speech` | Text-to-speech (Kokoro) |\n| `GET` | `/v1/models` | List loaded models |\n| `GET` | `/v1/cache/stats` | Cache statistics |\n| `GET` | `/health` | Server health check |\n\n**Ollama**\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `POST` | `/api/chat` | Chat completion (NDJSON streaming) |\n| `POST` | `/api/generate` | Text generation (NDJSON streaming) |\n| `GET` | `/api/tags` | List loaded models |\n| `POST` | `/api/show` | Model details |\n| `POST` | `/api/embeddings` | Generate embeddings |\n\n### curl Examples\n\n**Chat completion (streaming)**\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"local\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Explain quantum computing in 3 sentences.\"}],\n    \"stream\": true,\n    \"temperature\": 0.7\n  }'\n```\n\n**Chat completion with thinking mode**\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"local\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Solve: what is 23 * 47?\"}],\n    \"enable_thinking\": true,\n    \"stream\": true\n  }'\n```\n\n**Tool calling**\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"local\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"What is the weather in Tokyo?\"}],\n    \"tools\": [{\n      \"type\": \"function\",\n      \"function\": {\n        \"name\": \"get_weather\",\n        \"description\": \"Get current weather for a location\",\n        \"parameters\": {\n          \"type\": \"object\",\n          \"properties\": {\n            \"location\": {\"type\": \"string\", \"description\": \"City name\"}\n          },\n          \"required\": [\"location\"]\n        }\n      }\n    }]\n  }'\n```\n\n**Anthropic Messages API**\n\n```bash\ncurl http://localhost:8000/v1/messages \\\n  -H \"Content-Type: application/json\" \\\n  -H \"x-api-key: not-needed\" \\\n  -H \"anthropic-version: 2023-06-01\" \\\n  -d '{\n    \"model\": \"local\",\n    \"max_tokens\": 1024,\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n  }'\n```\n\n**Embeddings**\n\n```bash\ncurl http://localhost:8000/v1/embeddings \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"local\",\n    \"input\": \"The quick brown fox jumps over the lazy dog\"\n  }'\n```\n\n**Text-to-speech**\n\n```bash\ncurl http://localhost:8000/v1/audio/speech \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"kokoro\",\n    \"input\": \"Hello, welcome to vMLX!\",\n    \"voice\": \"af_heart\"\n  }' --output speech.wav\n```\n\n**Speech-to-text**\n\n```bash\ncurl http://localhost:8000/v1/audio/transcriptions \\\n  -F file=@audio.wav \\\n  -F model=whisper\n```\n\n**Image generation**\n\n```bash\ncurl http://localhost:8000/v1/images/generations \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"schnell\",\n    \"prompt\": \"A mountain landscape at sunset\",\n    \"size\": \"1024x1024\"\n  }'\n```\n\n**Reranking**\n\n```bash\ncurl http://localhost:8000/v1/rerank \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"local\",\n    \"query\": \"What is machine learning?\",\n    \"documents\": [\n      \"ML is a subset of AI\",\n      \"The weather is sunny today\",\n      \"Neural networks learn from data\"\n    ]\n  }'\n```\n\n**Cache stats**\n\n```bash\ncurl http://localhost:8000/v1/cache/stats\n```\n\n**Health check**\n\n```bash\ncurl http://localhost:8000/health\n```\n\n---\n\n## Desktop App\n\nvMLX includes a native macOS desktop app (MLX Studio) with 5 modes:\n\n| Mode | Description |\n|------|-------------|\n| **Chat** | Conversation interface with chat history, thinking mode, tool calling, agentic coding |\n| **Server** | Manage model sessions -- start, stop, configure, monitor |\n| **Image** | Text-to-image generation and image editing with Flux, Kontext, Qwen, and Fill models |\n| **Tools** | Model converter, GGUF-to-MLX, inspector, diagnostics |\n| **API** | Live endpoint reference with copy-pasteable code snippets |\n\n\u003ctable align=\"center\"\u003e\n\u003ctr\u003e\n\u003ctd align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/image-edit-tab.png\" width=\"450\" alt=\"Image generation and editing\" /\u003e\u003c/td\u003e\n\u003ctd align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/tools-tab.png\" width=\"450\" alt=\"Developer tools\" /\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"center\"\u003e\u003cem\u003eImage generation and editing with Flux models\u003c/em\u003e\u003c/td\u003e\n\u003ctd align=\"center\"\u003e\u003cem\u003eDeveloper tools -- model conversion and diagnostics\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/anthropic-api.png\" width=\"450\" alt=\"Anthropic API endpoint\" /\u003e\u003c/td\u003e\n\u003ctd align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/gguf-to-mlx.png\" width=\"450\" alt=\"GGUF to MLX conversion\" /\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"center\"\u003e\u003cem\u003eAnthropic Messages API endpoint -- full compatibility\u003c/em\u003e\u003c/td\u003e\n\u003ctd align=\"center\"\u003e\u003cem\u003eGGUF to MLX conversion -- bring your own models\u003c/em\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### Download\n\nGet the latest DMG from [MLX Studio Releases](https://github.com/jjang-ai/mlxstudio/releases/latest), or build from source:\n\n```bash\ngit clone https://github.com/jjang-ai/vmlx.git\ncd vmlx/panel\nnpm install \u0026\u0026 npm run build\nnpx electron-builder --mac dmg\n```\n\n### Menu Bar\n\nvMLX lives in your menu bar showing all running models, GPU memory usage, and quick controls.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/menu-bar.png\" width=\"300\" alt=\"Menu Bar\" /\u003e\n\u003c/p\u003e\n\n---\n\n## Advanced Quantization\n\nvMLX supports standard MLX quantization (4-bit, 8-bit uniform) out of the box. For users who want to push further, **JANG adaptive mixed-precision** assigns different bit widths to different layer types -- attention gets more bits, MLP layers get fewer -- achieving better quality at the same model size.\n\n### JANG Profiles\n\n| Profile | Attention | Embeddings | MLP | Avg Bits | Use Case |\n|---------|-----------|------------|-----|----------|----------|\n| `JANG_2M` | 8-bit | 4-bit | 2-bit | ~2.5 | Balanced compression |\n| `JANG_2L` | 8-bit | 6-bit | 2-bit | ~2.7 | Quality 2-bit |\n| `JANG_3M` | 8-bit | 3-bit | 3-bit | ~3.2 | **Recommended** |\n| `JANG_4M` | 8-bit | 4-bit | 4-bit | ~4.2 | Standard quality |\n| `JANG_6M` | 8-bit | 6-bit | 6-bit | ~6.2 | Near lossless |\n\n### Convert\n\n```bash\npip install vmlx[jang]\n\n# Standard MLX quantization\nvmlx convert my-model --bits 4\n\n# JANG adaptive quantization\nvmlx convert my-model --jang-profile JANG_3M\n\n# Activation-aware calibration (better at 2-3 bit)\nvmlx convert my-model --jang-profile JANG_2L --calibration-method activations\n\n# Serve the converted model\nvmlx serve ./my-model-JANG_3M --continuous-batching --use-paged-cache\n```\n\nPre-quantized JANG models are available at [JANGQ-AI on HuggingFace](https://huggingface.co/JANGQ-AI).\n\n### Smelt Mode (Partial Expert Loading)\n\nFor MoE models that don't fit in RAM, **Smelt** loads only a subset of experts per layer from SSD and keeps the backbone resident. Routing is biased toward the resident experts, so response quality stays coherent while RAM usage drops. Trade-off: throughput scales inversely with expert % loaded, because expert swaps hit SSD on the hot path.\n\n```bash\n# Load 50% of experts per layer (default when --smelt alone)\nvmlx serve ./MyMoE-JANG_4M --smelt --smelt-experts 50\n\n# Aggressive: load 25% — smallest RAM, slowest\nvmlx serve ./MyMoE-JANG_4M --smelt --smelt-experts 25\n```\n\n**Benchmarks on `Nemotron-Cascade-2-30B-A3B-JANG_4M`** (23 MoE layers × 128 experts, Apple M3 Ultra / 128 GB, dedicated machine, no parallel models):\n\n| `--smelt-experts` | Active RAM | Decode tok/s | RAM saving | Coherent |\n|---|---:|---:|---|---|\n| _off (baseline)_ | **17,408 MB** | **89.9** | — | ✓ |\n| `50` | 9,529 MB | **66.5** | **−45%** | ✓ |\n| `25` | 5,590 MB | * | **−68%** | ✓ |\n\n\\* Responses too short (2-5 tokens) for reliable steady-state tok/s measurement at 25 %. Subjectively responsive.\n\nAll three configurations produced coherent, non-looping output. No quality degradation observed — routing bias keeps the model on-topic.\n\n\u003e **Credit**: Smelt mode is inspired by [Anemll's **flash-moe**](https://github.com/Anemll/flash-moe) — a pure C / Objective‑C / Metal inference engine that showed huge MoE models (Qwen3.5-397B) can run on modest Apple Silicon hardware by streaming expert weights from SSD with `pread()` on demand. vMLX Smelt takes a different implementation path: Python/MLX, tied to the JANG quantization format, and loading a fixed subset of experts per layer at startup (backbone resident, routing biased toward the loaded subset) rather than on-demand per-token. It plugs into the full vMLX server with continuous batching, paged cache, and OpenAI-compatible API. Different engine, same core insight — thanks to the flash-moe team for validating the approach.\n\n**Smelt is mutually exclusive with VLM mode.** vMLX detects smelt and automatically disables `--is-mllm` (with a warning) because the vision tower is not wired through the partial-expert loader — image input on a smelt-loaded VLM would produce garbage logits. Use a text-only model when running smelt, or disable smelt when running a VLM.\n\nSmelt requires an MoE model in JANG format. Not compatible with dense models (no experts to partial-load) or with non-JANG formats.\n\n---\n\n## CLI Commands\n\n```bash\nvmlx serve \u003cmodel\u003e              # Start inference server\nvmlx convert \u003cmodel\u003e --bits 4   # MLX uniform quantization\nvmlx convert \u003cmodel\u003e -j JANG_3M # JANG adaptive quantization\nvmlx info \u003cmodel\u003e               # Model metadata and config\nvmlx doctor \u003cmodel\u003e             # Run diagnostics\nvmlx bench \u003cmodel\u003e              # Performance benchmarks\nvmlx-worker --secret \u003csecret\u003e   # Start distributed worker node\n```\n\n---\n\n## Configuration\n\n### Server Options\n\n```bash\nvmlx serve \u003cmodel\u003e \\\n  --host 0.0.0.0 \\              # Bind address (default: 0.0.0.0)\n  --port 8000 \\                 # Port (default: 8000)\n  --api-key sk-your-key \\       # Optional API key authentication\n  --continuous-batching \\       # Enable concurrent request handling\n  --enable-prefix-cache \\       # Reuse KV states for repeated prompts\n  --use-paged-cache \\           # Block-based KV cache with dedup\n  --kv-cache-quantization q8 \\  # Quantize cache: q4 or q8\n  --enable-disk-cache \\         # Persist cache to SSD\n  --enable-jit \\                # JIT Metal kernel compilation\n  --tool-call-parser auto \\     # Auto-detect tool call format\n  --reasoning-parser auto \\     # Auto-detect thinking format\n  --log-level INFO \\            # Logging: DEBUG, INFO, WARNING, ERROR\n  --max-model-len 8192 \\        # Max context length\n  --speculative-model \u003cmodel\u003e \\ # Draft model for speculative decoding\n  --enable-pld \\                # Prompt Lookup Decoding — no draft model, best for code/JSON/schemas\n  --distributed \\               # Enable multi-Mac pipeline parallelism\n  --cluster-secret \u003csecret\u003e \\   # Shared auth secret for workers\n  --distributed-mode pipeline \\ # pipeline (default) or tensor (coming soon)\n  --worker-nodes ip:port,... \\  # Manual worker IPs (overrides auto-discovery)\n  --cors-origins \"*\"            # CORS allowed origins\n```\n\n### Quantization Options\n\n```bash\nvmlx convert \u003cmodel\u003e \\\n  --bits 4 \\                    # Uniform quantization bits: 2, 3, 4, 6, 8\n  --group-size 64 \\             # Quantization group size (default: 64)\n  --output ./output-dir \\       # Output directory\n  --jang-profile JANG_3M \\      # JANG mixed-precision profile\n  --calibration-method activations  # Activation-aware calibration\n```\n\n### Image Generation \u0026 Editing Options\n\n```bash\npip install vmlx[image]\n\n# Generation models\nvmlx serve schnell \\            # or dev, z-image-turbo\n  --image-quantize 4 \\          # Quantization: 4, 8 (omit for full precision)\n  --port 8001\n\n# Editing models\nvmlx serve qwen-image-edit \\    # Instruction-based editing (full precision only)\n  --port 8001\n\n# Local model directory\nvmlx serve ~/.mlxstudio/models/image/FLUX.1-schnell-mflux-4bit\n```\n\n### Audio Options\n\nTTS and STT require the `mlx-audio` package:\n\n```bash\npip install mlx-audio\n\n# TTS: serve Kokoro model\nvmlx serve kokoro --port 8002\n\n# STT: serve Whisper model\nvmlx serve whisper --port 8003\n```\n\n### Optional Dependencies\n\n```bash\npip install vmlx              # Core: text LLMs, VLMs, embeddings, reranking\npip install vmlx[image]       # + Image generation (mflux)\npip install vmlx[jang]        # + JANG quantization tools\npip install vmlx[dev]         # + Development/testing tools\npip install vmlx[image,jang]  # Multiple extras\n```\n\n---\n\n## Architecture\n\n```\n+--------------------------------------------+\n|          Desktop App (Electron)             |\n|   Chat | Server | Image | Tools | API      |\n+--------------------------------------------+\n|          Session Manager (TypeScript)       |\n|   Process spawn | Health monitor | Tray     |\n+--------------------------------------------+\n|         vMLX Engine (Python / FastAPI)       |\n|  +--------+  +---------+  +-----------+    |\n|  |Simple  |  | Batched |  | ImageGen  |    |\n|  |Engine  |  | Engine  |  | Engine    |    |\n|  +---+----+  +----+----+  +-----+-----+    |\n|      |            |              |          |\n|  +---+------------+--+    +-----+-----+    |\n|  | mlx-lm / mlx-vlm  |    |  mflux    |    |\n|  +--------+-----------+    +-----------+    |\n|           |                                 |\n|  +--------+----------------------------+    |\n|  |       MLX Metal GPU Backend          |    |\n|  | quantized_matmul | KV cache | SDPA   |    |\n|  +--------------------------------------+    |\n+--------------------------------------------+\n|  L1: Prefix Cache (Memory-Aware / Paged)    |\n|  L2: Disk Cache (Persistent / Block Store)  |\n|  KV Quant: q4/q8 at storage boundary       |\n+--------------------------------------------+\n```\n\n---\n\n## Contributing\n\nContributions are welcome. Here is how to set up a development environment:\n\n```bash\ngit clone https://github.com/jjang-ai/vmlx.git\ncd vmlx\n\n# Python engine\npython -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -e \".[dev,jang,image]\"\npytest tests/ -k \"not Async\"    # 2000+ tests\n\n# Electron desktop app\ncd panel \u0026\u0026 npm install\nnpm run dev                      # Development mode with hot reload\nnpx vitest run                   # 1545+ tests\n```\n\n### Project Structure\n\n```\nvmlx/\n  vmlx_engine/          # Python inference engine (FastAPI server)\n  panel/                # Electron desktop app (React + TypeScript)\n    src/main/           # Electron main process\n    src/renderer/       # React frontend\n    src/preload/        # IPC bridge\n  tests/                # Python test suite\n  assets/               # Screenshots and logos\n```\n\n### Guidelines\n\n- Run the full test suite before submitting PRs\n- Follow existing code style and patterns\n- Include tests for new features\n- Update documentation for user-facing changes\n\n---\n\n## License\n\nApache License 2.0 -- see [LICENSE](LICENSE).\n\n---\n\n\u003cp align=\"center\"\u003e\n  Built by \u003ca href=\"https://github.com/jjang-ai\"\u003eJinho Jang\u003c/a\u003e (eric@jangq.ai)\u003cbr\u003e\n  \u003ca href=\"https://jangq.ai\"\u003eJANGQ AI\u003c/a\u003e \u0026bull; \u003ca href=\"https://pypi.org/project/vmlx/\"\u003ePyPI\u003c/a\u003e \u0026bull; \u003ca href=\"https://github.com/jjang-ai/vmlx\"\u003eGitHub\u003c/a\u003e \u0026bull; \u003ca href=\"https://github.com/jjang-ai/mlxstudio/releases\"\u003eDownloads\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## 한국어 (Korean)\n\n### vMLX — Apple Silicon을 위한 로컬 AI 엔진\n\nMac에서 LLM, VLM, 이미지 생성 및 편집 모델을 완전히 로컬로 실행하세요.\nOpenAI + Anthropic 호환 API. 클라우드 없음. API 키 불필요. 데이터가 기기를 떠나지 않습니다.\n\n### 빠른 시작\n\n```bash\npip install vmlx\nvmlx serve mlx-community/Llama-3.2-3B-Instruct-4bit\n```\n\n### 주요 기능\n\n| 기능 | 설명 |\n|------|------|\n| **텍스트 생성** | MLX 및 JANG 형식의 LLM 추론 |\n| **비전-언어 모델** | 이미지 + 텍스트 멀티모달 추론 |\n| **이미지 생성** | Flux Schnell/Dev, Z-Image Turbo (mflux 기반) |\n| **이미지 편집** | Qwen Image Edit (텍스트 지시 기반 이미지 편집) |\n| **5단계 캐싱** | 프리픽스, 페이지드, KV 양자화, 디스크, 메모리 인식 캐시 |\n| **연속 배칭** | 다중 동시 요청 처리 |\n| **에이전트 도구** | 30개 내장 도구 (파일, 웹 검색, Git, 터미널) |\n| **OpenAI API** | /v1/chat/completions, /v1/images/generations, /v1/images/edits |\n| **Anthropic API** | /v1/messages (스트리밍, 도구 호출, 시스템 프롬프트) |\n\n### 이미지 생성\n\n```bash\npip install vmlx[image]\nvmlx serve schnell          # 빠른 생성 (4 단계)\nvmlx serve dev              # 고품질 생성 (20 단계)\n```\n\n### 이미지 편집\n\n```bash\nvmlx serve qwen-image-edit  # 텍스트 지시 기반 이미지 편집\n```\n\n```bash\n# 이미지 편집 API\ncurl http://localhost:8000/v1/images/edits \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"qwen-image-edit\",\n    \"prompt\": \"배경을 해질녘으로 변경\",\n    \"image\": \"\u003cbase64 인코딩된 이미지\u003e\",\n    \"size\": \"1024x1024\",\n    \"strength\": 0.8\n  }'\n```\n\n### 데스크톱 앱 (MLX Studio)\n\nmacOS 네이티브 데스크톱 앱으로 5가지 모드를 제공합니다:\n\n| 모드 | 설명 |\n|------|------|\n| **채팅** | 대화 인터페이스, 채팅 기록, 도구 호출, 에이전트 코딩 |\n| **서버** | 모델 세션 관리 — 시작, 정지, 설정, 모니터링 |\n| **이미지** | 텍스트-이미지 생성 및 이미지 편집 (Flux, Qwen 모델) |\n| **도구** | 모델 변환기, GGUF-MLX 변환, 진단 |\n| **API** | 실시간 엔드포인트 참조 및 코드 스니펫 |\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/jjang-ai/vmlx/main/assets/image-edit-tab.png\" width=\"450\" alt=\"이미지 생성 및 편집\" /\u003e\n\u003c/p\u003e\n\n### 설치\n\n```bash\npip install vmlx              # 기본: 텍스트 LLM, VLM, 임베딩\npip install vmlx[image]       # + 이미지 생성/편집 (mflux)\npip install vmlx[jang]        # + JANG 양자화 도구\npip install vmlx[audio]       # + TTS/STT (mlx-audio)\n```\n\n### 라이선스\n\nApache License 2.0 — [LICENSE](LICENSE) 참조.\n\n---\n\n\u003cp align=\"center\"\u003e\n  개발자: \u003ca href=\"https://github.com/jjang-ai\"\u003e장진호\u003c/a\u003e (eric@jangq.ai)\u003cbr\u003e\n  \u003ca href=\"https://jangq.ai\"\u003eJANGQ AI\u003c/a\u003e \u0026bull;\n  \u003ca href=\"https://ko-fi.com/jangml\"\u003eKo-fi로 후원하기\u003c/a\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjjang-ai%2Fvmlx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjjang-ai%2Fvmlx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjjang-ai%2Fvmlx/lists"}