{"id":46542215,"url":"https://github.com/druide67/asiai","last_synced_at":"2026-04-01T22:33:13.792Z","repository":{"id":342673445,"uuid":"1169414686","full_name":"druide67/asiai","owner":"druide67","description":"Multi-engine LLM benchmark \u0026 monitoring CLI for Apple Silicon","archived":false,"fork":false,"pushed_at":"2026-03-30T08:33:36.000Z","size":11731,"stargazers_count":4,"open_issues_count":5,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-30T08:37:49.159Z","etag":null,"topics":["apple-silicon","benchmark","cli","inference","llm","lm-studio","mcp","mlx","monitoring","ollama","prometheus","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/druide67.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null},"funding":{"github":"druide67","custom":["https://asiai.dev"]}},"created_at":"2026-02-28T16:50:25.000Z","updated_at":"2026-03-30T08:33:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/druide67/asiai","commit_stats":null,"previous_names":["druide67/asiai"],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/druide67/asiai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druide67%2Fasiai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druide67%2Fasiai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druide67%2Fasiai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druide67%2Fasiai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/druide67","download_url":"https://codeload.github.com/druide67/asiai/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druide67%2Fasiai/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31292648,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T21:15:39.731Z","status":"ssl_error","status_checked_at":"2026-04-01T21:15:34.046Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","benchmark","cli","inference","llm","lm-studio","mcp","mlx","monitoring","ollama","prometheus","python"],"created_at":"2026-03-07T01:14:29.942Z","updated_at":"2026-04-01T22:33:13.784Z","avatar_url":"https://github.com/druide67.png","language":"Python","funding_links":["https://github.com/sponsors/druide67","https://asiai.dev"],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/logo.svg\" alt=\"asiai logo\" width=\"140\"\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003easiai\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eApple Silicon AI\u003c/strong\u003e — Multi-engine LLM benchmark \u0026 monitoring CLI\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/asiai/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/asiai.svg\" alt=\"PyPI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/asiai/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/dm/asiai.svg?color=brightgreen\" alt=\"Downloads\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/druide67/asiai/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/druide67/asiai/actions/workflows/ci.yml/badge.svg\" alt=\"CI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://codecov.io/gh/druide67/asiai\"\u003e\u003cimg src=\"https://codecov.io/gh/druide67/asiai/branch/main/graph/badge.svg\" alt=\"Coverage\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-Apache%202.0-blue.svg\" alt=\"License\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://python.org\"\u003e\u003cimg src=\"https://img.shields.io/badge/python-3.11%2B-blue.svg\" alt=\"Python\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://support.apple.com/en-us/116943\"\u003e\u003cimg src=\"https://img.shields.io/badge/macOS-Apple%20Silicon-black.svg\" alt=\"macOS\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/sponsors/druide67\"\u003e\u003cimg src=\"https://img.shields.io/badge/sponsor-%E2%9D%A4-pink.svg\" alt=\"Sponsor\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://api.asiai.dev/api/v1/badge/benchmarks\"\u003e\u003cimg src=\"https://api.asiai.dev/api/v1/badge/benchmarks\" alt=\"Benchmarks\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://api.asiai.dev/api/v1/badge/top-speed\"\u003e\u003cimg src=\"https://api.asiai.dev/api/v1/badge/top-speed\" alt=\"Top Speed\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.asiai.dev/agent/\"\u003e\u003cimg src=\"https://api.asiai.dev/api/v1/agent-badge\" alt=\"AI Agents\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/asiai-demo.gif\" alt=\"asiai bench demo\" width=\"720\"\u003e\n\u003c/p\u003e\n\n**asiai** compares inference engines side-by-side on your Mac. Load the same model on Ollama and LM Studio, run `asiai bench`, get the numbers. No guessing, no vibes — just tok/s, TTFT, power efficiency, and stability per engine.\n\nShare your results with the community (`--share`), compare against other Apple Silicon users (`asiai compare`), and get smart engine recommendations (`asiai recommend`).\n\nBorn from the OpenClaw project, where we needed hard data to pick the fastest engine for multi-agent swarms on Mac Mini M4 Pro.\n\n## Quick start\n\n```bash\npipx install asiai        # Recommended: isolated install\n```\n\nOr via Homebrew:\n\n```bash\nbrew tap druide67/tap\nbrew install asiai\n```\n\nOther options:\n\n```bash\nuvx asiai detect           # Run without installing (requires uv)\npip install asiai           # Standard pip install\n```\n\nThen benchmark and share:\n\n```bash\nasiai bench --quick --card --share    # Bench + shareable card in ~15 seconds\n```\n\n## Commands\n\n### `asiai detect`\n\nAuto-detect running inference engines across 7 ports.\n\n```\n$ asiai detect\n\nDetected engines:\n\n  ● ollama 0.17.4\n    URL: http://localhost:11434\n\n  ● lmstudio 0.4.5\n    URL: http://localhost:1234\n    Running: 1 model(s)\n      - qwen3.5-35b-a3b  MLX\n```\n\n### `asiai bench`\n\nCross-engine benchmark with standardized prompts. Runs 3 iterations per prompt by default, reports median tok/s (SPEC standard) with stability classification.\n\n```\n$ asiai bench -m qwen3.5 --runs 3 --power\n\n  Mac Mini M4 Pro — Apple M4 Pro  RAM: 64.0 GB (42% used)  Pressure: normal\n\nBenchmark: qwen3.5\n\n  Engine       tok/s (±stddev)    Tokens   Duration     TTFT       VRAM    Thermal\n  ────────── ───────────────── ───────── ────────── ──────── ────────── ──────────\n  lmstudio    72.6 ± 0.0 (stable)   435    6.20s    0.28s        —    nominal\n  ollama      30.4 ± 0.1 (stable)   448   15.28s    0.25s   26.0 GB   nominal\n\n  Winner: lmstudio (2.4x faster)\n  Power: lmstudio 13.2W (5.52 tok/s/W) — ollama 16.0W (1.89 tok/s/W)\n```\n\nOptions:\n\n```\n-m, --model MODEL          Model to benchmark (default: auto-detect)\n-e, --engines LIST         Filter engines (e.g. ollama,lmstudio,mlxlm)\n-p, --prompts LIST         Prompt types: code, tool_call, reasoning, long_gen\n-r, --runs N               Runs per prompt (default: 3, for median + stddev)\n    --power                Cross-validate power with sudo powermetrics (IOReport always-on)\n    --context-size SIZE    Context fill prompt: 4k, 16k, 32k, 64k\n    --share                Share results with the community (anonymous, opt-in)\n-Q, --quick                Quick benchmark: 1 prompt, 1 run (~15 seconds)\n    --card                 Generate shareable benchmark card (SVG + PNG with --share)\n-H, --history PERIOD       Show past benchmarks (e.g. 7d, 24h)\n```\n\nCross-model comparison — benchmark multiple models in one run and get a ranked summary:\n\n```bash\n# Cross-model comparison\nasiai bench --compare qwen3.5:4b deepseek-r1:7b -e ollama --card\n```\n\nThe runner resolves model names across engines automatically — `gemma2:9b` (Ollama) and `gemma-2-9b` (LM Studio) are matched as the same model.\n\n### `asiai models`\n\nList loaded models across all engines. Use `--json` for machine-readable output.\n\n```\n$ asiai models\n\nollama  http://localhost:11434\n  ● qwen3.5:35b-a3b                             26.0 GB Q4_K_M\n\nlmstudio  http://localhost:1234\n  ● qwen3.5-35b-a3b                                 MLX\n```\n\n### `asiai monitor`\n\nSystem and inference metrics snapshot, stored in SQLite. Use `--json` for machine-readable output.\n\n```\n$ asiai monitor\n\nSystem\n  Uptime:    3d 12h\n  CPU Load:  2.45 / 3.12 / 2.89  (1m / 5m / 15m)\n  Memory:    45.2 GB / 64.0 GB  71%\n  Pressure:  normal\n  Thermal:   nominal  (100%)\n\nInference  ollama 0.17.4\n  Models loaded: 1  VRAM total: 26.0 GB\n\n  Model                                        VRAM   Format  Quant\n  ──────────────────────────────────────── ────────── ──────── ──────\n  qwen3.5:35b-a3b                            26.0 GB     gguf Q4_K_M\n```\n\nOptions:\n\n```\n-w, --watch SEC            Refresh every SEC seconds\n-q, --quiet                Collect and store without output (for daemon use)\n    --json                 Output as JSON (for scripting)\n-H, --history PERIOD       Show history (e.g. 24h, 1h)\n-a, --analyze HOURS        Comprehensive analysis with trends\n-c, --compare TS TS        Compare two timestamps\n    --alert-webhook URL    POST alerts on state transitions (memory, thermal, engine down)\n```\n\n### `asiai doctor`\n\nDiagnose installation, engines, system health, and database.\n\n```\n$ asiai doctor\n\nDoctor\n\n  System\n    ✓ Apple Silicon       Mac Mini M4 Pro — Apple M4 Pro\n    ✓ RAM                 64 GB total, 42% used\n    ✓ Memory pressure     normal\n    ✓ Thermal             nominal (100%)\n\n  Engine\n    ✓ Ollama              v0.17.4 — 1 model(s): qwen3.5:35b-a3b\n    ✓ LM Studio           v0.4.5 — 1 model(s): qwen3.5-35b-a3b\n    ✗ mlx-lm              not installed\n    ✗ llama.cpp            not installed\n    ✗ vllm-mlx            not installed\n\n  Database\n    ✓ SQLite              2.4 MB, last entry: 1m ago\n\n  5 ok, 0 warning(s), 3 failed\n```\n\n### `asiai daemon`\n\nBackground monitoring via macOS launchd. Collects metrics every minute.\n\n```bash\nasiai daemon start              # Install and start the daemon\nasiai daemon start --interval 30  # Custom interval (seconds)\nasiai daemon status             # Check if running\nasiai daemon logs               # View recent logs\nasiai daemon stop               # Stop and uninstall\n```\n\n### `asiai web`\n\nWeb dashboard with real-time monitoring, benchmark controls, and interactive charts. Requires `pip install asiai[web]`.\n\n```bash\nasiai web                    # Opens browser at http://127.0.0.1:8899\nasiai web --port 9000        # Custom port\nasiai web --host 0.0.0.0     # Listen on all interfaces\nasiai web --no-open          # Don't auto-open browser\n```\n\nFeatures: system overview, engine status, live benchmark with SSE progress, history charts, doctor checks, dark/light theme.\n\n### `asiai leaderboard`\n\nBrowse community benchmarks. Filter by chip or model.\n\n```bash\nasiai leaderboard                      # All results\nasiai leaderboard --chip \"M4 Pro\"      # Filter by chip\nasiai leaderboard --model qwen2.5      # Filter by model\n```\n\n### `asiai compare`\n\nCompare your local results against community medians.\n\n```bash\nasiai compare --chip \"Apple M1 Max\" --model qwen2.5:7b\n```\n\n### `asiai recommend`\n\nGet engine recommendations based on your hardware and benchmarks.\n\n```bash\nasiai recommend                                # Best engine for your Mac\nasiai recommend --use-case latency             # Optimize for TTFT\nasiai recommend --model qwen2.5 --community    # Include community data\n```\n\n### `asiai setup`\n\nInteractive setup wizard — detects hardware, engines, models, and suggests next steps.\n\n```bash\nasiai setup\n```\n\n### `asiai mcp`\n\nStart the MCP server for AI agent integration. 11 tools, 3 resources.\n\n```bash\nasiai mcp                          # stdio (Claude Code, Cursor)\nasiai mcp --transport sse          # SSE (network agents)\n```\n\n### `asiai tui`\n\nInteractive terminal dashboard with auto-refresh. Requires `pip install asiai[tui]`.\n\n```bash\nasiai tui\n```\n\n## Benchmark Card — share your results\n\nGenerate a shareable benchmark card image with one flag:\n\n```bash\nasiai bench --card                    # SVG saved locally (zero dependencies)\nasiai bench --card --share            # SVG + PNG via community API\nasiai bench --quick --card --share    # Quick bench + card + share\n```\n\n![Benchmark card example](docs/assets/benchmark-card-example.png)\n\nA **1200x630 dark-themed card** with your model, chip, specs banner (quantization, RAM, GPU cores, context size), engine comparison bar chart, winner highlight, and metric chips (tok/s, TTFT, power, engine version). Optimized for Reddit, X, Discord, and GitHub READMEs.\n\nEvery shared card includes asiai branding — the [Speedtest.net model](https://www.speedtest.net) for local LLM inference.\n\n## Supported engines\n\n| Engine | Port | Install | API |\n|--------|------|---------|-----|\n| [Ollama](https://ollama.com) | 11434 | `brew install ollama` | Native |\n| [LM Studio](https://lmstudio.ai) | 1234 | `brew install --cask lm-studio` | OpenAI-compatible |\n| [mlx-lm](https://github.com/ml-explore/mlx-examples) | 8080 | `brew install mlx-lm` | OpenAI-compatible |\n| [llama.cpp](https://github.com/ggml-org/llama.cpp) | 8080 | `brew install llama.cpp` | OpenAI-compatible |\n| [oMLX](https://github.com/jundot/omlx) | 8000 | `brew tap jundot/omlx \u0026\u0026 brew install omlx` | OpenAI-compatible |\n| [vllm-mlx](https://github.com/vllm-project/vllm) | 8000 | `pip install vllm-mlx` | OpenAI-compatible |\n| [Exo](https://github.com/exo-explore/exo) | 52415 | `pip install exo` | OpenAI-compatible |\n\n## What it measures\n\n| Metric | Description |\n|--------|-------------|\n| **tok/s** | Generation speed (tokens/sec), excluding prompt processing (TTFT) |\n| **TTFT** | Time to first token — prompt processing latency |\n| **Power** | GPU, CPU, ANE, DRAM power in watts (IOReport, no sudo) |\n| **tok/s/W** | Energy efficiency — tokens per second per watt |\n| **Stability** | Run-to-run variance: stable (CV\u003c5%), variable (\u003c10%), unstable (\u003e10%) |\n| **VRAM** | Memory footprint — native API (Ollama, LM Studio) or `ri_phys_footprint` estimate (all other engines) |\n| **Thermal** | CPU throttling state and speed limit percentage |\n\nAll metrics stored in SQLite (`~/.local/share/asiai/metrics.db`) with 90-day retention and automatic regression detection.\n\n## Benchmark methodology\n\nFollowing [MLPerf](https://mlcommons.org/benchmarks/inference-server/), [SPEC CPU 2017](https://www.spec.org/cpu2017/), and [NVIDIA GenAI-Perf](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/benchmarking/genai_perf.html) standards:\n\n- **Warmup**: 1 non-timed generation per engine before measured runs\n- **Runs**: 3 iterations per prompt (configurable), median as primary metric\n- **Sampling**: `temperature=0` (greedy decoding) for deterministic results\n- **Power**: Always-on via IOReport (no sudo). Per-engine, not session-wide average\n- **Variance**: Pooled intra-prompt stddev (isolates run-to-run noise)\n- **Metadata**: Engine version, model quantization, hardware chip, macOS version stored per result\n\nSee [docs/benchmark-best-practices.md](docs/benchmark-best-practices.md) for the full conformance audit.\n\n## Benchmark prompts\n\nFour standardized prompts test different generation patterns:\n\n| Name | Tokens | Tests |\n|------|--------|-------|\n| `code` | 512 | Structured code generation (BST in Python) |\n| `tool_call` | 256 | JSON function calling / instruction following |\n| `reasoning` | 384 | Multi-step math problem |\n| `long_gen` | 1024 | Sustained throughput (bash script) |\n\nUse `--context-size 4k|16k|32k|64k` to test with large context fill prompts instead.\n\n## API \u0026 Prometheus\n\nWhen running `asiai web`, three REST API endpoints are available for programmatic access. Interactive API documentation (Swagger UI) is available at `http://localhost:8899/docs`.\n\n| Endpoint | Description |\n|----------|-------------|\n| `GET /api/status` | Lightweight health check (\u003c 500ms) — engine reachability, memory pressure, thermal |\n| `GET /api/snapshot` | Full system + engine snapshot with loaded models, VRAM, versions |\n| `GET /api/benchmarks` | Benchmark results with tok/s, TTFT, power, context_size, engine_version |\n| `GET /api/engine-history` | Engine status history (TCP, KV cache, tokens predicted) |\n| `GET /api/benchmark-process` | Process CPU/RSS metrics from benchmark runs (7d retention) |\n| `GET /api/metrics` | Prometheus exposition format — system, engine, model, benchmark gauges |\n\n### Prometheus integration\n\n```yaml\n# prometheus.yml\nscrape_configs:\n  - job_name: 'asiai'\n    static_configs:\n      - targets: ['localhost:8899']\n    metrics_path: '/api/metrics'\n    scrape_interval: 30s\n```\n\n### CLI JSON output\n\n```bash\nasiai monitor --json | jq '.mem_pressure'\nasiai models --json | jq '.engines[].models[].name'\n```\n\n## Requirements\n\n- macOS on Apple Silicon (M1 / M2 / M3 / M4 families)\n- Python 3.11+\n- At least one inference engine running locally\n\n## Zero dependencies\n\nThe core uses only the Python standard library — `urllib`, `sqlite3`, `subprocess`, `argparse`. No `requests`, no `psutil`, no `rich`. Just stdlib.\n\nOptional extras:\n- `asiai[web]` — FastAPI web dashboard with charts\n- `asiai[tui]` — Textual terminal dashboard\n- `asiai[all]` — Web + TUI\n- `asiai[dev]` — pytest, ruff\n\n## Roadmap\n\n| Version | Scope | Status |\n|---------|-------|--------|\n| **v0.1** | detect + bench + monitor + models (CLI, stdlib) | **Done** |\n| **v0.2** | mlx-lm + doctor + daemon + TUI (Textual) | **Done** |\n| **v0.3** | 5 engines, power metrics, multi-run variance, regression detection | **Done** |\n| **v0.4** | CI, MkDocs, export JSON, thermal drift, web dashboard | **Done** |\n| **v0.5** | REST API, Prometheus /metrics, CLI --json, engine uptime tracking | **Done** |\n| **v0.6** | Multi-service LaunchAgent (`daemon start web`), daemon status/logs/stop --all | **Done** |\n| **v0.7** | Alert webhooks, LM Studio VRAM, Ollama config in doctor | **Done** |\n| **v1.0** | Community Benchmark DB, smart recommendations, Exo engine, leaderboard | **Done** |\n| **v1.0.1** | MCP server (11 tools), benchmark card, `--quick` mode, setup wizard, agent integration | **Done** |\n| **v1.2** | Web dashboard redesign, shareable cards, Share on X/Reddit, community API | **Done** |\n| **v1.3** | Dark theme, self-hosted fonts, universal VRAM (phys_footprint), power in Monitor/History | **Done** |\n| v1.4 | Fleet mode (multi-Mac), notifications macOS, MCP prompts, bench methodology improvements | Planned |\n\n## License\n\nApache 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdruide67%2Fasiai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdruide67%2Fasiai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdruide67%2Fasiai/lists"}