{"id":51012657,"url":"https://github.com/jerrythomas/rust-embedding-bench","last_synced_at":"2026-06-21T05:30:46.084Z","repository":{"id":360936761,"uuid":"1252361276","full_name":"jerrythomas/rust-embedding-bench","owner":"jerrythomas","description":"Same model, same hardware: an apples-to-apples Rust embedding benchmark across fastembed, ort, candle, ollama, and llama-cpp-2.","archived":false,"fork":false,"pushed_at":"2026-05-28T13:39:19.000Z","size":67,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-28T14:22:17.597Z","etag":null,"topics":["benchmark","candle","embeddings","fastembed","llama-cpp","minilm","onnx","onnx-runtime","rust","sentence-transformers"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jerrythomas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-28T12:51:46.000Z","updated_at":"2026-05-28T13:39:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jerrythomas/rust-embedding-bench","commit_stats":null,"previous_names":["jerrythomas/rust-embedding-bench"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jerrythomas/rust-embedding-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerrythomas%2Frust-embedding-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerrythomas%2Frust-embedding-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerrythomas%2Frust-embedding-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerrythomas%2Frust-embedding-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jerrythomas","download_url":"https://codeload.github.com/jerrythomas/rust-embedding-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerrythomas%2Frust-embedding-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34596046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-21T02:00:05.568Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","candle","embeddings","fastembed","llama-cpp","minilm","onnx","onnx-runtime","rust","sentence-transformers"],"created_at":"2026-06-21T05:30:42.906Z","updated_at":"2026-06-21T05:30:46.075Z","avatar_url":"https://github.com/jerrythomas.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/banner.svg\" alt=\"rust-embedding-bench — throughput comparison across fastembed, ort, candle, ollama, llama-cpp-2 on Apple M4 Max\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n# rust-embedding-bench\n\nA reproducible benchmark harness for sentence-embedding throughput across the popular Rust ML libraries, using `sentence-transformers/all-MiniLM-L6-v2` as the common model.\n\nCompares **ort** (ONNX Runtime), **fastembed**, **candle**, **ollama** (HTTP), and **llama-cpp-2** (in-process llama.cpp). Sweeps sequence length, batch size, thread count, HTTP concurrency, and GPU offload. Every backend's output is validated against a Python `sentence-transformers` reference via cosine similarity before its latency numbers are trusted.\n\nFull methodology and findings: [**REPORT.md**](REPORT.md).\n\n## TL;DR\n\nOn an Apple M4 Max, fp32-or-equivalent precision, single-machine benchmark:\n\n| Backend | b=1 short (ms p50) | b=32 short (eps) | b=32 long (eps) |\n|---|---|---|---|\n| **llama-cpp-2 (Metal)** | 1.26 | **9676** | **948** |\n| ort fp32 (t=8) | 1.10 | 3052 | 261 |\n| fastembed | 1.71 | 2849 | 315 |\n| candle (t=8) | 8.15 | 603 | 31 |\n| ollama (HTTP, default) | 11.05 | 433 | 223 |\n| ollama (b=8, concurrency=16) | 19.7 | 553 | 260 |\n\nAll backends validated at cosine ≥ 0.999 vs Python reference, except naive dynamic int8 quantization (cosine 0.961).\n\nHeadline observations:\n\n- **Single-query latency:** ort fp32 or llama-cpp-2, both around 1–2 ms p50 across all sequence lengths.\n- **High-throughput indexing:** llama-cpp-2 with the `metal` feature flag. ~9.7k embeddings per second on short text.\n- **Threading parity matters.** A lot of \"library X is much faster than library Y\" comparisons resolve to \"X auto-detected cores; Y honored `--threads 1`.\"\n- **Ollama's HTTP layer is most of the cost** for single-query workloads. The same F16 GGUF loaded in-process via llama-cpp-2 is roughly 8x faster.\n- **Dynamic int8 quantization was a wash on Apple Silicon.** Same speed or slower than fp32 ONNX, cosine drops to 0.96. May be different on x86 with VNNI.\n\n## Quick start\n\nPrerequisites: Rust toolchain, [`uv`](https://docs.astral.sh/uv/) (or any way to make a Python 3.12 venv), CMake (build dep for llama-cpp-2). Optional: a local Ollama daemon if you want to benchmark it.\n\n```bash\ngit clone https://github.com/jerrythomas/rust-embedding-bench.git\ncd rust-embedding-bench\nmake bench\n```\n\n`make help` lists the other targets (`build`, `sweep`, `aggregate`, `correctness`, `clean`, `nuke`). The pipeline is idempotent. On a clean checkout it will:\n\n1. Create `.venv` and install Python deps (sentence-transformers, optimum, numpy)\n2. Pull `all-minilm` into Ollama (if the daemon is reachable; otherwise the ollama backend is skipped)\n3. Build all Rust runners in release mode\n4. Export the ONNX model and dynamic-int8-quantize it\n5. Generate Python reference embeddings\n6. Sweep every backend × length × batch × threads combination\n7. Run a correctness check (cosine vs reference) per backend\n8. Print an aggregated comparison table\n\nCold first run: ~10 minutes (model downloads + cargo build). Warm reruns: ~2–3 minutes.\n\n## What's measured\n\n**Backends**\n\n| Backend | Engine | Model file |\n|---|---|---|\n| fastembed | ONNX Runtime via `ort` | Qdrant pre-optimized fp32 ONNX (downloaded) |\n| ort | ONNX Runtime | Optimum-exported fp32 ONNX |\n| ort-qdrant | ONNX Runtime | Same model fastembed uses, called from our wrapper |\n| ort-int8 | ONNX Runtime | Dynamic int8 quantization of the fp32 ONNX |\n| candle | candle (pure Rust) | fp32 safetensors from HF |\n| ollama | llama.cpp via HTTP daemon | F16 GGUF |\n| llama-cpp-2 | llama.cpp in-process | Same F16 GGUF; CPU build by default, Metal optional |\n\n**Sweep dimensions**\n\n- Sequence length: short (5–20 tokens) / medium (50–100) / long (200–256)\n- Batch size: 1, 8, 32\n- Thread count: 1, 4, 8 (controlled per-runner; passed via env to libraries that honor `RAYON_NUM_THREADS` / `OMP_NUM_THREADS`)\n- Ollama HTTP concurrency: 1, 4, 8, 16 (parallel in-flight requests)\n- llama-cpp-2 GPU offload: enable via `features = [\"metal\"]` in `runners/llama_runner/Cargo.toml`\n\n## Configuration\n\nThe sweep is configured via environment variables or `make VAR=...`. Examples:\n\n```bash\n# Only fastembed and ort, short text, batch=32, all thread configs\nmake bench SKIP=\"candle ollama llama\" LENGTHS=short BATCHES=32 THREADS=\"1 4 8\"\n\n# Fast smoke test\nmake bench WARMUP=20 MEASURE=100\n\n# Remote Ollama\nOLLAMA_HOST=http://other-host:11434 make bench\n```\n\n| Variable | Default | Effect |\n|---|---|---|\n| `SKIP` | empty | Space-separated backends to skip (e.g. `\"ollama ort\"`) |\n| `LENGTHS` | `short medium long` | Length buckets to sweep |\n| `BATCHES` | `1 8 32` | Batch sizes to sweep |\n| `THREADS` | `1` | Thread counts to sweep |\n| `WARMUP` | 50 | Warmup embeddings per run (discarded) |\n| `MEASURE` | 500 | Measured embeddings per run |\n| `OLLAMA_HOST` | `http://localhost:11434` | Ollama daemon URL |\n\nIndividual runners can also be invoked directly:\n\n```bash\n./target/release/ort_runner --length medium --batch 32 --threads 8 \\\n    --warmup 50 --measure 500 --out my_run.json\n```\n\n## Project layout\n\n```\n.\n├── Makefile                    # entry point (`make bench`, `make help`, ...)\n├── REPORT.md                   # methodology + full results\n├── LICENSE                     # MIT\n├── Cargo.toml                  # Rust workspace\n├── corpus/sentences.json       # 29 deterministic test sentences\n├── reference/\n│   ├── generate_reference.py   # Python sentence-transformers reference\n│   ├── export_onnx.py          # ONNX export via optimum\n│   ├── quantize.py             # dynamic int8 quantization\n│   ├── download_qdrant.py      # Qdrant pre-optimized ONNX\n│   └── download_gguf.py        # F16 GGUF (Ollama cache or HF Hub)\n├── runners/\n│   ├── shared/                 # common CLI args, result schema, IO helpers\n│   ├── fastembed_runner/\n│   ├── ort_runner/             # supports --model, --tokenizer, --backend-label\n│   ├── candle_runner/\n│   ├── ollama_runner/          # supports --concurrency\n│   └── llama_runner/           # in-process llama-cpp-2\n└── analyze/\n    ├── compare.py              # aggregate JSON results into a table\n    └── correctness.py          # cosine similarity vs Python reference\n```\n\n## Adding your own backend\n\nEach runner is a small (~150 line) Rust binary that:\n\n1. Parses `shared::CommonArgs` (corpus, length, batch, threads, warmup, measure, out, save_vectors)\n2. Loads the model once at startup\n3. Runs a warmup loop\n4. Runs a measurement loop, tracking per-item latency\n5. Writes a `BenchResult` JSON via `shared::write_result`\n\nTo add a new library `foo`:\n\n```bash\nmkdir -p runners/foo_runner/src\ncp runners/fastembed_runner/Cargo.toml runners/foo_runner/Cargo.toml\ncp runners/fastembed_runner/src/main.rs runners/foo_runner/src/main.rs\n# Edit the dependencies and the embed loop. Add foo_runner to the workspace\n# members in the root Cargo.toml. Add \"foo\" to the BACKENDS list in the Makefile.\nmake bench\n```\n\nRun with `--save-vectors vectors/foo_short.bin` once and let the harness compare them to the reference vectors. The correctness check catches most \"the embedding pipeline is wired wrong\" bugs before any latency claim becomes load-bearing.\n\n## Hardware caveats\n\nThe numbers in this README and in REPORT.md were collected on a MacBook Pro with an Apple M4 Max (12 performance + 4 efficiency cores), 48 GB unified memory, macOS 26.3.1. Results on different hardware will differ. In particular:\n\n- x86 with AVX-512 VNNI is expected to do considerably better with int8 (the AVX-512 dot-product path is a real fast track that ARM NEON lacks an equivalent for).\n- NVIDIA GPU support is available in `ort` and `llama-cpp-2` via feature flags, but is not exercised here.\n- ARM NEON fp32 throughput is very competitive on Apple Silicon, narrowing the typical fp32-vs-int8 gap seen on x86.\n\nIf you re-run this on different hardware, the harness will produce JSON output you can drop into [REPORT.md](REPORT.md)-style tables.\n\n## License\n\n[MIT](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjerrythomas%2Frust-embedding-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjerrythomas%2Frust-embedding-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjerrythomas%2Frust-embedding-bench/lists"}