{"id":51280372,"url":"https://github.com/ruvnet/rupixel","last_synced_at":"2026-07-02T03:00:34.734Z","repository":{"id":367558579,"uuid":"1280664963","full_name":"ruvnet/rupixel","owner":"ruvnet","description":"Pixel-native visual RAG ported to Rust on the ruvector ANN substrate (HNSW + IVF-Flat) — screenshot/document retrieval over visual embeddings, a Rust port of PixelRAG, with a metaharness benchmark CLI: npx rupixel","archived":false,"fork":false,"pushed_at":"2026-06-26T15:47:45.000Z","size":5477,"stargazers_count":29,"open_issues_count":0,"forks_count":7,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-01T02:21:28.183Z","etag":null,"topics":["colpali","document-retrieval","embeddings","hnsw","ivf","metaharness","multimodal","ocr-free","pixelrag","rag","retrieval-augmented-generation","rust","ruvector","vector-search","visual-rag"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ruvnet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-25T20:17:50.000Z","updated_at":"2026-06-30T06:35:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ruvnet/rupixel","commit_stats":null,"previous_names":["ruvnet/rupixel"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ruvnet/rupixel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruvnet%2Frupixel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruvnet%2Frupixel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruvnet%2Frupixel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruvnet%2Frupixel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ruvnet","download_url":"https://codeload.github.com/ruvnet/rupixel/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruvnet%2Frupixel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35030921,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["colpali","document-retrieval","embeddings","hnsw","ivf","metaharness","multimodal","ocr-free","pixelrag","rag","retrieval-augmented-generation","rust","ruvector","vector-search","visual-rag"],"created_at":"2026-06-30T01:03:54.289Z","updated_at":"2026-07-02T03:00:34.708Z","avatar_url":"https://github.com/ruvnet.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rupixel — search documents by *meaning*, including pictures of pages\n\n[![npm](https://img.shields.io/npm/v/rupixel.svg)](https://www.npmjs.com/package/rupixel)\n[![license](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)\n[![status](https://img.shields.io/badge/status-early--stage%20(WIP)-orange.svg)](#how-finished-is-this)\n\n**rupixel finds the documents that match what you *mean*, not just the words you typed.**\nYou can search a pile of documents two ways:\n\n1. **By their text** — the classic approach.\n2. **By how the page *looks*** — rupixel takes a **screenshot of each page** and\n   searches the pictures. That matters when the text is locked inside a scan, an\n   image, a table, or a chart that ordinary text search can't read.\n\nTry it right now — all run **entirely in your browser**, no install, no server:\n\n- **▶ Real-time video search:** https://ruvnet.github.io/rupixel/live.html\n  — point a camera/screen at it, type what you're looking for, jump to the moment.\n- **▶ Visual search:** https://ruvnet.github.io/rupixel/visual.html\n- **▶ Text search:** https://ruvnet.github.io/rupixel/\n\n[![rupixel real-time video search — live CLIP keyframes in the browser](docs/assets/live-screenshot.png)](https://ruvnet.github.io/rupixel/live.html)\n\n### Real-time video search (live)\n\nA live feed (sample clip / webcam / screen) is sampled a few times a second.\nFrames that barely changed are **skipped** (a \"keyframe gate\"); the rest are\nembedded with CLIP so you can search the stream by meaning. Highlights:\n\n- **Runs on your GPU when available** (WebGPU via transformers.js v3), falling back\n  to CPU/WASM automatically — so frame embedding is fast enough to feel live.\n- **Runs on your GPU when available** (WebGPU via transformers.js v3), falling back\n  to CPU/WASM automatically.\n- **Optional live captions (like closed captions).** A streaming vision LLM\n  (default **Qwen3-VL**, a recent, low-cost Chinese vision model) narrates each\n  keyframe word-by-word: the current line shows as a **subtitle over the video**,\n  and every line is kept in a **timestamped transcript** below it.\n- **Built for motion.** When lots changes at once, it doesn't flood — it describes\n  only the **latest settled frame** (skipping the in-between ones, shown as `·+N`)\n  and feeds the **previous caption as context**, so the captions narrate *what\n  changed* (\"the screen has switched to…\") instead of restarting each time.\n- **Your API key is handled safely.** The public demo is **bring-your-own-key**:\n  your OpenRouter key is stored only in your browser tab (`sessionStorage`,\n  cleared when you close it), sent only to OpenRouter, never uploaded here or\n  committed. For a shared/managed key, run the included **server-side proxy**\n  (`describe-proxy.mjs`) that reads the key from an environment variable and paste\n  the proxy's URL instead — the key then never reaches the browser at all.\n\n[![rupixel live captions — streaming Qwen3-VL narration over a video feed](docs/assets/live-cc.png)](https://ruvnet.github.io/rupixel/live.html)\n\nDesign details: [ADR-265](docs/adr/ADR-265-real-time-video-visual-rag-rupixel.md)\n(pipeline), [ADR-266](docs/adr/ADR-266-midstream-streaming-frame-ingestion.md)\n(MidStream scale tier + the key-security proxy),\n[ADR-267](docs/adr/ADR-267-photonlayer-optical-front-end-video-frames.md)\n(experimental optical front-end).\n\n\u003e In the demo above, the question *\"the unseen monster lurking at a galaxy's center\"*\n\u003e brings back the **black-hole** page — even though the question never says the\n\u003e words \"black hole.\" That's searching by **meaning**.\n\n---\n\n## A 30-second explainer (no jargon)\n\n- **Searching by meaning.** A computer can't compare meanings directly, so each\n  document is turned into a long list of numbers that captures *what it's about*.\n  Things that mean similar things get similar numbers. To search, we turn your\n  question into numbers the same way and find the documents whose numbers are\n  closest. (The technical name for \"meaning as numbers\" is an **embedding**.)\n- **Two translators do this:**\n  - **Text → numbers:** a small, fast model called **MiniLM**.\n  - **Picture → numbers** *and* **text → numbers, in the same number-space:** a\n    model called **CLIP**. Because CLIP puts pictures and words in the *same*\n    space, you can type words and get back **images** — that's the visual search.\n- **It's fast and private.** The models are small enough to run on a normal CPU,\n  right inside your browser tab. Nothing is uploaded.\n- **What the numbers in the demo mean.** Each result shows a score like `0.27`.\n  That's a **similarity score** (how close the meanings are — higher is a better\n  match), **not** a time. Speed is reported separately, in **milliseconds**.\n\nThis idea — \"search the *picture* of the page\" — comes from a project called\n[PixelRAG](https://github.com/StarTrail-org/PixelRAG). rupixel is a fresh\nre-build of it in the **Rust** language, on top of\n[ruvector](https://github.com/ruvnet/ruvector) (a fast engine for storing and\nsearching those number-lists).\n\n\u003e **\"RAG\"?** It stands for *Retrieval-Augmented Generation*: first **find** the\n\u003e documents relevant to a question, then hand them to an AI to write the answer.\n\u003e rupixel is the **\"find\"** half — the search engine, not the answer-writer.\n\n---\n\n## How well does it work?\n\nWe tested it on **8 real Wikipedia pages** (black holes, the French Revolution,\nphotosynthesis, espresso, TCP/IP, baroque music, sunflowers, the Great Barrier\nReef) using **8 questions** phrased in everyday words that *don't* reuse the\npage's vocabulary — so it can only succeed by understanding meaning.\n\n| How we searched | Got the right page #1 | Speed per search |\n|---|---|---|\n| **By text** (MiniLM reads the words) | **8 out of 8** | ~0.6 ms |\n| **By picture** (CLIP looks at the screenshot) | **8 out of 8**\\* | ~0.5 ms |\n\n\\* 8/8 when run on the desktop/Rust side. The in-browser version gets **7 out of 8**\n(it draws the images slightly differently, which flips one near-tie: \"a coral\necosystem\" puts the reef page 2nd behind the photosynthesis page — both are green\nnature scenes).\n\n**The honest takeaway:** these 8 pages are *easy* — clean text, clearly different\ntopics — so **both** methods ace it and the test can't really tell them apart.\nThe point of visual search shows up on the *hard* stuff that text search chokes\non: scanned paper, screenshots, complex layouts, tables and charts. That harder\ntest is the next thing to build. Also, **CLIP is a modest, free, CPU-friendly\n\"eyes\" model** — a stronger one (like Qwen3-VL or ColPali) would do better, but\nneeds a graphics card (GPU). Full details and how to reproduce these numbers:\n**[docs/BENCHMARK.md](./docs/BENCHMARK.md)**.\n\n---\n\n## Try the command-line tool\n\n```bash\nnpx rupixel            # what it is + links\nnpx rupixel doctor     # check your setup\n```\n\n`npx rupixel` is a tiny helper (no install needed beyond Node 18+). It explains\nthe project and runs the benchmark harness. It does **not** compile the Rust code\nfor you — see [\"Run it yourself\"](#run-it-yourself) for that.\n\n---\n\n## How finished is this?\n\n**Early-stage, but everything described here actually runs — there is no fake or\nplaceholder code.** What works today:\n\n- ✅ **Text search** with a real model (MiniLM).\n- ✅ **Visual search** with a real model (CLIP) over real page screenshots.\n- ✅ **Page rendering** (turning a web page or PDF into a screenshot) using a real\n  headless browser.\n- ✅ **Two live in-browser demos** + a reproducible benchmark.\n\nWhat's **not** done yet (and is honestly described as future work, not faked):\n\n- A **stronger visual model** (Qwen3-VL / ColPali) for sharper results — needs a GPU.\n- A **bigger, harder test set** (scans, tables, charts) to really show where\n  visual search beats text search.\n\n---\n\n## What's inside (for developers)\n\nThe pipeline is simple: **turn each page into numbers → store them → find the\nclosest ones to your query.**\n\n```\npage  →  (text or screenshot)  →  numbers (MiniLM / CLIP)  →  stored in ruvector\nyour question  →  numbers  →  find the closest pages  →  ranked results\n```\n\nThree small Rust packages, all real code:\n\n| Package | What it does |\n|---|---|\n| `pixelrag-core` | the pipeline + storing/searching the number-lists (via ruvector) |\n| `pixelrag-encoder` | turns text/images into number-lists (MiniLM / CLIP) |\n| `pixelrag-cli` | runs the benchmarks and prints accuracy + speed |\n\nThe models run via small **Node.js helper scripts** (\"sidecars\") so they work on\na plain CPU with no special setup. Storage and search use two interchangeable\nindexing methods from ruvector (you can switch between them to trade memory for\nspeed).\n\n\u003e There's also an optional, **completely removable** tuning tool\n\u003e ([metaharness/darwin](https://www.npmjs.com/package/@metaharness/darwin)) that\n\u003e can automatically search for the fastest/most-accurate settings. The project\n\u003e works fine without it — details in [docs/BENCH.md](./docs/BENCH.md).\n\n---\n\n## Run it yourself\n\nThe Rust code lives inside the [ruvector](https://github.com/ruvnet/ruvector)\nproject (it reuses ruvector's search engine), so build it from a ruvector checkout:\n\n```bash\n# one-time: install the small model helpers\n( cd crates/pixelrag-cli/sidecar \u0026\u0026 npm install )\n\n# build\ncargo build -p pixelrag-core -p pixelrag-cli\n\n# search by text (MiniLM)\ncargo run -p pixelrag-cli -- benchmark --mode text  --embedder real \\\n  --ground-truth tests/fixtures/pixelrag/compare/text/ground-truth.json \\\n  --queries      tests/fixtures/pixelrag/compare/text/queries.json \\\n  --tiles        tests/fixtures/pixelrag/compare/text/tiles \\\n  --metrics ndcg,mrr,recall@10\n\n# search by picture (CLIP, over the page screenshots)\ncargo run -p pixelrag-cli -- benchmark --mode visual\n```\n\nMore detail in [`rust/README.md`](./rust/README.md).\n\n---\n\n## Credits \u0026 license\n\n- **The idea:** [StarTrail-org/PixelRAG](https://github.com/StarTrail-org/PixelRAG)\n  (Apache-2.0) — searching pages as pictures. rupixel is an independent Rust\n  rebuild; all code here is original.\n- **The search engine:** [ruvnet/ruvector](https://github.com/ruvnet/ruvector).\n- **The models:** MiniLM and CLIP, run in-browser/on-CPU via\n  [transformers.js](https://github.com/xenova/transformers.js).\n- **Optional tuner:** [@metaharness/darwin](https://www.npmjs.com/package/@metaharness/darwin).\n\nLicensed under [MIT](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruvnet%2Frupixel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fruvnet%2Frupixel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruvnet%2Frupixel/lists"}