{"id":48845437,"url":"https://github.com/dropbox/witchcraft","last_synced_at":"2026-04-15T05:02:46.561Z","repository":{"id":349662524,"uuid":"1190994999","full_name":"dropbox/witchcraft","owner":"dropbox","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-14T15:10:36.000Z","size":2928,"stargazers_count":66,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-04-14T16:30:27.449Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dropbox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-24T20:24:27.000Z","updated_at":"2026-04-14T15:10:52.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dropbox/witchcraft","commit_stats":null,"previous_names":["dropbox/witchcraft"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dropbox/witchcraft","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Fwitchcraft","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Fwitchcraft/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Fwitchcraft/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Fwitchcraft/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dropbox","download_url":"https://codeload.github.com/dropbox/witchcraft/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Fwitchcraft/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31826907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"online","status_checked_at":"2026-04-15T02:00:06.175Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-15T05:02:28.461Z","updated_at":"2026-04-15T05:02:46.553Z","avatar_url":"https://github.com/dropbox.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Witchcraft\n\n(formerly known as Rust-Warp)\n\nThis is a from-scratch reimplementation of Stanford's XTR-Warp semantic search\nengine ( https://github.com/jlscheerer/xtr-warp ) in safe rust, using a\nsingle-file SQLite database as backing storage, making it suitable for\nclient-side deployment. It runs completely stand-alone on your device, needs no\nAPI keys, no vector database, no chunking strategy, no fancy re-rankers, and it\nis lightning fast (21ms p.95 end-to-end search latency on NFCorpus, at 33%\nNDCG@10, on an Apple Macbook Pro M2 Max, more than twice as fast as the\noriginal XTR-WARP on server-class hardware, at similar accuracy.)\n\n![pickbrain](pickbrain.png)\n\n# Building and Running #\nTo run, you will need the XTR weights released by Google Deepmind, and\nthis repo contains python scripts for downloading them from Huggingface\nand quantizing to GGUF format. We can cheat by doing everything with Make:\n```\nmake warp-cli\n```\n## Creating an index: ##\n\nFor testing, we used the BEIR download script from XTR-Warp to download\nnfcorpus and check that we could replicate their results.\nFor your convenience, nfcorpus.tsv is included here, so you can run:\n```\n$ ./warp-cli readcsv datasets/nfcorpus.tsv\n```\nWith all the nfcorpus documents imported, we can now create embeddings for them, with:\n```\n$ ./warp-cli embed\n```\nNext we create the index over the embeddings, with:\n```\n$ ./warp-cli index\n```\n\nAll state gets persisted in mydb.sqlite, and you can abort the indexer and\nit will pick up where it left off. To start over, you can just delete\nmydb.sqlite.\n\n## Querying the index ##\n\nWhen you have the index, you can query it with:\n```\n$ ./warp-cli query \"does milk intake cause acne in teenagers?\"\n```\nAnd hopefully get a bunch of relevant answers. You can also try other\nvariations of this, instead of \"query\" you can also use \"hybrid\", which\ncombines semantic search with the BM25 search functionality that comes\nstandard with sqlite.\n\n# Pickbrain: semantic search over your AI coding sessions #\n\nIncluded as an example is **pickbrain** (screenshot above), a CLI that indexes\nyour Claude Code and OpenAI Codex session transcripts, memory files, and\nauthored documents into a Witchcraft database for fast semantic search. Ever\nwondered \"what was that conversation where I fixed the auth middleware?\" —\npickbrain finds it, and lets you resume the session directly.\n\n```\nmake pickbrain\n./pickbrain auth middleware fix    # search across all sessions (auto-ingests new sessions)\n./pickbrain --session \u003cUUID\u003e auth  # search within one session\n./pickbrain --dump \u003cUUID\u003e          # print full conversation\n```\n\nThe source lives in `examples/pickbrain/` and demonstrates how to use\nWitchcraft as a library: document ingestion, embedding, indexing, and hybrid\nsearch. To install pickbrain as a skill for both Claude Code and Codex:\n\n```\nmake pickbrain-install\n```\n\nThis puts the binary on your `PATH` and installs the skill definitions so\nyou can use pickbrain directly from either tool to answer questions requiring\nglobal knowledge of all your projects:\n\n![skill](skill.png)\n\n# More build info #\n## Feature flags ##\n\nWhen building, exactly one T5 backend must be enabled:\n- `t5-quantized` -- GGUF quantized weights via candle (default)\n- `t5-openvino` -- OpenVINO inference backend\n\nOther flags:\n- `metal` -- macOS GPU acceleration (Apple Silicon only)\n- `accelerate` -- macOS BLAS via Accelerate framework\n- `fbgemm` -- fbgemm-rs packed GEMM (bf16 weights, faster on x86)\n- `hybrid-dequant` -- F32 attention + Q4K FFN with fused gated-gelu (x86, requires `fbgemm`)\n- `napi` -- Node.js native module via napi-rs\n- `embed-assets` -- bake weights into binary\n- `progress` -- progress bars for CLI\n\nPlatform-specific recommended features:\n- **Apple Silicon**: `t5-quantized,metal,accelerate`\n- **Intel Mac (x86_64)**: `t5-quantized,accelerate,hybrid-dequant,fbgemm`\n- **Intel Windows (x86_64)**: `t5-openvino,fbgemm`\n\n## Using as Node module ##\n\n```\nmake module\n```\nBuilds target/release/warp.node. Which loads the compressed weights from the\n\"assets\" dir.\n\n## Unit tests and Code Coverage\n\n```\ncargo install cargo-nextest\ncargo install cargo-llvm-cov\ncargo install llvm-tools-preview\nmake test\n```\n\nNOTICE that nextest is necessary, simply using \"cargo test\" will lead\nto random test failures, because individual tests run in the same process,\nleading to \"history effects\".\n\n# License\n\nUnless otherwise noted:\n```\nCopyright (c) 2026 Dropbox Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdropbox%2Fwitchcraft","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdropbox%2Fwitchcraft","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdropbox%2Fwitchcraft/lists"}