{"id":49626625,"url":"https://github.com/Ricoledan/vectordb-from-scratch","last_synced_at":"2026-06-24T08:00:41.673Z","repository":{"id":336969496,"uuid":"1151868354","full_name":"Ricoledan/vectordb-from-scratch","owner":"Ricoledan","description":"A vector database built from scratch in Rust - learning project to understand database internals and HNSW indexing","archived":false,"fork":false,"pushed_at":"2026-02-13T16:06:15.000Z","size":58,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-26T15:53:30.036Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Ricoledan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-07T02:21:04.000Z","updated_at":"2026-02-13T16:06:29.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Ricoledan/vectordb-from-scratch","commit_stats":null,"previous_names":["ricoledan/vectordb-from-scratch"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Ricoledan/vectordb-from-scratch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ricoledan%2Fvectordb-from-scratch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ricoledan%2Fvectordb-from-scratch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ricoledan%2Fvectordb-from-scratch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ricoledan%2Fvectordb-from-scratch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Ricoledan","download_url":"https://codeload.github.com/Ricoledan/vectordb-from-scratch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ricoledan%2Fvectordb-from-scratch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34722710,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-24T02:00:07.484Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-05T08:00:23.459Z","updated_at":"2026-06-24T08:00:41.667Z","avatar_url":"https://github.com/Ricoledan.png","language":"Rust","funding_links":[],"categories":["Experimental \u0026 Learning Vector DBs"],"sub_categories":[],"readme":"# VectorDB From Scratch\n\nA vector database implementation in Rust, built from scratch as a learning project to understand database internals and systems programming.\n\n## Features\n\n- **Vector storage** with CRUD operations and string-based IDs\n- **Distance metrics**: Euclidean, Cosine, Dot Product\n- **Brute-force search** (FlatIndex) and **approximate nearest neighbor** search (HNSW)\n- **Metadata filtering** with composable filter expressions (eq, ne, exists, and, or)\n- **Batch operations** for bulk inserts and parallel searches\n- **Persistence** with write-ahead log (WAL), snapshots, and crash recovery\n- **HTTP API** (9 endpoints) powered by Axum\n- **Metrics collection** with latency percentiles and operation counters\n- **CLI** for direct interaction and running the HTTP server\n- **89 tests** — unit, integration, recall, and doc tests\n\n## Installation\n\nThis project uses [Nix](https://nixos.org/) with flakes for reproducible development environments.\n\n### Prerequisites\n1. Install Nix with flakes enabled\n2. Clone this repository\n\n### Enter Development Environment\n\n```bash\n# Enter the Nix development shell\nnix develop\n\n# Or use direnv for automatic activation\necho \"use flake\" \u003e .envrc\ndirenv allow\n```\n\nThe Nix shell provides:\n- Rust toolchain (rustc, cargo, clippy, rustfmt)\n- Development tools (cargo-watch, cargo-edit, cargo-nextest, cargo-criterion)\n\n## Usage\n\n### Build\n\n```bash\ncargo build --release\n```\n\n### Run Tests\n\n```bash\n# Run unit tests\ncargo test\n\n# Run with nextest (faster)\ncargo nextest run\n```\n\n### CLI\n\n```bash\n# Insert a vector\ncargo run -- insert v1 --vector \"1.0,2.0,3.0\"\n\n# Search for similar vectors (default k=5)\ncargo run -- search \"1.1,2.1,3.1\" --k 10\n\n# Delete a vector\ncargo run -- delete v1\n\n# List all vector IDs\ncargo run -- list\n\n# Use HNSW index instead of brute-force\ncargo run -- --index hnsw insert v1 --vector \"1.0,2.0,3.0\"\n\n# Enable persistence with a data directory\ncargo run -- --data-dir ./db insert v1 --vector \"1.0,2.0,3.0\"\n\n# Start the HTTP API server (default: 0.0.0.0:3000)\ncargo run -- serve\ncargo run -- serve --addr 127.0.0.1:8080\n```\n\n### HTTP API\n\nStart the server with `cargo run -- serve`, then interact via HTTP:\n\n#### Insert a vector\n\n```bash\ncurl -X POST http://localhost:3000/vectors \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"id\": \"v1\", \"vector\": [1.0, 2.0, 3.0], \"metadata\": {\"color\": \"red\"}}'\n```\n\n#### Batch insert\n\n```bash\ncurl -X POST http://localhost:3000/vectors/batch \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"vectors\": [\n    {\"id\": \"v1\", \"vector\": [1.0, 2.0, 3.0]},\n    {\"id\": \"v2\", \"vector\": [4.0, 5.0, 6.0], \"metadata\": {\"color\": \"blue\"}}\n  ]}'\n```\n\n#### Search\n\n```bash\ncurl -X POST http://localhost:3000/search \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"vector\": [1.1, 2.1, 3.1], \"k\": 5}'\n```\n\n#### Search with metadata filter\n\n```bash\ncurl -X POST http://localhost:3000/search \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"vector\": [1.1, 2.1, 3.1], \"k\": 5, \"filter\": {\"op\": \"eq\", \"field\": \"color\", \"value\": \"red\"}}'\n```\n\n#### Batch search\n\n```bash\ncurl -X POST http://localhost:3000/search/batch \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"queries\": [{\"vector\": [1.0, 2.0, 3.0], \"k\": 3}, {\"vector\": [4.0, 5.0, 6.0], \"k\": 3}]}'\n```\n\n#### Other endpoints\n\n```bash\n# List all vector IDs\ncurl http://localhost:3000/vectors\n\n# Get a specific vector\ncurl http://localhost:3000/vectors/v1\n\n# Delete a vector\ncurl -X DELETE http://localhost:3000/vectors/v1\n\n# Health check\ncurl http://localhost:3000/health\n\n# Metrics\ncurl http://localhost:3000/metrics\n```\n\n### API Reference\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `POST` | `/vectors` | Insert a vector (with optional metadata) |\n| `GET` | `/vectors` | List all vector IDs |\n| `GET` | `/vectors/:id` | Get a vector by ID |\n| `DELETE` | `/vectors/:id` | Delete a vector |\n| `POST` | `/vectors/batch` | Batch insert vectors |\n| `POST` | `/search` | Search for similar vectors (with optional filter) |\n| `POST` | `/search/batch` | Batch search queries |\n| `GET` | `/health` | Health check with vector count |\n| `GET` | `/metrics` | Query latency percentiles and operation counters |\n\n### Metadata Filters\n\nFilters are composable JSON expressions using a tagged `\"op\"` field:\n\n```json\n{\"op\": \"eq\", \"field\": \"color\", \"value\": \"red\"}\n{\"op\": \"ne\", \"field\": \"color\", \"value\": \"blue\"}\n{\"op\": \"exists\", \"field\": \"category\"}\n{\"op\": \"and\", \"filters\": [\n  {\"op\": \"eq\", \"field\": \"color\", \"value\": \"red\"},\n  {\"op\": \"exists\", \"field\": \"category\"}\n]}\n{\"op\": \"or\", \"filters\": [\n  {\"op\": \"eq\", \"field\": \"color\", \"value\": \"red\"},\n  {\"op\": \"eq\", \"field\": \"color\", \"value\": \"blue\"}\n]}\n```\n\n### Demo\n\nRun the interactive demo script to see every API endpoint in action:\n\n```bash\nbash examples/demo.sh\n```\n\nIt builds the project, starts the server, and walks through inserts, searches, filtered searches, batch operations, deletions, and metrics — with clear output at each step.\n\n### Benchmarks\n\n```bash\n# Run benchmarks (FlatIndex and HNSW comparison)\ncargo bench\n\n# Results will be in target/criterion/\n```\n\n## Architecture\n\n### Index Types\n\n- **FlatIndex** — Brute-force O(n) search. Exact results, simple and reliable.\n- **HnswIndex** — Approximate nearest neighbor search using [Hierarchical Navigable Small World](https://arxiv.org/abs/1603.09320) graphs. Achieves \u003e95% recall with significantly faster search on large datasets.\n\nHNSW default parameters: `m=16`, `ef_construction=200`, `ef_search=50`, `max_layers=16`.\n\n### Persistence\n\n- **Write-Ahead Log (WAL)** — All inserts and deletes are durably logged before being applied. Entries are length-prefixed bincode with CRC32 checksums.\n- **Snapshots** — Periodic checkpoints of the full dataset (default: every 1,000 WAL entries).\n- **Crash Recovery** — On startup, loads the latest snapshot and replays any WAL entries written after it.\n- **Memory-mapped I/O** — Optional mmap-based reads for snapshot files.\n\n### Metrics\n\nThe `/metrics` endpoint reports:\n- Total queries, inserts, and deletes\n- Average, p50, p95, and p99 query latency (microseconds)\n\n## Project Structure\n\n```\nvectordb-from-scratch/\n├── flake.nix                    # Nix development environment\n├── Cargo.toml                   # Rust dependencies\n├── src/\n│   ├── lib.rs                   # Library entry point, public API\n│   ├── main.rs                  # CLI application\n│   ├── vector.rs                # Vector type and operations\n│   ├── storage.rs               # VectorStore\u003cI: Index\u003e, metadata, search\n│   ├── distance.rs              # Distance metrics\n│   ├── index.rs                 # Index trait (abstract interface)\n│   ├── flat_index.rs            # Brute-force index\n│   ├── error.rs                 # Error types\n│   ├── metrics.rs               # Latency percentiles and counters\n│   ├── hnsw/\n│   │   ├── mod.rs               # HnswIndex public API\n│   │   ├── graph.rs             # HNSW graph and algorithm\n│   │   └── neighbor_queue.rs    # Priority queue helpers\n│   ├── persistence/\n│   │   ├── mod.rs               # Module exports\n│   │   ├── engine.rs            # StorageEngine (WAL + snapshots)\n│   │   ├── wal.rs               # Write-ahead log\n│   │   ├── snapshot.rs          # Snapshot manager\n│   │   ├── serialization.rs     # Bincode + CRC32 serialization\n│   │   └── mmap.rs              # Memory-mapped file I/O\n│   └── server/\n│       ├── mod.rs               # Server startup\n│       └── routes.rs            # HTTP endpoint handlers\n├── examples/\n│   └── demo.sh                  # Interactive API demo script\n├── tests/\n│   ├── integration_test.rs      # End-to-end workflow tests\n│   └── recall_test.rs           # HNSW recall benchmarks\n└── benches/\n    ├── search_bench.rs          # FlatIndex benchmarks\n    └── hnsw_bench.rs            # HNSW vs FlatIndex benchmarks\n```\n\n## Development Roadmap\n\n- [x] **Phase 1 — Foundation**: Vector type, distance metrics, VectorStore, CLI, tests\n- [x] **Phase 2 — Indexing**: Index trait, FlatIndex extraction, HNSW index\n- [x] **Phase 3 — Persistence**: WAL, snapshots, crash recovery, mmap storage\n- [x] **Phase 4 — HTTP API**: Axum server, metrics, parallel HNSW, CLI serve command\n- [x] **Phase 5 — Search Enhancements**: Metadata-filtered search, batch operations, API enhancements\n\n## Learning Resources\n\nThis project follows the tutorial:\n- [Write You a Vector Database](https://skyzh.github.io/write-you-a-vector-db/) by Alex Chi\n\nKey academic papers:\n- [HNSW (2016)](https://arxiv.org/abs/1603.09320) - Hierarchical Navigable Small World graphs\n- [Faiss (2024)](https://arxiv.org/abs/2401.08281) - Facebook's billion-scale similarity search\n\nReference implementations:\n- [hnswlib-rs](https://github.com/jean-pierreBoth/hnswlib-rs)\n- [pgvecto.rs](https://github.com/tensorchord/pgvecto.rs)\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\nCopyright (c) 2026 Ricardo Ledan\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRicoledan%2Fvectordb-from-scratch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRicoledan%2Fvectordb-from-scratch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRicoledan%2Fvectordb-from-scratch/lists"}