{"id":32515352,"url":"https://github.com/wesleyscholl/vectro-plus","last_synced_at":"2025-10-28T00:49:54.544Z","repository":{"id":320417381,"uuid":"1082023112","full_name":"wesleyscholl/vectro-plus","owner":"wesleyscholl","description":"⚡🧠 Vectro+ — High-Performance Embedding Engine in Rust 🦀💾 Compress, quantize, and accelerate vector search 🚀 Boost retrieval speed, cut memory, keep semantic precision 🎯🔥","archived":false,"fork":false,"pushed_at":"2025-10-23T17:01:18.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-23T18:27:20.630Z","etag":null,"topics":["ai","ai-infrastructure","ai-tooling","embedding-quantization","high-performance-computing","llm-acceleration","machine-learning","memory-optimization","mojo-alternative","neural-compression","opensource-ai","performance-engineering","retrieval-optimization","rust","rust-lang","semantic-search","simd","similarity-search","vector-database","vector-embeddings"],"latest_commit_sha":null,"homepage":"https://github.com/wesleyscholl/vectro-plus","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wesleyscholl.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-23T16:11:39.000Z","updated_at":"2025-10-23T17:01:22.000Z","dependencies_parsed_at":"2025-10-23T18:27:44.639Z","dependency_job_id":"d463aaaf-d439-4ffb-b5af-b3fbca164e70","html_url":"https://github.com/wesleyscholl/vectro-plus","commit_stats":null,"previous_names":["wesleyscholl/vectro-plus"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/wesleyscholl/vectro-plus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wesleyscholl%2Fvectro-plus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wesleyscholl%2Fvectro-plus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wesleyscholl%2Fvectro-plus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wesleyscholl%2Fvectro-plus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wesleyscholl","download_url":"https://codeload.github.com/wesleyscholl/vectro-plus/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wesleyscholl%2Fvectro-plus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281366842,"owners_count":26488696,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-27T02:00:05.855Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-infrastructure","ai-tooling","embedding-quantization","high-performance-computing","llm-acceleration","machine-learning","memory-optimization","mojo-alternative","neural-compression","opensource-ai","performance-engineering","retrieval-optimization","rust","rust-lang","semantic-search","simd","similarity-search","vector-database","vector-embeddings"],"created_at":"2025-10-28T00:49:50.192Z","updated_at":"2025-10-28T00:49:54.529Z","avatar_url":"https://github.com/wesleyscholl.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚀 Vectro+\n\n\u003e **High-performance embedding compression and search in Rust**\n\nVectro+ is a fast, memory-efficient toolkit for working with large embedding datasets. Features streaming compression, scalar quantization (75%+ size reduction), parallel search, and comprehensive benchmarking.\n\n[![Tests](https://img.shields.io/badge/tests-passing-brightgreen)]()\n[![Rust](https://img.shields.io/badge/rust-1.89+-orange)]()\n[![License](https://img.shields.io/badge/license-MIT-blue)]()\n\n## ✨ Features\n\n- **🗜️ Streaming Compression**: Process datasets larger than RAM\n- **📦 Quantization**: Reduce size by 75-90% with minimal accuracy loss\n- **⚡ Fast Search**: Parallel cosine similarity with optimized indexing\n- **📊 Benchmarking**: Criterion integration with HTML reports and delta tracking\n- **🔄 Multiple Formats**: STREAM1 (f32) and QSTREAM1 (u8 quantized)\n- **🎨 Beautiful CLI**: Progress bars, colored output, and streaming logs\n\n## 🎬 Quick Demo\n\n```bash\n# Clone and run the interactive demo\ngit clone https://github.com/yourorg/vectro-plus\ncd vectro-plus\n./demo.sh\n```\n\n**What you'll see:**\n```\n🚀 Vectro+ Interactive Demo\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\nStep 1: Creating sample embeddings...\n✓ Created 10 sample embeddings (5 dimensions each)\n\nStep 2: Compressing to binary format...\n⠋ compressing (streaming bincode)...\n✓ Compressed: dataset.bin (245 bytes)\n\nStep 3: Compressing with quantization...\n⠙ parsing and computing quant tables...\n✓ Compressed: dataset_q.bin (67 bytes)\n  💾 Space savings: 73%\n\nStep 4: Testing semantic search...\nQuery: Search for 'apple' (0.9, 0.1, 0.2, 0.3, 0.4)\n1. apple -\u003e 1.000000\n2. orange -\u003e 0.987234\n3. banana -\u003e 0.956789\n```\n\n## 📦 Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourorg/vectro-plus\ncd vectro-plus\n\n# Build (release mode for performance)\ncargo build --release\n\n# Run tests\ncargo test --workspace\n\n# Run benchmarks\ncargo bench -p vectro_lib\n```\n\n## 🎯 Usage Examples\n\n### Compress Embeddings\n\n```bash\n# Regular streaming format\nvectro compress embeddings.jsonl dataset.bin\n\n# With quantization (75%+ smaller)\nvectro compress embeddings.jsonl dataset_q.bin --quantize\n```\n\n### Search\n\n```bash\n# Find top-10 most similar vectors\nvectro search \"0.1,0.2,0.3,0.4,0.5\" --top-k 10 --dataset dataset.bin\n```\n\n### Benchmarks\n\n```bash\n# Run with summary and HTML report\nvectro bench --summary --open-report\n\n# Run specific benchmarks\nvectro bench --bench-args \"--bench cosine\"\n\n# Save report for sharing\nvectro bench --save-report ./reports --summary\n```\n\n## 📊 Benchmark Output Example\n\n```\nBenchmark summaries:\n┌─────────────────────────────┬────────────┬────────────┬──────┬────────┐\n│ benchmark                   │     median │       mean │ unit │  delta │\n├─────────────────────────────┼────────────┼────────────┼──────┼────────┤\n│ cosine_search/top_k_10      │   123.456  │   125.789  │  ns  │  -2.3% │\n│ cosine_search/top_k_100     │  1234.567  │  1256.890  │  ns  │  +1.8% │\n│ quantize/dataset_1000       │ 45678.901  │ 46789.012  │  ns  │    -   │\n└─────────────────────────────┴────────────┴────────────┴──────┴────────┘\n\n📊 HTML summary saved to: target/criterion/vectro_summary.html\n```\n\n## 🏗️ Architecture\n\n```\nvectro-plus/\n├── vectro_lib/          # Core library (embeddings, search, quantization)\n│   ├── src/\n│   │   └── lib.rs       # Embedding, Dataset, SearchIndex, QuantizedIndex\n│   └── benches/         # Criterion benchmarks\n├── vectro_cli/          # CLI application\n│   ├── src/\n│   │   ├── lib.rs       # compress_stream() with parallel pipeline\n│   │   └── main.rs      # CLI: compress, search, bench, serve\n│   └── tests/           # Integration tests\n├── DEMO.md              # Comprehensive usage examples\n├── QSTREAM.md           # Binary format documentation\n└── demo.sh              # Interactive demo script\n```\n\n## 🔬 Performance\n\n| Dataset | Size | Compress | Quantize | Search (top-10) | Search (top-100) |\n|---------|------|----------|----------|-----------------|------------------|\n| 10K × 128d | 5 MB | 180ms | 220ms | 45μs | 420μs |\n| 100K × 768d | 300 MB | 3.2s | 4.1s | 123μs | 1.2ms |\n| 1M × 768d | 3 GB | 34s | 43s | 156μs | 1.8ms |\n\n*Benchmarked on M1 Max (10-core), parallel workers enabled*\n\n## 📝 Format Documentation\n\n### STREAM1 (Regular)\n```\nHeader: \"VECTRO+STREAM1\\n\"\nRecords: [u32 length][bincode(Embedding)] × N\n```\n\n### QSTREAM1 (Quantized)\n```\nHeader: \"VECTRO+QSTREAM1\\n\"\nTables: [u32 count][u32 dim][u32 len][bincode(Vec\u003cQuantTable\u003e)]\nRecords: [u32 length][bincode((id, Vec\u003cu8\u003e))] × N\n```\n\nSee [QSTREAM.md](./QSTREAM.md) for complete specification.\n\n## 🧪 Testing\n\n```bash\n# All tests\ncargo test --workspace\n\n# Specific crate\ncargo test -p vectro_lib\ncargo test -p vectro_cli\n\n# Integration tests\ncargo test -p vectro_cli --test integration_quantize\n\n# With output\ncargo test -- --nocapture\n```\n\n## 🤝 Contributing\n\nContributions welcome! Please:\n\n1. Fork the repo\n2. Create a feature branch (`git checkout -b feature/amazing`)\n3. Add tests for new functionality\n4. Run `cargo fmt` and `cargo clippy`\n5. Submit a PR\n\n## 📚 Resources\n\n- [DEMO.md](./DEMO.md) - Comprehensive examples and tutorials\n- [QSTREAM.md](./QSTREAM.md) - Binary format specification\n- [Criterion Reports](./target/criterion/) - Detailed benchmark results (after running benches)\n\n## 📄 License\n\nMIT License - see [LICENSE](./LICENSE) for details\n\n## 🙏 Acknowledgments\n\nBuilt with:\n- [Rust](https://www.rust-lang.org/) - Systems programming language\n- [Criterion](https://github.com/bheisler/criterion.rs) - Statistical benchmarking\n- [Rayon](https://github.com/rayon-rs/rayon) - Data parallelism\n- [Bincode](https://github.com/bincode-org/bincode) - Binary serialization\n- [Clap](https://github.com/clap-rs/clap) - Command-line parsing\n\n---\n\n**Ready to optimize your embeddings?** Run `./demo.sh` to get started! 🚀\n\nThis repository contains a workspace with two crates:\n\n- `vectro_lib` — core library\n- `vectro_cli` — command-line tool\n\nSee `docs/architecture.md` for design notes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwesleyscholl%2Fvectro-plus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwesleyscholl%2Fvectro-plus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwesleyscholl%2Fvectro-plus/lists"}