{"id":28528789,"url":"https://github.com/oramasearch/vector_quantizer","last_synced_at":"2025-10-03T19:51:11.837Z","repository":{"id":264837480,"uuid":"894403476","full_name":"oramasearch/vector_quantizer","owner":"oramasearch","description":"Simple vector quantization utilities and functions.","archived":false,"fork":false,"pushed_at":"2024-12-26T16:41:28.000Z","size":71,"stargazers_count":15,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-03T04:43:55.106Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oramasearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-26T09:49:29.000Z","updated_at":"2025-07-04T05:31:27.000Z","dependencies_parsed_at":"2024-11-26T11:46:06.080Z","dependency_job_id":"817dee21-ca16-4eb6-bd90-017ac8453ac7","html_url":"https://github.com/oramasearch/vector_quantizer","commit_stats":null,"previous_names":["oramasearch/quantizer"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/oramasearch/vector_quantizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oramasearch%2Fvector_quantizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oramasearch%2Fvector_quantizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oramasearch%2Fvector_quantizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oramasearch%2Fvector_quantizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oramasearch","download_url":"https://codeload.github.com/oramasearch/vector_quantizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oramasearch%2Fvector_quantizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278219765,"owners_count":25950349,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-03T02:00:06.070Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-09T13:11:33.564Z","updated_at":"2025-10-03T19:51:11.832Z","avatar_url":"https://github.com/oramasearch.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vector Quantizer\n\n[![Tests](https://github.com/oramasearch/quantizer/actions/workflows/ci.yml/badge.svg)](https://github.com/oramasearch/quantizer/actions/workflows/ci.yml)\n\nSimple vector quantization utilities and functions.\n\n```shell\ncargo add vector_quantizer\n```\n\nExample usage:\n\n```rust\nuse anyhow::Result;\nuse ndarray::Array2;\nuse ndarray_rand::RandomExt;\nuse vector_quantizer::pq::PQ;\nuse rand_distr::StandardNormal;\n\nfn main() -\u003e Result\u003c()\u003e {\n    // Generate sample vectors to quantize\n    let num_vectors = 1000;\n    let dimension = 128;\n    let original_vectors = Array2::random((num_vectors, dimension), StandardNormal);\n\n    // Configure PQ parameters\n    let m = 8; // Number of subspaces (controls compression ratio)\n    let ks = 256; // Number of centroids per subspace (usually 256 for uint8)\n    let mut pq = PQ::try_new(m, ks)?;\n\n    // Train the quantizer on the data\n    println!(\"Training PQ model...\");\n    pq.fit(\u0026original_vectors, 20)?;\n\n    // Quantize the vectors\n    println!(\"Quantizing vectors...\");\n    let quantized_vectors = pq.compress(\u0026original_vectors)?;\n\n    // Print some statistics about the quantization\n    let compression_ratio = calc_compression_ratio(m, ks, dimension);\n    let mse = calc_mse(\u0026original_vectors, \u0026quantized_vectors);\n\n    println!(\"\\nQuantization Results:\");\n    println!(\"Original vector size: {} bytes\", dimension * 4); // 4 bytes per f32\n    println!(\"Quantized vector size: {} bytes\", m); // 1 byte per subspace with ks=256\n    println!(\"Compression ratio: {:.2}x\", compression_ratio);\n    println!(\"Mean Squared Error: {:.6}\", mse);\n\n    // Example of how to get the compact codes for storage\n    let compact_codes = pq.encode(\u0026original_vectors)?;\n    println!(\"\\nCompact codes shape: {:?}\", compact_codes.dim());\n\n    // Demonstrate reconstructing vectors from compact codes\n    let reconstructed = pq.decode(\u0026compact_codes)?;\n    assert_eq!(reconstructed.dim(), original_vectors.dim());\n\n    Ok(())\n}\n\n// Helper function to calculate compression ratio\nfn calc_compression_ratio(m: usize, ks: u32, dimension: usize) -\u003e f64 {\n    let original_size = dimension * 4; // 4 bytes per f32\n    let quantized_size = m; // 1 byte per subspace when ks=256\n    original_size as f64 / quantized_size as f64\n}\n\n// Helper function to calculate Mean Squared Error\nfn calc_mse(original: \u0026Array2\u003cf32\u003e, quantized: \u0026Array2\u003cf32\u003e) -\u003e f32 {\n    (\u0026(original - quantized))\n        .mapv(|x| x.powi(2))\n        .mean()\n        .unwrap()\n}\n```\n\nSee a more detailed example here: [/src/bin/example.rs](/src/bin/example.rs)\n\n## Performance Benchmarks\n\nThe PQ implementation was tested on datasets ranging from 1,000 to 1,000,000 vectors (128 dimensions each), using 16 subspaces and 256 centroids per subspace. Key findings:\n\n- **Memory Efficiency**: Consistently achieves 96.88% memory reduction across all dataset sizes\n- **Processing Speed**:\n    - Fitting: Scales linearly, processing 100k vectors in ~3.7s (1M vectors in ~38s)\n    - Compression: Very efficient, handling ~278k vectors per second (1M vectors in 3.57s)\n- **Quality Metrics**:\n    - Reconstruction Error: Remains low (0.013-0.021) across all dataset sizes\n    - Recall@10: Ranges from 0.40 (small datasets) to 0.18 (large datasets)\n\nThe benchmark was tested on a 2022 MacBook Pro, M2 Pro, 16GB RAM. Run your own tests by running:\n\n```sh\nmake quality_check\n```\n\n## Acknowledgements\nThe code in this repository is mostly adapted from [https://github.com/xinyandai/product-quantization](https://github.com/xinyandai/product-quantization), a great Python lib for vector quantization.\n\nThe original code and the one written in this repository is derived from \"Norm-Explicit Quantization: Improving Vector Quantization for Maximum Inner Product Search\" by Dai, Xinyan and Yan, Xiao and Ng, Kelvin KW and Liu, Jie and Cheng, James: [https://arxiv.org/abs/1911.04654](https://arxiv.org/abs/1911.04654)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foramasearch%2Fvector_quantizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foramasearch%2Fvector_quantizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foramasearch%2Fvector_quantizer/lists"}