https://github.com/finalfusion/reductive
Vector quantization
https://github.com/finalfusion/reductive
embeddings product-quantization quantizer
Last synced: 2 months ago
JSON representation
Vector quantization
- Host: GitHub
- URL: https://github.com/finalfusion/reductive
- Owner: finalfusion
- License: apache-2.0
- Created: 2019-09-04T09:48:00.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2021-12-03T11:08:21.000Z (over 4 years ago)
- Last Synced: 2026-03-06T12:39:06.465Z (3 months ago)
- Topics: embeddings, product-quantization, quantizer
- Language: Rust
- Homepage:
- Size: 96.7 KB
- Stars: 7
- Watchers: 1
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Reductive
## Training of optimized product quantizers
Training of *optimized* product quantizers requires a LAPACK implementation. For
this reason, training of the `Opq` and `GaussianOpq` quantizers is feature-gated
by the `opq-train` feature. This feature must be enabled if you want to use
`Opq` or `GaussianOpq`:
~~~toml
[dependencies]
reductive = { version = "0.7", features = ["opq-train"] }
~~~
This also requires that a crate that links a LAPACK library is added as a
dependency, e.g. `accelerate-src`, `intel-mkl-src`, `openblas-src`, or
`netlib-src`.
### Running tests
#### Linux
You can run all tests on Linux, including tests for optimized product
quantizers, using the `intel-mkl-test` feature:
~~~shell
$ cargo test --features intel-mkl-test
~~~
#### macOS
All tests can be run on macOS with the `accelerate-test` feature:
~~~shell
$ cargo test --features accelerate-test
~~~
### Multi-threaded OpenBLAS
`reductive` uses Rayon to parallelize quantizer training. However,
multi-threaded OpenBLAS is [known to
conflict](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded)
with application threading. Is you use OpenBLAS, ensure that threading
is disabled, for instance by setting the number of threads to 1:
~~~shell
$ export OPENBLAS_NUM_THREADS=1
~~~