Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/brian14708/tsar
Archive file format for storing large tensors efficiently
https://github.com/brian14708/tsar
archive compression onnx
Last synced: about 1 month ago
JSON representation
Archive file format for storing large tensors efficiently
- Host: GitHub
- URL: https://github.com/brian14708/tsar
- Owner: brian14708
- License: apache-2.0
- Created: 2022-06-26T13:29:39.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-09-10T05:23:37.000Z (3 months ago)
- Last Synced: 2024-09-10T07:55:40.200Z (3 months ago)
- Topics: archive, compression, onnx
- Language: Rust
- Homepage:
- Size: 122 KB
- Stars: 4
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Tensor Archive
Archive file format for storing tensors, with optional lossy compression for better storage efficiency.
Features:
- floating-point compression ([zfp](https://github.com/LLNL/zfp))
- storing data in lower precision format (bfloat16, ...)
- compressing mantissa and exponents separately
- tools for building archives from ONNX format
## Quick Start
Install `tsar-py` package from source (require Rust build environment):
```sh
pip install git+https://github.com/brian14708/tsar.git#subdirectory=tsar-py
```
### ONNX format
```sh
# lossless compression
tsar-pack -e 0 ".onnx" output.tsar
# lossy compression (with maximum error 1e-6)
tsar-pack -e 1e-6 ".onnx" output.tsar
# extract file to model/ directory
tsar-unpack output.tsar model/
```
### Python API
TODO
## Results
| Model | Compression | Size |
| ----------------------------------------------------------------------------------- | --------------- | --------- |
| [ResNet-152](https://github.com/onnx/models/tree/main/vision/classification/resnet) | none | 230.6 MiB |
| | gzip | 215.4 MiB |
| | tsar (lossless) | 197.4 MiB |
| | tsar (err=1e-6) | 129.8 MiB |
| | tsar (err=1e-5) | 108.7 MiB |
| | tsar (err=1e-4) | 87.8 MiB |
| | tsar (err=1e-3) | 60.6 MiB |