https://github.com/fuuzetsu/zstd-seekable-s3

zstd-compressed seekable S3 objects
https://github.com/fuuzetsu/zstd-seekable-s3

Last synced: 10 months ago
JSON representation

zstd-compressed seekable S3 objects

• Host: GitHub
• URL: https://github.com/fuuzetsu/zstd-seekable-s3
• Owner: Fuuzetsu
• Created: 2021-01-25T12:17:24.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2025-03-31T07:29:02.000Z (11 months ago)
• Last Synced: 2025-04-18T04:55:26.635Z (11 months ago)
• Language: Rust
• Size: 92.8 KB
• Stars: 10
• Watchers: 2
• Forks: 2
• Open Issues: 3
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# zstd-seekable-s3 [![Actions Status: build](https://github.com/Fuuzetsu/zstd-seekable-s3/workflows/build/badge.svg)](https://github.com/Fuuzetsu/zstd-seekable-s3/actions?query=workflow%3A"build")

This is a wrapper over zstd_compressed and rusoto packages. It retrofits

S3 objects with seeking via the HTTP range header and implements a

decompression layer on top of anything we can read/seek. The result is

that we can read S3 objects that were compressed with (seekable) zstd

and only fetch the parts of the file we're interested as if we were

working on uncompressed data but without actually paying any bandwidth

or storage costs that come with storing uncompressed data.

See the `examples` directory for a potential way to use it.

This package is currently in experimental state, do expect the API to change.