https://github.com/hexon/s2randomaccess

Random Access over an S2 compressed stream
https://github.com/hexon/s2randomaccess

compression golang golang-library s2

Last synced: 12 months ago
JSON representation

Random Access over an S2 compressed stream

• Host: GitHub
• URL: https://github.com/hexon/s2randomaccess
• Owner: hexon
• License: bsd-2-clause
• Created: 2024-06-25T13:43:16.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2024-12-22T10:29:17.000Z (over 1 year ago)
• Last Synced: 2025-03-12T01:34:05.317Z (over 1 year ago)
• Topics: compression, golang, golang-library, s2
• Language: Go
• Homepage:
• Size: 26.4 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Random Access over an S2 compressed stream

[![Go Reference](https://pkg.go.dev/badge/github.com/hexon/s2randomaccess.svg)](https://pkg.go.dev/github.com/hexon/s2randomaccess)

S2 is an extension of [Snappy](https://github.com/google/snappy). This package builds on top of the exellent [github.com/klauspost/compress/s2](https://pkg.go.dev/github.com/klauspost/compress/s2).

This package is similar to [s2.ReadSeeker](https://pkg.go.dev/github.com/klauspost/compress/s2#ReadSeeker), but optimized for concurrent requests and caching decompression of blocks.

This package requires the entire stream as a `[]byte`, which should not be modified during the lifetime of the `*Seeker`.

```go

var compressedStream []byte

func baseLibrary() {

	seeker, err := s2.NewReader(bytes.NewReader(compressedStream)).ReadSeeker(true, nil)

	_, err := seeker.ReadAt(buf, uncompressedOffset)

}

func withS2RandomAccess() {

	seeker := s2randomaccess.Open(compressedStream)

	buf, deref, err := seeker.Get(uncompressedOffset, uncompressedLength)

	use(buf)

	deref()

}

```

## Allocators

The default allocator simply uses `make([]byte, n)` and never reuses memory. `WithAllocator(&SyncPoolAllocator{})` can be used to reuse buffers using `sync.Pool`s. `WithAllocator(s2ramalloc.Allocator{})` can be used to use malloc(3) and free(3), but requires CGO.

When using the default allocator, the returned deref functions can be ignored. With other allocators they must be called exactly once after being done with the returned slice.

Returned slices might point into memory allocated by the chosen Allocator, or straight into the input data if it was in an uncompressed data block in the stream.

## Caching

This library caches the last 100 decoded blocks globally. The number of globally cached blocks can be changed with `SetGlobalLRUSize(1000)` at any time.