{"id":19728603,"url":"https://github.com/savetherbtz/zstd-seekable-format-go","last_synced_at":"2025-04-05T20:04:36.434Z","repository":{"id":37871482,"uuid":"447670501","full_name":"SaveTheRbtz/zstd-seekable-format-go","owner":"SaveTheRbtz","description":"Seekable ZSTD compression format implemented in Golang.","archived":false,"fork":false,"pushed_at":"2025-01-13T20:42:57.000Z","size":357,"stargazers_count":93,"open_issues_count":9,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-29T19:05:01.065Z","etag":null,"topics":["compression","go","golang","golang-library","zstd"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/SaveTheRbtz/zstd-seekable-format-go/pkg","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SaveTheRbtz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-13T16:26:42.000Z","updated_at":"2025-03-18T09:34:05.000Z","dependencies_parsed_at":"2023-02-13T02:46:28.224Z","dependency_job_id":"b79eab35-11fe-4858-8f79-d3ffbdfc2f2f","html_url":"https://github.com/SaveTheRbtz/zstd-seekable-format-go","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SaveTheRbtz%2Fzstd-seekable-format-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SaveTheRbtz%2Fzstd-seekable-format-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SaveTheRbtz%2Fzstd-seekable-format-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SaveTheRbtz%2Fzstd-seekable-format-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SaveTheRbtz","download_url":"https://codeload.github.com/SaveTheRbtz/zstd-seekable-format-go/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247393566,"owners_count":20931812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","go","golang","golang-library","zstd"],"created_at":"2024-11-12T00:04:45.952Z","updated_at":"2025-04-05T20:04:36.396Z","avatar_url":"https://github.com/SaveTheRbtz.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"  [![License][license-img]][license] [![GoDoc][doc-img]][doc] [![Build Status][ci-img]][ci] [![Go Report][report-img]][report]\n# ZSTD seekable compression format implementation in Go\n[Seekable ZSTD compression format](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md) implemented in Golang.\n\nThis library provides a random access reader (using uncompressed file offsets) for ZSTD-compressed streams.  This can be used for creating transparent compression layers.  Coupled with Content Defined Chunking (CDC) it can also be used as a robust de-duplication layer.\n## Installation\n\n`go get -u github.com/SaveTheRbtz/zstd-seekable-format-go/pkg`\n\n## Using the seekable format\n\nWriting is done through the `Writer` interface:\n```go\nimport (\n\t\"github.com/klauspost/compress/zstd\"\n\tseekable \"github.com/SaveTheRbtz/zstd-seekable-format-go/pkg\"\n)\n\nenc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest))\nif err != nil {\n\tlog.Fatal(err)\n}\ndefer enc.Close()\n\nw, err := seekable.NewWriter(f, enc)\nif err != nil {\n\tlog.Fatal(err)\n}\n\n// Write data in chunks.\nfor _, b := range [][]byte{[]byte(\"Hello\"), []byte(\" \"), []byte(\"World!\")} {\n\t_, err = w.Write(b)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n}\n\n// Close and flush seek table.\nerr = w.Close()\nif err != nil {\n\tlog.Fatal(err)\n}\n```\nNB! Do not forget to call `Close` since it is responsible for flushing the seek table.\n\nReading can either be done through `ReaderAt` interface:\n\n```go\ndec, err := zstd.NewReader(nil)\nif err != nil {\n\tlog.Fatal(err)\n}\ndefer dec.Close()\n\nr, err := seekable.NewReader(f, dec)\nif err != nil {\n\tlog.Fatal(err)\n}\ndefer r.Close()\n\nello := make([]byte, 4)\n// ReaderAt\nr.ReadAt(ello, 1)\nif !bytes.Equal(ello, []byte(\"ello\")) {\n\tlog.Fatalf(\"%+v != ello\", ello)\n}\n```\n\nOr through the `ReadSeeker`:\n```go\nworld := make([]byte, 5)\n// Seeker\nr.Seek(-6, io.SeekEnd)\n// Reader\nr.Read(world)\nif !bytes.Equal(world, []byte(\"World\")) {\n\tlog.Fatalf(\"%+v != World\", world)\n}\n```\n\nSeekable format utilizes [ZSTD skippable frames](https://github.com/facebook/zstd/blob/release/doc/zstd_compression_format.md#skippable-frames) so it is a valid ZSTD stream:\n\n```go\n// Standard ZSTD Reader\nf.Seek(0, io.SeekStart)\ndec, err := zstd.NewReader(f)\nif err != nil {\n\tlog.Fatal(err)\n}\ndefer dec.Close()\n\nall, err := io.ReadAll(dec)\nif err != nil {\n\tlog.Fatal(err)\n}\nif !bytes.Equal(all, []byte(\"Hello World!\")) {\n\tlog.Fatalf(\"%+v != Hello World!\", all)\n}\n```\n\n[doc-img]: https://pkg.go.dev/badge/github.com/SaveTheRbtz/zstd-seekable-format-go\n[doc]: https://pkg.go.dev/github.com/SaveTheRbtz/zstd-seekable-format-go/pkg\n[ci-img]: https://github.com/SaveTheRbtz/zstd-seekable-format-go/actions/workflows/go.yml/badge.svg\n[ci]: https://github.com/SaveTheRbtz/zstd-seekable-format-go/actions/workflows/go.yml\n[report-img]: https://goreportcard.com/badge/SaveTheRbtz/zstd-seekable-format-go\n[report]: https://goreportcard.com/report/SaveTheRbtz/zstd-seekable-format-go\n[license-img]: https://img.shields.io/badge/License-MIT-blue.svg\n[license]: https://opensource.org/licenses/MIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsavetherbtz%2Fzstd-seekable-format-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsavetherbtz%2Fzstd-seekable-format-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsavetherbtz%2Fzstd-seekable-format-go/lists"}