{"id":23240727,"url":"https://github.com/skbkontur/zstdnet","last_synced_at":"2025-05-16T07:03:49.649Z","repository":{"id":10587614,"uuid":"66259619","full_name":"skbkontur/ZstdNet","owner":"skbkontur","description":"Zstd wrapper for .NET","archived":false,"fork":false,"pushed_at":"2022-05-31T19:07:52.000Z","size":2089,"stargazers_count":310,"open_issues_count":8,"forks_count":52,"subscribers_count":39,"default_branch":"master","last_synced_at":"2025-05-16T01:11:20.088Z","etag":null,"topics":["compression","csharp","dotnet","zstandard","zstd"],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skbkontur.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-08-22T09:41:09.000Z","updated_at":"2025-04-23T06:30:51.000Z","dependencies_parsed_at":"2022-08-01T08:09:27.042Z","dependency_job_id":null,"html_url":"https://github.com/skbkontur/ZstdNet","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skbkontur%2FZstdNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skbkontur%2FZstdNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skbkontur%2FZstdNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skbkontur%2FZstdNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skbkontur","download_url":"https://codeload.github.com/skbkontur/ZstdNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254485053,"owners_count":22078767,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","csharp","dotnet","zstandard","zstd"],"created_at":"2024-12-19T05:13:48.351Z","updated_at":"2025-05-16T07:03:49.629Z","avatar_url":"https://github.com/skbkontur.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"ZstdNet\n=======\n\n[![NuGet](https://img.shields.io/nuget/v/ZstdNet.svg)](https://www.nuget.org/packages/ZstdNet/)\n\n**ZstdNet** is a wrapper of **Zstd** native library for .NET languages. It has the following features:\n\n* Compression and decompression of byte arrays\n* Streaming compression and decompression\n* Generation of Dictionaries from a collection of samples\n\nTake a look on a library reference or unit tests to explore its behavior in different situations.\n\nZstd\n----\n\n**Zstd**, short for Zstandard, is a fast lossless compression algorithm, which\nprovides both good compression ratio _and_ speed for your standard compression\nneeds. \"Standard\" translates into everyday situations which neither look for\nhighest possible ratio (which LZMA and ZPAQ cover) nor extreme speeds (which\nLZ4 covers). Zstandard is licensed under [BSD 3-Clause License](ZstdNet/build/LICENSE).\n\n**Zstd** is initially developed by Yann Collet and the source is available at:\nhttps://github.com/facebook/zstd\n\nThe motivation to develop of the algorithm, ways of use and its properties are\nexplained in the blog that introduces the library:\nhttp://fastcompression.blogspot.com/2015/01/zstd-stronger-compression-algorithm.html\n\nThe benefits of the dictionary mode are described here:\nhttp://fastcompression.blogspot.ru/2016/02/compressing-small-data.html\n\nReference\n---------\n\n### Requirements\n\n*ZstdNet* requires *libzstd* \u003e= v1.4.0. Both 32-bit and 64-bit versions are supported.\nThe corresponding DLLs are included in this repository cross-compiled using\n`(i686|x86_64)-w64-mingw32-gcc -DZSTD_MULTITHREAD -DZSTD_LEGACY_SUPPORT=0 -pthread -s`.\nNote that `ZSTD_LEGACY_SUPPORT=0` means \"do not support legacy formats\" to minimize the binary size.\n\n### Exceptions\n\nThe wrapper throws `ZstdException` in case of malformed data or an error inside *libzstd*.\nIf the given destination buffer is too small, `ZstdException` with `ZSTD_error_dstSize_tooSmall`\nerror code is thrown away.\nCheck [zstd_errors.h](https://github.com/facebook/zstd/blob/v1.4.5/lib/common/zstd_errors.h#L52) for more info.\n\n### Compressor class\n\nBlock compression implementation. Instances of this class are **not thread-safe**.\nConsider using `ThreadStatic` or pool of compressors for bulk processing.\n\n* Constructor allow specifying compression options. Otherwise, default values will be used for `CompressionOptions`.\n\n  ```c#\n  Compressor();\n  Compressor(CompressionOptions options);\n  ```\n\n  Options will be exposed in `Options` read-only field.\n\n  Note that `Compressor` class implements `IDisposable`.\n  If you use a lot of instances of this class,\n  it's recommended to call `Dispose` to avoid loading on the finalizer thread. For example:\n\n  ```c#\n  using var compressor = new Compressor();\n  var compressedData = compressor.Wrap(sourceData);\n  ```\n\n* `Wrap` compress data and save it in a new or an existing buffer (in such case a length of saved data will be returned).\n\n  ```c#\n  byte[] Wrap(byte[] src);\n  byte[] Wrap(ArraySegment\u003cbyte\u003e src);\n  byte[] Wrap(ReadOnlySpan\u003cbyte\u003e src);\n  int Wrap(byte[] src, byte[] dst, int offset);\n  int Wrap(ArraySegment\u003cbyte\u003e src, byte[] dst, int offset);\n  int Wrap(ReadOnlySpan\u003cbyte\u003e src, byte[] dst, int offset);\n  int Wrap(ReadOnlySpan\u003cbyte\u003e src, Span\u003cbyte\u003e dst);\n  ```\n\n  Note that on buffers close to 2GB `Wrap` tries its best, but if `src` is uncompressible and its size is too large,\n  `ZSTD_error_dstSize_tooSmall` will be thrown. `Wrap` method call will only be reliable for a buffer size such that\n  `GetCompressBoundLong(size) \u003c= 0x7FFFFFC7`. Consider using streaming compression API on large data inputs.\n\n* `GetCompressBound` returns required destination buffer size for source data of size `size`.\n\n  ```c#\n  static int GetCompressBound(int size);\n  static ulong GetCompressBoundLong(ulong size);\n  ```\n\n### CompressionStream class\n\nImplementation of streaming compression. The stream is write-only.\n\n* Constructor\n\n  ```c#\n  CompressionStream(Stream stream);\n  CompressionStream(Stream stream, int bufferSize);\n  CompressionStream(Stream stream, CompressionOptions options, int bufferSize = 0);\n  ```\n\n  Options:\n    - `Stream stream` \u0026mdash; output stream for writing compressed data.\n    - `CompressionOptions options`\n      Default is `CompressionOptions.Default` with default compression level.\n    - `int bufferSize` \u0026mdash; buffer size used for compression buffer.\n      Default is the result of calling `ZSTD_CStreamOutSize` which guarantees to successfully flush at least one complete compressed block (currently ~128KB).\n\n  The buffer for compression is allocated using `ArrayPool\u003cbyte\u003e.Shared.Rent()`.\n\n  Note that `CompressionStream` class implements `IDisposable` and `IAsyncDisposable`.\n  If you use a lot of instances of this class,\n  it's recommended to call `Dispose` or `DisposeAsync` to avoid loading on the finalizer thread. For example:\n\n  ```c#\n  await using var compressionStream = new CompressionStream(outputStream, zstdBufferSize);\n  await inputStream.CopyToAsync(compressionStream, copyBufferSize);\n  ```\n\n### CompressionOptions class\n\nStores compression options and \"digested\" (for compression) information\nfrom a compression dictionary, if present.\nInstances of this class **are thread-safe**. They can be shared across threads to avoid\nperformance and memory overhead.\n\n* Constructor\n\n  ```c#\n  CompressionOptions(int compressionLevel);\n  CompressionOptions(byte[] dict, int compressionLevel = DefaultCompressionLevel);\n  CompressionOptions(byte[] dict, IReadOnlyDictionary\u003cZSTD_cParameter, int\u003e advancedParams, int compressionLevel = DefaultCompressionLevel);\n  ```\n\n  Options:\n    - `byte[] dict` \u0026mdash; compression dictionary.\n      It can be read from a file or generated with `DictBuilder` class.\n      Default is `null` (no dictionary).\n    - `int compressionLevel` \u0026mdash; compression level.\n      Should be in range from `CompressionOptions.MinCompressionLevel` to `CompressionOptions.MaxCompressionLevel` (currently 22).\n      Default is `CompressionOptions.DefaultCompressionLevel` (currently 3).\n    - `IReadOnlyDictionary\u003cZSTD_cParameter, int\u003e advancedParams` \u0026mdash; advanced API provides a way\n      to set specific parameters during compression. For example, it allows you to compress with multiple threads,\n      enable long distance matching mode and more.\n      Check [zstd.h](https://github.com/facebook/zstd/blob/v1.4.5/lib/zstd.h#L265) for additional info.\n\n  Specified options will be exposed in read-only fields.\n\n  Note that `CompressionOptions` class implements `IDisposable`.\n  If you use a lot of instances of this class,\n  it's recommended to call `Dispose` to avoid loading on the finalizer thread. For example:\n\n  ```c#\n  using var options = new CompressionOptions(dict, compressionLevel: 5);\n  using var compressor = new Compressor(options);\n  var compressedData = compressor.Wrap(sourceData);\n  ```\n\n### Decompressor class\n\nBlock decompression implementation. Instances of this class are **not thread-safe**.\nConsider using `ThreadStatic` or pool of decompressors for bulk processing.\n\n* Constructor allow specifying decompression options. Otherwise, default values for `DecompressionOptions` will be used.\n\n  ```c#\n  new Decompressor();\n  new Decompressor(DecompressionOptions options);\n  ```\n\n  Options will be exposed in `Options` read-only field.\n\n  Note that `Decompressor` class implements `IDisposable`.\n  If you use a lot of instances of this class,\n  it's recommended to call `Dispose` to avoid loading on the finalizer thread. For example:\n\n  ```c#\n  using var decompressor = new Decompressor();\n  var decompressedData = decompressor.Unwrap(compressedData);\n  ```\n\n* `Unwrap` decompress data and save it in a new or an existing buffer (in such case a length of saved data will be returned).\n\n  ```c#\n  byte[] Unwrap(byte[] src, int maxDecompressedSize = int.MaxValue);\n  byte[] Unwrap(ArraySegment\u003cbyte\u003e src, int maxDecompressedSize = int.MaxValue);\n  byte[] Unwrap(ReadOnlySpan\u003cbyte\u003e src, int maxDecompressedSize = int.MaxValue);\n  int Unwrap(byte[] src, byte[] dst, int offset, bool bufferSizePrecheck = true);\n  int Unwrap(ArraySegment\u003cbyte\u003e src, byte[] dst, int offset, bool bufferSizePrecheck = true);\n  int Unwrap(ReadOnlySpan\u003cbyte\u003e src, byte[] dst, int offset, bool bufferSizePrecheck = true);\n  int Unwrap(ReadOnlySpan\u003cbyte\u003e src, Span\u003cbyte\u003e dst, bool bufferSizePrecheck = true);\n  ```\n\n  Data can be saved to a new buffer only if a field with decompressed data size\n  is present in compressed data. You can limit size of the new buffer with\n  `maxDecompressedSize` parameter (it's necessary to do this on untrusted data).\n\n  If `bufferSizePrecheck` flag is set and the decompressed field length is specified,\n  the size of the destination buffer will be checked before actual decompression.\n\n  Note that if this field is malformed (and is less than actual decompressed data size),\n  *libzstd* still doesn't allow a buffer overflow to happen during decompression.\n\n* `GetDecompressedSize` reads a field with decompressed data size stored in compressed data.\n\n  ```c#\n  static ulong GetDecompressedSize(byte[] src);\n  static ulong GetDecompressedSize(ArraySegment\u003cbyte\u003e src);\n  static ulong GetDecompressedSize(ReadOnlySpan\u003cbyte\u003e src);\n  ```\n\n### DecompressionStream class\n\nImplementation of streaming decompression. The stream is read-only.\n\n* Constructor\n\n  ```c#\n  DecompressionStream(Stream stream);\n  DecompressionStream(Stream stream, int bufferSize);\n  DecompressionStream(Stream stream, DecompressionOptions options, int bufferSize = 0);\n  ```\n\n  Options:\n    - `Stream stream` \u0026mdash; input stream for reading raw data.\n    - `DecompressionOptions options`\n      Default is `null` (no dictionary).\n    - `int bufferSize` \u0026mdash; buffer size used for decompression buffer.\n      Default is the result of calling `ZSTD_DStreamInSize` \u0026mdash; recommended size for input buffer (currently ~128KB).\n\n  The buffer for decompression is allocated using `ArrayPool\u003cbyte\u003e.Shared.Rent()`.\n\n  Note that `DecompressionStream` class implements `IDisposable` and `IAsyncDisposable`.\n  If you use a lot of instances of this class,\n  it's recommended to call `Dispose` or `DisposeAsync` to avoid loading on the finalizer thread. For example:\n\n  ```c#\n  await using var decompressionStream = new DecompressionStream(inputStream, zstdBufferSize);\n  await decompressionStream.CopyToAsync(outputStream, copyBufferSize);\n  ```\n\n### DecompressionOptions class\n\nStores decompression options and \"digested\" (for decompression) information\nfrom a compression dictionary, if present.\nInstances of this class **are thread-safe**. They can be shared across threads to avoid\nperformance and memory overhead.\n\n* Constructor\n\n  ```c#\n  DecompressionOptions();\n  DecompressionOptions(byte[] dict);\n  DecompressionOptions(byte[] dict, IReadOnlyDictionary\u003cZSTD_dParameter, int\u003e advancedParams);\n  ```\n\n  Options:\n    - `byte[] dict` \u0026mdash; compression dictionary.\n      It can be read from a file or generated with `DictBuilder` class.\n      Default is `null` (no dictionary).\n    - `IReadOnlyDictionary\u003cZSTD_dParameter, int\u003e advancedParams` \u0026mdash; advanced decompression API\n      that allows you to set parameters like maximum memory usage.\n      Check [zstd.h](https://github.com/facebook/zstd/blob/v1.4.5/lib/zstd.h#L513) for additional info.\n\n  Specified options will be exposed in read-only fields.\n\n  Note that `CompressionOptions` class implements `IDisposable`.\n  If you use a lot of instances of this class,\n  it's recommended to call `Dispose` to avoid loading on the finalizer thread. For example:\n\n  ```c#\n  using var options = new DecompressionOptions(dict);\n  using var decompressor = new Decompressor(options);\n  var decompressedData = decompressor.Unwrap(compressedData);\n  ```\n\n### DictBuilder static class\n\n* `TrainFromBuffer` generates a compression dictionary from a collection of samples.\n\n  ```c#\n  static byte[] TrainFromBuffer(IEnumerable\u003cbyte[]\u003e samples, int dictCapacity = DefaultDictCapacity);\n  ```\n\n  Options:\n    - `int dictCapacity` \u0026mdash; maximal dictionary size in bytes.\n      Default is `DictBuilder.DefaultDictCapacity`, currently 110 KiB (the default in zstd utility).\n\nWrapper Authors\n---------------\n\nCopyright (c) 2016-present [SKB Kontur](https://kontur.ru/eng/about)\n\n*ZstdNet* is distributed under [BSD 3-Clause License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskbkontur%2Fzstdnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskbkontur%2Fzstdnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskbkontur%2Fzstdnet/lists"}