{"id":13396813,"url":"https://github.com/klauspost/reedsolomon","last_synced_at":"2025-05-13T17:05:28.233Z","repository":{"id":33979504,"uuid":"37726333","full_name":"klauspost/reedsolomon","owner":"klauspost","description":"Reed-Solomon Erasure Coding in Go","archived":false,"fork":false,"pushed_at":"2025-02-03T09:25:51.000Z","size":1465,"stargazers_count":1940,"open_issues_count":2,"forks_count":254,"subscribers_count":52,"default_branch":"master","last_synced_at":"2025-04-23T22:57:43.274Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Assembly","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/klauspost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-06-19T14:29:40.000Z","updated_at":"2025-04-19T03:50:05.000Z","dependencies_parsed_at":"2023-02-19T09:01:32.658Z","dependency_job_id":"de4b97a8-274c-4c26-8e6f-e5156235edd5","html_url":"https://github.com/klauspost/reedsolomon","commit_stats":{"total_commits":293,"total_committers":38,"mean_commits":"7.7105263157894735","dds":0.2764505119453925,"last_synced_commit":"d4574a5be5062e5a07ffae17ad59789976baf674"},"previous_names":[],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klauspost%2Freedsolomon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klauspost%2Freedsolomon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klauspost%2Freedsolomon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klauspost%2Freedsolomon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/klauspost","download_url":"https://codeload.github.com/klauspost/reedsolomon/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253990460,"owners_count":21995774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T18:01:03.706Z","updated_at":"2025-05-13T17:05:28.206Z","avatar_url":"https://github.com/klauspost.png","language":"Assembly","funding_links":[],"categories":["Library","Assembly","Erasure Coding and Hashing"],"sub_categories":[],"readme":"# Reed-Solomon\n[![Go Reference](https://pkg.go.dev/badge/github.com/klauspost/reedsolomon.svg)](https://pkg.go.dev/github.com/klauspost/reedsolomon) [![Go](https://github.com/klauspost/reedsolomon/actions/workflows/go.yml/badge.svg)](https://github.com/klauspost/reedsolomon/actions/workflows/go.yml)\n\nReed-Solomon Erasure Coding in Go, with speeds exceeding 1GB/s/cpu core implemented in pure Go.\n\nThis is a Go port of the [JavaReedSolomon](https://github.com/Backblaze/JavaReedSolomon) library released by \n[Backblaze](http://backblaze.com), with some additional optimizations.\n\nFor an introduction on erasure coding, see the post on the [Backblaze blog](https://www.backblaze.com/blog/reed-solomon/).\n\nFor encoding high shard counts (\u003e256) a Leopard implementation is used.\nFor most platforms this performs close to the original Leopard implementation in terms of speed. \n\nPackage home: https://github.com/klauspost/reedsolomon\n\nGodoc: https://pkg.go.dev/github.com/klauspost/reedsolomon\n\n# Installation\nTo get the package use the standard:\n```bash\ngo get -u github.com/klauspost/reedsolomon\n```\n\nUsing Go modules is recommended.\n\n# Changes\n\n## 2024\n\n * Auto-generation of SVE and NEON routines for ARM based on AVX2 code. This results in a speedup of 2x for SVE (as measured using Graviton 3 on AWS) and a speedup of 1.5x as compared to the existing NEON-accelerated code.\n\n## 2022\n\n* [GFNI](https://github.com/klauspost/reedsolomon/pull/224) support for amd64, for up to 3x faster processing.\n* [Leopard GF8](https://github.com/klauspost/reedsolomon#leopard-gf8) mode added, for faster processing of medium shard counts.\n* [Leopard GF16](https://github.com/klauspost/reedsolomon#leopard-compatible-gf16) mode added, for up to 65536 shards. \n* [WithJerasureMatrix](https://pkg.go.dev/github.com/klauspost/reedsolomon?tab=doc#WithJerasureMatrix) allows constructing a [Jerasure](https://github.com/tsuraan/Jerasure) compatible matrix.\n\n## 2021\n\n* Use `GOAMD64=v4` to enable faster AVX2.\n* Add progressive shard encoding.\n* Wider AVX2 loops\n* Limit concurrency on AVX2, since we are likely memory bound.\n* Allow 0 parity shards.\n* Allow disabling inversion cache.\n* Faster AVX2 encoding.\n\n\u003cdetails\u003e\n\t\u003csummary\u003eSee older changes\u003c/summary\u003e\n\n## May 2020\n\n* ARM64 optimizations, up to 2.5x faster.\n* Added [WithFastOneParityMatrix](https://pkg.go.dev/github.com/klauspost/reedsolomon?tab=doc#WithFastOneParityMatrix) for faster operation with 1 parity shard.\n* Much better performance when using a limited number of goroutines.\n* AVX512 is now using multiple cores.\n* Stream processing overhaul, big speedups in most cases.\n* AVX512 optimizations\n\n## March 6, 2019\n\nThe pure Go implementation is about 30% faster. Minor tweaks to assembler implementations.\n\n## February 8, 2019\n\nAVX512 accelerated version added for Intel Skylake CPUs. This can give up to a 4x speed improvement as compared to AVX2.\nSee [here](https://github.com/klauspost/reedsolomon#performance-on-avx512) for more details.\n\n## December 18, 2018\n\nAssembly code for ppc64le has been contributed, this boosts performance by about 10x on this platform.\n\n## November 18, 2017\n\nAdded [WithAutoGoroutines](https://godoc.org/github.com/klauspost/reedsolomon#WithAutoGoroutines) which will attempt \nto calculate the optimal number of goroutines to use based on your expected shard size and detected CPU.\n\n## October 1, 2017\n\n* [Cauchy Matrix](https://godoc.org/github.com/klauspost/reedsolomon#WithCauchyMatrix) is now an option. \nThanks to [templexxx](https://github.com/templexxx) for the basis of this.\n\n* Default maximum number of [goroutines](https://godoc.org/github.com/klauspost/reedsolomon#WithMaxGoroutines) \nhas been increased for better multi-core scaling.\n\n* After several requests the Reconstruct and ReconstructData now slices of zero length but sufficient capacity to \nbe used instead of allocating new memory.\n\n## August 26, 2017\n\n*  The [`Encoder()`](https://godoc.org/github.com/klauspost/reedsolomon#Encoder) now contains an `Update` \nfunction contributed by [chenzhongtao](https://github.com/chenzhongtao).\n\n* [Frank Wessels](https://github.com/fwessels) kindly contributed ARM 64 bit assembly, \nwhich gives a huge performance boost on this platform.\n\n## July 20, 2017\n\n`ReconstructData` added to [`Encoder`](https://godoc.org/github.com/klauspost/reedsolomon#Encoder) interface. \nThis can cause compatibility issues if you implement your own Encoder. A simple workaround can be added:\n\n```Go\nfunc (e *YourEnc) ReconstructData(shards [][]byte) error {\n\treturn ReconstructData(shards)\n}\n```\n\nYou can of course also do your own implementation. \nThe [`StreamEncoder`](https://godoc.org/github.com/klauspost/reedsolomon#StreamEncoder) \nhandles this without modifying the interface. \nThis is a good lesson on why returning interfaces is not a good design.\n\n\u003c/details\u003e\n\n# Usage\n\nThis section assumes you know the basics of Reed-Solomon encoding. \nA good start is this [Backblaze blog post](https://www.backblaze.com/blog/reed-solomon/).\n\nThis package performs the calculation of the parity sets. The usage is therefore relatively simple.\n\nFirst of all, you need to choose your distribution of data and parity shards. \nA 'good' distribution is very subjective, and will depend a lot on your usage scenario. \n\nTo create an encoder with 10 data shards (where your data goes) and 3 parity shards (calculated):\n```Go\n    enc, err := reedsolomon.New(10, 3)\n```\nThis encoder will work for all parity sets with this distribution of data and parity shards. \n\nIf you will primarily be using it with one shard size it is recommended to use \n[`WithAutoGoroutines(shardSize)`](https://pkg.go.dev/github.com/klauspost/reedsolomon?tab=doc#WithAutoGoroutines)\nas an additional parameter. This will attempt to calculate the optimal number of goroutines to use for the best speed.\nIt is not required that all shards are this size. \n\nThen you send and receive data that is a simple slice of byte slices; `[][]byte`. \nIn the example above, the top slice must have a length of 13.\n\n```Go\n    data := make([][]byte, 13)\n```\nYou should then fill the 10 first slices with *equally sized* data, \nand create parity shards that will be populated with parity data. In this case we create the data in memory, \nbut you could for instance also use [mmap](https://github.com/edsrzf/mmap-go) to map files.\n\n```Go\n    // Create all shards, size them at 50000 each\n    for i := range input {\n      data[i] := make([]byte, 50000)\n    }\n    \n    // The above allocations can also be done by the encoder:\n    // data := enc.(reedsolomon.Extended).AllocAligned(50000)\n    \n    // Fill some data into the data shards\n    for i, in := range data[:10] {\n      for j:= range in {\n         in[j] = byte((i+j)\u00260xff)\n      }\n    }\n```\n\nTo populate the parity shards, you simply call `Encode()` with your data.\n```Go\n    err = enc.Encode(data)\n```\nThe only cases where you should get an error is, if the data shards aren't of equal size. \nThe last 3 shards now contain parity data. You can verify this by calling `Verify()`:\n\n```Go\n    ok, err = enc.Verify(data)\n```\n\nThe final (and important) part is to be able to reconstruct missing shards. \nFor this to work, you need to know which parts of your data is missing. \nThe encoder *does not know which parts are invalid*, so if data corruption is a likely scenario, \nyou need to implement a hash check for each shard. \n\nIf a byte has changed in your set, and you don't know which it is, there is no way to reconstruct the data set.\n\nTo indicate missing data, you set the shard to nil before calling `Reconstruct()`:\n\n```Go\n    // Delete two data shards\n    data[3] = nil\n    data[7] = nil\n    \n    // Reconstruct the missing shards\n    err := enc.Reconstruct(data)\n```\nThe missing data and parity shards will be recreated. If more than 3 shards are missing, the reconstruction will fail.\n\nIf you are only interested in the data shards (for reading purposes) you can call `ReconstructData()`:\n\n```Go\n    // Delete two data shards\n    data[3] = nil\n    data[7] = nil\n    \n    // Reconstruct just the missing data shards\n    err := enc.ReconstructData(data)\n```\n\nIf you don't need all data shards you can use `ReconstructSome()`:\n\n```Go\n    // Delete two data shards\n    data[3] = nil\n    data[7] = nil\n    \n    // Reconstruct just the shard 3\n    err := enc.ReconstructSome(data, []bool{false, false, false, true, false, false, false, false})\n```\n\nSo to sum up reconstruction:\n* The number of data/parity shards must match the numbers used for encoding.\n* The order of shards must be the same as used when encoding.\n* You may only supply data you know is valid.\n* Invalid shards should be set to nil.\n\nFor complete examples of an encoder and decoder see the \n[examples folder](https://github.com/klauspost/reedsolomon/tree/master/examples).\n\n# Splitting/Joining Data\n\nYou might have a large slice of data. \nTo help you split this, there are some helper functions that can split and join a single byte slice.\n\n```Go\n   bigfile, _ := os.Readfile(\"myfile.data\")\n   \n   // Split the file\n   split, err := enc.Split(bigfile)\n```\nThis will split the file into the number of data shards set when creating the encoder and create empty parity shards. \n\nAn important thing to note is that you have to *keep track of the exact input size*. \nIf the size of the input isn't divisible by the number of data shards, extra zeros will be inserted in the last shard.\n\nTo join a data set, use the `Join()` function, which will join the shards and write it to the `io.Writer` you supply: \n```Go\n   // Join a data set and write it to io.Discard.\n   err = enc.Join(io.Discard, data, len(bigfile))\n```\n\n## Aligned Allocations\n\nFor AMD64 aligned inputs can make a big speed difference.\n\nThis is an example of the speed difference when inputs are unaligned/aligned:\n\n```\nBenchmarkEncode100x20x10000-32    \t    7058\t    172648 ns/op\t6950.57 MB/s\nBenchmarkEncode100x20x10000-32    \t    8406\t    137911 ns/op\t8701.24 MB/s\n```\n\nThis is mostly the case when dealing with odd-sized shards. \n\nTo facilitate this the package provides an `AllocAligned(shards, each int) [][]byte`. \nThis will allocate a number of shards, each with the size `each`.\nEach shard will then be aligned to a 64 byte boundary.\n\nEach encoder also has a `AllocAligned(each int) [][]byte` as an extended interface which will return the same, \nbut with the shard count configured in the encoder.   \n\nIt is not possible to re-aligned already allocated slices, for example when using `Split`.\nWhen it is not possible to write to aligned shards, you should not copy to them.\n\n# Progressive encoding\n\nIt is possible to encode individual shards using EncodeIdx:\n\n```Go\n\t// EncodeIdx will add parity for a single data shard.\n\t// Parity shards should start out as 0. The caller must zero them.\n\t// Data shards must be delivered exactly once. There is no check for this.\n\t// The parity shards will always be updated and the data shards will remain the same.\n\tEncodeIdx(dataShard []byte, idx int, parity [][]byte) error\n```\n\nThis allows progressively encoding the parity by sending individual data shards.\nThere is no requirement on shards being delivered in order, \nbut when sent in order it allows encoding shards one at the time,\neffectively allowing the operation to be streaming. \n\nThe result will be the same as encoding all shards at once.\nThere is a minor speed penalty using this method, so send \nshards at once if they are available.\n\n## Example\n\n```Go\nfunc test() {\n    // Create an encoder with 7 data and 3 parity slices.\n    enc, _ := reedsolomon.New(7, 3)\n\n    // This will be our output parity.\n    parity := make([][]byte, 3)\n    for i := range parity {\n        parity[i] = make([]byte, 10000)\n    }\n\n    for i := 0; i \u003c 7; i++ {\n        // Send data shards one at the time.\n        _ = enc.EncodeIdx(make([]byte, 10000), i, parity)\n    }\n\n    // parity now contains parity, as if all data was sent in one call.\n}\n```\n\n# Streaming/Merging\n\nIt might seem like a limitation that all data should be in memory, \nbut an important property is that *as long as the number of data/parity shards are the same, \nyou can merge/split data sets*, and they will remain valid as a separate set.\n\n```Go\n    // Split the data set of 50000 elements into two of 25000\n    splitA := make([][]byte, 13)\n    splitB := make([][]byte, 13)\n    \n    // Merge into a 100000 element set\n    merged := make([][]byte, 13)\n    \n    for i := range data {\n      splitA[i] = data[i][:25000]\n      splitB[i] = data[i][25000:]\n      \n      // Concatenate it to itself\n\t  merged[i] = append(make([]byte, 0, len(data[i])*2), data[i]...)\n\t  merged[i] = append(merged[i], data[i]...)\n    }\n    \n    // Each part should still verify as ok.\n    ok, err := enc.Verify(splitA)\n    if ok \u0026\u0026 err == nil {\n        log.Println(\"splitA ok\")\n    }\n    \n    ok, err = enc.Verify(splitB)\n    if ok \u0026\u0026 err == nil {\n        log.Println(\"splitB ok\")\n    }\n    \n    ok, err = enc.Verify(merge)\n    if ok \u0026\u0026 err == nil {\n        log.Println(\"merge ok\")\n    }\n```\n\nThis means that if you have a data set that may not fit into memory, you can split processing into smaller blocks. \nFor the best throughput, don't use too small blocks.\n\nThis also means that you can divide big input up into smaller blocks, and do reconstruction on parts of your data. \nThis doesn't give the same flexibility of a higher number of data shards, but it will be much more performant.\n\n# Streaming API\n\nThere has been added support for a streaming API, to help perform fully streaming operations, \nwhich enables you to do the same operations, but on streams. \nTo use the stream API, use [`NewStream`](https://godoc.org/github.com/klauspost/reedsolomon#NewStream) function \nto create the encoding/decoding interfaces. \n\nYou can use [`WithConcurrentStreams`](https://godoc.org/github.com/klauspost/reedsolomon#WithConcurrentStreams) \nto ready an interface that reads/writes concurrently from the streams.\n\nYou can specify the size of each operation using \n[`WithStreamBlockSize`](https://godoc.org/github.com/klauspost/reedsolomon#WithStreamBlockSize).\nThis will set the size of each read/write operation.\n\nInput is delivered as `[]io.Reader`, output as `[]io.Writer`, and functionality corresponds to the in-memory API. \nEach stream must supply the same amount of data, similar to how each slice must be similar size with the in-memory API. \nIf an error occurs in relation to a stream, \na [`StreamReadError`](https://godoc.org/github.com/klauspost/reedsolomon#StreamReadError) \nor [`StreamWriteError`](https://godoc.org/github.com/klauspost/reedsolomon#StreamWriteError) \nwill help you determine which stream was the offender.\n\nThere is no buffering or timeouts/retry specified. If you want to add that, you need to add it to the Reader/Writer.\n\nFor complete examples of a streaming encoder and decoder see the \n[examples folder](https://github.com/klauspost/reedsolomon/tree/master/examples).\n\nGF16 (more than 256 shards) is not supported by the streaming interface. \n\n# Advanced Options\n\nYou can modify internal options which affects how jobs are split between and processed by goroutines.\n\nTo create options, use the WithXXX functions. You can supply options to `New`, `NewStream`. \nIf no Options are supplied, default options are used.\n\nExample of how to supply options:\n\n ```Go\n     enc, err := reedsolomon.New(10, 3, WithMaxGoroutines(25))\n ```\n\n# Leopard Compatible GF16\n\nWhen you encode more than 256 shards the library will switch to a [Leopard-RS](https://github.com/catid/leopard) implementation.\n\nThis allows encoding up to 65536 shards (data+parity) with the following limitations, similar to leopard:\n\n* The original and recovery data must not exceed 65536 pieces.\n* The shard size *must*  each be a multiple of 64 bytes.\n* Each buffer should have the same number of bytes.\n* Even the last shard must be rounded up to the block size.\n\n|                 | Regular | Leopard |\n|-----------------|---------|---------|\n| Encode          | ✓       | ✓       |\n| EncodeIdx       | ✓       | -       |\n| Verify          | ✓       | ✓       |\n| Reconstruct     | ✓       | ✓       |\n| ReconstructData | ✓       | ✓       |\n| ReconstructSome | ✓       | ✓ (+)   |\n| Update          | ✓       | -       |\n| Split           | ✓       | ✓       |\n| Join            | ✓       | ✓       |\n\n* (+) Same as calling `ReconstructData`.\n\nThe Split/Join functions will help to split an input to the proper sizes.\n\nSpeed can be expected to be `O(N*log(N))`, compared to the `O(N*N)`. \nReconstruction matrix calculation is more time-consuming, \nso be sure to include that as part of any benchmark you run.  \n\nFor now SSSE3, AVX2 and AVX512 assembly are available on AMD64 platforms.\n\nLeopard mode currently always runs as a single goroutine, since multiple \ngoroutines doesn't provide any worthwhile speedup.\n\n## Leopard GF8\n\nIt is possible to replace the default reed-solomon encoder with a leopard compatible one.\nThis will typically be faster when dealing with more than 20-30 shards.\nNote that the limitations listed above also applies to this mode. \nSee table below for speed with different number of shards.\n\nTo enable Leopard GF8 mode use `WithLeopardGF(true)`.\n\nBenchmark Encoding and Reconstructing *1KB* shards with variable number of shards.\nAll implementation use inversion cache when available.\nSpeed is total shard size for each operation. Data shard throughput is speed/2.\nAVX2 is used.\n\n| Encoder      | Shards      | Encode         | Recover All  | Recover One    |\n|--------------|-------------|----------------|--------------|----------------|\n| Cauchy       | 4+4         | 23076.83 MB/s  | 5444.02 MB/s | 10834.67 MB/s  |\n| Cauchy       | 8+8         | 15206.87 MB/s  | 4223.42 MB/s | 16181.62  MB/s |\n| Cauchy       | 16+16       | 7427.47 MB/s   | 3305.84 MB/s | 22480.41  MB/s |\n| Cauchy       | 32+32       | 3785.64 MB/s   | 2300.07 MB/s | 26181.31  MB/s |\n| Cauchy       | 64+64       | 1911.93 MB/s   | 1368.51 MB/s | 27992.93 MB/s  |\n| Cauchy       | 128+128     | 963.83 MB/s    | 1327.56 MB/s | 32866.86 MB/s  |\n| Leopard GF8  | 4+4         | 17061.28 MB/s  | 3099.06 MB/s | 4096.78 MB/s   |\n| Leopard GF8  | 8+8         | 10546.67 MB/s  | 2925.92 MB/s | 3964.00 MB/s   |\n| Leopard GF8  | 16+16       | 10961.37  MB/s | 2328.40 MB/s | 3110.22 MB/s   |\n| Leopard GF8  | 32+32       | 7111.47 MB/s   | 2374.61 MB/s | 3220.75 MB/s   |\n| Leopard GF8  | 64+64       | 7468.57 MB/s   | 2055.41 MB/s | 3061.81 MB/s   |\n| Leopard GF8  | 128+128     | 5479.99 MB/s   | 1953.21 MB/s | 2815.15 MB/s   |\n| Leopard GF16 | 256+256     | 6158.66 MB/s   | 454.14 MB/s  | 506.70 MB/s    |\n| Leopard GF16 | 512+512     | 4418.58 MB/s   | 685.75 MB/s  | 801.63 MB/s    |\n| Leopard GF16 | 1024+1024   | 4778.05 MB/s   | 814.51 MB/s  | 1080.19 MB/s   |\n| Leopard GF16 | 2048+2048   | 3417.05 MB/s   | 911.64 MB/s  | 1179.48 MB/s   |\n| Leopard GF16 | 4096+4096   | 3209.41 MB/s   | 729.13 MB/s  | 1135.06 MB/s   |\n| Leopard GF16 | 8192+8192   | 2034.11 MB/s   | 604.52 MB/s  | 842.13 MB/s    |\n| Leopard GF16 | 16384+16384 | 1525.88 MB/s   | 486.74 MB/s  | 750.01 MB/s    |\n| Leopard GF16 | 32768+32768 | 1138.67 MB/s   | 482.81 MB/s  | 712.73 MB/s    |\n\n\"Traditional\" encoding is faster until somewhere between 16 and 32 shards.\nLeopard provides fast encoding in all cases, but shows a significant overhead for reconstruction.\n\nCalculating the reconstruction matrix takes a significant amount of computation. \nWith bigger shards that will be smaller. Arguably, fewer shards typically also means bigger shards.\nDue to the high shard count caching reconstruction matrices generally isn't feasible for Leopard. \n\n# Performance\n\nPerformance depends mainly on the number of parity shards. \nIn rough terms, doubling the number of parity shards will double the encoding time.\n\nHere are the throughput numbers with some different selections of data and parity shards. \nFor reference each shard is 1MB random data, and 16 CPU cores are used for encoding.\n\n| Data | Parity | Go MB/s | SSSE3 MB/s | AVX2 MB/s |\n|------|--------|---------|------------|-----------|\n| 5    | 2      | 20,772  | 66,355     | 108,755   |\n| 8    | 8      | 6,815   | 38,338     | 70,516    |\n| 10   | 4      | 9,245   | 48,237     | 93,875    |\n| 50   | 20     | 2,063   | 12,130     | 22,828    |\n\nThe throughput numbers here is the size of the encoded data and parity shards.\n\nIf `runtime.GOMAXPROCS()` is set to a value higher than 1, \nthe encoder will use multiple goroutines to perform the calculations in `Verify`, `Encode` and `Reconstruct`.\n\n\nBenchmarking `Reconstruct()` followed by a `Verify()` (=`all`) versus just calling `ReconstructData()` (=`data`) gives the following result:\n```\nbenchmark                            all MB/s     data MB/s    speedup\nBenchmarkReconstruct10x2x10000-8     2011.67      10530.10     5.23x\nBenchmarkReconstruct50x5x50000-8     4585.41      14301.60     3.12x\nBenchmarkReconstruct10x2x1M-8        8081.15      28216.41     3.49x\nBenchmarkReconstruct5x2x1M-8         5780.07      28015.37     4.85x\nBenchmarkReconstruct10x4x1M-8        4352.56      14367.61     3.30x\nBenchmarkReconstruct50x20x1M-8       1364.35      4189.79      3.07x\nBenchmarkReconstruct10x4x16M-8       1484.35      5779.53      3.89x\n```\n\nThe package will use [GFNI](https://en.wikipedia.org/wiki/AVX-512#GFNI) instructions combined with AVX512 when these are available.\nThis further improves speed by up to 3x over AVX2 code paths.\n\n## ARM64 NEON\n\nBy exploiting NEON instructions the performance for ARM has been accelerated. \nBelow are the performance numbers for a single core on an EC2 m6g.16xlarge (Graviton2) instance (Amazon Linux 2):\n\n```\nBenchmarkGalois128K-64        119562     10028 ns/op        13070.78 MB/s\nBenchmarkGalois1M-64           14380     83424 ns/op        12569.22 MB/s\nBenchmarkGaloisXor128K-64      96508     12432 ns/op        10543.29 MB/s\nBenchmarkGaloisXor1M-64        10000    100322 ns/op        10452.13 MB/s\n```\n\n# Performance on ppc64le\n\nThe performance for ppc64le has been accelerated. \nThis gives roughly a 10x performance improvement on this architecture as can be seen below:\n\n```\nbenchmark                      old MB/s     new MB/s     speedup\nBenchmarkGalois128K-160        948.87       8878.85      9.36x\nBenchmarkGalois1M-160          968.85       9041.92      9.33x\nBenchmarkGaloisXor128K-160     862.02       7905.00      9.17x\nBenchmarkGaloisXor1M-160       784.60       6296.65      8.03x\n```\n\n# Legal\n\n\u003e None of section below is legal advice. Seek your own legal counsel.\n\u003e As stated by the [LICENSE](LICENSE) the authors will not be held reliable for any use of this library.\n\u003e Users are encouraged to independently verify they comply with all legal requirements. \n\nAs can be seen in [recent news](https://www.datanami.com/2023/10/16/cloudera-hit-with-240-million-judgement-over-erasure-coding/)\nthere has been lawsuits related to possible patents of aspects of erasure coding functionality.\n\nAs a possible mitigation it is possible to use the tag `nopshufb` when compiling any code which includes this package.\nThis will remove all inclusion and use of `PSHUFB` and equivalent on other platforms.\n\nThis is done by adding `-tags=nopshufb` to `go build` and similar commands that produce binary output.\n\nThe removed code may not be infringing and even after `-tags=nopshufb` there may still be infringing code left. \n\n# Links\n* [Backblaze Open Sources Reed-Solomon Erasure Coding Source Code](https://www.backblaze.com/blog/reed-solomon/).\n* [JavaReedSolomon](https://github.com/Backblaze/JavaReedSolomon). Compatible java library by Backblaze.\n* [ocaml-reed-solomon-erasure](https://gitlab.com/darrenldl/ocaml-reed-solomon-erasure). Compatible OCaml implementation.\n* [reedsolomon-c](https://github.com/jannson/reedsolomon-c). C version, compatible with output from this package.\n* [Reed-Solomon Erasure Coding in Haskell](https://github.com/NicolasT/reedsolomon). Haskell port of the package with similar performance.\n* [reed-solomon-erasure](https://github.com/darrenldl/reed-solomon-erasure). Compatible Rust implementation.\n* [go-erasure](https://github.com/somethingnew2-0/go-erasure). A similar library using cgo, slower in my tests.\n* [Screaming Fast Galois Field Arithmetic](https://www.snia.org/sites/default/files/files2/files2/SDC2013/presentations/NewThinking/EthanMiller_Screaming_Fast_Galois_Field%20Arithmetic_SIMD%20Instructions.pdf). Basis for SSE3 optimizations.\n* [Leopard-RS](https://github.com/catid/leopard) C library used as basis for GF16 implementation.\n* [reed-solomon-simd](https://github.com/AndersTrier/reed-solomon-simd) Leopard-RS Rust implementation.\n\n# License\n\nThis code, as the original [JavaReedSolomon](https://github.com/Backblaze/JavaReedSolomon) is published under an MIT license. See LICENSE file for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklauspost%2Freedsolomon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fklauspost%2Freedsolomon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklauspost%2Freedsolomon/lists"}