{"id":13608079,"url":"https://github.com/minio/md5-simd","last_synced_at":"2025-05-15T17:09:30.365Z","repository":{"id":46509193,"uuid":"257397381","full_name":"minio/md5-simd","owner":"minio","description":"Accelerate aggregated MD5 hashing performance up to 8x for AVX512 and 4x for AVX2. Useful for server applications that need to compute many MD5 sums in parallel.","archived":false,"fork":false,"pushed_at":"2025-04-02T07:55:34.000Z","size":719,"stargazers_count":194,"open_issues_count":2,"forks_count":22,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-07T23:04:44.026Z","etag":null,"topics":["assembly","avx2","avx512","golang","hashing","md5","performance","simd"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/minio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-20T20:34:52.000Z","updated_at":"2025-04-04T02:16:47.000Z","dependencies_parsed_at":"2022-07-19T22:33:53.567Z","dependency_job_id":null,"html_url":"https://github.com/minio/md5-simd","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minio%2Fmd5-simd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minio%2Fmd5-simd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minio%2Fmd5-simd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minio%2Fmd5-simd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/minio","download_url":"https://codeload.github.com/minio/md5-simd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254384989,"owners_count":22062422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["assembly","avx2","avx512","golang","hashing","md5","performance","simd"],"created_at":"2024-08-01T19:01:24.069Z","updated_at":"2025-05-15T17:09:25.356Z","avatar_url":"https://github.com/minio.png","language":"Go","funding_links":[],"categories":["Go","Hash"],"sub_categories":[],"readme":"\n# md5-simd\n\nThis is a SIMD accelerated MD5 package, allowing up to either 8 (AVX2) or 16 (AVX512) independent MD5 sums to be calculated on a single CPU core.\n\nIt was originally based on the [md5vec](https://github.com/igneous-systems/md5vec) repository by Igneous Systems, but has been made more flexible by amongst others supporting different message sizes per lane and adding AVX512.\n\n`md5-simd` integrates a similar mechanism as described in [minio/sha256-simd](https://github.com/minio/sha256-simd#support-for-avx512) for making it easy for clients to take advantages of the parallel nature of the MD5 calculation. This will result in reduced overall CPU load. \n\nIt is important to understand that `md5-simd` **does not speed up** a single threaded MD5 hash sum. \nRather it allows multiple __independent__  MD5 sums to be computed in parallel on the same CPU core, \nthereby making more efficient usage of the computing resources.\n\n## Usage\n\n[![Documentation](https://godoc.org/github.com/minio/md5-simd?status.svg)](https://pkg.go.dev/github.com/minio/md5-simd?tab=doc)\n\n\nIn order to use `md5-simd`, you must first create an `Server` which can be \nused to instantiate one or more objects for MD5 hashing. \n\nThese objects conform to the regular [`hash.Hash`](https://pkg.go.dev/hash?tab=doc#Hash) interface \nand as such the normal Write/Reset/Sum functionality works as expected. \n\nAs an example: \n```\n    // Create server\n    server := md5simd.NewServer()\n    defer server.Close()\n\n    // Create hashing object (conforming to hash.Hash)\n    md5Hash := server.NewHash()\n    defer md5Hash.Close()\n\n    // Write one (or more) blocks\n    md5Hash.Write(block)\n    \n    // Return digest\n    digest := md5Hash.Sum([]byte{})\n```\n\nTo keep performance both a [Server](https://pkg.go.dev/github.com/minio/md5-simd?tab=doc#Server) \nand individual [Hasher](https://pkg.go.dev/github.com/minio/md5-simd?tab=doc#Hasher) should \nbe closed using the `Close()` function when no longer needed.\n\nA Hasher can efficiently be re-used by using [`Reset()`](https://pkg.go.dev/hash?tab=doc#Hash) functionality.\n\nIn case your system does not support the instructions required it will fall back to using `crypto/md5` for hashing.\n\n## Limitations\n\nAs explained above `md5-simd` does not speed up an individual MD5 hash sum computation,\nunless some hierarchical tree construct is used but this will result in different outcomes.\nRunning a single hash on a server results in approximately half the throughput.\n\nInstead, it allows running multiple MD5 calculations in parallel on a single CPU core. \nThis can be beneficial in e.g. multi-threaded server applications where many go-routines \nare dealing with many requests and multiple MD5 calculations can be packed/scheduled for parallel execution on a single core.\n\nThis will result in a lower overall CPU usage as compared to using the standard `crypto/md5`\nfunctionality where each MD5 hash computation will consume a single thread (core).\n\nIt is best to test and measure the overall CPU usage in a representative usage scenario in your application\nto get an overall understanding of the benefits of `md5-simd` as compared to `crypto/md5`, ideally under heavy CPU load.\n\nAlso note that `md5-simd` is best meant to work with large objects, \nso if your application only hashes small objects of a few kilobytes \nyou may be better of by using `crypto/md5`.\n\n## Performance\n\nFor the best performance writes should be a multiple of 64 bytes, ideally a multiple of 32KB.\nTo help with that a [`buffered := bufio.NewWriterSize(hasher, 32\u003c\u003c10)`](https://golang.org/pkg/bufio/#NewWriterSize) \ncan be inserted if you are unsure of the sizes of the writes. \nRemember to [flush](https://golang.org/pkg/bufio/#Writer.Flush) `buffered` before reading the hash. \n\nA single 'server' can process 16 streams concurrently with 1 core (AVX-512) or 2 cores (AVX2). \nIn situations where it is likely that more than 16 streams are fully loaded it may be beneficial\nto use multiple servers.\n\nThe following chart compares the multi-core performance between `crypto/md5` vs the AVX2 vs the AVX512 code:\n\n![md5-performance-overview](chart/Multi-core-MD5-Aggregated-Hashing-Performance.png)\n\nCompared to `crypto/md5`, the AVX2 version is up to 4x faster:\n\n```\n$ benchcmp crypto-md5.txt avx2.txt \nbenchmark                     old MB/s     new MB/s     speedup\nBenchmarkParallel/32KB-4      2229.22      7370.50      3.31x\nBenchmarkParallel/64KB-4      2233.61      8248.46      3.69x\nBenchmarkParallel/128KB-4     2235.43      8660.74      3.87x\nBenchmarkParallel/256KB-4     2236.39      8863.87      3.96x\nBenchmarkParallel/512KB-4     2238.05      8985.39      4.01x\nBenchmarkParallel/1MB-4       2233.56      9042.62      4.05x\nBenchmarkParallel/2MB-4       2224.11      9014.46      4.05x\nBenchmarkParallel/4MB-4       2199.78      8993.61      4.09x\nBenchmarkParallel/8MB-4       2182.48      8748.22      4.01x\n```\n\nCompared to `crypto/md5`, the AVX512 is up to 8x faster (for larger block sizes):\n\n```\n$ benchcmp crypto-md5.txt avx512.txt\nbenchmark                     old MB/s     new MB/s     speedup\nBenchmarkParallel/32KB-4      2229.22      11605.78     5.21x\nBenchmarkParallel/64KB-4      2233.61      14329.65     6.42x\nBenchmarkParallel/128KB-4     2235.43      16166.39     7.23x\nBenchmarkParallel/256KB-4     2236.39      15570.09     6.96x\nBenchmarkParallel/512KB-4     2238.05      16705.83     7.46x\nBenchmarkParallel/1MB-4       2233.56      16941.95     7.59x\nBenchmarkParallel/2MB-4       2224.11      17136.01     7.70x\nBenchmarkParallel/4MB-4       2199.78      17218.61     7.83x\nBenchmarkParallel/8MB-4       2182.48      17252.88     7.91x\n```\n\nThese measurements were performed on AWS EC2 instance of type `c5.xlarge` equipped with a Xeon Platinum 8124M CPU at 3.0 GHz.\n\nIf only one or two inputs are available the scalar calculation method will be used for the \noptimal speed in these cases.\n\n## Operation\n\nTo make operation as easy as possible there is a “Server” coordinating everything. The server keeps track of individual hash states and updates them as new data comes in. This can be visualized as follows:\n\n![server-architecture](chart/server-architecture.png)\n\nThe data is sent to the server from each hash input in blocks of up to 32KB per round. In our testing we found this to be the block size that yielded the best results.\n\nWhenever there is data available the server will collect data for up to 16 hashes and process all 16 lanes in parallel. This means that if 16 hashes have data available all the lanes will be filled. However since that may not be the case, the server will fill less lanes and do a round anyway. Lanes can also be partially filled if less than 32KB of data is written.\n\n![server-lanes-example](chart/server-lanes-example.png)\n\nIn this example 4 lanes are fully filled and 2 lanes are partially filled. In this case the black areas will simply be masked out from the results and ignored. This is also why calculating a single hash on a server will not result in any speedup and hash writes should be a multiple of 32KB for the best performance.\n\nFor AVX512 all 16 calculations will be done on a single core, on AVX2 on 2 cores if there is data for more than 8 lanes.\nSo for optimal usage there should be data available for all 16 hashes. It may be perfectly reasonable to use more than 16 concurrent hashes.\n\n\n## Design \u0026 Tech\n\nmd5-simd has both an AVX2 (8-lane parallel), and an AVX512 (16-lane parallel version) algorithm to accelerate the computation with the following function definitions:\n```\n//go:noescape\nfunc block8(state *uint32, base uintptr, bufs *int32, cache *byte, n int)\n\n//go:noescape\nfunc block16(state *uint32, ptrs *int64, mask uint64, n int)\n```\n\nThe AVX2 version is based on the [md5vec](https://github.com/igneous-systems/md5vec) repository and is essentially unchanged except for minor (cosmetic) changes.\n\nThe AVX512 version is derived from the AVX2 version but adds some further optimizations and simplifications.\n\n### Caching in upper ZMM registers\n\nThe AVX2 version passes in a `cache8` block of memory (about 0.5 KB) for temporary storage of intermediate results during `ROUND1` which are subsequently used during `ROUND2` through to `ROUND4`.\n\nSince AVX512 has double the amount of registers (32 ZMM registers as compared to 16 YMM registers), it is possible to use the upper 16 ZMM registers for keeping the intermediate states on the CPU. As such, there is no need to pass in a corresponding `cache16` into the AVX512 block function.\n\n### Direct loading using 64-bit pointers\n\nThe AVX2 uses the `VPGATHERDD` instruction (for YMM) to do a parallel load of 8 lanes using (8 independent) 32-bit offets. Since there is no control over how the 8 slices that are passed into the (Golang) `blockMd5` function are laid out into memory, it is not possible to derive a \"base\" address and corresponding offsets (all within 32-bits) for all 8 slices.\n\nAs such the AVX2 version uses an interim buffer to collect the byte slices to be hashed from all 8 inut slices and passed this buffer along with (fixed) 32-bit offsets into the assembly code.\n\nFor the AVX512 version this interim buffer is not needed since the AVX512 code uses a pair of `VPGATHERQD` instructions to directly dereference 64-bit pointers (from a base register address that is initialized to zero).\n\nNote that two load (gather) instructions are needed because the AVX512 version processes 16-lanes in parallel, requiring 16 times 64-bit = 1024 bits in total to be loaded. A simple `VALIGND` and `VPORD` are subsequently used to merge the lower and upper halves together into a single ZMM register (that contains 16 lanes of 32-bit DWORDS).\n\n### Masking support\n\nDue to the fact that pointers are passed directly from the Golang slices, we need to protect against NULL pointers. \nFor this a 16-bit mask is passed in the AVX512 assembly code which is used during the `VPGATHERQD` instructions to mask out lanes that could otherwise result in segment violations.\n\n### Minor optimizations\n\nThe `roll` macro (three instructions on AVX2) is no longer needed for AVX512 and is replaced by a single `VPROLD` instruction.\n\nAlso several logical operations from the various ROUNDS of the AVX2 version could be combined into a single instruction using ternary logic (with the `VPTERMLOGD` instruction), resulting in a further simplification and speed-up.\n\n## Low level block function performance\n\nThe benchmark below shows the (single thread) maximum performance of the `block()` function for AVX2 (having 8 lanes) and AVX512 (having 16 lanes). Also the baseline single-core performance from the standard `crypto/md5` package is shown for comparison.\n\n```\nBenchmarkCryptoMd5-4                     687.66 MB/s           0 B/op          0 allocs/op\nBenchmarkBlock8-4                       4144.80 MB/s           0 B/op          0 allocs/op\nBenchmarkBlock16-4                      8228.88 MB/s           0 B/op          0 allocs/op\n```\n\n## License\n\n`md5-simd` is released under the Apache License v2.0. You can find the complete text in the file LICENSE.\n\n## Contributing\n\nContributions are welcome, please send PRs for any enhancements.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminio%2Fmd5-simd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fminio%2Fmd5-simd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminio%2Fmd5-simd/lists"}