{"id":16826626,"url":"https://github.com/burntsushi/memchr","last_synced_at":"2025-05-12T13:12:56.227Z","repository":{"id":33638286,"uuid":"37290652","full_name":"BurntSushi/memchr","owner":"BurntSushi","description":"Optimized string search routines for Rust.","archived":false,"fork":false,"pushed_at":"2025-03-21T21:01:16.000Z","size":12097,"stargazers_count":1078,"open_issues_count":15,"forks_count":109,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-05-12T13:12:38.783Z","etag":null,"topics":["bytes","memchr","rabin-karp","rust","simd","string","string-searching","twoway"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BurntSushi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["BurntSushi"]}},"created_at":"2015-06-11T23:01:31.000Z","updated_at":"2025-05-12T05:10:36.000Z","dependencies_parsed_at":"2023-02-16T00:30:42.249Z","dependency_job_id":"55cad25e-d9a6-430b-90e0-ebf24e7a7333","html_url":"https://github.com/BurntSushi/memchr","commit_stats":{"total_commits":241,"total_committers":44,"mean_commits":"5.4772727272727275","dds":0.3360995850622407,"last_synced_commit":"746182171d2e88600652de3e6768cd8aef77e585"},"previous_names":[],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BurntSushi%2Fmemchr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BurntSushi%2Fmemchr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BurntSushi%2Fmemchr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BurntSushi%2Fmemchr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BurntSushi","download_url":"https://codeload.github.com/BurntSushi/memchr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253745195,"owners_count":21957319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bytes","memchr","rabin-karp","rust","simd","string","string-searching","twoway"],"created_at":"2024-10-13T11:17:57.761Z","updated_at":"2025-05-12T13:12:56.192Z","avatar_url":"https://github.com/BurntSushi.png","language":"Rust","funding_links":["https://github.com/sponsors/BurntSushi"],"categories":[],"sub_categories":[],"readme":"memchr\n======\nThis library provides heavily optimized routines for string search primitives.\n\n[![Build status](https://github.com/BurntSushi/memchr/workflows/ci/badge.svg)](https://github.com/BurntSushi/memchr/actions)\n[![Crates.io](https://img.shields.io/crates/v/memchr.svg)](https://crates.io/crates/memchr)\n\nDual-licensed under MIT or the [UNLICENSE](https://unlicense.org/).\n\n\n### Documentation\n\n[https://docs.rs/memchr](https://docs.rs/memchr)\n\n\n### Overview\n\n* The top-level module provides routines for searching for 1, 2 or 3 bytes\n  in the forward or reverse direction. When searching for more than one byte,\n  positions are considered a match if the byte at that position matches any\n  of the bytes.\n* The `memmem` sub-module provides forward and reverse substring search\n  routines.\n\nIn all such cases, routines operate on `\u0026[u8]` without regard to encoding. This\nis exactly what you want when searching either UTF-8 or arbitrary bytes.\n\n### Compiling without the standard library\n\nmemchr links to the standard library by default, but you can disable the\n`std` feature if you want to use it in a `#![no_std]` crate:\n\n```toml\n[dependencies]\nmemchr = { version = \"2\", default-features = false }\n```\n\nOn `x86_64` platforms, when the `std` feature is disabled, the SSE2 accelerated\nimplementations will be used. When `std` is enabled, AVX2 accelerated\nimplementations will be used if the CPU is determined to support it at runtime.\n\nSIMD accelerated routines are also available on the `wasm32` and `aarch64`\ntargets. The `std` feature is not required to use them.\n\nWhen a SIMD version is not available, then this crate falls back to\n[SWAR](https://en.wikipedia.org/wiki/SWAR) techniques.\n\n### Minimum Rust version policy\n\nThis crate's minimum supported `rustc` version is `1.61.0`.\n\nThe current policy is that the minimum Rust version required to use this crate\ncan be increased in minor version updates. For example, if `crate 1.0` requires\nRust 1.20.0, then `crate 1.0.z` for all values of `z` will also require Rust\n1.20.0 or newer. However, `crate 1.y` for `y \u003e 0` may require a newer minimum\nversion of Rust.\n\nIn general, this crate will be conservative with respect to the minimum\nsupported version of Rust.\n\n\n### Testing strategy\n\nGiven the complexity of the code in this crate, along with the pervasive use\nof `unsafe`, this crate has an extensive testing strategy. It combines multiple\napproaches:\n\n* Hand-written tests.\n* Exhaustive-style testing meant to exercise all possible branching and offset\n  calculations.\n* Property based testing through [`quickcheck`](https://github.com/BurntSushi/quickcheck).\n* Fuzz testing through [`cargo fuzz`](https://github.com/rust-fuzz/cargo-fuzz).\n* A huge suite of benchmarks that are also run as tests. Benchmarks always\n  confirm that the expected result occurs.\n\nImprovements to the testing infrastructure are very welcome.\n\n\n### Algorithms used\n\nAt time of writing, this crate's implementation of substring search actually\nhas a few different algorithms to choose from depending on the situation.\n\n* For very small haystacks,\n  [Rabin-Karp](https://en.wikipedia.org/wiki/Rabin%E2%80%93Karp_algorithm)\n  is used to reduce latency. Rabin-Karp has very small overhead and can often\n  complete before other searchers have even been constructed.\n* For small needles, a variant of the\n  [\"Generic SIMD\"](http://0x80.pl/articles/simd-strfind.html#algorithm-1-generic-simd)\n  algorithm is used. Instead of using the first and last bytes, a heuristic is\n  used to select bytes based on a background distribution of byte frequencies.\n* In all other cases,\n  [Two-Way](https://en.wikipedia.org/wiki/Two-way_string-matching_algorithm)\n  is used. If possible, a prefilter based on the \"Generic SIMD\" algorithm\n  linked above is used to find candidates quickly. A dynamic heuristic is used\n  to detect if the prefilter is ineffective, and if so, disables it.\n\n\n### Why is the standard library's substring search so much slower?\n\nWe'll start by establishing what the difference in performance actually\nis. There are two relevant benchmark classes to consider: `prebuilt` and\n`oneshot`. The `prebuilt` benchmarks are designed to measure---to the extent\npossible---search time only. That is, the benchmark first starts by building a\nsearcher and then only tracking the time for _using_ the searcher:\n\n```\n$ rebar rank benchmarks/record/x86_64/2023-08-26.csv --intersection -e memchr/memmem/prebuilt -e std/memmem/prebuilt\nEngine                       Version                   Geometric mean of speed ratios  Benchmark count\n------                       -------                   ------------------------------  ---------------\nrust/memchr/memmem/prebuilt  2.5.0                     1.03                            53\nrust/std/memmem/prebuilt     1.73.0-nightly 180dffba1  6.50                            53\n```\n\nConversely, the `oneshot` benchmark class measures the time it takes to both\nbuild the searcher _and_ use it:\n\n```\n$ rebar rank benchmarks/record/x86_64/2023-08-26.csv --intersection -e memchr/memmem/oneshot -e std/memmem/oneshot\nEngine                      Version                   Geometric mean of speed ratios  Benchmark count\n------                      -------                   ------------------------------  ---------------\nrust/memchr/memmem/oneshot  2.5.0                     1.04                            53\nrust/std/memmem/oneshot     1.73.0-nightly 180dffba1  5.26                            53\n```\n\n**NOTE:** Replace `rebar rank` with `rebar cmp` in the above commands to\nexplore the specific benchmarks and their differences.\n\nSo in both cases, this crate is quite a bit faster over a broad sampling of\nbenchmarks regardless of whether you measure only search time or search time\nplus construction time. The difference is a little smaller when you include\nconstruction time in your measurements.\n\nThese two different types of benchmark classes make for a nice segue into\none reason why the standard library's substring search can be slower: API\ndesign. In the standard library, the only APIs available to you require\none to re-construct the searcher for every search. While you can benefit\nfrom building a searcher once and iterating over all matches in a single\nstring, you cannot reuse that searcher to search other strings. This might\ncome up when, for example, searching a file one line at a time. You'll need\nto re-build the searcher for every line searched, and this can [really\nmatter][burntsushi-bstr-blog].\n\n**NOTE:** The `prebuilt` benchmark for the standard library can't actually\navoid measuring searcher construction at some level, because there is no API\nfor it. Instead, the benchmark consists of building the searcher once and then\nfinding all matches in a single string via an iterator. This tends to\napproximate a benchmark where searcher construction isn't measured, but it\nisn't perfect. While this means the comparison is not strictly\napples-to-apples, it does reflect what is maximally possible with the standard\nlibrary, and thus reflects the best that one could do in a real world scenario.\n\nWhile there is more to the story than just API design here, it's important to\npoint out that even if the standard library's substring search were a precise\nclone of this crate internally, it would still be at a disadvantage in some\nworkloads because of its API. (The same also applies to C's standard library\n`memmem` function. There is no way to amortize construction of the searcher.\nYou need to pay for it on every call.)\n\nThe other reason for the difference in performance is that\nthe standard library has trouble using SIMD. In particular, substring search\nis implemented in the `core` library, where platform specific code generally\ncan't exist. That's an issue because in order to utilize SIMD beyond SSE2\nwhile maintaining portable binaries, one needs to use [dynamic CPU feature\ndetection][dynamic-cpu], and that in turn requires platform specific code.\nWhile there is [an RFC for enabling target feature detection in\n`core`][core-feature], it doesn't yet exist.\n\nThe bottom line here is that `core`'s substring search implementation is\nlimited to making use of SSE2, but not AVX.\n\nStill though, this crate does accelerate substring search even when only SSE2\nis available. The standard library could therefore adopt the techniques in this\ncrate just for SSE2. The reason why that hasn't happened yet isn't totally\nclear to me. It likely needs a champion to push it through. The standard\nlibrary tends to be more conservative in these things. With that said, the\nstandard library does use some [SSE2 acceleration on `x86-64`][std-sse2] added\nin [this PR][std-sse2-pr]. However, at the time of writing, it is only used\nfor short needles and doesn't use the frequency based heuristics found in this\ncrate.\n\n**NOTE:** Another thing worth mentioning is that the standard library's\nsubstring search routine requires that both the needle and haystack have type\n`\u0026str`. Unless you can assume that your data is valid UTF-8, building a `\u0026str`\nwill come with the overhead of UTF-8 validation. This may in turn result in\noverall slower searching depending on your workload. In contrast, the `memchr`\ncrate permits both the needle and the haystack to have type `\u0026[u8]`, where\n`\u0026[u8]` can be created from a `\u0026str` with zero cost. Therefore, the substring\nsearch in this crate is strictly more flexible than what the standard library\nprovides.\n\n[burntsushi-bstr-blog]: https://blog.burntsushi.net/bstr/#motivation-based-on-performance\n[dynamic-cpu]: https://doc.rust-lang.org/std/arch/index.html#dynamic-cpu-feature-detection\n[core-feature]: https://github.com/rust-lang/rfcs/pull/3469\n[std-sse2]: https://github.com/rust-lang/rust/blob/bf9229a2e366b4c311f059014a4aa08af16de5d8/library/core/src/str/pattern.rs#L1719-L1857\n[std-sse2-pr]: https://github.com/rust-lang/rust/pull/103779\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fburntsushi%2Fmemchr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fburntsushi%2Fmemchr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fburntsushi%2Fmemchr/lists"}