{"id":28455099,"url":"https://github.com/d-rickyy-b/rexamine","last_synced_at":"2025-10-26T08:33:08.333Z","repository":{"id":223133604,"uuid":"759402530","full_name":"d-Rickyy-b/rexamine","owner":"d-Rickyy-b","description":"A lightweight Go library designed for scanning vast volumes of data using regex","archived":false,"fork":false,"pushed_at":"2025-05-02T22:22:16.000Z","size":108,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-06T21:15:15.868Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/d-Rickyy-b.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-18T13:56:42.000Z","updated_at":"2025-05-02T20:59:13.000Z","dependencies_parsed_at":"2024-02-18T14:48:41.893Z","dependency_job_id":"646a3861-9b67-484c-b5ce-b1aedf3007c2","html_url":"https://github.com/d-Rickyy-b/rexamine","commit_stats":null,"previous_names":["d-rickyy-b/rexamine"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/d-Rickyy-b/rexamine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-Rickyy-b%2Frexamine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-Rickyy-b%2Frexamine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-Rickyy-b%2Frexamine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-Rickyy-b%2Frexamine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/d-Rickyy-b","download_url":"https://codeload.github.com/d-Rickyy-b/rexamine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-Rickyy-b%2Frexamine/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262460217,"owners_count":23314684,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-06T21:15:16.735Z","updated_at":"2025-10-26T08:33:08.272Z","avatar_url":"https://github.com/d-Rickyy-b.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./docs/img/rexamine.png\" height=\"220\" /\u003e\n\u003c/p\u003e\n\n# rexamine\n\n[![build](https://github.com/d-Rickyy-b/rexamine/actions/workflows/test.yml/badge.svg)](https://github.com/d-Rickyy-b/rexamine/actions/workflows/test.yml)\n[![Go Reference](https://pkg.go.dev/badge/github.com/d-Rickyy-b/rexamine.svg)](https://pkg.go.dev/github.com/d-Rickyy-b/rexamine)\n\nRexamine, pronounced as [ɹɛɡˈzæmən], is a lightweight Go library designed for scanning vast volumes of data using regex. This library was created in order to avoid the excessive memory usage encountered when searching within large files, which traditionally required loading the entire file into memory.\n\nThis tool aims to fix that issue by either passing a reader (e.g. from an open file) or by streaming data to the `io.Writer` interface of `RegexWriter`.\nThat makes it to scan streams of data with regex.\n\nTo find out more about the project, check out the [blog post](https://blog.rico-j.de/rexamine-golang-stream-regex).\n\n## Usage\n\nInstall the library by using\n`go get github.com/d-Rickyy-b/rexamine`.\n\nThen you can use it in your project like this.\n\n```Go\npackage main\n\nimport (\n    \"fmt\"\n    \"os\"\n    \"regexp\"\n    \"strings\"\n\n    \"github.com/d-Rickyy-b/rexamine\"\n)\n\nfunc main() {\n    pattern := regexp.MustCompile(`[\\w\\-+\\.%]+@[\\w-]+\\.[a-zA-Z]{2,24}`)\n\n    targetFile, openErr := os.Open(\"sample.txt\")\n    if openErr != nil {\n        os.Exit(1)\n    }\n\n    newReader := rexamine.NewRegexReader(targetFile, pattern)\n\n    matches, err := newReader.FindAllMatches()\n    if err != nil {\n        return\n    }\n\n    fmt.Println(strings.Join(matches, \"\\n\"))\n}\n```\n\n## Pitfalls and Drawbacks\n\nThis library was created as a proof of concept.\nIt hasn't been tested excessively, so there might still be bugs.\nUse at your own risk.\n\nTo search through large files via regex, rexamine caches data in a buffer.\nIf the number of characters matched by a given regex exceeds the chosen buffer size, the full match cannot be extracted.\nThis can easily happen by using unlimited quantifiers like `*`, `+` or `{3,}`.\nTo prevent issues, make sure to limit the length of a match by either specifically defining the quantity `{5}` or at least by setting an upper bound.\n\nAn example use case could be the matching of long lines: `.*\\n`. If a line exceeds the buffer length, the line will not be matched and extracted.\n\nSo instead of matching \"any number of matches\" with `*`, use `{,10}` to match \"any number of matches up to 10\".\nAnd instead of matching \"one or more matches\" with `+`, use `{1,10}` to match \"one or more matches up to 10\".\n\n## Benchmark\n\nThe benchmarks show that rexamine is about **5-7%** slower than reading all the file's content into memory and searching it with regex all at once.\nOn the other hand, it uses only about **19 KB** of memory.\n\n### Preparation\n\nFor the benchmark, we need to generate some binary and text files (haystack) in order to compare the different approaches.\nTo do so, we used the following method:\n\nBinary file:\n`dd if=/dev/urandom bs=100M count=1 iflag=fullblock of=sample.bin`\n\nText file:\n`dd if=/dev/urandom iflag=fullblock | base64 -w 0 | head -c 100M \u003e sample.txt`\n\nTo insert some data (needle) we want to find with rexamine:\n\n```bash\necho -n \"MyData\" | dd bs=1 seek=1000 of=sample.txt\necho -n \"MyData\" | dd bs=1 seek=10000 of=sample.txt\necho -n \"MyData\" | dd bs=1 seek=100000 of=sample.txt\n```\n\nAfter that we only need to compile the four test binaries.\n\n```bash\ngo build .\\cmd\\iocopy\ngo build .\\cmd\\ioreadall\ngo build .\\cmd\\rexamine\ngo build .\\cmd\\rexaminewriter\n```\n\n### hyperfine\n\nWith the generated files in place, we can now run rexamine on these files and compare different approaches.\nTo do this efficiently, we can use [hyperfine](https://github.com/sharkdp/hyperfine).\n\n```bash\nrexamine\u003e hyperfine -w 2 -r 6 'iocopy.exe -file 500mb.txt -regex \"...\"' 'ioreadall.exe -file 500mb.txt -regex \"...\"' 'rexamine.exe -file 500mb.txt -regex \"...\"' 'rexaminewriter.exe -file 500mb.txt -regex \"...\"'\nBenchmark 1: iocopy.exe -file 500mb.txt -regex [\\w\\-+\\.%]+@[\\w-]+\\.[a-zA-Z]{2,24}\n  Time (mean ± σ):      8.045 s ±  0.279 s    [User: 2.852 s, System: 0.049 s]\n  Range (min … max):    7.887 s …  8.612 s    6 runs\n\nBenchmark 2: ioreadall.exe -file 500mb.txt -regex [\\w\\-+\\.%]+@[\\w-]+\\.[a-zA-Z]{2,24}\n  Time (mean ± σ):      8.085 s ±  0.042 s    [User: 3.263 s, System: 0.042 s]\n  Range (min … max):    8.023 s …  8.135 s    6 runs\n\nBenchmark 3: rexamine.exe -file 500mb.txt -regex [\\w\\-+\\.%]+@[\\w-]+\\.[a-zA-Z]{2,24}\n  Time (mean ± σ):      8.630 s ±  0.083 s    [User: 3.729 s, System: 0.104 s]\n  Range (min … max):    8.572 s …  8.753 s    6 runs\n\nBenchmark 4: rexaminewriter.exe -file 500mb.txt -regex [\\w\\-+\\.%]+@[\\w-]+\\.[a-zA-Z]{2,24}\n  Time (mean ± σ):      8.601 s ±  0.041 s    [User: 1.391 s, System: 0.062 s]\n  Range (min … max):    8.551 s …  8.669 s    6 runs\n\nSummary\n  iocopy.exe -file 500mb.txt -regex [\\w\\-+\\.%]+@[\\w-]+\\.[a-zA-Z]{2,24} ran\n    1.00 ± 0.04 times faster than ioreadall.exe -file 500mb.txt -regex [\\w\\-+\\.%]+@[\\w-]+\\.[a-zA-Z]{2,24}\n    1.07 ± 0.04 times faster than rexamine.exe -file 500mb.txt -regex [\\w\\-+\\.%]+@[\\w-]+\\.[a-zA-Z]{2,24}\n    1.07 ± 0.04 times faster than rexaminewriter.exe -file 500mb.txt -regex [\\w\\-+\\.%]+@[\\w-]+\\.[a-zA-Z]{2,24}\n```\n\nWe can see that rexamine is about 7% slower than reading the full file into memory and processing it via regex.\n\n### Memory footprint\n\nSince rexamine was specifically developed to decrease the memory footprint, the used memory is much more important than execution speed.\nWe can use Go's benchmarking tooling to get data on memory usage.\n\n```bash\nrexamine\u003e go test -bench=.\\ -benchmem -run=^$ -bench ^Benchmark.+$ -count 5\ncpu: AMD Ryzen 9 7900 12-Core Processor\nBenchmarkIOCopy-24                     1        1618671100 ns/op        268449152 B/op        76 allocs/op\nBenchmarkIOReadAll-24                  1        1570844100 ns/op        615242440 B/op       106 allocs/op\nBenchmarkRexamine-24                   1        1751799700 ns/op           19464 B/op         58 allocs/op\nBenchmarkRexamineWriter-24             1        1735913500 ns/op           53440 B/op         68 allocs/op\n```\n\n`io.ReadAll` needs by far the most memory allocations. For a 100 MB file, it allocates (and frees) more than 600 MB.\n`io.Copy` requires less than that, but still around 270 MB.\nrexamine completely crushes it with only 19 KB of allocated memory.\n\n```text\ncpu: AMD Ryzen 9 7900 12-Core Processor\n                  │   sec/op    │\nIOCopy-24            1.572 ± 4%\nIOReadAll-24         1.594 ± 1%\nRexamine-24          1.761 ± 1%\nRexamineWriter-24    1.760 ± 3%\ngeomean              1.669\n\n                  │     B/op      │\nIOCopy-24           256.0Mi ±  0%\nIOReadAll-24        586.7Mi ±  0%\nRexamine-24         18.95Ki ±  0%\nRexamineWriter-24   51.37Ki ± 11%\ngeomean             3.436Mi\n\n                  │  allocs/op  │\nIOCopy-24           69.00 ± 10%\nIOReadAll-24        94.50 ±  4%\nRexamine-24         55.00 ±  4%\nRexamineWriter-24   62.00 ± 52%\ngeomean             68.67\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-rickyy-b%2Frexamine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fd-rickyy-b%2Frexamine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-rickyy-b%2Frexamine/lists"}