{"id":13581959,"url":"https://github.com/zix99/rare","last_synced_at":"2026-01-28T05:36:16.046Z","repository":{"id":39096604,"uuid":"210987929","full_name":"zix99/rare","owner":"zix99","description":"Create console histograms, bar graphs, tables, heatmaps and more in realtime using regex and expressions.","archived":false,"fork":false,"pushed_at":"2025-05-15T02:39:12.000Z","size":4282,"stargazers_count":299,"open_issues_count":4,"forks_count":17,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-15T03:28:18.694Z","etag":null,"topics":["analyzer","apache","awk","cli","console","grep","heatmaps","histogram","histograms","log-parser","nginx","nginx-logs","regex","regex-extractor","sed","visualisation"],"latest_commit_sha":null,"homepage":"https://rare.zdyn.net/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zix99.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-09-26T03:04:44.000Z","updated_at":"2025-05-11T02:18:08.000Z","dependencies_parsed_at":"2024-06-07T01:55:11.660Z","dependency_job_id":"eaf3fc3c-4447-45ff-a569-b47bbf87e56d","html_url":"https://github.com/zix99/rare","commit_stats":null,"previous_names":[],"tags_count":51,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zix99%2Frare","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zix99%2Frare/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zix99%2Frare/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zix99%2Frare/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zix99","download_url":"https://codeload.github.com/zix99/rare/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254544146,"owners_count":22088807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analyzer","apache","awk","cli","console","grep","heatmaps","histogram","histograms","log-parser","nginx","nginx-logs","regex","regex-extractor","sed","visualisation"],"created_at":"2024-08-01T15:02:21.138Z","updated_at":"2026-01-28T05:36:16.040Z","avatar_url":"https://github.com/zix99.png","language":"Go","funding_links":[],"categories":["Files and Directories","Go","cli","\u003ca name=\"text-processing\"\u003e\u003c/a\u003eText processing","Text Search"],"sub_categories":["Search"],"readme":"# rare\n\n[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/zix99/rare/main.yml?branch=master)](https://github.com/zix99/rare/actions)\n[![GitHub release (latest by date)](https://img.shields.io/github/v/release/zix99/rare)](https://github.com/zix99/rare/releases)\n[![codecov](https://codecov.io/gh/zix99/rare/branch/master/graph/badge.svg)](https://codecov.io/gh/zix99/rare)\n![GitHub all releases](https://img.shields.io/github/downloads/zix99/rare/total)\n![GitHub](https://img.shields.io/github/license/zix99/rare)\n\nA fast text scanner/regex extractor and realtime summarizer. Quickly search, reformat and visualize text files\nsuch as logs, csv, json, etc.\n\nSupports various CLI-based graphing and metric formats (filter (grep-like), histogram, table, bargraph, heatmap, reduce).\n\n`rare` is a play on \"more\" and \"less\", but can also stand for \"realtime aggregated regular expressions\".\n\nSee [rare.zdyn.net](https://rare.zdyn.net) or the [docs/ folder](docs/) for the full documentation.\n\n![rare gif](docs/images/rare.gif)\n\n## Features\n\n * Multiple summary formats including: filter (like grep), histogram, bar graphs, tables, heatmaps, reduce, and numerical analysis\n * Parse using regex (`-m`) or dissect tokenizer (`-d`)\n * Manipulate results with handlebars-like expressions (`-e`)\n * File glob expansions (eg `/var/log/*` or `/var/log/*/*.log`) and recursion `-R` with path filters (include/exclude patterns)\n * Optional gzip decompression (with `-z`)\n * Following `-f` or re-open following `-F` (use `--poll` to poll, and `--tail` to tail)\n * Ignoring lines that match an expression (with `-i`)\n * Aggregating and realtime summary (Don't have to wait for all data to be scanned)\n * Multi-threaded reading, parsing, and aggregation (It's fast)\n * Color-coded outputs (optionally)\n * Pipe support (stdin for reading, stdout will disable realtime, and `--csv` formatting) eg. `tail -f | rare ... \u003e out`\n\nTake a look at [examples](docs/usage/examples.md) to see more of what *rare* does.\n\n### Output Formats\n\nOutput formats include:\n\n* `filter` is grep-like, in that each line will be processed and the extracted key will be output directly to stdout\n* `histogram` will count instances of the extracted key\n* `table` will count the key in 2 dimensions\n* `heatmap` will display a color-coded version of the strength of a cell in a dense format\n* `sparkline` draws rows of sparklines for the data\n* `bargraph` will create either a stacked or non-stacked bargraph based on 2 dimensions\n* `analyze` will use the key as a numeric value and compute mean/median/mode/stddev/percentiles\n* `reduce` allows evaluating data using expressions, and grouping/sorting the output\n\nMore details on various output formats and aggregators (including examples) can be found in [aggregators](docs/usage/aggregators.md)\n\n## Installation\n\n### Bash Script\n\nThis script downloads the latest version from github, and installs it to `~/.local/bin` (`/usr/bin` if *root*).\n\n**Warning:** You should always verify the [script](docs/install.sh) before running it locally.\n\n```sh\ncurl -sfL https://rare.zdyn.net/install.sh | bash\n```\n\n### Manual (Prebuilt Binary)\n\nDownload appropriate binary or package from [Releases](https://github.com/zix99/rare/releases)\n\n### Homebrew\n\n```sh\nbrew tap zix99/rare\nbrew install rare\n```\n\n### Community Contributed\n\nThe below install methods have been contributed by the community, and aren't maintained directly.\n\n#### MacPorts\n\n```sh\nsudo port selfupdate\nsudo port install rare\n```\n\n### From code\n\nClone the repo, and:\n\nRequires GO 1.23 or higher\n\n```sh\ngo mod download\n\n# Build binary\ngo build .\n\n# OR, with experimental features\ngo build -tags experimental .\n```\n\nAvailable tags:\n\n* `experimental` Enable experimental features (eg. fuzzy search)\n* `pcre2` Enables PCRE 2 (v10) where able. Currently linux only\n* `rare_no_pprof` Disables profiling capabilities, which reduces binary size\n* `urfave_cli_no_docs` Disables man and markdown documentation generation, which reduces binary size\n\n**A Note on PCRE (Perl Compatible Regular Expressions)**\n\nBesides your standard OS versions, there is an additional `pcre` build which is ~4x faster than go's `re2` implementation in moderately complex cases.  In order to use this, you must make sure that **libpcre2** is installed (eg `apt install libpcre2-8-0` or `dnf install pcre2`).  Right now, it is only bundled with the linux distribution.\n\nPCRE2 also comes with pitfalls, two of the most important are:\n1. That *rare* is now dynamically linked, meaning that you need to have libc and libpcre installed\n2. That pcre is an exponential-time algorithm (re2 is linear).  While it can be significantly faster than go's `re2`, it can also be catastrophically slower in some situations. There is a good post [here](https://swtch.com/~rsc/regexp/regexp1.html) that talks about regexp timings.\n\nI will leave it up to the user as to which they find suitable to use for their situation.  Generally, if you know what *rare* is getting as an input, the pcre version is perfectly safe and can be much faster.\n\n## Documentation\n\nAll documentation may be found here, in the [docs/](docs/) folder, by running `rare docs` (embedded docs/ folder), or on the website [rare.zdyn.net](https://rare.zdyn.net)\n\nYou can also see a dump of the CLI options at [cli-help.md](docs/cli-help.md)\n\n## Example\n\n### Create histogram from sample data\n\n```sh\n$ cat input.txt\n1\n2\n1\n3\n1\n0\n\n$ rare histo input.txt\n1                   3         \n0                   1         \n2                   1         \n3                   1         \n\nMatched: 6 / 6 (Groups: 4)\n```\n\n### Extract status and size from nginx logs\n```sh\n$ rare filter -n 4 -m \"(\\d{3}) (\\d+)\" -e \"{1} {2}\" access.log\n404 169\n404 169\n404 571\n404 571\nMatched: 4 / 4\n```\n\n### Extract status codes from nginx logs\n\n```sh\n$ rare histo \\\n    -m '\"(\\w{3,4}) ([A-Za-z0-9/.]+).*\" (\\d{3})' \\ # The regex that extracts match-groups\n    -e '{3} {1}' \\ # The expression will be the key, referencing the match-groups\n    access.log     # One or more files (or -R for recursion)\n\n200 GET                          160663\n404 GET                          857\n304 GET                          53\n200 HEAD                         18\n403 GET                          14\n```\n\n### More Examples\n\nFor more examples, check out the [docs](docs/usage/examples.md) or [the website](https://rare.zdyn.net/usage/examples/)\n\n\n## Performance Benchmarking\n\nI know there are different solutions, and rare accomplishes summarization in a way\nthat grep, awk, etc can't, however I think it's worth analyzing the performance of this\ntool vs standard tools to show that it's at least as good.\n\nSee [benchmarks](docs/benchmarks.md) for comparisons between common tools like `grep | wc`,\nsilversearcher-ag, etc.\n\n\n## Development\n\nNew additions to `rare` should pass the following checks\n\n- Documentation for any new functionality or expression changes\n- Before and after CPU and memory benchmarking for core additions (Expressions, aggregation, benchmarking, and rendering)\n- Limit memory allocations (preferably 0!) in the high-throughput functions\n- Tests, and if it makes sense, benchmarks of a given function\n\n### Running/Testing\n\n```bash\ngo run .\ngo test ./...\n```\n\n### Profiling\n\nNew high-throughput changes should be performance benchmarked.\n\nTo Benchmark:\n\n```bash\ngo run . --profile out \u003cyour test code\u003e\ngo tool pprof -http=:8080 out.cpu.prof # CPU\ngo tool pprof -http=:8080 out_num.prof # Memory\n```\n\n### Documentation\n\nNew functionality should be well-documented in `docs/` folder.  The docs are\nboth published to the website via `mkdocs` and embedded in the application.\n\nTesting and publishing docs:\n\n```bash\n# Install mkdocs version\npip3 install -r requirements.txt\n\n# Test locally\nmkdocs serve\n\n# Test cli docs\ngo run . docs\n\n# If CLI arguments change, publish updated docs\ngo run . _gendoc \u003e docs/cli-help.md\n```\n\n## License\n\n    Copyright (C) 2019  Christopher LaPointe\n\n    This program is free software: you can redistribute it and/or modify\n    it under the terms of the GNU General Public License as published by\n    the Free Software Foundation, either version 3 of the License, or\n    (at your option) any later version.\n\n    This program is distributed in the hope that it will be useful,\n    but WITHOUT ANY WARRANTY; without even the implied warranty of\n    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n    GNU General Public License for more details.\n\n    You should have received a copy of the GNU General Public License\n    along with this program.  If not, see \u003chttps://www.gnu.org/licenses/\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzix99%2Frare","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzix99%2Frare","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzix99%2Frare/lists"}