{"id":51558500,"url":"https://github.com/git001/mergelog-rs","last_synced_at":"2026-07-10T08:01:45.317Z","repository":{"id":344515819,"uuid":"1182043152","full_name":"git001/mergelog-rs","owner":"git001","description":"mergelog is a small but handy tool: it takes multiple Apache log files from servers behind a round-robin DNS and merges them into a single chronologically sorted stream. ","archived":false,"fork":false,"pushed_at":"2026-03-24T00:57:06.000Z","size":56,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-25T00:20:39.282Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/git001.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-15T00:46:42.000Z","updated_at":"2026-03-24T00:56:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/git001/mergelog-rs","commit_stats":null,"previous_names":["git001/mergelog-rs"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/git001/mergelog-rs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fmergelog-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fmergelog-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fmergelog-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fmergelog-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/git001","download_url":"https://codeload.github.com/git001/mergelog-rs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fmergelog-rs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35325188,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-10T08:01:44.638Z","updated_at":"2026-07-10T08:01:45.310Z","avatar_url":"https://github.com/git001.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mergelog-rs\n\n[![License: GPL-3.0-or-later](https://img.shields.io/badge/license-GPL--3.0--or--later-blue.svg)](LICENSE)\n\nMerge and sort HTTP log files chronologically — Rust rewrite of\n[mergelog 4.5](https://mergelog.sourceforge.net/) by Bertrand Demiddelaer.\n\nReads multiple HTTP access log files (**NCSA Combined Log Format** and **Caddy\nJSON**) and writes a single chronologically sorted CLF stream to stdout.\nDesigned for consolidating logs from multiple servers behind a round-robin DNS.\n\n\u003e Full background and benchmarks: **[Alek's Blog — mergelog-rs](https://blog.none.at/blog/2026/2026-03-15-mergelog-rs/)**\n\n---\n\n## Features\n\n- **Caddy JSON support** — auto-detects Caddy JSON access logs per file and converts them to CLF on the fly\n- **Mixed input** — CLF and Caddy JSON files can be merged in a single run; format is detected per file\n- **Auto-detect compression** via magic bytes — gzip, bzip2, xz, zstd, and plain text, no file extension needed\n- **Timezone-aware** — timestamps are converted to UTC before comparison; logs from servers in different timezones sort correctly\n- **O(N log K) k-way heap merge** — scales with the number of lines, not the time span of the logs\n- **stdin support** — pass `-` as a filename to read from stdin\n- **Statistics** — `--stats` reports per-file line counts, time ranges and malformed lines\n- **2.26× faster** than the original C binary on the same workload\n\n---\n\n## Installation\n\n### Pre-built binaries\n\nDownload the latest release for your platform from the\n[Releases](https://github.com/git001/mergelog-rs/releases) page:\n\n| Platform | Archive |\n|----------|---------|\n| Linux x86_64 (glibc) | `mergelog-rs-v1.0.0-x86_64-unknown-linux-gnu.tar.gz` |\n| Linux x86_64 (static musl) | `mergelog-rs-v1.0.0-x86_64-unknown-linux-musl.tar.gz` |\n| Linux arm64 (static musl) | `mergelog-rs-v1.0.0-aarch64-unknown-linux-musl.tar.gz` |\n| macOS Apple Silicon | `mergelog-rs-v1.0.0-aarch64-apple-darwin.tar.gz` |\n| macOS Intel | `mergelog-rs-v1.0.0-x86_64-apple-darwin.tar.gz` |\n\n### Container image\n\n```sh\npodman pull ghcr.io/git001/mergelog-rs:latest\n# or\ndocker pull ghcr.io/git001/mergelog-rs:latest\n```\n\n### Build from source\n\nRequires Rust 1.94+.\n\n```sh\ngit clone https://github.com/git001/mergelog-rs\ncd mergelog-rs\ncargo build --release --locked\n# binary at target/release/mergelog-rs\n```\n\n### Install binary + man page\n\n```sh\nmake install                                      # to /usr/local\nmake install PREFIX=/usr                          # custom prefix\nmake install DESTDIR=/tmp/pkg PREFIX=/usr         # for packaging\n\n# or with just:\njust install\nPREFIX=/usr just install\n```\n\nIf pandoc is not in `PATH`, specify it explicitly:\n\n```sh\nmake install PANDOC=/path/to/pandoc\nPANDOC=/path/to/pandoc just install\n```\n\n---\n\n## Usage\n\n```\nmergelog-rs [OPTIONS] \u003cFILE\u003e...\n```\n\n### Options\n\n| Option | Description |\n|--------|-------------|\n| `\u003cFILE\u003e...` | Log files to merge. Use `-` for stdin. |\n| `--progress` | Print running line count to stderr every 10,000 lines. |\n| `--stats` | Print per-file statistics to stderr after the merge. |\n| `-o, --output \u003cFILE\u003e` | Write to a file instead of stdout. |\n| `--format \u003cFORMAT\u003e` | Force log format: `clf` or `caddy`. Default: auto-detect per file. |\n| `-h, --help` | Print help. |\n| `-V, --version` | Print version. |\n\n### Examples\n\n```sh\n# Merge CLF log files\nmergelog-rs access1.log access2.log access3.log \u003e merged.log\n\n# Merge Caddy JSON logs (auto-detected, output is CLF)\nmergelog-rs caddy1.log caddy2.log \u003e merged.log\n\n# Mix CLF and Caddy JSON — format detected per file\nmergelog-rs nginx.log caddy.log \u003e merged.log\n\n# Force Caddy format for all files (e.g. multiple pods)\nmergelog-rs --format caddy pod1.log pod2.log \u003e merged.log\n\n# Merge compressed files — compression auto-detected via magic bytes\nmergelog-rs access.log.gz archive.log.bz2 old.log.xz backup.log.zst \u003e merged.log\n\n# Read from stdin (e.g. over SSH), merge with a local file\nssh web01 \"cat /var/log/caddy/access.log\" | mergelog-rs - /var/log/caddy/access.log\n\n# Write to a file and show statistics\nmergelog-rs --stats -o merged.log access1.log access2.log\n\n# Run in a container\npodman run --rm -v /var/log/caddy:/logs:ro ghcr.io/git001/mergelog-rs \\\n  /logs/access1.log /logs/access2.log \u003e merged.log\n```\n\n### Statistics output\n\n```\n──── merge statistics ──────────────────────────────────────\n  Files    : 3\n  Lines    : 12,450,000  (0 malformed)\n  Range    : 2024-01-01 00:00:00 UTC → 2024-12-31 23:59:59 UTC\n────────────────────────────────────────────────────────────\n  access-eu.log    4,150,000 lines  2024-01-01 00:00:00 UTC → 2024-12-31 22:00:00 UTC\n  access-us.log    4,150,000 lines  2024-01-01 05:00:00 UTC → 2024-12-31 23:59:59 UTC\n  access-ap.log    4,150,000 lines  2024-01-01 09:00:00 UTC → 2024-12-31 21:00:00 UTC\n────────────────────────────────────────────────────────────\n```\n\n---\n\n## Development Tools\n\n### `src/bin/profile_timing.rs`\n\nA standalone timing binary used during development to identify where CPU time\nis spent. Since `perf` was locked down on the target system\n(`perf_event_paranoid = 4`), this binary instruments each phase of the merge\nmanually using `std::time::Instant`:\n\n```sh\ncargo build --release --bin profile_timing\ntarget/release/profile_timing access1.log access2.log ... \u003e /dev/null\n```\n\nExample output:\n\n```\n──── profile_timing (41.65M lines) ──────────────────\n  I/O   (read_line)       3.57 s   47.1%\n  Parse (hand-rolled)     1.47 s   19.5%\n  Heap  (push/pop)        1.87 s   24.7%\n  Write (write_all)       0.65 s    8.7%\n  ─────────────────────────────────────────────────\n  Σ measured              7.56 s\n```\n\nThis tool was how the bottlenecks (`jiff::strptime` at 45.7%, then I/O at\n47.1%) were found and quantified — driving the hand-rolled parser, buffer\nreuse, and 4 MiB buffer optimizations. See the\n[blog post](https://blog.none.at/blog/2026/2026-03-15-mergelog-rs/) for the\nfull story.\n\n---\n\n## Performance\n\nMeasured on 7 × 1 GiB log files (41.65 million lines, 7 GiB total):\n\n| Version | Wall time | Peak RSS |\n|---------|-----------|----------|\n| mergelog-4.5 (C, 2001) | 9.98 s | 1.6 MiB |\n| **mergelog-rs 1.0.0** | **4.43 s** | **30.7 MiB** |\n\n**2.26× faster.** The higher RSS comes from 4 MiB read buffers per input file\n(7 × 4 MiB = 28 MiB) — a deliberate speed/memory trade-off.\n\nKey optimizations over a naive Rust port:\n\n1. K-way heap merge instead of second-by-second iteration\n2. Hand-rolled CLF timestamp parser (3.7× faster than `jiff::strptime`)\n3. `String` buffer reuse across heap iterations\n4. mimalloc as the global allocator\n5. 4 MiB `BufReader` + SIMD `memchr` newline search\n\n---\n\n## Log Formats\n\n### CLF — NCSA Combined Log Format\n\n`%h %l %u %t \"%r\" %\u003es %b \"%{Referer}i\" \"%{User-agent}i\"` — produced by Apache, nginx, and others:\n\n```\n93.184.216.34 - alice [15/Mar/2026:12:00:00 +0100] \"GET / HTTP/1.1\" 200 4096 \"https://example.com/\" \"Mozilla/5.0\"\n```\n\nSee the [Apache mod_log_config documentation](https://httpd.apache.org/docs/2.4/mod/mod_log_config.html)\nfor the full format reference.\n\n### Caddy JSON\n\nOne JSON object per line, as written by Caddy's `json` log encoder:\n\n```json\n{\"ts\":\"2026-03-22T14:50:51.525Z\",\"request\":{\"client_ip\":\"1.2.3.4\",\"method\":\"GET\",\"uri\":\"/\",\"proto\":\"HTTP/1.1\",\"headers\":{\"User-Agent\":[\"Mozilla/5.0\"]}},\"status\":200,\"size\":4096,\"user_id\":\"\"}\n```\n\nThe `ts` field may be an ISO 8601 string or a Unix float. Non-request lines\n(error/info/debug entries) are skipped silently. Output is always CLF.\n\n---\n\n## Acknowledgements\n\nFull respect and honour to **Bertrand Demiddelaer**, who created mergelog in\n2000. Writing a tool this fast, correct, and compact in C — with no external\ndependencies beyond zlib — is genuinely impressive. mergelog-rs stands on his\nshoulders.\n\n---\n\n## License\n\nGPL-3.0-or-later.\nBased on mergelog 4.5 © 2000–2001 Bertrand Demiddelaer (GPL-2.0-or-later).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgit001%2Fmergelog-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgit001%2Fmergelog-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgit001%2Fmergelog-rs/lists"}