{"id":43105245,"url":"https://github.com/pixelistik/request_log_analyzer","last_synced_at":"2026-01-31T17:52:53.879Z","repository":{"id":11282208,"uuid":"68523891","full_name":"pixelistik/request_log_analyzer","owner":"pixelistik","description":"Extract some performance metrics from the request.log of a CQ/AEM instance. It can be used as an alternative to or in addition to `rlog.jar`.","archived":false,"fork":false,"pushed_at":"2023-10-17T11:45:07.000Z","size":372,"stargazers_count":30,"open_issues_count":3,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2023-10-17T14:10:06.667Z","etag":null,"topics":["adobe-cq","aem","graphite","influxdb","logfile","performance","prometheus","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pixelistik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-09-18T13:40:20.000Z","updated_at":"2023-03-14T20:06:10.000Z","dependencies_parsed_at":"2023-01-12T11:01:09.369Z","dependency_job_id":"d85f289f-8f30-4abe-8a8f-23ffe3926b9c","html_url":"https://github.com/pixelistik/request_log_analyzer","commit_stats":null,"previous_names":[],"tags_count":32,"template":null,"template_full_name":null,"purl":"pkg:github/pixelistik/request_log_analyzer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelistik%2Frequest_log_analyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelistik%2Frequest_log_analyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelistik%2Frequest_log_analyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelistik%2Frequest_log_analyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pixelistik","download_url":"https://codeload.github.com/pixelistik/request_log_analyzer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelistik%2Frequest_log_analyzer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28948823,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T14:26:55.697Z","status":"ssl_error","status_checked_at":"2026-01-31T14:26:52.545Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adobe-cq","aem","graphite","influxdb","logfile","performance","prometheus","rust"],"created_at":"2026-01-31T17:52:53.797Z","updated_at":"2026-01-31T17:52:53.871Z","avatar_url":"https://github.com/pixelistik.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/pixelistik/request_log_analyzer.svg?branch=master)](https://travis-ci.org/pixelistik/request_log_analyzer)\n[![codecov](https://codecov.io/gh/pixelistik/request_log_analyzer/branch/master/graph/badge.svg)](https://codecov.io/gh/pixelistik/request_log_analyzer)\n\n## Installation\nDownload and unzip one of the 64-bit Linux / macOS / Windows binaries from\nthe [releases page](https://github.com/pixelistik/request_log_analyzer/releases/latest)\n(built on Travis/Appveyor servers). The program is a single binary that you run in\nthe terminal.\n\nAlternatively you can build from source, if you have a Rust toolchain set up:\nClone the repository and run `cargo build --release`.\n\n## Usage\n\n    Request.log Analyzer\n\n    USAGE:\n        request_log_analyzer [OPTIONS] [--] [FILES]...\n\n    FLAGS:\n        -h, --help       Prints help information\n        -q, --quiet      Don't output results to stdout\n        -V, --version    Prints version information\n\n    OPTIONS:\n        --exclude \u003cTERM\u003e...                          Exclude lines that contain one of these terms\n        --graphite-port \u003cGRAPHITE_PORT\u003e               [default: 2003]\n        --graphite-prefix \u003cGRAPHITE_PREFIX\u003e\n        Prefix for Graphite key, e.g. 'servers.prod.publisher1'\n\n        --graphite-server \u003cGRAPHITE_SERVER\u003e\n        Send values to this Graphite server instead of stdout\n\n        --include \u003cTERM\u003e...\n        Only include lines that contain one of these terms\n\n        --influxdb-write-url \u003cINFLUXDB_WRITE_URL\u003e\n        base URL of InfluxDB to send metrics to, e.g. 'http://localhost:8086/write?db=mydb'\n\n        --influxdb-tags \u003cINFLUXDB_TAGS\u003e\n        tags for the submitted measurement, e.g. 'host=prod3' or 'host=prod3,type=worker'\n\n        --prometheus-listen \u003cBINDING_ADDRESS\u003e\n        Address and port to bind Prometheus HTTP server to, e.g. 'localhost:9898'\n\n        -t \u003cMINUTES\u003e                                     Limit to the last n minutes\n\n    ARGS:\n        \u003cFILES\u003e...    Log files to analyze, defaults to stdin\n\n## Example output\n    $ request_log_analyzer crx-quickstart/logs/request.log\n    count:\t54510\n    time.avg:\t127\n    time.min:\t1\n    time.median:\t6\n    time.90percent:\t27\n    time.99percent:\t3614\n    time.max:\t15747\n    error.client_error_4xx_rate:\t0.023\n    error.server_error_5xx_rate:\t0.0002\n\n## Getting started\n\n### Analyze an entire log file\n\nRun the tool on a request.log file\n\n\t$ request_log_analyzer crx-quickstart/logs/request.log\n\tcount:  1221\n\ttime.avg:       840\n\ttime.min:       0\n\ttime.median:    841\n\ttime.90percent: 1537\n\ttime.99percent: 1650\n\ttime.max:       1709\n    error.client_error_4xx_rate:\t0.02\n    error.server_error_5xx_rate:\t0.01\n\nLet's run through the results:\n\nThere are 1221 request/response pairs in the log file.  \nThe average response time was 840ms.  \nThe fastest response was 0ms.  \nThe median response time was 841ms.  \nThe 90 percentile response time was 1537ms. That means that 90% of all requests were finished after 1537ms.  \nThe 99 percentile response time was 1650ms. That means that 99% of all requests were finished after 1650ms.  \nThe slowest response was 1709ms.  \n2% of all requests have failed with a 4xx HTTP error code (client error).  \n1% of all requests have failed with a 5xx HTTP error code (server error).  \n\n### Include only certain requests\n\nLet's say we only care about the rendering of HTML pages, so we want to ignore anything else.\n\n\t$ request_log_analyzer --include \"text/html\" crx-quickstart/logs/request.log\n\nNow the result only refers to data where either request or response line contains the specified `--include` term, in this case the MIME type \"text/html\"\n\n### Restrict to latest period\n\nNow we specifically want to look at the latest hour, because we suspect a recent problem:\n\n\t$ request_log_analyzer -t 60 crx-quickstart/logs/request.log\n\nWith the `-t` param, only the most recent _n_ minutes will be taken into account.\n\n### Combine everything\n\n\t$ request_log_analyzer --include \"text/html\" \\\n\t\t--include \"content/dam/\" \\\n\t\t--exclude \"POST\" \\\n\t\t-t 180 \\\n\t\tcrx-quickstart/logs/request.log\n\nWe look at request/response lines that contain \"text/html\" (the MIME type) or a path from the DAM, but we exclude POST requests. Also, we are only interested in the latest 3 hours.\n\n### Piped log data\n\nIf the built-in filtering options are not enough, we can use other tools for filtering the log lines and the pipe them into the tool for analysis:\n\n\t$ grep \"09:..:..\" crx-quickstart/logs/request.log | request_log_analyzer\n\nHere we only look at the request/response lines from a specific hour.\n\n## Continuous monitoring\n\n### Graphite\n\nIn the examples above, we used `request_log_analyzer` for individual insights.\n\nBut it can also be used to continuously feed data into a Graphite, InfluxDB or Prometheus data store, in order to be used for monitoring.\n\n\t$ request_log_analyzer -t 5 \\\n\t\t--graphite-server localhost \\\n\t\t--graphite-prefix my-app.production.5min \\\n\t\tcrx-quickstart/logs/request.log\n\nThis will analyze the latest 5 minutes of the log file, then push it to a Graphite server running on localhost, storing the results under the keys\n\n\tmy-app.production.5min.requests.count\n\tmy-app.production.5min.requests.time.max\n\tmy-app.production.5min.requests.time.min\n\netc.\n\nIf you set this command up as a cronjob to run every 1 minute, you can constantly monitor the data for the previous 5 minute window.\n\n### InfluxDB\n\n    $ request_log_analyzer -t 5 \\\n        --quiet \\\n        --influxdb-write-url \"http://localhost:8086/write?db=metrics\" \\\n        --influxdb-tags type=publisher,time=5min \\\n        crx-quickstart/logs/request.log\n\nThis will analyze the latest 5 minutes of the log file, then push it to an InfluxDB server running on localhost, storing the results in the database `metrics` as the measurement `request_log` with the fields `time_max`, `time_min` etc. The result\nwill not be displayed in the terminal.\n\n`--influxdb-tags` are an optional way to identify and categorize your measurements, in this example the tags `type=publisher` and `time=5min` are set.\n\n### Prometheus\n\n\t$ request_log_analyzer -t 5 --prometheus-listen localhost:9898 crx-quickstart/logs/request.log\n\nThis will start a Prometheus endpoint (a small HTTP server) on port 9898. Whenever the Prometheus server queries this endpoint, the latest 5 minutes of the `request.log` will be analyzed and the results will be provided under the keys\n\n\trequest_count\n\trequest_time_max\n\trequest_time_min\n\netc. If you set up your Prometheus server to pull data from this endpoint, you can constantly monitor the data for the previous 5 minute window.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpixelistik%2Frequest_log_analyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpixelistik%2Frequest_log_analyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpixelistik%2Frequest_log_analyzer/lists"}