{"id":25084152,"url":"https://github.com/ncounter/logorator","last_synced_at":"2026-05-04T12:38:52.227Z","repository":{"id":42108152,"uuid":"111088139","full_name":"ncounter/logorator","owner":"ncounter","description":"A log analyzer to extract analytic reports of usage","archived":false,"fork":false,"pushed_at":"2023-01-25T08:55:30.000Z","size":2430,"stargazers_count":0,"open_issues_count":21,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-31T11:25:08.752Z","etag":null,"topics":["frequent-patterns","json","log-analyzer","python","regex","stats","tomcat-log","visit-count","web-dashboard"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ncounter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-17T10:01:45.000Z","updated_at":"2018-07-17T14:35:45.000Z","dependencies_parsed_at":"2023-02-14T06:16:00.199Z","dependency_job_id":null,"html_url":"https://github.com/ncounter/logorator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ncounter/logorator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncounter%2Flogorator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncounter%2Flogorator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncounter%2Flogorator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncounter%2Flogorator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ncounter","download_url":"https://codeload.github.com/ncounter/logorator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncounter%2Flogorator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32608169,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-04T10:08:07.713Z","status":"ssl_error","status_checked_at":"2026-05-04T10:08:02.005Z","response_time":58,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["frequent-patterns","json","log-analyzer","python","regex","stats","tomcat-log","visit-count","web-dashboard"],"created_at":"2025-02-07T06:46:23.992Z","updated_at":"2026-05-04T12:38:52.212Z","avatar_url":"https://github.com/ncounter.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# logorator\nA log grinder to extract statistics and more\n\n# What it does\n\n1. A backend python log grinder to analyze, extract and compute information, generating JSON files with different content like:\n  - a **distinct list  of URLs** with a **count visit** number taken from the log files. See [the algorithm](https://github.com/ncounter/logorator/blob/master/backend/count_urls.py) here\n  - a **distinct list of URLs** with the **count visit** number taking the list from a given *URLs router map* file and counting the occurrencies of those URLs within the log files. See [the algorithm](https://github.com/ncounter/logorator/blob/master/backend/known_urls.py) here\n  - a *from-to* pattern pair of urls starting from the log files, extracting relevant URLs grouped by the same user/ip, and counting the occurrencies of each *from-to* pattern. See [the algorithm](https://github.com/ncounter/logorator/blob/master/backend/elaborate_patterns.py) here\n\n2. A web dashboard to view results from the backend analysis with some  filtering options\n\n\n\n# How to use\n\n## Get the source project\n```\ngit clone git@github.com:ncounter/logorator.git\ncd logorator\n```\n\n\n## Configuration\nYou may want to configure parameters in the `logorator.conf` file first.\n\n```\n...\ntomcat_log_path=\u003cTOMCAT_LOGS_PATH\u003e\nstats_file=./dashboard/public/stats.json\nknown_urls_output=./dashboard/public/known-urls.json\npattern_file=./dashboard/public/patterns.json\n...\n```\n**Important**: changing location of files within this application, `stats_file` for instance, will make the **dashboard** web page unable to load data. Please make sure you will update the `json source file name` accordingly for each page.\n\n\n## Backend: python analyzer\n\n### Important\nThe `count_urls.py` program makes an assumption about your [log format](https://github.com/ncounter/logorator/blob/master/backend/utils.py#L14) expecting log lines with **the request** information containing the **url** wrapped between `\"\"`, something like the following:\n\n`192.168.1.101 - - [19/Oct/2017:11:00:16 +0200] \"POST /my/url/path HTTP/1.1\" 200 334`\n\nThe `known_urls.py` relies on the output of `count_urls.py` and on the output it generates itself from a given `.xml` map files with\n```\n  \u003caction-mappings\u003e\n    \u003caction path=\"\"\u003e\u003c/action\u003e\n  \u003c/action-mappings\u003e\n```\nelement tags for the list of known routes.\n\nThe `elaborate_patterns.py` relies on more than one assumption instead:\n- the `ip` address\n- the `date and time` format\n- the `url`\n\n\n### Running\n\n#### Everything in one shot\n```\ncd logorator\npython backend/logorator.py\n```\n\n*Note: in case you miss to populate some `configuration parameter` values, the related elaboration will not be executed\"\n\n\n## Frontend: dashboard web page\n\nAvailable pages:\n- Raw Stats - *How many times your urls have been hitten?*\n- Known route Stats - *Which known routes have been hit the most, and which never?*\n- Refined Patterns - *Frequent patterns workflow*\n\n* Functionalities - [[see here](https://github.com/ncounter/logorator#what-it-does)]\n\n### Requirements\n\n1. Nodejs\n2. Yarn\n\n### Running\n```\ncd logorator/dashboard\nyarn install\nyarn start\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncounter%2Flogorator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fncounter%2Flogorator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncounter%2Flogorator/lists"}