{"id":16875045,"url":"https://github.com/dlsniper/logalyzer","last_synced_at":"2025-08-25T19:04:36.473Z","repository":{"id":9584617,"uuid":"11501549","full_name":"dlsniper/logalyzer","owner":"dlsniper","description":"Log file analyzer for nginx and CloudFront written in GO","archived":false,"fork":false,"pushed_at":"2014-04-09T15:39:53.000Z","size":3807,"stargazers_count":12,"open_issues_count":3,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-11T12:03:53.900Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dlsniper.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-07-18T11:47:18.000Z","updated_at":"2021-11-19T17:25:41.000Z","dependencies_parsed_at":"2022-07-30T01:47:52.307Z","dependency_job_id":null,"html_url":"https://github.com/dlsniper/logalyzer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dlsniper/logalyzer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlsniper%2Flogalyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlsniper%2Flogalyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlsniper%2Flogalyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlsniper%2Flogalyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dlsniper","download_url":"https://codeload.github.com/dlsniper/logalyzer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlsniper%2Flogalyzer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272116857,"owners_count":24876268,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-25T02:00:12.092Z","response_time":1107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T15:34:57.154Z","updated_at":"2025-08-25T19:04:36.446Z","avatar_url":"https://github.com/dlsniper.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"logalyzer\n=========\n\nCloudFront and Nginx log analyzer\n\nThis was originally created to help parsing the log files of CloudFront and get some statistics out of them.\n\n\nParameters\n---\n\nParameter | Description\n----|------\n```fmt```  | Input file format. Can be: ```nginx```, ```cloudfront```. Default: ```nginx```\n```f```    | Name of the file to be parsed, default empty ``` ```\n```fr```   | Regex of the files to be parsed, default all ```(.*)```. If used it will override ```-f```. Must be used with ```-dir``` option\n```dir```  | Directory name of where the files should be loaded from, default empty ``` ```. If used it will override ```-f```. Must be used with ```-fr``` option\n```url```  | Regex of the urls that will be matched. Default all ```(.*)```\n```iqs```  | Ignore the query string of the url. Default ```false```\n```rt```   | Type of the request: ```GET```, ```POST```, ```PUT``` and so on. If empty, all request will be processed. Default: empty ``` ```\n```cfrt``` | Type of the CloudFront request : ```Hit```, ```RefreshHit```, ```Miss```, ```Pass```(```RefreshHit, Miss```), ```LimitExceded```, ```CapacityExceeded```, ```Exceed```(```LimitExceded, CapacityExceeded```), ```Error```. If empty, all request will be processed. Default: empty ``` ```\n```l```    | Number of lines to be parsed. Default, all, ```0```\n```h```    | Show the hits for the urls. Default ```false```\n```p```    | Set the prefix for the urls to be displayed. Default empty ``` ```\n```s```    | Compute statistics for hits of the urls. Default ```false```\n```hs```   | Show statistics in human format. Default ```true```\n```fd```   | Just extract the whole urls and do nothing else to process them. Overrides all other switches but ```limit``` and ```-p```\n```a```    | Aggregate data from all input files. Must be used with ```-dir``` option\n```af```   | Aggregate data from the chunks of N files. If 0 is passed then all files will be aggregated. This must be used with ```-a```. Default ```0```\n```ab```   | Aggregate by: ```url```, ```hm``` (hits/minute), ```uhm``` (url hits / minute for a specific url). Default ```url```\n```tu```   | Display only the first N accessed URLs. If 0 is passed then all URLs will be shown. This must be used with ```-s```. Default ```0```\n```su```   | Display a separator every Nth accessed URLs. If 0 is passed then all fallback to default. This must be used with ```-s```. Default ```100```\n```v```    | Verbose. Default ```false```\n\n\nUsage\n---\n\n- Get the list of all urls sorted by total number of hits ignoring query string\n```bash\nlogalyzer -a=true -fmt=\"cloudfront\" -dir=\"/path/\" -p=\"http://localhost\" -cfrt=\"Pass\" -iqs=true\n```\n\n- Get the list of all urls sorted by total number of hits ignoring query string and display the statistics for them, in human readable format\n```bash\nlogalyzer -a=true -fmt=\"cloudfront\" -dir=\"/path/\" -p=\"http://localhost\" -cfrt=\"Pass\" -s -hs=true -iqs=true\n```\n\n- Get list of the highest 10 minutes hits for a specified url pattern (phpinfo.php) from CloudFront in human readable format\n```bash\nlogalyzer -a=true -fmt=\"cloudfront\" -dir=\"/path/\" -p=\"http://localhost\" -cfrt=\"Pass\" -s -hs=true -ab=\"uhm\" -url=\"(?i)/phpinfo\\.php\" -iqs=true -tu=10\n```\n\nPerformance\n---\n\nThis should be highly performant due to GO mostly but it's also depending on the used hardware.\nOn my machine, Intel Core i7-3635QM, 12GB and SSD for parsing:\n\ntype | value\n----|------\nfiles | 5382\ntotal size | 14GB\navg load before test | 0.40\n\nusing:\n```bash\nlogalyzer -a=true -fmt=\"cloudfront\" -dir=\"/path/\" -p=\"http://localhost\" -s -hs=true -iqs=true\n```\n\ntype | value\n----|------\nmax load | 1.40\ntime logalyzer | 200,25s user 5,99s system 97% cpu 3:31,00 total\nTotal unique URLs | 69321\nTotal URLs accesed | 50573946\n\nI have the following:\n```bash\nlogalyzer -a=true -fmt=\"cloudfront\" -dir=\"/path/\" -p=\"http://localhost\" -cfrt=\"Pass\" -s -hs=true -iqs=true\n```\n\ntype | value\n----|------\nmax load | 1.30\ntime logalyzer | 187,30s user 6,10s system 98% cpu 3:17,06 total\nTotal unique URLs | 65377\nTotal URLs accesed | 12243949\n\n\nDisclaimer\n----\n\nPlease note that I've generated the above files for demo purposes only and they don't reflect in any way, shape or form any real data that I may or may not have access to.\n\n\nThank you\n---\n\nA big thank you goes to [motain GmbH](http://motain.de) for allowing this to be open sourced.\n\n\nFinal word\n---\n\nSome of the CloudFront analyzers on the Internet can cost a lot of money. If you do happen to use this, please help me improving it by submitting pull requests or just sending me a thank you on Twitter: ```@dlsniper```\nIf you make changes on your own environment, please stick to the license and add them back here as well. Thanks.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlsniper%2Flogalyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdlsniper%2Flogalyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlsniper%2Flogalyzer/lists"}