{"id":19341792,"url":"https://github.com/elb4rto/logdoctor","last_synced_at":"2025-04-23T03:32:04.057Z","repository":{"id":63641712,"uuid":"507397927","full_name":"elB4RTO/LogDoctor","owner":"elB4RTO","description":"Apache2 / Nginx / IIS logs analyzer: parse access logs and view dynamically generated statistics","archived":false,"fork":false,"pushed_at":"2024-11-27T20:57:48.000Z","size":6834,"stargazers_count":4,"open_issues_count":4,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-02T07:11:32.278Z","etag":null,"topics":["analyzer","apache2","apache2-logs","cpp","gui","iis","iis-logs","logs","logs-parser","nginx","nginx-logs","qt","statistics","web-servers-logs"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elB4RTO.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-25T19:17:10.000Z","updated_at":"2024-10-25T06:39:00.000Z","dependencies_parsed_at":"2023-02-16T20:30:54.112Z","dependency_job_id":"e210b4fb-e925-4df2-990b-6c8a4692ee64","html_url":"https://github.com/elB4RTO/LogDoctor","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elB4RTO%2FLogDoctor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elB4RTO%2FLogDoctor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elB4RTO%2FLogDoctor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elB4RTO%2FLogDoctor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elB4RTO","download_url":"https://codeload.github.com/elB4RTO/LogDoctor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250365750,"owners_count":21418742,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analyzer","apache2","apache2-logs","cpp","gui","iis","iis-logs","logs","logs-parser","nginx","nginx-logs","qt","statistics","web-servers-logs"],"created_at":"2024-11-10T03:32:32.474Z","updated_at":"2025-04-23T03:32:02.168Z","avatar_url":"https://github.com/elB4RTO.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eLogDoctor\u003c/h1\u003e\n  \u003cp\u003eParse Apache2 / Nginx / IIS logs and view dynamically generated statistics\u003c/p\u003e\n  \u003cbr/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/version-4.02-fff\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/C%2B%2B-23-blue\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Qt-6-blue\"/\u003e\n  \u003cbr/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Linux-supported-31c653\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/BSD-supported-31c653\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Windows-supported-31c653\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Mac%20OS%20X-supported-31c653\"/\u003e\n  \u003cbr/\u003e\n  \u003cimg src=\"https://github.com/elB4RTO/LogDoctor/actions/workflows/cmake.yml/badge.svg\"/\u003e\n\u003c/div\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Table of contents\n\n- [Overview](#overview)\n- [Installation and usage](#installation-and-usage)\n  - [Requirements / dependencies](#requirements--dependencies)\n  - [Usage without installation](#usage-without-installation)\n  - [Usage with installation](#usage-with-installation)\n  - [How to compile](#how-to-compile)\n- [Updates](#updates)\n  - [Version check](#version-check)\n  - [How to update](#how-to-update)\n- [Before to start](#before-to-start)\n- [Logs data](#logs-data)\n  - [Storage](#storage)\n  - [Examined fields](#examined-fields)\n  - [Logs options](#logs-options)\n    - [Usage control](#usage-control)\n    - [Logs path](#logs-path)\n    - [Logs format](#logs-format)\n      - [Apache2](#apache2)\n      - [Nginx](#nginx)\n      - [IIS](#iis)\n    - [Blacklist](#blacklist)\n    - [Warnlist](#warnlist)\n- [Statistics](#statistics)\n  - [Warnings](#warnings)\n  - [Counts](#counts)\n  - [Speed](#speed)\n  - [Time of day](#time-of-day)\n  - [Relational](#relational)\n- [Extra features](#extra-features)\n  - [Log files viewer](#log-files-viewer)\n  - [Block note](#block-note)\n  - [Games](#games)\n- [Final considerations](#final-considerations)\n  - [Backups](#backups)\n  - [Estimated working speed](#estimated-working-speed)\n- [Languages](#languages)\n- [Contributions](#contributions)\n  - [Translations](#translations)\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Overview\n\nLogDoctor is a web servers' access logs parser which allows to view dynamic satistics of the collected data.\u003cbr/\u003e\nSupported web servers are **Apache2**, **Nginx** and **IIS**.\n\n\u003cbr/\u003e\n\n![screenshot](https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/log_files.png)\n\n![screenshot](https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/make_stats.png)\n\n\u003cbr/\u003e\u003cbr/\u003e\n\nLogDoctor is a hard fork of [Craplog](https://github.com/elB4RTO/CRAPLOG).\n\n\u003cbr/\u003e\n\n## Installation and usage\n\n### Requirements / Dependencies\n\n- **From binary**:\n  - C++ 20\n  - Qt6 *(Framework 6.6+, Linguist, Widgets, Charts, Sql, Network)*\u003cbr/\u003e\u003cbr/\u003e\n- **From source**:\n  - *all the above*\n  - CMake\n  - gcc / clang / msvc\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\n- **As Docker**:\n  - Docker\u003cbr/\u003e\u003cbr/\u003e\n\n\u003cbr/\u003e\n\n### Usage without installation\n\n- Download a pre-compiled [Release](https://github.com/elB4RTO/LogDoctor/releases)\n  \u003cbr/\u003e*or*\u003cbr/\u003e\n  Follow the step-by-step guide in [HOW_TO_COMPILE.md](https://github.com/elB4RTO/LogDoctor/blob/main/HOW_TO_COMPILE.md)\n\n- Run the executable\n\n\u003cbr/\u003e\n\n### Usage with installation\n\nSee [HOW_TO_INSTALL.md](https://github.com/elB4RTO/LogDoctor/blob/main/HOW_TO_INSTALL.md)\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n### How to compile\n\nSee [HOW_TO_COMPILE.md](https://github.com/elB4RTO/LogDoctor/blob/main/HOW_TO_COMPILE.md)\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Updates\n\n### Version check\n\nTo check for updates, open the menu `Utilities`→`Version check`.\u003cbr/\u003e\u003cbr/\u003e\n\n### How to update\n\nSee [HOW_TO_UPDATE.md](https://github.com/elB4RTO/LogDoctor/blob/main/HOW_TO_UPDATE.md)\n\n\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\n\n## Before to start\n\nWhen you run LogDoctor for the first time, you will most likely see an empty list of log files.\u003cbr/\u003e\nHead to the **configurations** section and give a look at least at the [logs format](#logs-format) settings. Only files containings logs that match the given format will be shown in the list.\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Logs data\n\nArchived (**gzipped**) log files can be used as well as normal files.\n\n\u003cbr/\u003e\n\n### Storage\n\nParsed data will be stored in an [SQLite](https://www.sqlite.org/about.html) database, which makes it easy to transport/view/edit it as you please.\u003cbr/\u003e\nIf LogDoctor's funcionalities aren't enough for your needs, you can always use a *DB manager* or the SQLite *API* to make your own queries and retrieve the data you need.\n\n\u003cbr/\u003e\n\n### Examined fields\n\nNot all the available log fields (expecially for *Apache2* and *Nginx*) are taken into consideration.\u003cbr/\u003e\nThe considered fields are:\n- **Date** and **Time**\n- Request stuff: **Protocol**, **Method**, **URI** and **Query**\n- Server stuff: **Bytes received**, **Bytes sent** and **Time taken**\n- Client stuff: **User-agent**, **IP address**, **Cookie** and **Referrer site**\n\nFurther informations can be found in the [wiki](https://github.com/elB4RTO/LogDoctor/wiki/Examined-fields) or while running LogDoctor.\n\n\u003cbr/\u003e\n\n### Logs options\n\nVarious options can be configured about log files.\n\n\u003cbr/\u003e\n\n#### Usage control\n\nWhen you parse a file, it will be hashed using the **SHA256** algorithm and the hash will be stored in another database, to keep track of which files you've already used and help you not parsing them twice.\u003cbr/\u003e\n\n##### Note\n\nThe *SHA256* algorithm produces an irreversible hash, which means that no information about the file can be retrieved from the hash.\u003cbr/\u003e\nLogDoctor will **never** grab and/or use any information about you or the usage you make of it.\n\n\u003cbr/\u003e\n\n#### Logs path\n\nA different logs path can be used for any of the three supported *Web Servers*.\u003cbr/\u003e\nIt can be the default system folder or any folder you decide to use, just set it in the options.\n\n\u003cbr/\u003e\n\n#### Logs format\n\nBefore to start parsing logs, you must set-up the *loga format*.\u003cbr/\u003e\nHead to the **configurations** section, under `Logs` select the **Web Server** you want to configure and tap `Format`.\u003cbr/\u003e\nOnce inside the **Format** section, you can insert the *log format string* you're using. Don't forget to use the `Generete preview` button to generate a *log line sample* and **check the correctness** of the format!\u003cbr/\u003e\n\nFor reliability reasons, LogDoctor **does not** support the usage of the **Carriage Return** inside the log format string.\n\n\u003cbr/\u003e\n\n##### Apache2\n\nThe log format string must be specified. Any format is supported, if valid.\u003cbr/\u003e\nTo retrieve your format string:\n- open the configuration file `/etc/apache2/apache2.conf`\n- *usually*, the line you're looking for is the one starting with `LogFormat` and ending with `combined`. It should be somewhere near to the end of the file.\n- you must not paste the whole line, just the part holding the *format string*.\u003cbr/\u003e\n  Example:\u003cbr/\u003e\n  - this is the whole line:\u003cbr/\u003e\n    ```\n    LogFormat \"%h %l %u %t \\\"%r\\\" %\u003es %b \\\"%{Referer}i\\\" \\\"%{User-agent}i\\\"\" combined\n    ```\n  - this is the *format string*:\u003cbr/\u003e\n    ```\n    %h %l %u %t \\\"%r\\\" %\u003es %b \\\"%{Referer}i\\\" \\\"%{User-agent}i\\\"\n    ```\n    please notice that you have to remove the enclosing quotes/apostrophes as well\u003cbr/\u003e\n\nMore informations can be found in the [wiki](https://github.com/elB4RTO/LogDoctor/wiki/Apache2) or while setting the format.\n\n\u003cbr/\u003e\n\n##### Nginx\n\nThe log format string must be specified. Any format is supported, if valid.\u003cbr/\u003e\nTo retrieve your format string:\u003cbr/\u003e\n- open the configuration file `/usr/local/etc/nginx/nginx.conf`\n- *usually*, the line you're looking for is the one starting with `log_format main`. It should be somwehere in the middle of the file\n- one **important** thing: don't paste the indentations and new lines! The default line is usualy declared in consecutive lines, and indented. You must reduce it to a one consecutive string (by also removing the *apostrophes* in the middle of it). The best way is to do this job inside the configuration file, then save and restart Nginx to see if any error is thrown.\u003cbr/\u003e\n  Example:\n  - this is the whole line:\u003cbr/\u003e\n    ```\n    log_format main '$remote_addr - $remote_user [$time_local] '\n                    '\"$request\" $status $body_bytes_sent '\n                    '\"$http_referer\" \"$http_user_agent\" \"$gzip_ratio\"';\n    ```\n  - this is the resulting *format string*:\u003cbr/\u003e\n    ```\n    $remote_addr - $remote_user [$time_local] \"$request\" $status $bytes_sent \"$http_referer\" \"$http_user_agent\" \"$gzip_ratio\"\n    ```\n    please notice that you have to remove the enclosing apostrophes/quotes as well\u003cbr/\u003e\n\nMore informations can be found in the [wiki](https://github.com/elB4RTO/LogDoctor/wiki/Nginx) or while setting the format.\n\n\u003cbr/\u003e\n\n##### IIS\n\nSupported log formats are: **W3C**, **NCSA** and **IIS**.\u003cbr/\u003e\n\nThe *NCSA* and *IIS* modules doesn't allow any modification from the user, so nothing more have to be specified.\n\nThe *W3C* module instead allows the user to decide which fields to log, and thus you must declare the *log format string* you're using.\nTo retrieve your format string (for the *W3C* module only):\n- open any of the log files which have been generated by this module\n- the line you're looking for is the one starting with `#Fields:`, usually at the beginning of the file.\u003cbr/\u003e\n  Example:\u003cbr/\u003e\n  - this is the whole line:\u003cbr/\u003e\n    ```\n    #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken\n    ```\n  - this is the *format string*:\u003cbr/\u003e\n    ```\n    date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken\n    ```\n\nMore informations can be found in the [wiki](https://github.com/elB4RTO/LogDoctor/wiki/IIS) or while setting the format.\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n#### Blacklist\n\nYou can add elements to the **blacklist** to avoid storing the lines containing those elements.\n\nEach web server has its own list.\n\n\u003cbr/\u003e\n\n#### Warnlist\n\nAs for the *blacklist*, you can add elements to the **warnlist**.\u003cbr/\u003e\n*Warnlists* will mark with a **warning** the lines triggering them. Warnings can be viewed in the relative [statistics](#warnings) section.\n\nEach web server has its own lists.\n\n\u003cbr/\u003e\n\n## Statistics\n\nMost of the *statistics sections* allows you to set filters to the log fields, to skim data by only including lines matching those parameters.\u003cbr/\u003e\n\n\u003cbr/\u003e\n\n### Warnings\n\nIn the *warning* section you can view the lines which are triggering a warning.\u003cbr/\u003e\nWarnings are generated dinamically depending on your [warnlists](#warnlist): changing the elements in the *warnlists* will produce different warnings.\u003cbr/\u003e\n\n![screenshot](https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/stats_warnings.png)\n\n\u003cbr/\u003e\n\n### Speed\n\nIn the *speed* section you can view how fast has been your server at serving contents (if you logged the *time taken*, of course).\u003cbr/\u003e\n\n![screenshot](https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/stats_speed.png)\n\n\u003cbr/\u003e\n\n### Counts\n\nThe *count* section is very simple. It just shows the recurrence of the elements for a specific field.\u003cbr/\u003e\n\n![screenshot](https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/stats_count.png)\n\n\u003cbr/\u003e\n\n### Time of day\n\nIn the *time of day* section you can see the traffic, in terms of number of requests logged.\u003cbr/\u003e\nWhen viewing a period of time, the mean value (of all the logged days in that period) is shown.\u003cbr/\u003e\n\n![screenshot](https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/stats_daytime.png)\n\n\u003cbr/\u003e\n\n### Relational\n\nIn the *relational* section you can view how many times a specific field brought to another.\u003cbr/\u003e\nThis section is more suited for long periods of time.\u003cbr/\u003e\n\n![screenshot](https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/stats_relational.png)\n\n\u003cbr/\u003e\n\n### Globals\n\nIn the *globals* section you can have an overview of your logs history.\u003cbr/\u003e\n\n![screenshot](https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/stats_globals.png)\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Extra features\n\n### Log files viewer\n\nUse the built-in logs viewer to inspect the content of your log files.\u003cbr/\u003e\nColor schemes will be applied using the currently set log format.\n\n\u003cbr/\u003e\n\n### Block-note\n\nA block-note utility is available at `Tools`→`BlockNote` which can be used to temporary write text, notes, etc.\n\n\u003cbr/\u003e\n\n### Games\n\nSimple mini-games to kill the time.\u003cbr/\u003e\u003cbr/\u003e\n\n#### CrissCross\n\n\u003cimg height=\"300px\" src=\"https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/game_crisscross.png\" /\u003e\u003cbr/\u003e\u003cbr/\u003e\n\n#### Snake\n\n\u003cimg height=\"350px\" src=\"https://raw.githubusercontent.com/elB4RTO/screenshots/main/LogDoctor/game_snake.png\" /\u003e\u003cbr/\u003e\u003cbr/\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Final considerations\n\n### Backups\n\nLogDoctor can automatically do a backup of your **logs database** file, so you can recover your data in case something goes wrong.\u003cbr/\u003e\n\nMove inside LogDoctor's folder (if you don't know/remember the path, open the `Utilities`→`Infos`\u003e`Paths` menu to view it) and open the folder named \"**backups**'.\u003cbr/\u003e\nHere you will find the backups with an increasing index, where '.1' represents the newest.\n\nA new backup is made every time you quit LogDoctor after doing a job which affected the database in any way.\n\n#### Note\n\nOnly the *logs-data database* will be backed-up, the *hashes database* **won't**.\u003cbr/\u003e\nThis is because it is unlikely (supposedly impossible) that a hash equals another, therefore they're supposed to be useful for a short period of time (that is, until you or your web server delete the original log files).\n\n\u003cbr/\u003e\n\n### Estimated working speed\n\n10~200 MB/s\n\nTake this estimation with a grain of salt, it may be even higher or lower depending on a variety of factors, like: the build type, your hardware, the complexity of the logs, the complexity of the blacklist, the workload of your system during the execution...\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Languages\n\nLogDoctor is available in multiple languages, most of which are automatically translated. *Wanna [contribute](https://github.com/elB4RTO/LogDoctor/blob/main/TRANSLATING.md) to improve them?*)\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Contributions\n\nLogDoctor is under constant development.\n\nIf you have suggestions about how to improve it, please open an [issue](https://github.com/elB4RTO/LogDoctor/issues).\n\nIf you want to contribute to the code, please read the [Contribution Guidelines](https://github.com/elB4RTO/LogDoctor/blob/main/CONTRIBUTING.md).\n\nIf you want to contribute to the translation, please read the [Translation Guidelines](https://github.com/elB4RTO/LogDoctor/blob/main/TRANSLATING.md).\n\n\u003cbr/\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felb4rto%2Flogdoctor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felb4rto%2Flogdoctor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felb4rto%2Flogdoctor/lists"}