{"id":19341789,"url":"https://github.com/elb4rto/craplog-cli","last_synced_at":"2025-10-13T21:32:25.791Z","repository":{"id":36993372,"uuid":"354508849","full_name":"elB4RTO/craplog-CLI","owner":"elB4RTO","description":"Parse Apache2 logs to create statistics","archived":false,"fork":false,"pushed_at":"2023-02-11T23:24:34.000Z","size":479,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-24T08:45:47.730Z","etag":null,"topics":["apache2","apache2-logs","cli","logs","python3","statistics"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elB4RTO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-04T09:55:26.000Z","updated_at":"2024-05-19T18:10:51.000Z","dependencies_parsed_at":"2024-11-10T03:32:33.101Z","dependency_job_id":"b4370435-0b05-45b0-aa4f-54a0e2ea7336","html_url":"https://github.com/elB4RTO/craplog-CLI","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/elB4RTO/craplog-CLI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elB4RTO%2Fcraplog-CLI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elB4RTO%2Fcraplog-CLI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elB4RTO%2Fcraplog-CLI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elB4RTO%2Fcraplog-CLI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elB4RTO","download_url":"https://codeload.github.com/elB4RTO/craplog-CLI/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elB4RTO%2Fcraplog-CLI/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017032,"owners_count":26085951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache2","apache2-logs","cli","logs","python3","statistics"],"created_at":"2024-11-10T03:32:28.281Z","updated_at":"2025-10-13T21:32:25.759Z","avatar_url":"https://github.com/elB4RTO.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Craplog CLI\nParse Apache2 logs to create statistics\n\n\u003cbr/\u003e\n\n![logo](https://raw.githubusercontent.com/elB4RTO/screenshots/main/Craplog/craplogo.png)\n\n\u003cbr/\u003e\n\n## Table of contents\n\n- [Overview](#overview)\n- [Installation and execution](#installation-and-execution)\n  - [Dependencies](#dependencies)\n  - [Run without installation](#run-without-installation)\n  - [Run with installation](#run-with-installation)\n- [Usage](#usage)\n  - [Arguments](#arguments)\n  - [Examples](#examples)\n  - [Tools examples](#tools-examples)\n  - [Output control](#output-control)\n- [How to configure](#how-to-configure)\n  - [Crapset](#crapset)\n  - [Configuration files](#configuration-files)\n  - [Hardcoded values](#hardcoded-values)\n- [How to update](#how-to-update)\n  - [Crapup](#crapup)\n  - [Self-service](#self-service)\n- [Logs](#logs)\n  - [Usage control](#usage-control)\n  - [Log files](#log-files)\n  - [Logs path](#logs-path)\n  - [Logs structure](#logs-structure)\n- [Statistics](#statistics)\n  - [Storage](#storage)\n  - [Examined fields](#examined-fields)\n  - [Sessions statistics](#sessions-statistics)\n  - [Global statistics](#global-statistics)\n  - [Whitelist](#whitelist)\n- [View statistics](#view-statistics)\n  - [Crapview](#crapview)\n- [Extra features](#extra-features)\n  - [Crapstats converter](#crapstats-converter)\n- [Final considerations](#final-considerations)\n  - [Estimated working speed](#estimated-working-speed)\n  - [Backups](#backups)\n- [Contributions](#contributions)\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Overview\n\nCraplog is a tool that takes Apache2 logs in their default form, parses them and creates simple statistics.\n\n\u003cbr/\u003e\n\nWelcome to the **command line** version\n\n![screenshot](https://raw.githubusercontent.com/elB4RTO/screenshots/main/Craplog/CLI/craplog.png)\n\n\u003cbr/\u003e\n\nSearching for something different? Try the [other versions of CRAPLOG](https://github.com/elB4RTO/CRAPLOG).\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Installation and execution\n\n### Dependencies\n*None*\n\n\u003cbr/\u003e\n\n### Run without installation\n\n- Download and un-archive this repo\n  \u003cbr/\u003e*or*\u003cbr/\u003e\n  ```\n  git clone https://github.com/elB4RTO/craplog-CLI.git\n  ```\n  \u003cbr/\u003e\n- Open a terminal inside \"*craplog-CLI-main/*\"\n  \u003cbr/\u003e*or*\u003cbr/\u003e\n  ```\n  cd craplog-CLI/\n  ```\n  \u003cbr/\u003e\n- Run craplog using python's environment:\n  \u003cbr/\u003e\n  \n  ```\n  python3 craplog/craplog.py --help\n  ```\n  \u003cbr/\u003e\n\n\u003cbr/\u003e\n\n### Run with installation\n\n- Download and un-archive this repo\n  \u003cbr/\u003e*or*\u003cbr/\u003e\n  ```\n  git clone https://github.com/elB4RTO/craplog-CLI.git\n  ```\n  \u003cbr/\u003e\n- Open a terminal inside \"*craplog-CLI-main*\"\n  \u003cbr/\u003e*or*\u003cbr/\u003e\n  ```\n  cd craplog-CLI/\n  ```\n  \u003cbr/\u003e\n- Run the installation script:\n  \u003cbr/\u003e\n  \n  ```\n  chmod +x ./install.sh \u0026\u0026 ./install.sh\n  ```\n  \u003cbr/\u003e\n- You can now run craplog from terminal, as any other application (you don't need to be in craplog's folder):\n  \u003cbr/\u003e\n  \n  ```\n  craplog --help\n  ```\n  \u003cbr/\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Usage\n\n### Syntax\n\n`craplog [TOOL] {[OPTION] [ARGUMENT]}`\n\n\u003cbr/\u003e\n\n### Tools\n\n\u003cbr/\u003e\n\n|   Tool | Description |\n| -----: | :---------- |\n|    *log* | Craplog: make statistics from the logs\u003cbr/\u003e*Implicit, can be omitted* |\n|   view | Crapview: view your statistics |\n|  setup | Crapset: configure these tools |\n| update | Crapup: check for updates |\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n### Arguments\n\n\u003cbr/\u003e\n\n| Abbr. |           Option | Additional     | Description |\n| :-: | -----------------: | :------------- | :-- |\n| -h  |             --help |                | prints an help screen |\n|     |         --examples |                | prints usage examples |\n| -l  |             --less |                | less output on screen |\n| -m  |             --more |                | more output on screen |\n| -p  |      --performance |                | prints performances data |\n|     |        --no-colors |                | prints text without using colors |\n|     |      --auto-delete |                | auto-chooses to delete files/folders |\n|     |       --auto-merge |                | auto-chooses to merges sessions having the same date |\n| -e  |           --errors |                | make sstatistics of error logs too |\n| -eO |      --only-errors |                | only uses error logs (doesn't parse access logs) |\n| -gO |     --only-globals |                | only updates globals (doesn't store sessions) |\n| -gA |    --avoid-globals |                | does not update global statistics |\n| -b  |           --backup |                | stores a backup copy of the original logs file |\n| -bT |       --backup-tar |                | stores the backup as a compressed tar.gz archive |\n| -bZ |       --backup-zip |                | stores the backup as a compressed zip archive |\n| -dO | --delete-originals |                | deletes the original log files when done |\n|     |            --trash | *\u0026lt;path\u0026gt;* | moves files to Trash instead of remove\u003cbr/\u003e*\u0026lt;path\u0026gt;* is optional: if omitted, default will be used |\n|     |            --shred |                | uses shred on files instead of remove |\n| -P  |        --logs-path | *\u0026lt;path\u0026gt;* | path of the directory where the logs are located |\n| -F  |        --log-files | *\u0026lt;list\u0026gt;* | list of log files to use (names, NOT paths)\u003cbr/\u003e*\u0026lt;list\u0026gt;*: whitespace-separated file names |\n| -A  |    --access-fields | *\u0026lt;list\u0026gt;* | list of fields to use while parsing access logs\u003cbr/\u003e*\u0026lt;list\u0026gt;*: whitespace-separated fields |\n| -W  |     --ip-whitelist | *\u0026lt;list\u0026gt;* | doesn't parse log lines from these IPs\u003cbr/\u003e*\u0026lt;list\u0026gt;*: whitespace-separated IPs |\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n### Examples\n\n\u003cbr/\u003e\n\n- Uses default log files as input, including errors (access logs are used by default). Stores a backup copy the original files as a *tar.gz* compressed archive, without deleting them. Moves files to trash if needed (instead of complete deletion). Global statistics will be updated by default.\n \n  ```\n  craplog -e -bT --trash\n  ```\n  \u003cbr/\u003e\n- As the above one, but only parses errors (not access logs). Stores a backup copy the original files as a *zip* compressed archive, without deleting them. Shreds files if needed (instead of normal deletion). Global statistics will be updated by default.\n  \n  ```\n  craplog -eO -bZ --shred\n  ```\n  \u003cbr/\u003e\n- Uses user-defined access and/or error logs files from an alternative logs path. Automatically merges sessions having the same date if needed.\n  \n  ```\n  craplog -e -P /your/logs/path -F file.log.2 file.log.3.gz --auto-merge\n  ```\n  \u003cbr/\u003e\n- Uses default log files for both access and error logs. Uses a whitelist for IPs and a selection of which access fields to parse.\n  \n  ```\n  craplog -e -W ::1 192.168. -A REQ RES\n  ```\n  \u003cbr/\u003e\n- Print more informations on screen, including performance details. Use the default access logs file but only update globals, not sessions.\n  \n  ```\n  craplog -m -p -gO\n  ```\n  \u003cbr/\u003e\n- Print less informations on screen, including performance details but without using colors. Use the default access and error logs files, but do not update globals. Make a backup copy of the original files used and delete them when done.\n  \n  ```\n  craplog -l -p --no-colors -e -gA -b -dO\n  ```\n\n\u003cbr/\u003e\n\n### Tools examples\n\n***Warning***: *the following syntax is only suited for the usage **with installation**. If you're using Craplog without installing it, you'll have to run the tools individually. Further informations can be found in the relative sections.*\n\n\u003cbr/\u003e\n\n- Make a *version check* query.\n  \n  ```\n   craplog update\n  ```\n  \u003cbr/\u003e\n- View your statistics.\n  \n  ```\n   craplog view\n  ```\n  \u003cbr/\u003e\n- Set-up Craplog's tools.\n  \n  ```\n   craplog setup\n  ```\n  \u003cbr/\u003e\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n### Output control\n\nYou can control the output on screen, like: quantity of informations printed, performance details and the use of colors.\u003cbr/\u003e\u003cbr/\u003e\n\n![output diffs](https://raw.githubusercontent.com/elB4RTO/screenshots/main/Craplog/CLI/output_diff.png)\n*Normal output vs Less output*\n\n\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\n\n## How to configure\n\nSometimes is annoying to keep remembering and passing arguments, I know. This is why Craplog gives you the possibility to customize the way it gets ready to do its job.\n\nThere's actually more then one way you can customize Craplog's settings: using the [configuration tool](#crapset), editing the [configuration files](#configuration-files) or the [hardcoded values](#hardcoded-values).\n\n\u003cbr/\u003eThe configuration file will override the hardcoded values and will be overridden by the command-line arguments, so that the configurations **hierarchy** results as follows:\n  - Hardcoded values\n  - Configuration files\n  - Command-line arguments\n\n\u003cbr/\u003eYou can also **lock** a configuration method to avoid it, like discarding any command-line argument or not reading the configuration files. Further informations can be found while following one of the procedures listed above.\n\n\u003cbr/\u003e\n\n### Crapset\n\n**Crapset** is an utility to easily and safely customize Craplog.\u003cbr/\u003e\u003cbr/\u003e\n\nWith Craplog installed:\n```\ncraplog setup\n```\n\u003cbr/\u003e\n\nWithout Craplog installed *(from the main folder)*:\n```\npython3 crapset/crapset.py\n```\n\n\u003cbr/\u003e\n\n### Configuration files\n\nManually editing the [configuration files](https://github.com/elB4RTO/craplog-CLI/tree/main/crapconfs) you need.\u003cbr/\u003e\n\nFiles can be found inside **craplog-cli/crapconfs/**\n\n\u003cbr/\u003e\n\n### Hardcoded values\n\nDirectly modifying the script's hardcoded variables, to set pre-defined initialization values:\n  - **Craplog**  -\u003e line [**117**](https://github.com/elB4RTO/craplog-CLI/blob/main/craplog/craplog.py#L117) **@** *craplog-CLI/craplog/craplog.py*\n  - **Crapview** -\u003e line [**21**](https://github.com/elB4RTO/craplog-CLI/blob/main/crapview/crapview.py#L21) **@** *craplog-CLI/cragview/crapview.py*\n  - **Crapup**   -\u003e line [**66**](https://github.com/elB4RTO/craplog-CLI/blob/main/crapup/crapup.py#L66) **@** *craplog-CLI/cragup/crapup.py*\n  - **Crapset**  -\u003e line [**32**](https://github.com/elB4RTO/craplog-CLI/blob/main/crapset/crapset.py#L32) **@** *craplog-CLI/cragset/crapset.py*\n\n\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\n\n## How to update\n\nUpdates can be checked with the [updater tool](#crapup) or, in alternative, you can always do a [manual update](#self-service).\n\n\u003cbr/\u003e\n\n### Crapup\n\n**Crapup** allows you to query the updates in two different ways: a simple [version check](#version-check) query, or an effective update through [git pull](#git-pull).\u003cbr/\u003e\u003cbr/\u003e\n\nWith Craplog installed:\n```\ncraplog update\n```\n\u003cbr/\u003e\n\nWithout Craplog installed *(from the main folder)*:\n```\n`python3 crapup/crapup.py\n```\n\n\u003cbr/\u003e\n\n#### Version check\n\nThis is the **default** method.\n\n**Crapup** will check for a version-update through a simple GET request to the [version check file](https://github.com/elB4RTO/craplog-CLI/blob/main/version_check) in this page.\n\nNothing will be downloaded or updated, it will only queries this repo's version and gives back a response. You'll have to [manually download and apply the changes](#self-service).\u003cbr/\u003e\u003cbr/\u003e\n\n#### Git pull\n\nThis is the **suggested** method.\n\n**Crapup** will update your local version by directly fetching this repo. This is the suggested method since it's fast, reliable and easy.\n\nYou can also perform this procedure manually if you want, by following the [update with git](#git-update) guide.\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n### Self-service\n\nA self-served update of Craplog can be done in the well-known two ways.\u003cbr/\u003e\u003cbr/\u003e\n\n#### Manual update\n\nTo manually update Craplog, please download the new version of this repo and run the [update script](https://github.com/elB4RTO/craplog-CLI/blob/main/update.sh).\n\nOr alternatively manually *copy-paste* this list of files/folders in your Craplog installation directory:\u003cbr/\u003e\n`craplog/`, `crapset/`, `crapup/`, `crapview/`, `README.md`, `LICENSE`.\u003cbr/\u003e\nIf you opted for the manual *copy-paste*, please make sure the operation fully replace the old content, meaning that you have to check that no old entry (maybe with a different, old name) is left there.\u003cbr/\u003e\u003cbr/\u003e\n\n#### Git update\n\nTo update Craplog with **git** you'll need to have a local clone of this repo.\u003cbr/\u003e\nIf you downloaded Craplog using the `git clone` method, you should be ready to go.\n\nFollow these steps:\n\n- Make sure you're in Craplog's main folder with your terminal\u003cbr/\u003e\n  *You should see \"craplog\" as output*\u003cbr/\u003e\n  \n  ```\n  ls | grep craplog\n  ```\n  \u003cbr/\u003e\n- Make sure you have *git* installed in your system\u003cbr/\u003e\n  *This should output the path of your git executable*\u003cbr/\u003e\n  \n  ```\n  which git\n  ```\n  \u003cbr/\u003e\n- Test if a git repository is already initialized in the current directory\u003cbr/\u003e\n  *No error message should be shown*\u003cbr/\u003e\n  \n  ```\n  git status\n  ```\n  If you get an error message, follow these steps to initialize a git:\u003cbr/\u003e\n  \n  - Initialize the git repo, using `main` as local branch name\u003cbr/\u003e\n    \n    ```\n    git init -b main\n    ```\n    \n  - Configure it\u003cbr/\u003e\n    \n    ```\n    git config core.filemode false\n    git config remote.origin.url https://github.com/elB4RTO/craplog-CLI.git\n    git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*\n    git config remote.origin.prune true\n    git config branch.main.remote origin\n    git config branch.main.merge refs/heads/main\n    git config pull.rebase false\n    ```\n    \n  - Add Craplog's files to the git index\u003cbr/\u003e\n    \n    ```\n    git add craplib/ craplog/ crapview/ crapup/ crapset/ README.md LICENSE\n    ```\n    \n  - Make a `.gitignore` file to ignore the local *configurations* and *statistics*\u003cbr/\u003e\n    \n    ```\n    echo \"/crapconfs/\" \u003e\u003e .gitignore\n    echo \"*.crapconf\" \u003e\u003e .gitignore\n    echo \"/crapstats/\" \u003e\u003e .gitignore\n    echo \"*.crapstat\" \u003e\u003e .gitignore\n    ```\n    \u003cbr/\u003e\n    \n- Your local repo is ready to pull the updates from the remote:\u003cbr/\u003e\n  \n  - You can directly download and apply any modification with just one command:\u003cbr/\u003e\n    \n    ```\n    git pull origin main\n    ```\n    \u003cbr/\u003e\n  - Or you can split the process in steps:\u003cbr/\u003e\n    \n    - Download the informations about the new version's changes\u003cbr/\u003e\n      \n      ```\n      git fetch origin\n      ```\n      \n    - Inspect any modification\u003cbr/\u003e\n      \n      ```\n      git diff origin/main\n      ```\n      \n    - Finally apply the changes (if you want so)\u003cbr/\u003e\n      \n      ```\n      git merge origin/main\n      ```\n      \u003cbr/\u003e\u003cbr/\u003e\n    \n- If you're having troubles updating for refs/code mismatches, follow the following:\u003cbr/\u003e\n  \n  - Make a backup copy of the `crapstats` and `crapconfs` folders (and whatever else you care about).\u003cbr/\u003e\n    *Nothing should happen to the non-indexed files/folders, but who knows, right?*\n    \u003cbr/\u003e\n  \n  - Reset your local git, removing the indexed content\n    \n    ```\n    git reset --hard\n    ```\n  \n  - Pull a fresh copy of this repository\n    \n    ```\n    git pull origin main\n    ```\n  \n  - Restore your backups if required\u003cbr/\u003e\n    *Hopefully you shouldn't need to*\n\n\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\n\n## Logs\n\nAt the moment, it still only supports **Apache2** log files in their **default** form.\n\nArchived (**gzipped**) log files can be used as well as normal files.\n\n\u003cbr/\u003e\n\n### Usage control\nThis version of Craplog keeps track of the log files which have been used.\n\nWhen a file is parsed succesfully, its **sha256** checksum is stored.\u003cbr/\u003e\nThe stored checksums will be checked every time a file is given as input, to help preventing parsing the same files twice.\n\nHashes will be stored in **craplog/crapstats/.hashes**\n\n\u003cbr/\u003e\n\n### Log files\n\nIf not specified, the files to be used will be **access.log.1** *and/or* **error.log.1**\n\nDifferent file/s can be used by passing their names with `-F \u003cnames\u003e` / `--log-files \u003cnames\u003e`\n\nPlease notice that only **file names** have to be specified, NOT full paths.\n\n\u003cbr/\u003e\n\n### Logs path\n\nIf not specified, the default path will be **/var/log/apache2/**\n\nA different path can be used by passing it with `-P \u003cpath\u003e` / `--logs-path \u003cpath\u003e`\n\n\u003cbr/\u003e\n\n### Logs structure\n\nAt the moment of writing, this is the only supported logs structure.\u003cbr/\u003e\u003cbr/\u003e\n\n#### access.log.*\n\nIP - - [DATE:TIME] \"REQUEST URI\" RESPONSE \"FROM URI\" \"USER AGENT\"\n\n*123.123.123.123 - - [01/01/2000:00:10:20 +0000] \"GET /style.css HTTP/1.1\" 200 321 \"/index.php\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Firefox/86.0\"*\n\n\u003cbr/\u003e\n\n#### error.log.*\n\n[DATE TIME] [LOG LEVEL] [PID] ERROR REPORT\n\n*[Mon Jan 01 10:20:30.456789 2000] [headers:trace2] [pid 12345] [client 123.123.123.123:45678] AH00128: File does not exist: /var/www/html/domain/readme.txt*\n\n\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\n\n## Statistics\n\n### Storage\n\nStatistics will be stored in Craplog's main folder: **craplog-CLI/crapstats/**\u003cbr/\u003e\nPlease refer to the [statistics viewer tool](#view-statistics) to view your crapstats.\n\n\u003cbr/\u003e\n\n### Examined fields\n\n#### Access logs\n\nFour fields can be examined while parsing **access** logs:\n\n- IP address of the client\n- User-agent of the client\n- Requested page/URL\n- Response code from the server\n\n\u003cbr/\u003eYou can select which fields to use by passing them with `-A \u003cfields\u003e` / `--access-fields \u003cfields\u003e`\u003cbr/\u003e\nAvailable fields choices are: **IP**, **UA**, **REQ**, **RES**\u003cbr/\u003e\n\nYou can avoid parsing access logs by passing `-eO` / `--only-errors`\n\n\u003cbr/\u003e\n\n#### Error logs\n\nWhile parsing **error** logs, only two fields will be used:\n\n- Log level\n- Error report\n\n\u003cbr/\u003eBy default error logs won't be used, but you can parse them by passing `-e` / `--error-logs`\n\n\u003cbr/\u003e\n\n### Sessions statistics\n\n**Sessions** are made by grouping statistics depending on the **date** of the single lines and will be stored consequently: new content will be made if that date is not present in the *crapstats* yet, or it will be merged if the date already exists.\n\nOlny '**\\*.log.\\***' files will be considered valid as input. This is because these files (usually) contain the full logs stack of an entire (*past*) day.\u003cbr/\u003e\nRunning it against a *today*'s file (which is not complete yet) may lead to re-running it in the future on the same file, parsing the same lines twice.\n\nYou can avoid storing sessions by passing `-gO` / `--only-globals`\n\n\u003cbr/\u003e\n\n### Global statistics\n\nAdditionally, **global statistics** will be created and/or updated *consequently*.\u003cbr/\u003e\nThese statistics are identical to the session ones, in fact they're just merged sessions, for a larger view.\n\nYou can avoid updating globals by passing `-gA` / `--avoid-globals`\n\n\u003cbr/\u003e\n\n### Whitelist\n\nYou can add IP addresses to this list (may them be full *IPs*, only the *net-ID* part or just a portion of your choice), in order to skip the relative lines by whitelisting (or blacklisting..?) them, in both **access** and **error** logs.\n\nPlease notice that the given sequence must be the **starting part**: it's not possible (at the moment, and more likely also in future versions) to skip IPs ending or just containing that sequence.\n\nAs an example, if you insert \"123\", then only IP addresses starting with that sequence will be skipped.\u003cbr/\u003e\nIf you insert \".1\", then nothing will be skipped, since no IP will ever start with a dot.\u003cbr/\u003e\nBut the shortcut \"::1\" is used by Apache2 for internal connections and will therefore be valid to skip those lines.\n\nThe **default** is to only skip logs from **::1**, but different sequences can be passed with `-W \u003cIPs\u003e` / `--ip-whitelist \u003cIPs\u003e`\n\nPlease notice that using a custom list will overwrite the default one, not appending to it. When passing a custom list as argument, you should include the default *::1* in order to keep whitelisting the relative lines.\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## View statistics\n\nCraplog saves statistics as plain-text files, so you can directly view them, but you will agree that this ain't the best way to do that.\u003cbr/\u003e\u003cbr/\u003e\n\n### Crapview\n\n**Crapview** is a cursed application that lets you easily view your crapstats.\u003cbr/\u003e\u003cbr/\u003e\n\n![performance diffs](https://raw.githubusercontent.com/elB4RTO/screenshots/main/Craplog/CLI/crapview.png)\u003cbr/\u003e\n*Viewing statistics*\u003cbr/\u003e\u003cbr/\u003e\n\n#### Run crapview\n\nWith Craplog installed:\n```\ncraplog view\n```\n\u003cbr/\u003e\n\nWithout Craplog installed *(from the main folder)*:\n```\npython3 crapview/crapview.py\n```\n\u003cbr/\u003e\n\n#### Use crapview\n\nIt is pretty straight forward: use `TAB` to switch between windows, `ENTER` to interact, the arrow keys `←` `↑` `→` `↓` to move arownd and the letters to write in the **cli** or jump in the index of the **tree**.\u003cbr/\u003e\u003cbr/\u003e\n\nOn the left side you can see the **tree** of your *statistics*, as it is in your system.\n\nOn the right side you can **view** the selected *statistics file*. You can see the elements *(in white)* and their counts *(in green)*. The bars will show the percentage compared to the other elements in the same file.\n\nLast but not least, at the bottom you can find the **cli**, which is not a real cli, but more like a search box.\u003cbr/\u003e\nAvailable **keywords** ar the following:\n- `quit` : quits the program\n  \n- `clear \u003celement\u003e` : clears an element of the window. If no element is supplied, it will take effect on each one.\u003cbr/\u003e\n  Available elements are: `cli`, `tree`, `view`\n  \n- `\u003celement\u003e`: Directly jump to the relative element\n  \n- `\u003ctree path\u003e` : Directly jump to the relative position in the tree.\u003cbr/\u003e\n  Path must be composed by whitespace-separated words, as they are in the tree.\u003cbr/\u003e\n  *Example: see the Requests statistics of a particulare day:* `sessions access \u003cyear\u003e \u003cmonth\u003e \u003cday\u003e REQ`\n\n\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\n\n## Extra features\n\n### Crapstats converter\n\n***COMING SOON***\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Final considerations\n\n### Estimated working speed\n\n1~15 MB/s\n\nMay be higher or lower depending on the complexity of the logs, the complexity of the stored statistics (in case of merge), your hardware and the workload of your system during the execution.\u003cbr/\u003e\nUsually, if it takes more than 10 seconds to parse 10 MB of data, it means you've probably been tested in some way (better to check).\u003cbr/\u003e\u003cbr/\u003e\n\n![performance diffs](https://github.com/elB4RTO/CRAPLOG/blob/main/crapshots/CLI/perf_diff.png)\u003cbr/\u003e\n*Normal vs Scanned*\u003cbr/\u003e\u003cbr/\u003e\n\nThe above image shows the difference in performances between two different sessions, having the same number of lines and very similar data sizes.\u003cbr/\u003e\nOn the left side, the parsed logs resulted from a webserver with normal activity.\u003cbr/\u003e\nOn the right side, the parsed logs resulted from a webserver which have been scanned with tools like **sqlmap** and **nikto** (not nmap)\n\n\u003cbr/\u003e\n\n### Backups\n\nCraplog will automatically make backups of **global statistics** files (in case of fire).\u003cbr/\u003e\nIf something goes wrong and you lose your actual globals, you can recover them (at least the last backup taken).\n\nMove inside the folder you choose to store statistics in, open the \"**globals**\" folder, show hidden files and open the folder named \"**.backups**'.\u003cbr/\u003e\nThe complete path should look like `/\u003cyour_path\u003e/craplog/crapstats/globals/.backups/`\u003cbr/\u003e\nHere you will find the last 3 backups taken. Folder named '3' is always the oldest and '1' the newest.\n\nA new backup is made every time you run Craplog *successfully* over globals.\n\nPlease notice that *session* statistics will **not** be backed-up\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Contributions\n\nCraplog is under development\n\nIf you have suggestions about how to improve it please open an ![issue](https://github.com/elB4RTO/craplog-CLI/issues) or make a ![pull request](https://github.com/elB4RTO/craplog-CLI/pulls)\n\nIf you're not running Apache, but you like this tool: same as the above (bring a sample of a log file)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felb4rto%2Fcraplog-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felb4rto%2Fcraplog-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felb4rto%2Fcraplog-cli/lists"}