{"id":16366670,"url":"https://github.com/centminmod/centminmod-ultimate-bad-bot-blocker","last_synced_at":"2025-03-23T02:32:39.873Z","repository":{"id":80279700,"uuid":"127793934","full_name":"centminmod/centminmod-ultimate-bad-bot-blocker","owner":"centminmod","description":"Ultimate Bad Bot Blocker are for Centmin Mod 123.09beta01","archived":false,"fork":false,"pushed_at":"2018-04-04T11:42:01.000Z","size":14,"stargazers_count":12,"open_issues_count":3,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T16:53:00.543Z","etag":null,"topics":["bot-blocker","bots","centminmod","centos","nginx","referrer-spam","referrer-spam-blocker","spam-blocker","spam-referrer-blocker"],"latest_commit_sha":null,"homepage":"https://centminmod.com/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/centminmod.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-02T18:10:50.000Z","updated_at":"2023-04-14T11:01:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"dfe19f37-ea97-4999-ac71-02086c0b9d55","html_url":"https://github.com/centminmod/centminmod-ultimate-bad-bot-blocker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fcentminmod-ultimate-bad-bot-blocker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fcentminmod-ultimate-bad-bot-blocker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fcentminmod-ultimate-bad-bot-blocker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fcentminmod-ultimate-bad-bot-blocker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/centminmod","download_url":"https://codeload.github.com/centminmod/centminmod-ultimate-bad-bot-blocker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245047977,"owners_count":20552430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bot-blocker","bots","centminmod","centos","nginx","referrer-spam","referrer-spam-blocker","spam-blocker","spam-referrer-blocker"],"created_at":"2024-10-11T02:47:13.938Z","updated_at":"2025-03-23T02:32:39.857Z","avatar_url":"https://github.com/centminmod.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Contents\n\n* [General Notes](#general-notes)\n  * [Support](#support)\n* [Install](#install)\n* [Post Install Checks](#post-install-checks)\n  * [Check Nginx.conf Added Includes](#check-nginxconf-added-include-files)\n    * [nginx.conf fix](#nginxconf-fix)\n  * [Check Vhosts Added Includes](#check-vhosts-added-include-files)\n    * [quick diff](#quick-diff)\n    * [extended diff](#extended-diff)\n* [cronjob](#cronjob)\n* [Example Outputs](#example-output-from-install-commands)\n  * [dry runs](#dry-runs)\n  * [live runs](#live-runs)\n    * [actual installed files](#actual-installed-files)\n* [Testing](#testing)\n* [Rate Limits](#rate-limits)\n* [White Listing](#white-listing)\n* [Black Listing](#black-listing)\n* [Bad Referrers](#bad-referrers)\n* [blockbots.conf](#blockbotsconf)\n* [Customize Configurations](#customize-configurations)\n* [globalblacklist.conf](#globalblacklistconf)\n* [ngxtop](#ngxtop)\n    \n## General Notes\n\nInstallation commands for Mitchell Krog developed [Ultimate Bad Bot Blocker](https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker) are for [Centmin Mod 123.09beta01](https://centminmod.com/) or higher LEMP stack on CentOS 6/7 specifically due to the differences in [Centmin Mod's Nginx structure](https://centminmod.com/configfiles.html). If you have existing [bad bot blocking \u0026 rate limiting](https://community.centminmod.com/threads/blocking-bad-or-aggressive-bots.6433/) setup, you will need to remove or comment out those include files with hash # in front of them first for\n\n* `include /usr/local/nginx/conf/botlimit.conf;` in `/usr/local/nginx/conf/nginx.conf`\n* `include /usr/local/nginx/conf/blockbots.conf;` within each of your Centmin Mod Nginx vhost config files within directory at `/usr/local/nginx/conf/conf.d`\n\nAlso `update-ngxblocker` script and cronjob doesn't seem to update itself from [https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/157#issuecomment-378546793](https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/157#issuecomment-378546793) so for now need to manually update it using command and may need to setup a cronjob for that.\n\n```\nwget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/update-ngxblocker -O /usr/local/sbin/update-ngxblocker \u0026\u0026 chmod 700 /usr/local/sbin/update-ngxblocker\n```\n\n### Support\n\nInstructions below are provided as is with no support provided by me. For issues with false postives blocks etc, you will need to contact the official developer on their [Ultimate Bad Bot Blocker issue tracker](https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues).\n\nThere's a mailing list and Slack Channel too:\n\n* [https://groups.google.com/forum/#!forum/nginx-ultimate-bad-bot-blocker](https://groups.google.com/forum/#!forum/nginx-ultimate-bad-bot-blocker)\n* \u003ca href=\"mailto:mitchellkrog@gmail.com?subject=Please Add Me To the Slack Channel (Nginx Bad Bot Blocker)\"\u003eAsk For an Invite to\u003cimg src=\"https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/.assets/slack.png\"/\u003e\u003c/a\u003e\n\n## Install\n\nActual install commands for Nginx Ultimate Bad Bot Blocker installed at `/usr/local/nginx/conf/ultimate-badbot-blocker` and where the global bad bot blacklisting is contained in `/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf`.\n\n```\n# download and install\nwget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker -O /usr/local/sbin/install-ngxblocker\nchmod +x /usr/local/sbin/install-ngxblocker\nmkdir -p /usr/local/nginx/conf/ultimate-badbot-blocker\n# backup nginx.conf and conf.d directory before install\ncp -a /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf-backup-b4-badbot\ncp -a /usr/local/nginx/conf/conf.d/ /usr/local/nginx/conf/conf.d-backup-b4-badbot\n\n# dry run\ninstall-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\n# live run\ninstall-ngxblocker -x -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\n# fix duplicate directives\nsed -i 's|^server_names_hash_|#server_names_hash_|g' /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf\n\n# dry run\nsetup-ngxblocker -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf\n# live run\nsetup-ngxblocker -x -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf\n```\n\nI'll probably have to write up a script to check and notify me when the Ultimate Bad Bot Blocker installed scripts in `/usr/local/sbin/` are modified from their initial installed versions just to keep track when cron updates change these files\n\n* `/usr/local/sbin/install-ngxblocker`\n* `/usr/local/sbin/setup-ngxblocker`\n* `/usr/local/sbin/update-ngxblocker`\n\nMaybe via checking md5sum hashes\n\n```\nmd5sum /usr/local/sbin/install-ngxblocker | tee install-ngxblocker.md5\n938734ed5ae1001f90930f4b01fa51f8  /usr/local/sbin/install-ngxblocker\n\nmd5sum -c install-ngxblocker.md5\n/usr/local/sbin/install-ngxblocker: OK\n\nmd5sum /usr/local/sbin/setup-ngxblocker | tee setup-ngxblocker.md5\nc5cb4d154caeb956413dadf59072c7aa  /usr/local/sbin/setup-ngxblocker\n\nmd5sum -c setup-ngxblocker.md5\n/usr/local/sbin/setup-ngxblocker: OK\n\nmd5sum /usr/local/sbin/update-ngxblocker | tee update-ngxblocker.md5\n47375490d11e99505931bc3b753ba338  /usr/local/sbin/update-ngxblocker\n\nmd5sum -c update-ngxblocker.md5\n/usr/local/sbin/update-ngxblocker: OK\n```\n\n## Post Install Checks\n\n### check nginx.conf added include files\n\n\nYou can check nginx.conf config file at `/usr/local/nginx/conf/nginx.conf` against the backed up copy at `/usr/local/nginx/conf/nginx.conf-backup-b4-badbot` which you would of backed up in initial install commands above.\n\n`setup-ngxblocker` doesn't setup `/usr/local/nginx/conf/nginx.conf` includes properly as it inserted include files in wrong place in centmin mod nginx.conf outside of http{] context\n\n```\nnginx -t\nnginx: [emerg] \"server_names_hash_bucket_size\" directive is not allowed here in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf:15\nnginx: configuration file /usr/local/nginx/conf/nginx.conf test failed\n```\n\nsdiff side by side compare of backed up copy versus live copy of nginx.conf\n\n```\nsdiff -w 200 -s /usr/local/nginx/conf/nginx.conf-backup-b4-badbot /usr/local/nginx/conf/nginx.conf\n                                                                                                   \u003e            # Bad Bot Blocker\n                                                                                                   \u003e            include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf; \n                                                                                                   \u003e            include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;\n                                                                                                   \u003e  \n```\n\nuniversal diff compare with 10 line context output of backed up copy versus live copy of nginx.conf\n\n```\ndiff -U10 /usr/local/nginx/conf/nginx.conf-backup-b4-badbot /usr/local/nginx/conf/nginx.conf \n--- /usr/local/nginx/conf/nginx.conf-backup-b4-badbot   2018-03-07 01:39:33.843255502 +0000\n+++ /usr/local/nginx/conf/nginx.conf    2018-04-02 16:52:06.659257479 +0000\n@@ -1,11 +1,15 @@\n user              nginx nginx;\n+       # Bad Bot Blocker\n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf; \n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;\n+ \n worker_processes 2;\n worker_priority -10;\n \n worker_rlimit_nofile 260000;\n timer_resolution 100ms;\n \n pcre_jit on;\n include /usr/local/nginx/conf/dynamic-modules.conf;\n```\n\n#### nginx.conf fix\n\nfix is to move the 3 lines within http{} context after existing `variables_hash_max_size 2048;` line\n\n```\nhttp {\n map_hash_bucket_size 128;\n map_hash_max_size 4096;\n server_names_hash_bucket_size 128;\n server_names_hash_max_size 2048;\n variables_hash_max_size 2048;\n\n # Bad Bot Blocker\n include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;\n include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;\n```\n\nthen recheck and test nginx conf and you will get a new error for duplicate `server_names_hash_bucket_size` as `/usr/local/nginx/conf/nginx.conf` also lists this directive and it's being duplicated in thie bad bot installer's include file on line 15 of `/usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf`\n\n```\nnginx -t\nnginx: [emerg] \"server_names_hash_bucket_size\" directive is duplicate in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf:15\nnginx: configuration file /usr/local/nginx/conf/nginx.conf test failed\n```\n\n#### fix duplicate directives\n\ncommented instructions in `/usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf` already mention to comment out line 15 if it conflicts with your existing nginx.conf settings\n\n```\ncat -n /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf\n     1  ##############################################################################                                                                \n     2  #       _  __     _                                                          #\n     3  #      / |/ /__ _(_)__ __ __                                                 #\n     4  #     /    / _ `/ / _ \\\\ \\ /                                                 #\n     5  #    /_/|_/\\_, /_/_//_/_\\_\\                                                  #\n     6  #       __/___/      __   ___       __     ___  __         __                #\n     7  #      / _ )___ ____/ /  / _ )___  / /_   / _ )/ /__  ____/ /_____ ____      #\n     8  #     / _  / _ `/ _  /  / _  / _ \\/ __/  / _  / / _ \\/ __/  '_/ -_) __/      #\n     9  #    /____/\\_,_/\\_,_/  /____/\\___/\\__/  /____/_/\\___/\\__/_/\\_\\\\__/_/         #\n    10  #                                                                            #\n    11  ##############################################################################                                                                \n    12\n    13  # Version 1.1\n    14\n    15  server_names_hash_bucket_size 128;\n    16  server_names_hash_max_size 4096;\n    17  limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;\n    18  limit_conn_zone $binary_remote_addr zone=addr:50m;\n    19\n    20  # ****************************************************************************\n    21  # NOTE: IF you are using a system like Nginx-Proxy from @JWilder\n    22  # ****************************************************************************\n    23  # Repo URL: https://github.com/jwilder/nginx-proxy\n    24  # You will need to comment out the first line here as follows. \n    25  #     #server_names_hash_bucket_size 128;\n    26  # You will also need to modify the nginx.tmpl file to add the default include\n    27  #     include /etc/nginx/conf.d/*\n    28  # ****************************************************************************\n```\n\nSo edit `/usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf` line 15 to comment it out with a hash in front. Also comment out `server_names_hash_max_size` on line 16 as nginx.conf also already has that. Above instructions have been updated for sed edit and commenting out so you wouldn't need to manually do this anyway.\n\n```\nsed -i 's|^server_names_hash_|#server_names_hash_|g' /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf\n```\n\n```\ncat -n /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf\n     1  ##############################################################################                                                                \n     2  #       _  __     _                                                          #\n     3  #      / |/ /__ _(_)__ __ __                                                 #\n     4  #     /    / _ `/ / _ \\\\ \\ /                                                 #\n     5  #    /_/|_/\\_, /_/_//_/_\\_\\                                                  #\n     6  #       __/___/      __   ___       __     ___  __         __                #\n     7  #      / _ )___ ____/ /  / _ )___  / /_   / _ )/ /__  ____/ /_____ ____      #\n     8  #     / _  / _ `/ _  /  / _  / _ \\/ __/  / _  / / _ \\/ __/  '_/ -_) __/      #\n     9  #    /____/\\_,_/\\_,_/  /____/\\___/\\__/  /____/_/\\___/\\__/_/\\_\\\\__/_/         #\n    10  #                                                                            #\n    11  ##############################################################################                                                                \n    12\n    13  # Version 1.1\n    14\n    15  #server_names_hash_bucket_size 128;\n    16  #server_names_hash_max_size 4096;\n    17  limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;\n    18  limit_conn_zone $binary_remote_addr zone=addr:50m;\n    19\n    20  # ****************************************************************************\n    21  # NOTE: IF you are using a system like Nginx-Proxy from @JWilder\n    22  # ****************************************************************************\n    23  # Repo URL: https://github.com/jwilder/nginx-proxy\n    24  # You will need to comment out the first line here as follows. \n    25  #     #server_names_hash_bucket_size 128;\n    26  # You will also need to modify the nginx.tmpl file to add the default include\n    27  #     include /etc/nginx/conf.d/*\n    28  # ****************************************************************************\n```\n\nthen edit `/usr/local/nginx/conf/nginx.conf` to raise the value of `server_names_hash_max_size` from 2048 to 4096\n\n```\nhttp {\n map_hash_bucket_size 128;\n map_hash_max_size 4096;\n server_names_hash_bucket_size 128;\n server_names_hash_max_size 4096;\n variables_hash_max_size 2048;\n```\n\nrecheck nginx config\n\n```\nnginx -t\nnginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok\nnginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful\n```\n\n### check vhosts added include files\n\nYou can check vhosts' config files in `/usr/local/nginx/conf/conf.d` against the backed up copy of them at `/usr/local/nginx/conf/conf.d-backup-b4-badbot` which you would of backed up in initial install commands above.\n\n#### quick diff\n\n```\ndiff -qr /usr/local/nginx/conf/conf.d-backup-b4-badbot /usr/local/nginx/conf/conf.d   \nFiles /usr/local/nginx/conf/conf.d-backup-b4-badbot/demodomain.com.conf and /usr/local/nginx/conf/conf.d/demodomain.com.conf differ\nFiles /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain1.com.conf and /usr/local/nginx/conf/conf.d/domain1.com.conf differ\nFiles /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain2.com.conf and /usr/local/nginx/conf/conf.d/domain2.com.conf differ\nFiles /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.conf and /usr/local/nginx/conf/conf.d/http2.domain1.com.conf differ\nFiles /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.ssl.conf and /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf differ\nFiles /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub1.domain1.com.conf and /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf differ\nFiles /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub2.domain2.com.conf and /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf differ\nFiles /usr/local/nginx/conf/conf.d-backup-b4-badbot/virtual.conf and /usr/local/nginx/conf/conf.d/virtual.conf differ\n```\n\n#### extended diff\n\nfor the most part centmin mod nginx generated vhosts had the include files added in correct place above the first `location /` context with exception of 2 vhost files where added include files weren't added directly above `location /` but were added further up in vhost config files below. Though technically it is still correct so nothing needed to correct.\n\n* `/usr/local/nginx/conf/conf.d/demodomain.com.conf`\n* `/usr/local/nginx/conf/conf.d/virtual.conf`\n\ndiff compare\n\n```\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot /usr/local/nginx/conf/conf.d\n```\n\ndiff compare output\n\n```\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot /usr/local/nginx/conf/conf.d\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/demodomain.com.conf /usr/local/nginx/conf/conf.d/demodomain.com.conf\n--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/demodomain.com.conf   2018-03-07 01:39:33.105393742 +0000\n+++ /usr/local/nginx/conf/conf.d/demodomain.com.conf    2018-04-02 16:52:06.858264720 +0000\n@@ -10,8 +10,12 @@\n server {\n             listen   80;\n             server_name www.demodomain.com;\n \n+       # Bad Bot Blocker\n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; \n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;\n+ \n # limit_conn limit_per_ip 16;\n # ssi  on;\n \n             access_log /home/nginx/domains/demodomain.com/log/access.log ;\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain1.com.conf /usr/local/nginx/conf/conf.d/domain1.com.conf\n--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain1.com.conf      2018-04-02 16:20:00.996652290 +0000\n+++ /usr/local/nginx/conf/conf.d/domain1.com.conf       2018-04-02 16:52:07.268279631 +0000\n@@ -36,8 +36,12 @@\n   # server and/or vhost site\n   #include /usr/local/nginx/conf/cloudflare.conf;\n   include /usr/local/nginx/conf/503include-main.conf;\n \n+       # Bad Bot Blocker\n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; \n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;\n+ \n   location / {\n   include /usr/local/nginx/conf/503include-only.conf;\n \n # block common exploits, sql injections etc\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain2.com.conf /usr/local/nginx/conf/conf.d/domain2.com.conf\n--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/domain2.com.conf      2018-04-02 16:20:11.412014562 +0000\n+++ /usr/local/nginx/conf/conf.d/domain2.com.conf       2018-04-02 16:52:07.466286830 +0000\n@@ -36,8 +36,12 @@\n   # server and/or vhost site\n   #include /usr/local/nginx/conf/cloudflare.conf;\n   include /usr/local/nginx/conf/503include-main.conf;\n \n+       # Bad Bot Blocker\n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; \n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;\n+ \n   location / {\n   include /usr/local/nginx/conf/503include-only.conf;\n \n # block common exploits, sql injections etc\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.conf /usr/local/nginx/conf/conf.d/http2.domain1.com.conf\n--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.conf        2018-04-02 16:24:20.198637890 +0000\n+++ /usr/local/nginx/conf/conf.d/http2.domain1.com.conf 2018-04-02 16:52:08.071308824 +0000\n@@ -36,8 +36,12 @@\n   # server and/or vhost site\n   #include /usr/local/nginx/conf/cloudflare.conf;\n   include /usr/local/nginx/conf/503include-main.conf;\n \n+       # Bad Bot Blocker\n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; \n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;\n+ \n   location / {\n   include /usr/local/nginx/conf/503include-only.conf;\n \n # block common exploits, sql injections etc\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.ssl.conf /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf\n--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/http2.domain1.com.ssl.conf    2018-04-02 16:24:20.205638109 +0000\n+++ /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf     2018-04-02 16:52:08.285316598 +0000\n@@ -65,8 +65,12 @@\n   # server and/or vhost site\n   #include /usr/local/nginx/conf/cloudflare.conf;\n   include /usr/local/nginx/conf/503include-main.conf;\n \n+       # Bad Bot Blocker\n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; \n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;\n+ \n   location / {\n   include /usr/local/nginx/conf/503include-only.conf;\n \n # block common exploits, sql injections etc\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub1.domain1.com.conf /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf\n--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub1.domain1.com.conf 2018-04-02 16:20:26.498539308 +0000\n+++ /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf  2018-04-02 16:52:07.664294028 +0000\n@@ -36,8 +36,12 @@\n   # server and/or vhost site\n   #include /usr/local/nginx/conf/cloudflare.conf;\n   include /usr/local/nginx/conf/503include-main.conf;\n \n+       # Bad Bot Blocker\n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; \n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;\n+ \n   location / {\n   include /usr/local/nginx/conf/503include-only.conf;\n \n # block common exploits, sql injections etc\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub2.domain2.com.conf /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf\n--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/sub2.domain2.com.conf 2018-04-02 16:20:39.443989582 +0000\n+++ /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf  2018-04-02 16:52:07.870301518 +0000\n@@ -36,8 +36,12 @@\n   # server and/or vhost site\n   #include /usr/local/nginx/conf/cloudflare.conf;\n   include /usr/local/nginx/conf/503include-main.conf;\n \n+       # Bad Bot Blocker\n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; \n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;\n+ \n   location / {\n   include /usr/local/nginx/conf/503include-only.conf;\n \n # block common exploits, sql injections etc\ndiff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot/virtual.conf /usr/local/nginx/conf/conf.d/virtual.conf\n--- /usr/local/nginx/conf/conf.d-backup-b4-badbot/virtual.conf  2018-03-07 01:39:33.106393555 +0000\n+++ /usr/local/nginx/conf/conf.d/virtual.conf   2018-04-02 16:52:07.060272069 +0000\n@@ -1,8 +1,12 @@\n server {\n             listen 80 default_server backlog=2048 reuseport fastopen=256;\n             server_name centos7.localdomain;\n             root   html;\n+       # Bad Bot Blocker\n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf; \n+       include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;\n+ \n \n         access_log              /var/log/nginx/localhost.access.log     combined buffer=8k flush=1m;\n         error_log               /var/log/nginx/localhost.error.log      error;\n \n```\n\nOnce all fixed you can restart nginx server\n\n```\nservice nginx restart\n```\n\nor centmin mod cmd shortcut\n\n```\nngxrestart\n```\n\n## cronjob\n\nsetup cronjob via `cronjob -e` command to invoke nano text editor replacing `yourname@youremail.com` with your email address for update notifications. Note there's currently a bug with using emails with Gmail type `+alias` usernames I logged to their issue tracker [https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/157](https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/157)\n\n```\n00 */8 * * * /usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e yourname@youremail.com\n```\n\nyou can do a manual update check too\n\nwithout email notification add `-n`\n\n\n```\n/usr/local/sbin/update-ngxblocker -n -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\n```\n\nexample without email notifictaion\n\n```\n/usr/local/sbin/update-ngxblocker -n -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\n\nLOCAL Version: 3.2018.04.1080\nUpdated: Mon Apr  2 15:35:14 SAST 2018\n\nREMOTE Version: 3.2018.04.1080\nUpdated: Mon Apr  2 15:35:14 SAST 2018\n\nLatest Blacklist Already Installed: 3.2018.04.1080\n```\n\nwith email notification\n\n```\n/usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e yourname@youremail.com\n```\n\nexample with email notification\n\n```\n/usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e yourname@youremail.com\n\nLOCAL Version: 3.2018.04.1080\nUpdated: Mon Apr  2 15:35:14 SAST 2018\n\nREMOTE Version: 3.2018.04.1080\nUpdated: Mon Apr  2 15:35:14 SAST 2018\n\nLatest Blacklist Already Installed: 3.2018.04.1080\n\nEmailing report to: yourname@youremail.com\n```\n\nwhen updates are available\n\n```\n/usr/local/sbin/update-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -e yourname@youremail.com\n\nLOCAL Version: 3.2018.04.1080\nUpdated: Mon Apr  2 15:35:14 SAST 2018\n\nREMOTE Version: 3.2018.04.1082\nUpdated: Tue Apr  3 09:40:52 SAST 2018\n\nUpdate Available =\u003e 3.2018.04.1082\n\nDownloading: globalblacklist.conf ...[OK]\n\nChecking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt\n\nNothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker\nNothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\nNothing to update for directory: /usr/local/sbin\nSetting mode: 700 =\u003e /usr/local/sbin/install-ngxblocker\nSetting mode: 700 =\u003e /usr/local/sbin/setup-ngxblocker\nSetting mode: 700 =\u003e /usr/local/sbin/update-ngxblocker\nUpdating bots.d path: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d =\u003e /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf\n\nReloading NGINX configuration...[OK]\nEmailing report to: yourname@youremail.com\n```\n\n## example output from install commands\n\n### dry runs\n\n```\ninstall-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\nChecking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt\n\n** Dry Run ** | not updating files | run  as 'install-ngxblocker -x' to install files.\n\nCreating directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\n\nREPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master\n\nDownloading [FROM]=\u003e  [REPO]/conf.d/globalblacklist.conf            [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf\nDownloading [FROM]=\u003e  [REPO]/conf.d/botblocker-nginx-settings.conf  [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf\n\nREPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master\n\nDownloading [FROM]=\u003e  [REPO]/bots.d/blockbots.conf              [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf\nDownloading [FROM]=\u003e  [REPO]/bots.d/ddos.conf                   [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf\nDownloading [FROM]=\u003e  [REPO]/bots.d/custom-bad-referrers.conf   [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/custom-bad-referrers.conf\nDownloading [FROM]=\u003e  [REPO]/bots.d/bad-referrer-words.conf     [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/bad-referrer-words.conf\nDownloading [FROM]=\u003e  [REPO]/bots.d/blacklist-domains.conf      [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-domains.conf\nDownloading [FROM]=\u003e  [REPO]/bots.d/blacklist-ips.conf          [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-ips.conf\nDownloading [FROM]=\u003e  [REPO]/bots.d/blacklist-user-agents.conf  [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-user-agents.conf\nDownloading [FROM]=\u003e  [REPO]/bots.d/whitelist-domains.conf      [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-domains.conf\nDownloading [FROM]=\u003e  [REPO]/bots.d/whitelist-ips.conf          [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf\n\nREPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master\n\nDownloading [FROM]=\u003e  [REPO]/setup-ngxblocker      [TO]=\u003e  /usr/local/sbin/setup-ngxblocker\nDownloading [FROM]=\u003e  [REPO]/update-ngxblocker     [TO]=\u003e  /usr/local/sbin/update-ngxblocker\n```\n\n```\nsetup-ngxblocker -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf\nChecking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt\n\n** Dry Run ** | not updating files | run  as 'setup-ngxblocker -x' to setup files.\n\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;            =\u003e /usr/local/nginx/conf/nginx.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;  =\u003e /usr/local/nginx/conf/nginx.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/demodomain.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/demodomain.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/virtual.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/virtual.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/domain2.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/domain2.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/http2.domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/http2.domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf\n\nWhitelisting ip:  xxx.xxx.xxx.xxx   =\u003e /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf\n\nWeb directory not found ('/var/www'): not whitelisting domains.\n\nChecking for missing includes:\n\nChecking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt\n\nNothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker\nNothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\nNothing to update for directory: /usr/local/sbin\nSetting mode: 700 =\u003e /usr/local/sbin/install-ngxblocker\nSetting mode: 700 =\u003e /usr/local/sbin/setup-ngxblocker\nSetting mode: 700 =\u003e /usr/local/sbin/update-ngxblocker\n```\n\n### live runs\n\n```\ninstall-ngxblocker -x -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\nChecking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt\n\nCreating directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\n\nREPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master\n\nDownloading [FROM]=\u003e  [REPO]/conf.d/globalblacklist.conf            [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf...OK\nDownloading [FROM]=\u003e  [REPO]/conf.d/botblocker-nginx-settings.conf  [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf...OK\n\nREPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master\n\nDownloading [FROM]=\u003e  [REPO]/bots.d/blockbots.conf              [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf...OK\nDownloading [FROM]=\u003e  [REPO]/bots.d/ddos.conf                   [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf...OK\nDownloading [FROM]=\u003e  [REPO]/bots.d/custom-bad-referrers.conf   [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/custom-bad-referrers.conf...OK\nDownloading [FROM]=\u003e  [REPO]/bots.d/bad-referrer-words.conf     [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/bad-referrer-words.conf...OK\nDownloading [FROM]=\u003e  [REPO]/bots.d/blacklist-domains.conf      [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-domains.conf...OK\nDownloading [FROM]=\u003e  [REPO]/bots.d/blacklist-ips.conf          [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-ips.conf...OK\nDownloading [FROM]=\u003e  [REPO]/bots.d/blacklist-user-agents.conf  [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-user-agents.conf...OK\nDownloading [FROM]=\u003e  [REPO]/bots.d/whitelist-domains.conf      [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-domains.conf...OK\nDownloading [FROM]=\u003e  [REPO]/bots.d/whitelist-ips.conf          [TO]=\u003e  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf...OK\n\nREPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master\n\nDownloading [FROM]=\u003e  [REPO]/setup-ngxblocker      [TO]=\u003e  /usr/local/sbin/setup-ngxblocker...OK\nDownloading [FROM]=\u003e  [REPO]/update-ngxblocker     [TO]=\u003e  /usr/local/sbin/update-ngxblocker...OK\nSetting mode: 700 =\u003e /usr/local/sbin/install-ngxblocker\nSetting mode: 700 =\u003e /usr/local/sbin/setup-ngxblocker\nSetting mode: 700 =\u003e /usr/local/sbin/update-ngxblocker\n```\n\n```\nsetup-ngxblocker -x -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf\nChecking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt\n\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;            =\u003e /usr/local/nginx/conf/nginx.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;  =\u003e /usr/local/nginx/conf/nginx.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/demodomain.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/demodomain.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/virtual.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/virtual.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/domain2.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/domain2.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/sub1.domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/sub2.domain2.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/http2.domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/http2.domain1.com.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           =\u003e /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf\ninserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                =\u003e /usr/local/nginx/conf/conf.d/http2.domain1.com.ssl.conf\n\nWhitelisting ip:  xxx.xxx.xxx.xxx   =\u003e /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf\n\nWeb directory not found ('/var/www'): not whitelisting domains.\n\nChecking for missing includes:\n\nChecking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt\n\nNothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker\nNothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d\nNothing to update for directory: /usr/local/sbin\nSetting mode: 700 =\u003e /usr/local/sbin/install-ngxblocker\nSetting mode: 700 =\u003e /usr/local/sbin/setup-ngxblocker\nSetting mode: 700 =\u003e /usr/local/sbin/update-ngxblocker\n```\n\n#### actual installed files\n\n```\nls -lah /usr/local/sbin/ | grep ngxblocker\n-rwx------   1 root root 9.5K Apr  2 16:17 install-ngxblocker\n-rwx------   1 root root  13K Apr  2 16:27 setup-ngxblocker\n-rwx------   1 root root  12K Apr  2 16:27 update-ngxblocker\n```\n\n```\nls -lah /usr/local/nginx/conf/ultimate-badbot-blocker\ntotal 240K\ndrwxr-xr-x  3 root root   83 Apr  2 16:27 .\ndrwxr-xr-x. 8 root root 4.0K Apr  2 16:25 ..\n-rw-------  1 root root 1.8K Apr  2 16:27 botblocker-nginx-settings.conf\ndrwxr-xr-x  2 root root 4.0K Apr  2 16:27 bots.d\n-rw-------  1 root root 228K Apr  2 16:27 globalblacklist.conf\n```\n```\nls -lah /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/\ntotal 44K\ndrwxr-xr-x 2 root root 4.0K Apr  2 16:27 .\ndrwxr-xr-x 3 root root   83 Apr  2 16:27 ..\n-rw------- 1 root root 3.5K Apr  2 16:27 bad-referrer-words.conf\n-rw------- 1 root root 2.4K Apr  2 16:27 blacklist-domains.conf\n-rw------- 1 root root 7.0K Apr  2 16:27 blacklist-ips.conf\n-rw------- 1 root root 3.3K Apr  2 16:27 blacklist-user-agents.conf\n-rw------- 1 root root 2.1K Apr  2 16:27 blockbots.conf\n-rw------- 1 root root 2.6K Apr  2 16:27 custom-bad-referrers.conf\n-rw------- 1 root root 1.8K Apr  2 16:27 ddos.conf\n-rw------- 1 root root 2.5K Apr  2 16:27 whitelist-domains.conf\n-rw------- 1 root root 1.7K Apr  2 16:27 whitelist-ips.conf\n```\n\n## Testing\n\nRun the following commands one by one from a terminal on another linux machine against your own domain name. substitute http://domain1.com in the examples below with your REAL domain name. Should respond with HTTP 200 status OK code\n\n```\ncurl -I -A \"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\" http://domain1.com\n```\n\n```\ncurl -I -A \"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\" http://domain1.com\nHTTP/1.1 200 OK\nDate: Mon, 02 Apr 2018 17:54:54 GMT\nContent-Type: text/html; charset=utf-8\nContent-Length: 6040\nLast-Modified: Mon, 02 Apr 2018 16:20:00 GMT\nConnection: keep-alive\nVary: Accept-Encoding\nETag: \"5ac25830-1798\"\nServer: nginx centminmod\nX-Powered-By: centminmod\nAccept-Ranges: bytes\n```\n\nShould respond with: curl: (52) Empty reply from server for bad referrer domain `100dollars-seo.com` which is listed in `/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf`\n\n```\ncurl -I http://domain1.com -e http://100dollars-seo.com\n```\n\n```\ncurl -I http://domain1.com -e http://100dollars-seo.com\ncurl: (52) Empty reply from server\n```\n\n```\ngrep 100dollars /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf\n        \"~*\\b100dollars\\-seo\\.com\\b\"    1;\n```\n\nTest the rate limiting of a bot with user agent = `Baidu` which is on the aggressive rate limiting profile of 2 requests/second specified in `/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf`\n\n\nin `/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf`\n\n```\n# limits for Zone $bad_bot = 2\n# this rate limiting will only take effect if you change any of the bots and change\n# their block value from 1 to 2.\n        limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;\n        limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;\n```\n\nSiege load test with user agent `Baidu` rate limited with a HTTP 503 status code served up to rated limited requests\n\n```\nsiege -b -c10 -r100 -A \"Baidu\" http://domain1.com/\n\nLifting the server siege...\nTransactions:                    277 hits\nAvailability:                  51.87 %\nElapsed time:                 138.37 secs\nData transferred:               1.10 MB\nResponse time:                  4.90 secs\nTransaction rate:               2.00 trans/sec\nThroughput:                     0.01 MB/sec\nConcurrency:                    9.80\nSuccessful transactions:         277\nFailed transactions:             257\nLongest transaction:            5.01\nShortest transaction:           0.00\n```\n\n## Rate Limits\n\nRate limiting configuration settings are located in `/usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf` and `/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf`\n\n* `limit_req_zone` is setup with zone named `flood` with 90 requests/s rate limit with nodelay burst of 200 requests for normal rate limits set to `1` while there's a more aggressive rate limit of 2 requests per second for items set to `2` contained in `/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf`\n* `limit_conn_zone` is setup with connection limit set in `/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf` at 200 simultaneous IP connections\n\nThe global bad bot blacklisting is contained in `/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf`. You can see the source master list at [https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/conf.d/globalblacklist.conf](https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/conf.d/globalblacklist.conf).\n\nNext to each entry is a number 0, 1, 2 or 3 which denote the following:\n\n```\n### Note that: \n### 0 = allowed - no limits\n### 1 = allowed or rate limited less restrictive\n### 2 = rate limited more\n### 3 = block completely\n```\n\nat end of `/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf`\n\n```\ncat /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf\n\n# BAD BOT RATE LIMITING ZONE\n# limits for Zone $bad_bot = 1\n# Nothing Set - you can set a different zone limiter here if you like\n# We issue a 444 response instead to all bad bots.  \n\n# limits for Zone $bad_bot = 2\n# this rate limiting will only take effect if you change any of the bots and change\n# their block value from 1 to 2.\n        limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;\n        limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;\n\n```\n\n```\ncat /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf\n##############################################################################                                                                \n#       _  __     _                                                          #\n#      / |/ /__ _(_)__ __ __                                                 #\n#     /    / _ `/ / _ \\\\ \\ /                                                 #\n#    /_/|_/\\_, /_/_//_/_\\_\\                                                  #\n#       __/___/      __   ___       __     ___  __         __                #\n#      / _ )___ ____/ /  / _ )___  / /_   / _ )/ /__  ____/ /_____ ____      #\n#     / _  / _ `/ _  /  / _  / _ \\/ __/  / _  / / _ \\/ __/  '_/ -_) __/      #\n#    /____/\\_,_/\\_,_/  /____/\\___/\\__/  /____/_/\\___/\\__/_/\\_\\\\__/_/         #\n#                                                                            #\n##############################################################################                                                                \n\n# Version 1.1\n\n#server_names_hash_bucket_size 128;\n#server_names_hash_max_size 4096;\nlimit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;\nlimit_conn_zone $binary_remote_addr zone=addr:50m;\n\n# ****************************************************************************\n# NOTE: IF you are using a system like Nginx-Proxy from @JWilder\n# ****************************************************************************\n# Repo URL: https://github.com/jwilder/nginx-proxy\n# You will need to comment out the first line here as follows. \n#     #server_names_hash_bucket_size 128;\n# You will also need to modify the nginx.tmpl file to add the default include\n#     include /etc/nginx/conf.d/*\n# ****************************************************************************\n```\n\n```\ncat /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf\n#######################################################################\n\n### VERSION INFORMATION #\n###################################################\n### Version: V3.2017.01\n### Updated: Sun Jan  29 11:35:32 SAST 2017\n###################################################\n### VERSION INFORMATION ##\n\n##############################################################################                                                                \n#       _  __     _                                                          #\n#      / |/ /__ _(_)__ __ __                                                 #\n#     /    / _ `/ / _ \\\\ \\ /                                                 #\n#    /_/|_/\\_, /_/_//_/_\\_\\                                                  #\n#       __/___/      __   ___       __     ___  __         __                #\n#      / _ )___ ____/ /  / _ )___  / /_   / _ )/ /__  ____/ /_____ ____      #\n#     / _  / _ `/ _  /  / _  / _ \\/ __/  / _  / / _ \\/ __/  '_/ -_) __/      #\n#    /____/\\_,_/\\_,_/  /____/\\___/\\__/  /____/_/\\___/\\__/_/\\_\\\\__/_/         #\n#                                                                            #\n##############################################################################                                                                \n\n# Author: Mitchell Krog \u003cmitchellkrog@gmail.com\u003e - https://github.com/mitchellkrogza/\n\n# Include this in a vhost file within a server {} block using and include statement like below\n\n# server {\n#                       #Config stuff here\n#                       include /etc/nginx/bots.d/blockbots.conf\n#                       include /etc/nginx/bots.d/ddos.conf\n#                       #Other config stuff here\n#                }\n\n#######################################################################\n\nlimit_conn addr 200;\nlimit_req zone=flood burst=200 nodelay;\n```\n\n## White Listing\n\nDomain and IP whitelisting are set in respective include files at:\n\n* `/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-domains.conf`\n* `/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf`\n\n## Black Listing\n\nDomain and IP blacking are set in respective include files at:\n\n* `/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-domains.conf`\n* `/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-ips.conf`\n\n## Bad Referrers\n\ncustom bad referrers and bad referrer words are set in respective include files at:\n\n* `/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/custom-bad-referrers.conf`\n* `/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/bad-referrer-words.conf`\n\n## Blockbots.conf\n\nThe bad bot blocking logic is located in `/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf`\n\n```\n# Author: Mitchell Krog \u003cmitchellkrog@gmail.com\u003e - https://github.com/mitchellkrogza/\n\n### VERSION INFORMATION #\n###################################################\n### Version: V3.2017.02\n### Updated: Mon Aug  21 11:29:32 SAST 2017\n###################################################\n### VERSION INFORMATION ##\n\n\n##############################################################################                                                                \n#       _  __     _                                                          #\n#      / |/ /__ _(_)__ __ __                                                 #\n#     /    / _ `/ / _ \\\\ \\ /                                                 #\n#    /_/|_/\\_, /_/_//_/_\\_\\                                                  #\n#       __/___/      __   ___       __     ___  __         __                #\n#      / _ )___ ____/ /  / _ )___  / /_   / _ )/ /__  ____/ /_____ ____      #\n#     / _  / _ `/ _  /  / _  / _ \\/ __/  / _  / / _ \\/ __/  '_/ -_) __/      #\n#    /____/\\_,_/\\_,_/  /____/\\___/\\__/  /____/_/\\___/\\__/_/\\_\\\\__/_/         #\n#                                                                            #\n##############################################################################                                                                \n\n# Include this in a vhost file within a server {} block using and include statement like below\n\n# server {\n#                       #Config stuff here\n#                       include /etc/nginx/bots.d/blockbots.conf\n#                       include /etc/nginx/bots.d/ddos.conf\n#                       #Other config stuff here\n#                }\n\n#######################################################################\n\n# BOTS\n# ****\n#limit_conn bot1_connlimit 100;\nlimit_conn bot2_connlimit 10;\n#limit_req  zone=bot1_reqlimitip burst=50;\nlimit_req  zone=bot2_reqlimitip burst=10;\nif ($bad_bot = '3') {\n  return 444;\n  }\n\n# BAD REFER WORDS\n# ***************\nif ($bad_words) {\n  return 444;\n}\n\n\n# REFERERS\n# ********\nif ($bad_referer) {\n  return 444;\n}\n\n# IP BLOCKS\n# *********\nif ($validate_client) {\n  return 444;\n}\n\n#######################################################################\n```\n\n## Customize Configurations\n\nYou can now customize any of the following files below to suit your environment or requirements. These include files never get modified during an update using the auto update script `/usr/local/sbin/update-ngxblocker` outlined in above [cronjob](#cronjob) section so whatever customizations you do here will never be overwritten during an update.\n\n```\n/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf\n/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-domains.conf\n/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-user-agents.conf\n/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-ips.conf\n/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/bad-referrer-words.conf\n/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/custom-bad-referrers.conf\n```\n\n## globalblacklist.conf\n\nNginx Ultimate Bad Bot Blocker is installed at `/usr/local/nginx/conf/ultimate-badbot-blocker` and where the global bad bot blacklisting is contained in `/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf`. You can see the source master list at [https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/conf.d/globalblacklist.conf](https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/conf.d/globalblacklist.conf).\n\nExample of Good bots allowed\n\n```\n# ***********************************************\n# Allow Good User-Agent Strings We Know and Trust\n# ***********************************************\n\n# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###\n  \"~*\\badidxbot\\b\"    0;\n  \"~*\\bAdsBot-Google\\b\"   0;\n  \"~*\\baolbuild\\b\"    0;\n  \"~*\\bbingbot\\b\"   0;\n  \"~*\\bbingpreview\\b\"   0;\n  \"~*\\bDoCoMo\\b\"    0;\n  \"~*\\bduckduckgo\\b\"    0;\n  \"~*\\bfacebookexternalhit\\b\"   0;\n  \"~*\\bFeedfetcher-Google\\b\"    0;\n  \"~*\\bGooglebot\\b\"   0;\n  \"~*\\bGooglebot-Image\\b\"   0;\n  \"~*\\bGooglebot-Mobile\\b\"    0;\n  \"~*\\bGooglebot-News\\b\"    0;\n  \"~*\\bGooglebot/Test\\b\"    0;\n  \"~*\\bGooglebot-Video\\b\"   0;\n  \"~*\\bGoogle-HTTP-Java-Client\\b\"   0;\n  \"~*\\bGravityscan\\b\"   0;\n  \"~*\\bgsa-crawler\\b\"   0;\n  \"~*\\bJakarta\\ Commons\\b\"    0;\n  \"~*\\bKraken/0.1\\b\"    0;\n  \"~*\\bLinkedInBot\\b\"   0;\n  \"~*\\bMediapartners-Google\\b\"    0;\n  \"~*\\bmsnbot\\b\"    0;\n  \"~*\\bmsnbot-media\\b\"    0;\n  \"~*\\bSAMSUNG\\b\"   0;\n  \"~*\\bSlackbot\\b\"    0;\n  \"~*\\bSlackbot-LinkExpanding\\b\"    0;\n  \"~*\\bslurp\\b\"   0;\n  \"~*\\bteoma\\b\"   0;\n  \"~*\\bTwitterBot\\b\"    0;\n  \"~*\\bWordpress\\b\"   0;\n  \"~*\\byahoo\\b\"   0;\n# END GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###\n```\n\nUser Agent strings allowed but rated limited\n\n```\n# ***************************************************\n# User-Agent Strings Allowed Through but Rate Limited\n# ***************************************************\n\n# Some people block libwww-perl, it used widely in many valid (non rogue) agents\n# I allow libwww-perl as I use it for monitoring systems with Munin but it is rate limited\n\n# START ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###\n  \"~*\\bjetmon\\b\"    1;\n  \"~*\\blibwww-perl\\b\"   1;\n  \"~*\\bLynx\\b\"    1;\n  \"~*\\bmunin\\b\"   1;\n  \"~*\\bPresto\\b\"    1;\n  \"~*\\bWget/1.15\\b\"   1;\n# END ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###\n```\n\nUser Agent strings allowed but more aggressive/restrictive rate limiting\n\n```\n# **************************************************************\n# Rate Limited User-Agents who get a bit aggressive on bandwidth\n# **************************************************************\n\n# START LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###\n  \"~*\\bAlexa\\b\"   2;\n  \"~*\\barchive.org\\b\"   2;\n  \"~*\\bBaidu\\b\"   2;\n  \"~*\\bBUbiNG\\b\"    2;\n  \"~*\\bFlipboardProxy\\b\"    2;\n  \"~*\\bia_archiver\\b\"   2;\n  \"~*\\bMSIE\\ 7.0\\b\"   2;\n  \"~*\\bProximic\\b\"    2;\n  \"~*\\bR6_CommentReader\\b\"    2;\n  \"~*\\bR6_FeedFetcher\\b\"    2;\n  \"~*\\bRED/1\\b\"   2;\n  \"~*\\bRPT-HTTPClient\\b\"    2;\n  \"~*\\bsfFeedReader/0.9\\b\"    2;\n  \"~*\\bSpaidu\\b\"    2;\n  \"~*\\bUptimeRobot/2.0\\b\"   2;\n  \"~*\\bYandexBot\\b\"   2;\n  \"~*\\bYandexImages\\b\"    2;\n# END LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###\n```\n\nbad bots listed in `/usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf` section under title:\n\n```\n# *********************************************\n# Bad User-Agent Strings That We Block Outright\n# *********************************************\n```\n\n## ngxtop\n\nYou can use [ngxtop](https://github.com/lebinh/ngxtop) to analyse your Centmin Mod Nginx access logs and keep an eye on Nginx HTTP 444 status codes and also other user agent strings, HTTP status codes etc as outlined [here](https://community.centminmod.com/threads/ngxtop-real-time-metrics-for-nginx.285/).\n\nCentOS 7 ngxtop install\n\n```\nyum -y install python-pip\npip install --upgrade pip\npip install ngxtop\n```\n\nCheck for HTTP 444 status codes `/home/nginx/domains/domain1.com/log/access.log` access log\n\n```\ngrep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow\n```\n\nOnly one entry for 444 exists due to test `domain1.com` test command `curl -I http://domain1.com -e http://100dollars-seo.com` only ran once\n\n```\ngrep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow\nrunning for 0 seconds, 1 records processed: 2434.30 req/sec\n\nSummary:\n|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|---------+------------------+-------+-------+-------+-------|\n|       1 |            0.000 |     0 |     0 |     1 |     0 |\n\nDetailed:\n| request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|----------------+---------+------------------+-------+-------+-------+-------|\n| /              |       1 |            0.000 |     0 |     0 |     1 |     0 |\n```\n\nalternatively you can use native `-i 'status == 444'` flag\n\n```\ncat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444'\n```\n\n```\ncat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444'\nrunning for 0 seconds, 1 records processed: 369.97 req/sec\n\nSummary:\n|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|---------+------------------+-------+-------+-------+-------|\n|       1 |            0.000 |     0 |     0 |     1 |     0 |\n\nDetailed:\n| request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|----------------+---------+------------------+-------+-------+-------+-------|\n| /              |       1 |            0.000 |     0 |     0 |     1 |     0 |\n```\n\nprint request url, HTTP Status code and http user agent\n\n```\ncat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444' print request status http_user_agent\n```\n\n```\ncat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444' print request status http_user_agent\nrunning for 0 seconds, 1 records processed: 335.92 req/sec\n\nrequest, status, http_user_agent:\n| request         |   status | http_user_agent   |\n|-----------------+----------+-------------------|\n| HEAD / HTTP/1.1 |      444 | curl/7.29.0       |\n```\n\ngroup by remote IP address via `--group-by remote_addr`\n\n```\ngrep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by remote_addr\n```\n\noutput where `192.168.0.1` is the visitor ip address\n\n```\ngrep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by remote_addr\nrunning for 0 seconds, 1 records processed: 1597.22 req/sec\n\nSummary:\n|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|---------+------------------+-------+-------+-------+-------|\n|       1 |            0.000 |     0 |     0 |     1 |     0 |\n\nDetailed:\n| remote_addr   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|---------------+---------+------------------+-------+-------+-------+-------|\n| 192.168.0.1 |       1 |            0.000 |     0 |     0 |     1 |     0 |\n```\n\ngroup by user agent string via `--group-by http_user_agent`\n\n```\ngrep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent\n```\n\nthe test command was run via curl hence curl user agent string\n\n```\ngrep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent\nrunning for 0 seconds, 1 records processed: 1402.31 req/sec\n\nSummary:\n|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|---------+------------------+-------+-------+-------+-------|\n|       1 |            0.000 |     0 |     0 |     1 |     0 |\n\nDetailed:\n| http_user_agent   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|-------------------+---------+------------------+-------+-------+-------+-------|\n| curl/7.29.0       |       1 |            0.000 |     0 |     0 |     1 |     0 |\n```\n\nfilter by specific date using grep i.e. April 2nd would filter by `'02/Apr'`\n\n```\ngrep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | grep '02/Apr' | ngxtop --no-follow --group-by http_user_agent\n```\n\n```\ngrep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | grep '02/Apr' | ngxtop --no-follow --group-by http_user_agent\nrunning for 0 seconds, 1 records processed: 1742.54 req/sec\n\nSummary:\n|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|---------+------------------+-------+-------+-------+-------|\n|       1 |            0.000 |     0 |     0 |     1 |     0 |\n\nDetailed:\n| http_user_agent   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|-------------------+---------+------------------+-------+-------+-------+-------|\n| curl/7.29.0       |       1 |            0.000 |     0 |     0 |     1 |     0 |\n```\n\nWith Baidu user agent string rate limited results for HTTP 503 status codes in acces log\n\n```\ncat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow\nrunning for 1 seconds, 6081 records processed: 7868.07 req/sec\n\nSummary:\n|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|---------+------------------+-------+-------+-------+-------|\n|    6081 |         2060.460 |   373 |     0 |    81 |  5627 |\n\nDetailed:\n| request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|----------------+---------+------------------+-------+-------+-------+-------|\n| /              |    6081 |         2060.460 |   373 |     0 |    81 |  5627 |\n```\n\n```\ncat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent   \nrunning for 1 seconds, 6153 records processed: 7625.33 req/sec\n\nSummary:\n|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|---------+------------------+-------+-------+-------+-------|\n|    6153 |         2056.997 |   432 |     0 |    87 |  5634 |\n\nDetailed:\n| http_user_agent   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |\n|-------------------+---------+------------------+-------+-------+-------+-------|\n| Baidu             |    6151 |         2057.666 |   431 |     0 |    86 |  5634 |\n| curl/7.29.0       |       2 |            0.000 |     1 |     0 |     1 |     0 |\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentminmod%2Fcentminmod-ultimate-bad-bot-blocker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcentminmod%2Fcentminmod-ultimate-bad-bot-blocker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentminmod%2Fcentminmod-ultimate-bad-bot-blocker/lists"}