{"id":29195422,"url":"https://github.com/spithash/modsecurity-trusted-bot-ips","last_synced_at":"2026-04-13T21:05:58.728Z","repository":{"id":301526309,"uuid":"1009520171","full_name":"spithash/modsecurity-trusted-bot-ips","owner":"spithash","description":"A Bash script that automatically fetches and parses JSON-based IP range data for trusted search engine bots (Googlebot, Bingbot, and others), ideal for use with ModSecurity and other web application firewalls and web servers.","archived":false,"fork":false,"pushed_at":"2025-06-27T10:20:14.000Z","size":24,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-27T10:34:19.029Z","etag":null,"topics":["apache","bash","bingbot","googlebot","ip-ranges","modsecurity","nginx","security","waf","web-application-firewall"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spithash.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-27T09:09:05.000Z","updated_at":"2025-06-27T10:20:18.000Z","dependencies_parsed_at":"2025-06-27T10:34:20.307Z","dependency_job_id":"4b663b55-d1e0-428c-814d-600466fb62c7","html_url":"https://github.com/spithash/modsecurity-trusted-bot-ips","commit_stats":null,"previous_names":["spithash/modsecurity-trusted-bot-ips"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/spithash/modsecurity-trusted-bot-ips","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spithash%2Fmodsecurity-trusted-bot-ips","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spithash%2Fmodsecurity-trusted-bot-ips/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spithash%2Fmodsecurity-trusted-bot-ips/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spithash%2Fmodsecurity-trusted-bot-ips/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spithash","download_url":"https://codeload.github.com/spithash/modsecurity-trusted-bot-ips/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spithash%2Fmodsecurity-trusted-bot-ips/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263077630,"owners_count":23410167,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache","bash","bingbot","googlebot","ip-ranges","modsecurity","nginx","security","waf","web-application-firewall"],"created_at":"2025-07-02T05:04:23.124Z","updated_at":"2026-04-13T21:05:53.692Z","avatar_url":"https://github.com/spithash.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Modsecurity Trusted Bot IPs JSON Parser and Whitelist Generator\n\nA Bash script that fetches and parses JSON-based IP range data for trusted search engine bots (Googlebot, Bingbot, and others), ideal for use with ModSecurity and other web application firewalls and web servers (nginx, apache).\nI use this for creating a .txt file and then whitelist all the entries in modsecurity's whitelist.conf\n\n---\n\n## Features\n\n- Retrieves the official IP ranges published by Google and Bing.\n- Handles both IPv4 and IPv6 addresses.\n- Creates a clean, deduplicated list that works perfectly for ModSecurity IP whitelisting.\n- Includes helpful debug information like HTTP response codes and how many IPs were found.\n- Designed to be easy to automate using cron jobs or systemd timers.\n\n---\n\n## Requirements\n\n- `bash` (tested on Linux)\n- `curl` — for crawling the JSON urls and handling http statuses.\n- `jq` — for parsing JSON (install via your package manager, e.g. `sudo apt install jq`)\n\n---\n\n## Installation\n\n1. Clone the repository:\n\n   ```bash\n   git clone https://github.com/spithash/modsecurity-trusted-bot-ips.git\n   cd modsecurity-trusted-bot-ips\n2. Make the script executable:\n\n   ```bash\n   chmod +x update-trusted-bot-ips\n## Usage\n\nRun the script manually:\n\n```bash\nsudo ./update-trusted-bot-ips\n```\n\n## Usage in your whitelist.conf (works also for apache or nginx)\nMake sure you change the id '1007' to a unique id in case you have the same somewhere in your rules.\n```bash\nSecRule REMOTE_ADDR \"@pmFromFile /etc/modsecurity/google_bing_googlebot_ips.txt\" \"phase:1,nolog,allow,ctl:ruleEngine=Off,id:1007\"\n```\n\n## Automate with Cron\n\nTo keep your IP whitelist up to date, add a cron job:\n\n```cron\n0 2 * * * /usr/local/bin/update-trusted-bot-ips \u003e\u003e /var/log/update-trusted-bot-ips.log 2\u003e\u00261\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspithash%2Fmodsecurity-trusted-bot-ips","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspithash%2Fmodsecurity-trusted-bot-ips","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspithash%2Fmodsecurity-trusted-bot-ips/lists"}