Projects in Awesome Lists tagged with googlebot

https://github.com/lord-alfred/ipranges

🔨 List all IP ranges from: Google (Cloud & GoogleBot), Bing (Bingbot), Amazon (AWS), Microsoft, Oracle (Cloud), GitHub, Facebook (Meta), OpenAI (GPTBot) and other with daily updates.

amazon-aws azure bing bingbot bug-bounty bugbounty cidr cidr-ranges digitalocean facebook google-cloud googlebot ip-ranges iplist microsoft network-security oracle osint pentesting twitter

Last synced: 11 Apr 2025

https://github.com/dhruvldrp9/google-meet-bot

This project is a Python bot that automates the process of logging into Gmail, joining a Google Meet, recording the audio of the meeting, and then generating a summary, key points, action items, and sentiment analysis of the meeting.

audio-processing google google-api googlebot googlemeet googlemeetbot googlemeetingbot meet meetbot

Last synced: 16 Aug 2025

https://github.com/mantoufan/yzhanip

Crawl, match, parse IP or IP range, check if IP or range is in another range. Support IPv4, IPv6, IP Interval, Wildcard and CIDR. Check if IP is Cloudflare node IP, Google bot IP. 爬取，正则匹配，解析 IP 和 IP 范围，检测 IP 或范围是否在另一个范围中。支持 IPv4，IPv6，区间、通配符或 CIDR 表示的 IP 范围。检测 IP 是否是 Cloudflare 节点或 Google 漫游器 IP

cloudflare googlebot ipv4 ipv6

Last synced: 07 Sep 2025

https://github.com/devsnek/googlebot

google for discord, yo

discord-api google googlebot

Last synced: 20 Sep 2025

https://github.com/alaz/legitbot

🤔 Is this Web request from a real search engine🕷 or from an impersonating agent 🕵️‍♀️?

bot detect-crawlers fake googlebot impersonation protection ruby ruby-gem search-engine security

Last synced: 25 Apr 2026

https://github.com/chinmayrane16/scraping-amazon-for-mobile-details-with-scrapy

Scraping Amazon website using Proxies for extracting Mobile details

amazon-scraper crawler googlebot json proxy pycharm pypiwin32 scrapy user-agents

Last synced: 18 Mar 2025

https://github.com/heyxyz/sitemap

Hey sitemap

bingbot googlebot seo sitemap webmaster yandexbot

Last synced: 04 Mar 2025

https://github.com/macropin/django-googlebot

Django middleware to allow googlebot to index logged-in accessible content.

django googlebot python

Last synced: 05 Sep 2025

https://github.com/mcnaveen/googlebot-nginx-telegram-notifier

Shell Script for Monitoring GoogleBot Crawls via Nginx log file and Sending notification to Telegram

bash bash-script googlebot hacktoberfest hactoberfest2021 nginx shell shell-script telegram telegram-api

Last synced: 02 Sep 2025

https://github.com/danielmorell/se_bot_checker

Validate search engine user agents and IP addresses.

crawler googlebot python search-engine spider

Last synced: 15 Apr 2025

https://github.com/scottstraughan/simple-python-url-crawler

Super simple Python3 website URL scraper/crawler. Multi-threaded.

crawler googlebot lightweight link-collection multi-threaded python python3 scraper simple

Last synced: 29 Jul 2025

https://github.com/sefinek/known-bots-ip-whitelist

A whitelist of trusted IP addresses used by legitimate crawlers and services such as Googlebot, Bingbot, AhrefsBot, UptimeRobot, Pingdom, Cloudflare, Bunny CDN, Stripe, Shodan, FacebookBot, TelegramBot, etc.

bot bots crawler firewall goodbot goodbots googlebot ip-address ip-addresses ipset safe safe-bots safety security whitelist whitelist-bot whitelists

Last synced: 21 Jun 2025

https://github.com/infinitnet/wordpress-404-redirect

WordPress plugin to redirect 404 URLs to a specified URL.

googlebot seo wordpress-plugin wordpress-seo

Last synced: 30 Oct 2025

https://github.com/ipanalytics/crawlerscope

Interactive crawler IP intelligence dashboard for search, AI, and user-triggered fetchers.

ai-bots ai-crawlers bingbot bot-detection cidr crawler crawler-detection data-visualization github-pages googlebot gptbot ip-ranges nginx open-data osint robots-txt threat-intelligence waf web-security

Last synced: 09 Jun 2026

https://github.com/mcnaveen/googlebot-webhook-trigger

Fire a webhook when GoogleBot visits your website.

bash bash-script googlebot googlecrawl hacktoberfest webhook

Last synced: 05 May 2026

https://github.com/cnlangzi/botrate

🛡️ An GEO/SEO-friendly rate limiter for Go. Protects your server from abuse/DDoS while safely whitelisting verified Search Engines & AI Bots (Google, Bing, GPTBot, etc.) via IP/RDNS verification.

ai ai-robots geo-robots googlebot gptbot ratelimit ratelimiter seo-robots seo-tools

Last synced: 23 Jan 2026

https://github.com/nzambello/volto-staticrender-bots

Volto addon that adds an express middleware for static rendering for bots

bots googlebot plone puppeteer ssr volto volto-addon

Last synced: 16 Apr 2026

https://github.com/spithash/modsecurity-trusted-bot-ips

A Bash script that automatically fetches and parses JSON-based IP range data for trusted search engine bots (Googlebot, Bingbot, and others), ideal for use with ModSecurity and other web application firewalls and web servers.

apache bash bingbot googlebot ip-ranges modsecurity nginx security waf web-application-firewall