https://github.com/commoncrawl/ccbot-blocking-analysis

https://github.com/commoncrawl/ccbot-blocking-analysis

Last synced: 9 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/commoncrawl/ccbot-blocking-analysis
• Owner: commoncrawl
• Created: 2026-06-17T19:39:42.000Z (9 days ago)
• Default Branch: main
• Last Pushed: 2026-06-17T21:13:47.000Z (9 days ago)
• Last Synced: 2026-06-17T23:13:08.903Z (9 days ago)
• Language: Python
• Size: 21.5 KB
• Stars: 0
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# ccbot blocking analysis

This repo contains a prototype that analyzes crawl output, looking to

quantify how much blocking of CCBot is due to bot defenses, vs.

disallows in robots.txt files.

Please see OPEN-ATHENA-PILOT.md for an example analysis.