https://github.com/eliasdabbas/robotstxt_app

Visual App for Testing URLs and User-agents blocked by robots.txt Files
https://github.com/eliasdabbas/robotstxt_app

dashboard plotly-dash python robots-parser robots-txt

Last synced: 2 months ago
JSON representation

Visual App for Testing URLs and User-agents blocked by robots.txt Files

• Host: GitHub
• URL: https://github.com/eliasdabbas/robotstxt_app
• Owner: eliasdabbas
• Created: 2020-06-04T15:14:14.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2022-12-08T10:07:13.000Z (about 2 years ago)
• Last Synced: 2023-03-01T11:11:38.804Z (almost 2 years ago)
• Topics: dashboard, plotly-dash, python, robots-parser, robots-txt
• Language: Python
• Homepage: https://www.dashboardom.com/robotstxt
• Size: 24.4 KB
• Stars: 4
• Watchers: 2
• Forks: 0
• Open Issues: 7
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# robots.txt Tester App

An App for testing URLs and User-agents blocked by robots.txt Files.  

Given a certain robots.txt file, check whether or not any of the provided URLs can be fetched by all the user-agents in the robots.txt file (checks all the combinations).

[Start testing!](https://www.dashboardom.com/robotstxt)