https://github.com/cobbzilla/botalyzer
analyze robots.txt files across a domain
https://github.com/cobbzilla/botalyzer
Last synced: 4 months ago
JSON representation
analyze robots.txt files across a domain
- Host: GitHub
- URL: https://github.com/cobbzilla/botalyzer
- Owner: cobbzilla
- License: apache-2.0
- Created: 2022-07-27T15:31:43.000Z (almost 4 years ago)
- Default Branch: master
- Last Pushed: 2022-08-02T15:01:14.000Z (almost 4 years ago)
- Last Synced: 2024-12-31T09:47:15.211Z (over 1 year ago)
- Language: Shell
- Size: 50.8 KB
- Stars: 2
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
Robots Analyzer
===============
Given a domain name, this tool will:
* Discover subdomains
* Look for a robots.txt file on each subdomain
* Generate some analytics on the contents of the robots.txt files
# Setup
* In the `scripts` directory, copy the `botalyzer.example.env` to `botalyzer.env`
* Edit this file with the settings for your preferred DNS service
# Command Line Usage
./scripts/analyze.sh
# Node Express App
First build the docker image:
npm run docker-build
Then run the docker container:
npm run docker-run
To generate the JSON summary for a domain, request the URL `http://127.0.0.1:3000/example.com`
For example:
curl http://127.0.0.1:3000/example.com