https://github.com/kant01ne/robotstxt

"robotstxt" is a tool that will build URLs based on the robots.txt file of the host provided.
https://github.com/kant01ne/robotstxt

Last synced: 22 days ago
JSON representation

"robotstxt" is a tool that will build URLs based on the robots.txt file of the host provided.

• Host: GitHub
• URL: https://github.com/kant01ne/robotstxt
• Owner: kant01ne
• License: mit
• Created: 2020-05-20T14:06:11.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2020-06-03T12:59:32.000Z (over 4 years ago)
• Last Synced: 2024-04-24T20:45:30.853Z (8 months ago)
• Language: Go
• Size: 4.88 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# robotstxt

`robotstxt` is a tool that will build URLs based on the robots.txt file of the host provided.

# Install

```

▶ go get -u github.com/NkxxkN/robotstxt

```

# Basic Usage

```

▶ echo "https://yahoo.com" | robotstxt

https://yahoo.com/p/

https://yahoo.com/r/

https://yahoo.com/bin/

https://yahoo.com/includes/

https://yahoo.com/blank.html

https://yahoo.com/_td_api

https://yahoo.com/_tdpp_api

https://yahoo.com/_remote

https://yahoo.com/_multiremote

https://yahoo.com/_tdhl_api

https://yahoo.com/digest

https://yahoo.com/fpjs

https://yahoo.com/myjs

```

## Concurrency

You can set the concurrency level with the -c flag (default = 20):

```

▶ cat domains.txt | robotstxt -c 50

```

## Timeout

You can change the timeout by using the -t flag and specifying a timeout in milliseconds (default = 10000):

```

▶ cat domains.txt | robotstxt -t 20000

```

# Credits

This tool was 100% inspired from [TomNomNom](https://github.com/tomnomnom) tools <3