https://github.com/awnumar/spider
https://github.com/awnumar/spider
Last synced: 12 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/awnumar/spider
- Owner: awnumar
- Created: 2021-01-15T02:12:15.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2021-01-21T21:00:13.000Z (over 5 years ago)
- Last Synced: 2025-03-18T11:12:29.271Z (over 1 year ago)
- Language: Go
- Size: 9.77 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Spider
An aggressive, multi-threaded spider for discovering all reachable HTTP endpoints on a host.
## Usage
1. `go build .`
2. `./spider -help`
## Notes
- Some sites or WAFs may block this tool for sending too many requests too quickly. A mitigation strategy is to use the `-frequency` flag to limit the rate of requests.
- Side effect free functions are unit tested but integration tests would also be useful if this was to be deployed in production.
- The current status indication features are not great. There's a debug output that shows requests being made and any errors, and results are printed after crawling is finished. A better way would be to refresh the screen with information such as links visited, links in queue, unique links discovered, etc. Also, a sitemap could be generated instead of links being printed out.