Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/itszeeshan/crawlinit
A web crawler written in python3
https://github.com/itszeeshan/crawlinit
appsec bugbounty bugbounty-tool bugbountytips crawler crawler-python enumeration infosec python recon reconnaissance scanner url web
Last synced: 3 months ago
JSON representation
A web crawler written in python3
- Host: GitHub
- URL: https://github.com/itszeeshan/crawlinit
- Owner: itszeeshan
- Created: 2022-05-26T13:06:16.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-02-21T13:36:03.000Z (10 months ago)
- Last Synced: 2024-02-21T14:41:58.355Z (10 months ago)
- Topics: appsec, bugbounty, bugbounty-tool, bugbountytips, crawler, crawler-python, enumeration, infosec, python, recon, reconnaissance, scanner, url, web
- Language: Python
- Homepage:
- Size: 23.4 KB
- Stars: 4
- Watchers: 2
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# About Crawl in it
Crawl_in_it is a python tool designed to discover URLs. It will help penteration tester and bugbounty hunters to save time, so that they don't have to go manually through all the pages. This tool will generate logs which can helps penetration tester and bugbounty hunters retrieve the results of actions they performed in the past and also will allow them to see the status codes and length of the pages, the tool has crawled through.
# Screeshots
# Installation
```
git clone https://github.com/itszeeshan/crawlinit.git
```
# Recommended Python
Crawl in it currently only supports python3.
* The recommended version for python3 is 3.10.x
# Installing Dependencies
### Installing on Windows
```
python -m pip install -r requirements.txt
```
### Installing on Linux
```
sudo pip install -r requirements.txt
```
### Installing Modules independently
```
pip install numpy
pip install argparse
pip install torch
pip install traitlets
pip install bs4
pip install requests
pip install typing_extensions
```
# Usage
Short | Long | Description
------|-----------|-------------
-d | --domain | Domain to crawl into
-o | --output | Save results to file
-h | --help | Shows help message and exit
### For Example
* to generate help message
```
python Crawler.py -h
```
* Specifying domain for crawling
```
python Crawler.py -d hackerone.com
```
* Specifying domain for crawling and saving results into a file
```
python Crawler.py -d hackerone.com -o hackerone.txt
```
### The tool will automatically generate logs!