https://github.com/zebbern/dezcrwl

🕷️ | dezcrwl is a website history crawler gather hidden information and check vulnerabilities for extracted .js endpoints & much more!
https://github.com/zebbern/dezcrwl

crawl crawler crawler-python crawlers ctf-tools hacking historical-data information information-gathering information-retrieval information-security infosec osint osint-tool pentesting-tools python reconnaissance tool web website

Last synced: about 1 year ago
JSON representation

🕷️ | dezcrwl is a website history crawler gather hidden information and check vulnerabilities for extracted .js endpoints & much more!

• Host: GitHub
• URL: https://github.com/zebbern/dezcrwl
• Owner: zebbern
• License: mit
• Created: 2025-02-06T17:46:07.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-02-20T18:40:37.000Z (over 1 year ago)
• Last Synced: 2025-04-13T16:51:09.048Z (about 1 year ago)
• Topics: crawl, crawler, crawler-python, crawlers, ctf-tools, hacking, historical-data, information, information-gathering, information-retrieval, information-security, infosec, osint, osint-tool, pentesting-tools, python, reconnaissance, tool, web, website
• Language: Python
• Homepage:
• Size: 105 KB
• Stars: 4
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          




Domain History Crawler

![Python](https://img.shields.io/badge/Python-3.x-blue)

![Status](https://img.shields.io/badge/Status-Active-green)

![License](https://img.shields.io/badge/License-MIT-brightgreen)

dezcrwl is a `Web History Osint` tool for gathering URLs, subdomains, JavaScript endpoints, and sensitive information using various OSINT sources like Wayback Machine, Common Crawl, and VirusTotal.**


---





Features ⁀➴ 






 

- Fetches URLs from Common Crawl, Wayback Machine, and VirusTotal













 

- Detects sensitive information such as API keys and JWT tokens













 

- Generates detailed reports in TXT or JSON format













 

- Extracts JavaScript endpoints and hidden APIs













 

- Discovers subdomains automatically













 

- Supports status code filtering







│


│


🢃






 

$  [Installation For Linux](#linux)


$  [Installation For Windows](#windows)

$  [Usage](#usage) 

$  [Configuration](#configuration) 

$  [Showcase](#showcase)





🢇⁀🐧 ˗˗˗ˏˋ🖳ˎˊ˗˗˗ 🪟⁀🡾

  





Linux/macOS


### Run these commands:





```

git clone https://github.com/zebbern/dezcrwl.git

cd dezcrawl

chmod +x linuxinstall.sh

./linuxinstall.sh

```









 

Windows


### Run these commands:





```

git clone https://github.com/zebbern/dezcrwl.git

cd dezcrawl

windowsinstall.sh

```







### Now run `dezcrwl -h` anywhere in terminal






## Usage:

### Run the script with a domain:

```

python3 dezcrwl.py -t example.com -cw -js -sum -o output.txt

```

### Extract JavaScript endpoints:

```

python3 dezcrwl.py -t example.com -js

```

### Fetch results from Wayback and VirusTotal:

 ```

python3 dezcrwl.py -t example.com -cw -vt YOUR_VIRUSTOTAL_API_KEY

```

### Filter specific file types:

```

python3 dezcrwl.py -t example.com -f "(\.json|\.env|\.bak|\.sql)"

```

### How its supposed to run & with what:

```

dezCrwl target.com -dir -cw -js -ws -sum -o output.txt -f "(\.json|\.env|\.bak|\.backup|\.old|\.git|\.svn|\.swp|\.sql|\.db|\.sqlite|\.log|\.txt|\.zip|\.rar|\.tar\.gz|\.7z|\.pdf|\.docx|\.xlsx|\.conf|\.ini|\.yml|\.yaml|\.dump|\.sql\.dump|\.session|\.pem|\.key|\.crt|\.tmp)"

```

## Configuration:



## `config.yaml`:

- 
verbose: Enables detailed logging (true/false)

 

- 
output_format: Choose "txt" or "json"

 

- 
API keys: Configure URLScan and CertSpotter API keys




## Showcase

Coming....

Developer:

- GitHub: https://github.com/zebbern