https://github.com/omarhashem123/venom

Tool designed for fast crawl and extract endpoints
https://github.com/omarhashem123/venom

crawler python python3 spider

Last synced: 21 days ago
JSON representation

Tool designed for fast crawl and extract endpoints

• Host: GitHub
• URL: https://github.com/omarhashem123/venom
• Owner: omarhashem123
• Created: 2021-01-23T13:00:56.000Z (almost 4 years ago)
• Default Branch: main
• Last Pushed: 2021-04-05T11:24:13.000Z (over 3 years ago)
• Last Synced: 2024-08-05T17:45:16.376Z (4 months ago)
• Topics: crawler, python, python3, spider
• Language: Python
• Homepage: https://eg.linkedin.com/in/omar-1-hashem
• Size: 155 KB
• Stars: 15
• Watchers: 1
• Forks: 4
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-hacking-lists - omarhashem123/venom - Tool designed for fast crawl and extract endpoints (Python)

README

        
# venom

Tool designed for fast crawl and extract endpoints

# Features

###### 1- Fast web crawler for endpoint or unlike other tools you can crawl big list of endpoints

###### 2- Extract allowed and disallowed endpoints from robots.txt for big list of endpoints that you can use disallowed endpoints with other tools to bypass 403 Status_Code to get quick money easily

###### 3- Fast extracting endpoints from sitemap.xml for list of endpoints 

 

###### 4- So if you faced any problem with tool or your machine have connection problem with internet or you closed tool you don't need to crawl all endpoints again but you can go to file that contain failed threads and complete your crawl where tool stopped from failed threads file

###### 5- Any failed or lack threads you will find it in failed threads file so you can crawl it again

###### 6- randomize User-Agent 

###### 7- Extract subdomains from crt.sh and certspotter.com and crawl it

###### 8- Extract endpoints from Wayback Machine for subdomain or list of subdomain

###### 9- if you used a lot of threads and WAF detect it you can bypass it by taking control of threads and time to delay between threads

###### 10- Crawl enpoints authenticated or unauthenticated

###### 11- You can use it in linux or windows

# Installation

#### 1- Install python3 

#### 2- Run this commands

    git clone https://github.com/omarhashem123/venom.git

    pip install -r requirements.txt

# Usage

    $ python3 venom.py

    usage: 

    to extract endpoints from crawl:

    python3 venom.py -f  -t  -o  -l  -s  -time  -spider

    OR

    python3 venom.py -f  -t  -o  -l  -s  -time  -cookie  -spider

    to extract disallowed endpoinds and allowed endpoints from robots.txt:

    python3 venom.py -f  -t  -o  -time  -robots

    to extract endpoints from sitemap.xml:

    python3 venom.py -f  -t  -o  -time  -sitemap

    to extract endpoints from wayback machine:

    python3 venom.py -d  -o  -archive

    OR

    python3 venom.py -f  -t  -o  -archive

    to extract subdomains from crt.sh:

    python3 venom.py -d  -o  -crtsh

    to extract subdomains from certspotter:

    python3 venom.py -d  -o  -certspotter

    description:Tool that spider subdomains and extract endpoints

    optional arguments:

  

    -h, --help            show this help message and exit

  

    -f F                  

  

    -d D                  

  

    -s S                  

  

    -time [TIME]          

  

    -t [T]                

  

    -l L                  

  

    -o [O]                

  

    -spider [SPIDER]      

  

    -cookie COOKIE        

  

    -robots [ROBOTS]      

  

    -sitemap [SITEMAP]    

  

    -archive [ARCHIVE]    

  

    -crtsh [CRTSH]        

  

    -certspotter [CERTSPOTTER]

                        

# Screenshot

![](/Screenshot.png)