https://github.com/jonaslejon/lolcrawler

Headless web crawler for bugbounty and penetration-testing/redteaming
https://github.com/jonaslejon/lolcrawler

bugbounty crawler docker penetration-testing penetration-testing-tools redteam redteam-tools redteaming

Last synced: 3 months ago
JSON representation

Headless web crawler for bugbounty and penetration-testing/redteaming

• Host: GitHub
• URL: https://github.com/jonaslejon/lolcrawler
• Owner: jonaslejon
• Archived: true
• Created: 2021-05-28T19:36:12.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2024-03-25T20:29:57.000Z (7 months ago)
• Last Synced: 2024-06-06T22:47:22.230Z (5 months ago)
• Topics: bugbounty, crawler, docker, penetration-testing, penetration-testing-tools, redteam, redteam-tools, redteaming
• Language: Python
• Homepage:
• Size: 29.3 KB
• Stars: 39
• Watchers: 5
• Forks: 9
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-hacking-lists - jonaslejon/lolcrawler - Headless web crawler for bugbounty and penetration-testing/redteaming (Python)

README

        


# lolcrawler 🕷

[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/) 

Lolcrawler is a headless browser crawler created for BugBounty and penetration-testing/RedTeaming. Beware, this code is really slow but should be able to run for several days and find some really interesting urls/paths. 

The crawler is using several different methods for trying to find links and urls such as BeautifulSoup, jsbeautifier, urlextract and linkfinder.py.

You need to install docker to run this crawler due to the headless selenium browser. 

## Demo

Animated GIF running the crawler in verbose mode:



## Run the crawler

### Option 1 

Download and run from Docker Hub:

```

docker pull jonaslejon/lolcrawler

mkdir output

docker run --rm -v `pwd`/output:/output -it jonaslejon/lolcrawler -u https://tor.triop.se -o /output/crawl.log

```

### Option 2

Download, build and run:

```

git clone https://github.com/jonaslejon/lolcrawler.git

cd lolcrawler

docker build -t jonaslejon/lolcrawler .

docker run --rm -v `pwd`/output:/output -it jonaslejon/lolcrawler -u https://tor.triop.se -o /output/crawl.log

```

## Build docker image

Run the following command in the folder:

`docker rmi jonaslejon/lolcrawler ; docker build -t jonaslejon/lolcrawler .`

## About

I was looking for a tool similar to the Burp Suite crawler that I can run and forget about. Also parts of this code is powering the backend of [WPSec.com](https://wpsec.com).