{"id":13845539,"url":"https://github.com/jonaslejon/lolcrawler","last_synced_at":"2025-07-12T02:31:55.339Z","repository":{"id":151325127,"uuid":"371801911","full_name":"jonaslejon/lolcrawler","owner":"jonaslejon","description":"Headless web crawler for bugbounty and penetration-testing/redteaming","archived":true,"fork":false,"pushed_at":"2024-03-25T20:29:57.000Z","size":30,"stargazers_count":41,"open_issues_count":0,"forks_count":9,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-08-05T17:44:42.266Z","etag":null,"topics":["bugbounty","crawler","docker","penetration-testing","penetration-testing-tools","redteam","redteam-tools","redteaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonaslejon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-05-28T19:36:12.000Z","updated_at":"2024-07-30T23:25:55.000Z","dependencies_parsed_at":"2023-07-03T10:03:14.940Z","dependency_job_id":null,"html_url":"https://github.com/jonaslejon/lolcrawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonaslejon%2Flolcrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonaslejon%2Flolcrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonaslejon%2Flolcrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonaslejon%2Flolcrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonaslejon","download_url":"https://codeload.github.com/jonaslejon/lolcrawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225784736,"owners_count":17523699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bugbounty","crawler","docker","penetration-testing","penetration-testing-tools","redteam","redteam-tools","redteaming"],"created_at":"2024-08-04T17:03:27.784Z","updated_at":"2024-11-21T18:32:13.846Z","avatar_url":"https://github.com/jonaslejon.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cimg src=\"https://triop.se/wp-content/uploads/2021/05/lolcrawler.png\" alt=\"LOLCrawler!\"\u003e\n\n# lolcrawler 🕷\n\n[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/) \u003cimg alt=\"Docker\" src=\"https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge\u0026logo=docker\u0026logoColor=white\"/\u003e\n\nLolcrawler is a headless browser crawler created for BugBounty and penetration-testing/RedTeaming. Beware, this code is really slow but should be able to run for several days and find some really interesting urls/paths. \n\nThe crawler is using several different methods for trying to find links and urls such as BeautifulSoup, jsbeautifier, urlextract and linkfinder.py.\n\nYou need to install docker to run this crawler due to the headless selenium browser. \n\n## Demo\n\nAnimated GIF running the crawler in verbose mode:\n\n\u003cimg src=\"https://triop.se/wp-content/uploads/2021/05/lolcrawler.gif\" alt=\"lolcrawler demo\" width=\"500\" height=\"300\"\u003e\n\n## Run the crawler\n\n### Option 1 \n\nDownload and run from Docker Hub:\n\n```\ndocker pull jonaslejon/lolcrawler\nmkdir output\ndocker run --rm -v `pwd`/output:/output -it jonaslejon/lolcrawler -u https://tor.triop.se -o /output/crawl.log\n```\n\n### Option 2\n\nDownload, build and run:\n\n```\ngit clone https://github.com/jonaslejon/lolcrawler.git\ncd lolcrawler\ndocker build -t jonaslejon/lolcrawler .\ndocker run --rm -v `pwd`/output:/output -it jonaslejon/lolcrawler -u https://tor.triop.se -o /output/crawl.log\n```\n\n## Build docker image\n\nRun the following command in the folder:\n\n`docker rmi jonaslejon/lolcrawler ; docker build -t jonaslejon/lolcrawler .`\n\n\n## About\n\nI was looking for a tool similar to the Burp Suite crawler that I can run and forget about. Also parts of this code is powering the backend of [WPSec.com](https://wpsec.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonaslejon%2Flolcrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonaslejon%2Flolcrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonaslejon%2Flolcrawler/lists"}