{"id":20591009,"url":"https://github.com/eduardozepeda/go-web-crawler","last_synced_at":"2026-04-16T20:33:08.017Z","repository":{"id":107452201,"uuid":"567431760","full_name":"EduardoZepeda/go-web-crawler","owner":"EduardoZepeda","description":"A concurrent web crawler written in go that looks for exposed .git and .env uris.","archived":false,"fork":false,"pushed_at":"2024-02-16T18:37:48.000Z","size":26,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-05T01:35:23.742Z","etag":null,"topics":["crawler","environment-variables","git","go","pentesting","security-audit"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EduardoZepeda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-11-17T19:25:37.000Z","updated_at":"2023-11-03T17:38:00.000Z","dependencies_parsed_at":"2024-02-16T19:50:30.373Z","dependency_job_id":null,"html_url":"https://github.com/EduardoZepeda/go-web-crawler","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/EduardoZepeda/go-web-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EduardoZepeda%2Fgo-web-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EduardoZepeda%2Fgo-web-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EduardoZepeda%2Fgo-web-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EduardoZepeda%2Fgo-web-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EduardoZepeda","download_url":"https://codeload.github.com/EduardoZepeda/go-web-crawler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EduardoZepeda%2Fgo-web-crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31903718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"ssl_error","status_checked_at":"2026-04-16T18:21:47.142Z","response_time":69,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","environment-variables","git","go","pentesting","security-audit"],"created_at":"2024-11-16T07:39:03.353Z","updated_at":"2026-04-16T20:33:07.959Z","avatar_url":"https://github.com/EduardoZepeda.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Go crawler for .env and .git\n\nThis crawler was inspired by [scanning-26-million-domains-for-exposed-env-files](https://hackernoon.com/scanning-26-million-domains-for-exposed-env-files) article. It uses go's concurrency and worker pool pattern to crawl a list of domains and check for exposed .env and .git uris, in plain or www subdomains.\n\n## Quickstart\n\n### Clone the project and install dependencies\n\nAs you would with any go project, you can run it directly or compile it to produce a binary file.\n\n ```bash\ngit clone https://github.com/eduardoZepeda/go-web-crawler\ncd go-web-crawler/\ngo mod tidy\n ```\n\n### You need a file with urls\n\nYou will need a file with domain names separated by newlines. The default expected file name is *urls.txt*, located at the root of the project. Please see flag options to specify another file name.\n\nFor instance, consider a file named *urls.txt* at the root of the project or executable with the following domains:\n\n ```bash\nexample.org\ndomain.org\n ```\n \n The crawler will request the following urls:\n \n ```bash\nhttps://www.example.org/.env\nhttps://www.example.org/.git\nhttps://example.org/.env\nhttps://example.org/.git\n# ...\n ```\n\n ### Run the crawler\n\n Once you got the file set, you can start crawling with:\n\n ```bash\ngo run main.go\n ```\n\n You can also execute the previously generated binary file\n\n ### Getting the results\n\n A list of all vulnerable sites will be printed at the end of the code execution. If nothing was found, no output will be shown.\n \n ## Acceptance criteria\n \n The crawler consider a successful response any response with a code status between 200 and 300. The crawler ignores any redirection.\n\n Note: I'm aware this could lead to some false positives. Please modify the code according to your own needs.\n \n ## Flag options\n \n The following options can be used to customize the crawler behaviour.\n \n - logLevel: The log Level. Valid values: from 1 to 6, ascending verbosity\n - concurrent: Max number of concurrent requests. Default to 150\n - reqTimeout: Timeout (in seconds) before http request is aborted. Default to 5\n - connTimeout: Timeout (in seconds) before opening a new http connection. Default to 10\n - sleep: Timeout (in seconds) to sleep per worker or connection before the crawler makes a a new request.\n - file: Route of the file containing the urls to crawl, separated by newlines. Default to *urls.txt* at root of the project.\n - showResults: Show a summary of the urls with possible exposed .git or .env uris at the end of the execution process.\n\n ## Quick start with Docker\n\n To run this project using docker just follow the usual procedure. Make sure the urls file exists at the root of the project.\n\n ```bash\n docker build . -t \u003cname\u003e\n ``` \n\n After that you can run the crawler. You're free to pass any number of flags and its corresponding values.\n\n ```bash\n docker run --rm \u003cname\u003e [\u003cflag\u003e=\u003cvalue\u003e...]\n ``` \n\n ## Disclaimer\n \n I do not endorse or encourage the use of this crawler to engage in illegal pentesting. Before using this crawler make sure you got the proper written permission from the website's owner and make sure to consult with your lawyer.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feduardozepeda%2Fgo-web-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feduardozepeda%2Fgo-web-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feduardozepeda%2Fgo-web-crawler/lists"}