{"id":13541933,"url":"https://github.com/jaeles-project/gospider","last_synced_at":"2025-05-14T10:08:15.592Z","repository":{"id":37019495,"uuid":"235503516","full_name":"jaeles-project/gospider","owner":"jaeles-project","description":"Gospider - Fast web spider written in Go","archived":false,"fork":false,"pushed_at":"2024-04-21T08:12:13.000Z","size":11014,"stargazers_count":2685,"open_issues_count":56,"forks_count":325,"subscribers_count":51,"default_branch":"master","last_synced_at":"2025-04-10T04:22:05.675Z","etag":null,"topics":["bugbounty","crawler","go","gospider","spider"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaeles-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":null,"patreon":"j3ssie","open_collective":"jaeles-project","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":["https://paypal.me/j3ssiejjj"]}},"created_at":"2020-01-22T05:13:57.000Z","updated_at":"2025-04-07T14:26:02.000Z","dependencies_parsed_at":"2023-01-17T13:45:16.168Z","dependency_job_id":"3f89e6e0-be43-4f0d-8f08-74999bf3587a","html_url":"https://github.com/jaeles-project/gospider","commit_stats":{"total_commits":101,"total_committers":9,"mean_commits":"11.222222222222221","dds":"0.28712871287128716","last_synced_commit":"f6cc9a78d709e088e55ce62e5e92ab063f1a184b"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaeles-project%2Fgospider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaeles-project%2Fgospider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaeles-project%2Fgospider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaeles-project%2Fgospider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaeles-project","download_url":"https://codeload.github.com/jaeles-project/gospider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248339270,"owners_count":21087213,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bugbounty","crawler","go","gospider","spider"],"created_at":"2024-08-01T10:00:59.075Z","updated_at":"2025-04-11T03:41:06.596Z","avatar_url":"https://github.com/jaeles-project.png","language":"Go","funding_links":["https://patreon.com/j3ssie","https://opencollective.com/jaeles-project","https://paypal.me/j3ssiejjj"],"categories":["Recon","Go","Tools","Weapons","Repositories","Reconnaissance \u0026 OSINT"],"sub_categories":["Content Discovery","Tools"],"readme":"# GoSpider\n\n**GoSpider** - Fast web spider written in Go\n\n## Painless integrate Gospider into your recon workflow?\n\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"OsmedeusEngine\" src=\"https://raw.githubusercontent.com/osmedeus/assets/main/logo-transparent.png\" height=\"200\" /\u003e\n  \u003cp align=\"center\"\u003e\n    this project was part of Osmedeus Engine. Check out how it was integrated at \u003ca href=\"https://twitter.com/OsmedeusEngine\"\u003e@OsmedeusEngine\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n\n## Installation\n\n### GO install\n```\nGO111MODULE=on go install github.com/jaeles-project/gospider@latest\n```\n\n### Docker\n\n```bash\n# Clone the repo\ngit clone https://github.com/jaeles-project/gospider.git\n# Build the contianer\ndocker build -t gospider:latest gospider\n# Run the container\ndocker run -t gospider -h\n```\n\n## Features\n\n* Fast web crawling\n* Brute force and parse sitemap.xml\n* Parse robots.txt\n* Generate and verify link from JavaScript files\n* Link Finder\n* Find AWS-S3 from response source\n* Find subdomains from response source\n* Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault\n* Format output easy to Grep\n* Support Burp input\n* Crawl multiple sites in parallel\n* Random mobile/web User-Agent\n\n## Showcases\n\n[![asciicast](https://asciinema.org/a/301827.svg)](https://asciinema.org/a/301827)\n\n## Usage\n\n```shell\nFast web spider written in Go - v1.1.5 by @thebl4ckturtle \u0026 @j3ssiejjj\n\nUsage:\n  gospider [flags]\n\nFlags:\n  -s, --site string               Site to crawl\n  -S, --sites string              Site list to crawl\n  -p, --proxy string              Proxy (Ex: http://127.0.0.1:8080)\n  -o, --output string             Output folder\n  -u, --user-agent string         User Agent to use\n                                  \tweb: random web user-agent\n                                  \tmobi: random mobile user-agent\n                                  \tor you can set your special user-agent (default \"web\")\n      --cookie string             Cookie to use (testA=a; testB=b)\n  -H, --header stringArray        Header to use (Use multiple flag to set multiple header)\n      --burp string               Load headers and cookie from burp raw http request\n      --blacklist string          Blacklist URL Regex\n      --whitelist string          Whitelist URL Regex\n      --whitelist-domain string   Whitelist Domain\n  -t, --threads int               Number of threads (Run sites in parallel) (default 1)\n  -c, --concurrent int            The number of the maximum allowed concurrent requests of the matching domains (default 5)\n  -d, --depth int                 MaxDepth limits the recursion depth of visited URLs. (Set it to 0 for infinite recursion) (default 1)\n  -k, --delay int                 Delay is the duration to wait before creating a new request to the matching domains (second)\n  -K, --random-delay int          RandomDelay is the extra randomized duration to wait added to Delay before creating a new request (second)\n  -m, --timeout int               Request timeout (second) (default 10)\n  -B, --base                      Disable all and only use HTML content\n      --js                        Enable linkfinder in javascript file (default true)\n      --subs                      Include subdomains\n      --sitemap                   Try to crawl sitemap.xml\n      --robots                    Try to crawl robots.txt (default true)\n  -a, --other-source              Find URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com)\n  -w, --include-subs              Include subdomains crawled from 3rd party. Default is main domain\n  -r, --include-other-source      Also include other-source's urls (still crawl and request)\n      --debug                     Turn on debug mode\n      --json                      Enable JSON output\n  -v, --verbose                   Turn on verbose\n  -l, --length                    Turn on length\n  -L, --filter-length             Turn on length filter\n  -R, --raw                       Turn on raw\n  -q, --quiet                     Suppress all the output and only show URL\n      --no-redirect               Disable redirect\n      --version                   Check version\n  -h, --help                      help for gospider\n\n```\n\n## Example commands\n\n### Quite output\n\n```\ngospider -q -s \"https://google.com/\"\n```\n\n#### Run with single site\n\n```\ngospider -s \"https://google.com/\" -o output -c 10 -d 1\n```\n\n#### Run with site list\n\n```\ngospider -S sites.txt -o output -c 10 -d 1\n```\n\n#### Run with 20 sites at the same time with 10 bot each site\n\n```\ngospider -S sites.txt -o output -c 10 -d 1 -t 20\n```\n\n#### Also get URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com)\n\n```\ngospider -s \"https://google.com/\" -o output -c 10 -d 1 --other-source\n```\n\n#### Also get URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com) and include subdomains\n\n```\ngospider -s \"https://google.com/\" -o output -c 10 -d 1 --other-source --include-subs\n```\n\n#### Use custom header/cookies\n\n```\ngospider -s \"https://google.com/\" -o output -c 10 -d 1 --other-source -H \"Accept: */*\" -H \"Test: test\" --cookie \"testA=a; testB=b\"\n\ngospider -s \"https://google.com/\" -o output -c 10 -d 1 --other-source --burp burp_req.txt\n```\n\n#### Blacklist url/file extension.\n\n**P/s**: gospider blacklisted `.(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico)` as default\n\n```\ngospider -s \"https://google.com/\" -o output -c 10 -d 1 --blacklist \".(woff|pdf)\"\n```\n\n#### Show and Blacklist file length.\n\n```\ngospider -s \"https://google.com/\" -o output -c 10 -d 1 --length --filter-length \"6871,24432\"   \n```\n\n## License\n\n`Gospider` is made with ♥ by [@j3ssiejjj](https://twitter.com/j3ssiejjj)\n\u0026 [@thebl4ckturtle](https://twitter.com/thebl4ckturtle) and it is released under the MIT license.\n\n## Donation\n\n[![paypal](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://paypal.me/j3ssiejjj)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaeles-project%2Fgospider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaeles-project%2Fgospider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaeles-project%2Fgospider/lists"}