{"id":37141798,"url":"https://github.com/shellhunter/gocewl","last_synced_at":"2026-01-14T16:38:33.514Z","repository":{"id":57509604,"uuid":"131399251","full_name":"shellhunter/gocewl","owner":"shellhunter","description":"gocewl is a commandline tool to generate custom wordlists by crawling webpages","archived":false,"fork":false,"pushed_at":"2020-04-28T10:13:12.000Z","size":20,"stargazers_count":44,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-06-19T00:38:26.990Z","etag":null,"topics":["cli","dictionary-attack","golang","security-tools","wordlist"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shellhunter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-28T10:17:49.000Z","updated_at":"2024-02-17T07:27:58.000Z","dependencies_parsed_at":"2022-09-26T17:51:18.958Z","dependency_job_id":null,"html_url":"https://github.com/shellhunter/gocewl","commit_stats":null,"previous_names":["kevin-ott/gocewl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/shellhunter/gocewl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shellhunter%2Fgocewl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shellhunter%2Fgocewl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shellhunter%2Fgocewl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shellhunter%2Fgocewl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shellhunter","download_url":"https://codeload.github.com/shellhunter/gocewl/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shellhunter%2Fgocewl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28426117,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T16:32:27.303Z","status":"ssl_error","status_checked_at":"2026-01-14T16:28:36.419Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","dictionary-attack","golang","security-tools","wordlist"],"created_at":"2026-01-14T16:38:32.983Z","updated_at":"2026-01-14T16:38:33.508Z","avatar_url":"https://github.com/shellhunter.png","language":"Go","readme":"# goCeWL version 0.2\n\nClone of digininja's [CeWL](https://github.com/digininja/CeWL) written in Golang.\n\n- Crawl websites concurrently and extract words into a wordlist\n- Should be faster as the original CeWL, as requests and parsing are performed concurrently. \n- static binary available, so no dependencies required\n- lower memory fooprint\n\n**Note**: This repo is experimental. Cosider it pre-alpha. The api / cli can change at any time.\n\n\n## Installation\n\nNote: Currently there are no tagged releases or pre-compiled binaries. This will change in the future. \n\nTo compile and and install goCeWL, Go needs to be installed on your system. If that's not yet the case, please follow the installation instructions [here](https://golang.org/doc/install).\n\nIf you have Go installed, run `go get github.com/shellhunter/gocewl`. This will download all dependencies and install the binary to `$GOPATH/bin`. \n\n## Usage\nRun `gocewl --help` to display the commandline options.\n\n```\ngocewl is a commandline tool to generate custom wordlists by crawling webpages. It is based on CewL by digininja.\n\nUsage:\n  gocewl URL [flags]\n\nFlags:\n  -A, --allow stringArray   Domains in scope for the crawler. Provide as comma sperated list.\n  -d, --depth int           Maximum depth for crawling (default 2)\n  -h, --help                help for gocewl\n  -k, --insecure            Ignore self-signed certificates\n      --max-word int        Maximum word length (default 15)\n  -c, --min-count int       Minimum number of times that the word was found (default 1)\n      --min-word int        Mininum word length (default 3)\n  -O, --offsite             Allow the crawler to visit offsite domains\n  -p, --proxy string        Proxy to use: http[s]://[user:pass@]proxy.example.com[:8080]\n  -q, --quiet               No output, except for words\n  -t, --threads int         Amount of threads for crawling (default 10)\n  -u, --url string          URL to start crawling\n      --user-agent string   Custom user agent (default \"gocewl/0.1\")\n      --version             version for gocewl\n  -w, --write string        filename to write the wordlist to. If no file is provided, print to stdout (default \"wordlist.txt\")\n```\n\n\n## Examples\nCrawl https://en.wikipedia.org with default parameters.\n```gocewl https://en.wikipedia.org```\n\nCrawl https://en.wikipedia.org with depth of 2, 10 threads and write the output to wiki.txt\n```gocewl -d 2 -t 10 -w wiki.txt https://en.wikipedia.org```\n\n## Todos\n\n### Parity with CeWL\n- [x] Set minimum word length (defaults to 5)\n- [x] Set crawling depth (defaults to 2)\n- [x] Allow offsite crawling\n- [x] Proxy support\n- [ ] HTTP Basic / NTLM Auth support\n- [ ] Include E-Mails\n- [ ] Include metadata\n- [ ] Headers \n- [x] User-agent\n\n### Planned features\n- [ ] Cookie support\n- [ ] Sort wordlist by wordcount\n- [ ] --top-words cli switch to only print top X words (by count)\n\n### Other \n- [ ] Performance optimizations\n- [ ] Improved error handling\n- [ ] Improved cli\n\n## Changelog\n\n### 0.2\n- Performance improvements\n- Changed sync.Map to regular map with a mutex\n- Fixed a race consdition when counting requests and error\n- Fixed display of statistics\n\n### 0.1\n- Initial release to github\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshellhunter%2Fgocewl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshellhunter%2Fgocewl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshellhunter%2Fgocewl/lists"}