{"id":18395996,"url":"https://github.com/cyclone-github/spider","last_synced_at":"2025-04-07T03:35:19.723Z","repository":{"id":158459359,"uuid":"634037688","full_name":"cyclone-github/spider","owner":"cyclone-github","description":"URL Spider - web crawler and wordlist / ngram generator","archived":false,"fork":false,"pushed_at":"2025-03-19T15:20:20.000Z","size":60,"stargazers_count":14,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-22T13:22:31.554Z","etag":null,"topics":["cewl","crawler","cyclone","generator","gramify","n-gram","ngram","ngram-generator","scaping","scraper","spider","url","url-crawler","url-spider","web","web-crawler","web-scraping","wordlist","wordlist-generator"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cyclone-github.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-28T21:51:28.000Z","updated_at":"2025-03-19T15:20:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"fd2850ed-10fe-4f0f-8de9-3373cbf12dee","html_url":"https://github.com/cyclone-github/spider","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyclone-github%2Fspider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyclone-github%2Fspider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyclone-github%2Fspider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyclone-github%2Fspider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cyclone-github","download_url":"https://codeload.github.com/cyclone-github/spider/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247589800,"owners_count":20963022,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cewl","crawler","cyclone","generator","gramify","n-gram","ngram","ngram-generator","scaping","scraper","spider","url","url-crawler","url-spider","web","web-crawler","web-scraping","wordlist","wordlist-generator"],"created_at":"2024-11-06T02:12:34.391Z","updated_at":"2025-04-07T03:35:19.717Z","avatar_url":"https://github.com/cyclone-github.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Readme Card](https://github-readme-stats.vercel.app/api/pin/?username=cyclone-github\u0026repo=spider\u0026theme=gruvbox)](https://github.com/cyclone-github/spider/)\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/cyclone-github/spider)](https://goreportcard.com/report/github.com/cyclone-github/spider)\n[![GitHub issues](https://img.shields.io/github/issues/cyclone-github/spider.svg)](https://github.com/cyclone-github/spider/issues)\n[![License](https://img.shields.io/github/license/cyclone-github/spider.svg)](LICENSE)\n[![GitHub release](https://img.shields.io/github/release/cyclone-github/spider.svg)](https://github.com/cyclone-github/spider/releases)\n[![Go Reference](https://pkg.go.dev/badge/github.com/cyclone-github/spider.svg)](https://pkg.go.dev/github.com/cyclone-github/spider)\n\n# Cyclone's URL Spider\n\u003c!-- ![image](https://i.imgur.com/Z6RjlUv.png) --\u003e\n```\n ---------------------- \n| Cyclone's URL Spider |\n ---------------------- \n\nCrawling URL:   https://forum.hashpwn.net\nBase domain:    forum.hashpwn.net\nCrawl depth:    2\nngram len:      1-3\nCrawl delay:    0ms (increase this to avoid rate limiting, ex: -delay 100)\nURLs crawled:   51\nProcessing...   [====================] 100.00%\nUnique words:   1983\nUnique ngrams:  11030\nWriting...      [====================] 100.00%\nOutput file:    forum.hashpwn.net_wordlist.txt\nRAM used:       0.03 GB\nRuntime:        4.949s\n```\n\nWordlist \u0026 ngram creation tool to crawl a given url and create wordlists and/or ngrams (depending on flags given).\n### Usage Instructions:\n- To create a simple wordlist from a specified url (will save deduplicated wordlist to url_wordlist.txt):\n  - `./spider.bin -url https://github.com/cyclone-github`\n- To set url crawl url depth of 2 and create ngrams len 1-5, use flag \"-crawl 2\" and \"-ngram 1-5\"\n  - `./spider.bin -url https://github.com/cyclone-github -crawl 2 -ngram 1-5`\n- To set a custom output file, use flag \"-o filename\"\n  - `./spider.bin -url https://github.com/cyclone-github -o wordlist.txt`\n- To set a delay to keep from being rate-limited, use flag \"-delay nth\" where nth is time in milliseconds\n  - `./spider.bin -url https://github.com/cyclone-github -delay 100`\n- Run `./spider.bin -help` to see a list of all options\n\n### Compile from source:\n- If you want the latest features, compiling from source is the best option since the release version may run several revisions behind the source code.\n- This assumes you have Go and Git installed\n  - `git clone https://github.com/cyclone-github/spider.git`   # clone repo\n  - `cd spider`                                                # enter project directory\n  - `go mod init spider`                                       # initialize Go module (skips if go.mod exists)\n  - `go mod tidy`                                              # download dependencies\n  - `go build -ldflags=\"-s -w\" .`                              # compile binary in current directory\n  - `go install -ldflags=\"-s -w\" .`                            # compile binary and install to $GOPATH\n- Compile from source code how-to:\n  - https://github.com/cyclone-github/scripts/blob/main/intro_to_go.txt\n### Change Log:\n- https://github.com/cyclone-github/spider/blob/main/CHANGELOG.md\n### Mentions:\n- Go Package Documentation: https://pkg.go.dev/github.com/cyclone-github/spider\n- Softpedia: https://www.softpedia.com/get/Internet/Other-Internet-Related/Cyclone-s-URL-Spider.shtml\n\n### Antivirus False Positives:\n- Several antivirus programs on VirusTotal incorrectly detect compiled Go binaries as a false positive. This issue primarily affects the Windows executable binary, but is not limited to it. If this concerns you, I recommend carefully reviewing the source code, then proceed to compile the binary yourself.\n- Uploading your compiled binaries to https://virustotal.com and leaving an up-vote or a comment would be helpful as well.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyclone-github%2Fspider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcyclone-github%2Fspider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyclone-github%2Fspider/lists"}