{"id":13621302,"url":"https://github.com/dwisiswant0/galer","last_synced_at":"2025-04-12T20:46:09.236Z","repository":{"id":38040429,"uuid":"316116251","full_name":"dwisiswant0/galer","owner":"dwisiswant0","description":"A fast tool to fetch URLs from HTML attributes by crawl-in.","archived":false,"fork":false,"pushed_at":"2025-01-23T23:51:40.000Z","size":72,"stargazers_count":252,"open_issues_count":2,"forks_count":39,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-04T00:11:28.449Z","etag":null,"topics":["crawler","devtool","extractor","galer","go","golang","spider","url-extractor","url-parser","waybackurls"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dwisiswant0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["dwisiswant0"],"custom":["https://paypal.me/dw1s","https://saweria.co/dwisiswant0","https://unstoppabledomains.com/d/dwisiswant0.crypto"]}},"created_at":"2020-11-26T03:45:15.000Z","updated_at":"2025-03-31T08:55:00.000Z","dependencies_parsed_at":"2024-06-20T05:04:24.176Z","dependency_job_id":"e0e3fa4c-ab63-4fa9-bee4-c8f89e7252fb","html_url":"https://github.com/dwisiswant0/galer","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwisiswant0%2Fgaler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwisiswant0%2Fgaler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwisiswant0%2Fgaler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwisiswant0%2Fgaler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dwisiswant0","download_url":"https://codeload.github.com/dwisiswant0/galer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631702,"owners_count":21136559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","devtool","extractor","galer","go","golang","spider","url-extractor","url-parser","waybackurls"],"created_at":"2024-08-01T21:01:04.454Z","updated_at":"2025-04-12T20:46:09.213Z","avatar_url":"https://github.com/dwisiswant0.png","language":"Go","funding_links":["https://github.com/sponsors/dwisiswant0","https://paypal.me/dw1s","https://saweria.co/dwisiswant0","https://unstoppabledomains.com/d/dwisiswant0.crypto"],"categories":["Go","Shell (473)","Shell","spider","Pentesting"],"sub_categories":["OSINT - Open Source INTelligence"],"readme":"# galer\n\n[![made-with-Go](https://img.shields.io/badge/made%20with-Go-blue.svg)](http://golang.org)\n[![issues](https://img.shields.io/github/issues/dwisiswant0/galer?color=blue)](https://github.com/dwisiswant0/galer/issues)\n\n```txt\n             __\n   __    _ _(_ )   __  _ __ \n /'_ '\\/'_' )| | /'__'( '__)\n( (_) ( (_| || |(  ___| |\n'\\__  '\\__,_(___'\\____(_)\n( )_) |\n \\___/'  @dwisiswant0\n```\n\nA fast tool to fetch URLs from HTML attributes by crawl-in. Inspired by the [@omespino Tweet](https://twitter.com/omespino/status/1318605084989837312), which is possible to extract `src`, `href`, `url` and `action` values by evaluating JavaScript through Chrome DevTools Protocol.\n\n---\n\n## Resources\n\n- [Installation](#installation)\n\t- [from Binary](#from-binary)\n\t- [from Source](#from-source)\n\t- [from GitHub](#from-github)\n- [Usage](#usage)\n\t- [Basic Usage](#basic-usage)\n\t- [Flags](#flags)\n\t- [Examples](#examples)\n\t\t- [Single URL](#single-url)\n\t\t- [URLs from list](#urls-from-list)\n\t\t- [from Stdin](#from-stdin)\n\t- [Library](#library)\n- [TODOs](#todos)\n- [Help \u0026 Bugs](#help--bugs)\n- [License](#license)\n- [Version](#version)\n- [Acknowledgement](#acknowledgement)\n\n## Installation\n\n### from Binary\n\nThe installation is easy. You can download a prebuilt binary from [releases page](https://github.com/dwisiswant0/galer/releases), unpack and run! or with\n\n```bash\n▶ (sudo) curl -sSfL https://git.io/galer | sh -s -- -b /usr/local/bin\n```\n\n### from Source\n\nIf you have go1.22+ compiler installed and configured:\n\n```bash\n▶ go install -v github.com/dwisiswant0/galer@latest\n```\n\n### from GitHub\n\n```bash\n▶ git clone https://github.com/dwisiswant0/galer\n▶ cd galer\n▶ go build .\n▶ (sudo) install galer /usr/local/bin\n```\n\n## Usage\n\n### Basic Usage\n\nSimply, galer can be run with:\n\n```bash\n▶ galer -u \"http://domain.tld\"\n```\n\n### Flags\n\n![galer](https://user-images.githubusercontent.com/25837540/100824601-0ee53b80-3489-11eb-878d-a58d1ec3489d.jpg)\n\nThis will display help for the tool. Here are all the options it supports.\n\n```console\n$ galer -h\n\n             __   v0.2.0\n   __    _ _(_ )   __  _ __\n /'_ '\\/'_' )| | /'__'( '__)\n( (_) ( (_| || |(  ___| |\n'\\__  '\\__,_(___'\\____(_)\n( )_) |\n \\___/'  @dwisiswant0\n\nA fast tool to fetch URLs from HTML attributes by crawl-in\n\nUsage:\n  galer -u [URL|URLs.txt] -o [output.txt]\n\nOptions:\n  -u, --url \u003cURL/FILE\u003e        Target to fetches (single target URL or list)\n  -e, --extension \u003cEXT\u003e       Show only certain extensions (comma-separated, e.g. js,php)\n  -c, --concurrency \u003cN\u003e       Concurrency level (default: 50)\n  -w, --wait \u003cN\u003e              Wait N seconds before evaluate (default: 1)\n  -d, --depth \u003cN\u003e             Max. depth for crawling (levels of links to follow)\n      --same-host             Same host only\n      --same-root             Same root (eTLD+1) only (takes precedence over --same-host)\n  -o, --output \u003cFILE\u003e         Save fetched URLs output into file\n  -T, --template \u003cstring\u003e     Format for output template (e.g., \"{{scheme}}://{{host}}{{path}}\")\n                              Valid variables are: \"raw_url\", \"scheme\", \"user\", \"username\",\n                              \"password\", \"host\", \"hostname\", \"port\", \"path\", \"raw_path\",\n                              \"escaped_path\", \"raw_query\", \"fragment\", \"raw_fragment\".\n  -t, --timeout \u003cN\u003e           Max. time (seconds) allowed for connection (default: 60)\n  -s, --silent                Silent mode (suppress an errors)\n  -v, --verbose               Verbose mode show error details unless you weren't use silent\n  -h, --help                  Display its helps\n```\n\n### Examples\n\n#### Single URL\n\n```bash\n▶ galer -u \"http://domain.tld\"\n```\n\n#### URLs from list\n\n```bash\n▶ galer -u /path/to/urls.txt\n```\n\n#### from Stdin\n\n```bash\n▶ cat urls.txt | galer\n```\n\nIn case you want to chained with other tools:\n\n```bash\n▶ subfinder -d domain.tld -silent | httpx -silent | galer\n```\n\n### Library\n\n[![godoc](https://img.shields.io/badge/godoc-reference-blue.svg)](https://godoc.org/github.com/dwisiswant0/galer/pkg/galer)\n\nYou can use **galer** as library.\n\n```\n▶ go get github.com/dwisiswant0/galer/pkg/galer@latest\n```\n\nFor example:\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/dwisiswant0/galer/pkg/galer\"\n)\n\nfunc main() {\n\tcfg := \u0026galer.Config{\n\t\tTimeout: 60,\n\t}\n\tcfg = galer.New(cfg)\n\n\trun, err := cfg.Crawl(\"https://twitter.com\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tfor _, url := range run {\n\t\tfmt.Println(url)\n\t}\n}\n```\n\n## TODOs\n\n- [ ] Enable to set extra HTTP headers\n- [ ] Provide randomly User-Agent\n- [ ] Bypass headless browser\n- [ ] Add exception for specific extensions\n\n## Help \u0026 Bugs\n\n[![contributions welcome](https://img.shields.io/badge/contributions-welcome-blue.svg)](https://github.com/dwisiswant0/galer/issues)\n\nIf you are still confused or found a bug, please [open the issue](https://github.com/dwisiswant0/galer/issues). All bug reports are appreciated, some features have not been tested yet due to lack of free time.\n\n## Status\n\n\u003e [!CAUTION]\n\u003e galer has NOT reached 1.0 yet. Therefore, this library is currently not supported and does not offer a stable API; use at your own risk.\n\nThere are no guarantees of stability for the APIs in this library, and while they are not expected to change dramatically. API tweaks and bug fixes may occur.\n\n## Pronunciation\n\n`id_ID` • **/gäˈlər/** — kalau _galer_ jangan dicium baunya, langsung cuci tangan, _bego_!\n\n## Acknowledgement\n\n- [Omar Espino](https://twitter.com/omespino) for the idea, that's why this tool was made!\n\n### License\n\n`sebel` is released by **@dwisiswant0** under the MIT license. See [LICENSE](/LICENSE).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdwisiswant0%2Fgaler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdwisiswant0%2Fgaler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdwisiswant0%2Fgaler/lists"}