Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dwisiswant0/galer
A fast tool to fetch URLs from HTML attributes by crawl-in.
https://github.com/dwisiswant0/galer
crawler devtool extractor galer go golang spider url-extractor url-parser waybackurls
Last synced: about 2 months ago
JSON representation
A fast tool to fetch URLs from HTML attributes by crawl-in.
- Host: GitHub
- URL: https://github.com/dwisiswant0/galer
- Owner: dwisiswant0
- License: mit
- Created: 2020-11-26T03:45:15.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2024-06-18T23:16:40.000Z (3 months ago)
- Last Synced: 2024-06-20T05:49:02.364Z (3 months ago)
- Topics: crawler, devtool, extractor, galer, go, golang, spider, url-extractor, url-parser, waybackurls
- Language: Go
- Homepage:
- Size: 42 KB
- Stars: 208
- Watchers: 6
- Forks: 33
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
- project-awesome - dwisiswant0/galer - A fast tool to fetch URLs from HTML attributes by crawl-in. (Go)
- awesome-hacking-lists - dwisiswant0/galer - A fast tool to fetch URLs from HTML attributes by crawl-in. (Go)
README
# galer
[](http://golang.org)
[](https://github.com/dwisiswant0/galer/issues)
```txt
__
__ _ _(_ ) __ _ __
/'_ '\/'_' )| | /'__'( '__)
( (_) ( (_| || |( ___| |
'\__ '\__,_(___'\____(_)
( )_) |
\___/' @dwisiswant0
```
A fast tool to fetch URLs from HTML attributes by crawl-in. Inspired by the [@omespino Tweet](https://twitter.com/omespino/status/1318605084989837312), which is possible to extract `src`, `href`, `url` and `action` values by evaluating JavaScript through Chrome DevTools Protocol.
---
## Resources
- [Installation](#installation)
- [from Binary](#from-binary)
- [from Source](#from-source)
- [from GitHub](#from-github)
- [Usage](#usage)
- [Basic Usage](#basic-usage)
- [Flags](#flags)
- [Examples](#examples)
- [Single URL](#single-url)
- [URLs from list](#urls-from-list)
- [from Stdin](#from-stdin)
- [Library](#library)
- [TODOs](#todos)
- [Help & Bugs](#help--bugs)
- [License](#license)
- [Version](#version)
- [Acknowledgement](#acknowledgement)
## Installation
### from Binary
The installation is easy. You can download a prebuilt binary from [releases page](https://github.com/dwisiswant0/galer/releases), unpack and run! or with
```bash
▶ (sudo) curl -sSfL https://git.io/galer | sh -s -- -b /usr/local/bin
```
### from Source
If you have go1.15+ compiler installed and configured:
```bash
▶ GO111MODULE=on go get github.com/dwisiswant0/galer
```
### from GitHub
```bash
▶ git clone https://github.com/dwisiswant0/galer
▶ cd galer
▶ go build .
▶ (sudo) mv galer /usr/local/bin
```
## Usage
### Basic Usage
Simply, galer can be run with:
```bash
▶ galer -u "http://domain.tld"
```
### Flags
```bash
▶ galer -h
```
This will display help for the tool. Here are all the switches it supports.
| **Flag** | **Description** |
|------------------- |----------------------------------------------------------------- |
| -u, --url | Target to fetches _(single target URL or list)_ |
| -e, --extension | Show only certain extensions _(comma-separated, e.g. js,php)_ |
| -c, --concurrency | Concurrency level _(default: 50)_ |
| --in-scope | Show in-scope URLs/same host only |
| -o, --output | Save fetched URLs output into file |
| -t, --timeout | Maximum time _(seconds)_ allowed for connection _(default: 60)_ |
| -s, --silent | Silent mode _(suppress an errors)_ |
| -v, --verbose | Verbose mode show error details unless you weren't use silent |
| -h, --help | Display its helps |
### Examples
#### Single URL
```bash
▶ galer -u "http://domain.tld"
```
#### URLs from list
```bash
▶ galer -u /path/to/urls.txt
```
#### from Stdin
```bash
▶ cat urls.txt | galer
```
In case you want to chained with other tools:
```bash
▶ subfinder -d domain.tld -silent | httpx -silent | galer
```
### Library
[](https://godoc.org/github.com/dwisiswant0/galer/pkg/galer)
You can use **galer** as library.
```
▶ go get github.com/dwisiswant0/galer/pkg/galer
```
For example:
```go
package main
import (
"fmt"
"github.com/dwisiswant0/galer/pkg/galer"
)
func main() {
cfg := &galer.Config{
Timeout: 60,
}
cfg = galer.New(cfg)
run, err := cfg.Crawl("https://twitter.com")
if err != nil {
panic(err)
}
for _, url := range run {
fmt.Println(url)
}
}
```
## TODOs
- [ ] Enable to set extra HTTP headers
- [ ] Provide randomly User-Agent
- [ ] Bypass headless browser
- [ ] Add exception for specific extensions
## Help & Bugs
[](https://github.com/dwisiswant0/galer/issues)
If you are still confused or found a bug, please [open the issue](https://github.com/dwisiswant0/galer/issues). All bug reports are appreciated, some features have not been tested yet due to lack of free time.
## License
[](https://opensource.org/licenses/MIT)
**galer** released under MIT. See `LICENSE` for more details.
## Version
**Current version is 0.0.2** and still development.
## Pronunciation
`id_ID` • **/gäˈlər/** — kalau _galer_ jangan dicium baunya, langsung cuci tangan, _bego_!
## Acknowledgement
- [Omar Espino](https://twitter.com/omespino) for the idea, that's why this tool was made!