https://github.com/tomnomnom/concurl
Make concurrent requests with the curl command-line tool
https://github.com/tomnomnom/concurl
Last synced: about 2 months ago
JSON representation
Make concurrent requests with the curl command-line tool
- Host: GitHub
- URL: https://github.com/tomnomnom/concurl
- Owner: tomnomnom
- Created: 2018-05-01T15:52:14.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2019-10-09T09:48:01.000Z (over 5 years ago)
- Last Synced: 2025-03-18T07:51:34.674Z (about 2 months ago)
- Language: Go
- Size: 3.91 KB
- Stars: 73
- Watchers: 3
- Forks: 15
- Open Issues: 0
-
Metadata Files:
- Readme: README.mkd
Awesome Lists containing this project
README
# concurl
Concurrently request URLs provided on `stdin` using the `curl` command line utility (with per-domain rate limiting).
## Install
With Go:
```
▶ go get -u github.com/tomnomnom/concurl
```
Or [download a release for your platform](https://github.com/tomnomnom/concurl/releases).
## Usage
Basic usage:
```
▶ cat urls.txt
https://example.com/path?one=1&two=2
https://example.com/pathtwo?two=2&one=1
https://example.net/a/path?two=2&one=1
▶ cat urls.txt | concurl -c 3
out/example.com/6ad33f150c6a17b4d51bb3a5425036160e18643c https://example.com/path?one=1&two=2
out/example.net/33ce069e645b0cb190ef0205af9200ae53b57e53 https://example.net/a/path?two=2&one=1
out/example.com/5657622dd56a6c64da72459132d576a8f89576e2 https://example.com/pathtwo?two=2&one=1
▶ head -n 7 out/example.net/33ce069e645b0cb190ef0205af9200ae53b57e53
cmd: curl --silent https://example.net/a/path?two=2&one=1
------
Example Domain
```
### Curl Options
Supply options to the `curl` command after a `--`:
```
▶ echo "https://httpbin.org/anything" | concurl -c 5 -- -H'User-Agent: concurl' -H'X-Foo: bar'
out/httpbin.org/391256f9956ce947c3bcb9af616fe0725a35ff4e https://httpbin.org/anything
▶ cat out/httpbin.org/391256f9956ce947c3bcb9af616fe0725a35ff4e
cmd: curl --silent https://httpbin.org/anything -HUser-Agent: concurl -HX-Foo: bar
------
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "concurl",
"X-Foo": "bar"
},
"json": null,
"method": "GET",
"url": "https://httpbin.org/anything"
}
```
## Help
```
▶ concurl -h
Usage of concurl:
-c int
Concurrency level (default 20)
-d int
Delay between requests to the same domain (default 5000)
-o string
Output directory (default "out")
```