{"id":16824883,"url":"https://github.com/tomnomnom/concurl","last_synced_at":"2025-07-20T08:40:03.150Z","repository":{"id":41271421,"uuid":"131732148","full_name":"tomnomnom/concurl","owner":"tomnomnom","description":"Make concurrent requests with the curl command-line tool","archived":false,"fork":false,"pushed_at":"2019-10-09T09:48:01.000Z","size":4,"stargazers_count":73,"open_issues_count":0,"forks_count":15,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-18T07:51:34.674Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tomnomnom.png","metadata":{"files":{"readme":"README.mkd","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-01T15:52:14.000Z","updated_at":"2024-12-23T14:53:11.000Z","dependencies_parsed_at":"2022-09-20T23:51:18.804Z","dependency_job_id":null,"html_url":"https://github.com/tomnomnom/concurl","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomnomnom%2Fconcurl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomnomnom%2Fconcurl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomnomnom%2Fconcurl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomnomnom%2Fconcurl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tomnomnom","download_url":"https://codeload.github.com/tomnomnom/concurl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244902929,"owners_count":20529114,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T11:12:18.485Z","updated_at":"2025-03-22T03:31:48.232Z","avatar_url":"https://github.com/tomnomnom.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# concurl\n\nConcurrently request URLs provided on `stdin` using the `curl` command line utility (with per-domain rate limiting).\n\n## Install\n\nWith Go:\n\n```\n▶ go get -u github.com/tomnomnom/concurl\n```\n\nOr [download a release for your platform](https://github.com/tomnomnom/concurl/releases).\n\n## Usage\n\nBasic usage:\n\n```\n▶ cat urls.txt\nhttps://example.com/path?one=1\u0026two=2\nhttps://example.com/pathtwo?two=2\u0026one=1\nhttps://example.net/a/path?two=2\u0026one=1\n\n▶ cat urls.txt | concurl -c 3\nout/example.com/6ad33f150c6a17b4d51bb3a5425036160e18643c https://example.com/path?one=1\u0026two=2\nout/example.net/33ce069e645b0cb190ef0205af9200ae53b57e53 https://example.net/a/path?two=2\u0026one=1\nout/example.com/5657622dd56a6c64da72459132d576a8f89576e2 https://example.com/pathtwo?two=2\u0026one=1\n\n▶ head -n 7 out/example.net/33ce069e645b0cb190ef0205af9200ae53b57e53\ncmd: curl --silent https://example.net/a/path?two=2\u0026one=1\n------\n\n\u003c!doctype html\u003e\n\u003chtml\u003e\n\u003chead\u003e\n    \u003ctitle\u003eExample Domain\u003c/title\u003e\n```\n\n### Curl Options\n\nSupply options to the `curl` command after a `--`:\n\n```\n▶ echo \"https://httpbin.org/anything\" | concurl -c 5 -- -H'User-Agent: concurl' -H'X-Foo: bar'\nout/httpbin.org/391256f9956ce947c3bcb9af616fe0725a35ff4e https://httpbin.org/anything\n\n▶ cat out/httpbin.org/391256f9956ce947c3bcb9af616fe0725a35ff4e\ncmd: curl --silent https://httpbin.org/anything -HUser-Agent: concurl -HX-Foo: bar\n------\n\n{\n  \"args\": {}, \n  \"data\": \"\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Connection\": \"close\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"concurl\", \n    \"X-Foo\": \"bar\"\n  }, \n  \"json\": null, \n  \"method\": \"GET\", \n  \"url\": \"https://httpbin.org/anything\"\n}\n```\n\n## Help\n\n```\n▶ concurl -h\nUsage of concurl:\n  -c int\n    \tConcurrency level (default 20)\n  -d int\n    \tDelay between requests to the same domain (default 5000)\n  -o string\n    \tOutput directory (default \"out\")\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomnomnom%2Fconcurl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomnomnom%2Fconcurl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomnomnom%2Fconcurl/lists"}