{"id":13453213,"url":"https://github.com/htcat/htcat","last_synced_at":"2025-03-24T01:31:14.034Z","repository":{"id":9891344,"uuid":"11896487","full_name":"htcat/htcat","owner":"htcat","description":"Parallel and Pipelined HTTP GET Utility","archived":false,"fork":false,"pushed_at":"2019-02-26T22:54:07.000Z","size":426,"stargazers_count":553,"open_issues_count":4,"forks_count":28,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-01-06T20:43:14.444Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/htcat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-08-05T11:17:01.000Z","updated_at":"2024-11-28T16:27:41.000Z","dependencies_parsed_at":"2022-09-14T21:20:37.041Z","dependency_job_id":null,"html_url":"https://github.com/htcat/htcat","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/htcat%2Fhtcat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/htcat%2Fhtcat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/htcat%2Fhtcat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/htcat%2Fhtcat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/htcat","download_url":"https://codeload.github.com/htcat/htcat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245194202,"owners_count":20575724,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T08:00:34.892Z","updated_at":"2025-03-24T01:31:13.633Z","avatar_url":"https://github.com/htcat.png","language":"Go","readme":"# htcat #\n\n`htcat` is a utility to perform parallel, pipelined execution of a\nsingle HTTP `GET`.  `htcat` is intended for the purpose of\nincantations like:\n\n    htcat https://host.net/file.tar.gz | tar -zx\n\nIt is tuned (and only really useful) for faster interconnects:\n\n    $ htcat http://test.com/file | pv -a \u003e /dev/null\n    [ 109MB/s]\n\nThis is on a gigabit network, between an AWS EC2 instance and S3.\nThis represents 91% use of the theoretical maximum of gigabit (119.2\nMiB/s).\n\n## Installation ##\n\nThis program depends on Go 1.1 or later.  One can use `go get` to\ndownload and compile it from source:\n\n    $ go get github.com/htcat/htcat/cmd/htcat\n\n## Help and Reporting Bugs ##\n\nFor correspondence of all sorts, write to \u003chtcat@googlegroups.com\u003e.\nBugs can be filed at\n[htcat's GitHub Issues page](https://github.com/htcat/htcat/issues).\n\n## Approach ##\n\n`htcat` works by determining the size of the `Content-Length` of the\nURL passed, and then partitioning the work into a series of `GET`s\nthat use the `Range` header in the request, with the notable exception\nof the first issued `GET`, which has no `Range` header and is used to\nboth start the transfer and attempt to determine the size of the URL.\n\nUnlike most programs that do similar `Range`-based splitting, the\nrequests that are performed in parallel are limited to some bytes\nahead of the data emitted so far instead of splitting the entire byte\nstream evenly.  The purpose of this is to emit those bytes as soon as\nreasonably possible, so that pipelined execution of another tool can,\ntoo, proceed in parallel.\n\nThese requests may complete slightly out of order, and are held in\nreserve until contiguous bytes can be emitted by a defragmentation\nroutine, that catenates together the complete, consecutive payloads in\nmemory for emission.\n\nTweaking the number of simultaneous transfers and the size of each\n`GET` makes a trade-off between latency to fill the output pipeline,\nmemory usage, and churn in requests and connections and incurring\ntheir associated start-up costs.\n\nIf `htcat`'s peer on the server side processes `Range` requests more\nslowly than regular `GET` without a `Range` header, then, `htcat`'s\nperformance can suffer relative to a simpler, single-stream `GET`.\n\n## Numbers ##\n\nThese are measurements falling well short of real benchmarks that are\nintended to give a rough sense of the performance improvements that\nmay be useful to you.  These were taken via an AWS EC2 instance\nconnecting to S3, and there is definitely some variation in runs,\nsometimes very significant, especially at the higher speeds.\n\n|Tool       | TLS | Rate     |\n|-----------|-----|----------|\n|htcat      | no  | 109 MB/s |\n|curl       | no  | 36 MB/s  |\n|aria2c -x5 | no  | 113 MB/s |\n|htcat      | yes | 59 MB/s  |\n|curl       | yes | 5 MB/s   |\n|aria2c -x5 | yes | 17 MB/s  |\n\nOn somewhat small files, the situation changes: `htcat` chooses\nsmaller parts, as to still get some parallelism.\n\nBelow are results while performing a 13MB transfer from S3 (Seattle)\nto an EC2 instance in Virginia.  Notably, TLS being on or off did not\nseem to matter, perhaps in this case it was not a bottleneck.\n\n| Tool   | Time     |\n|--------|----------|\n| curl   | 5.20s    |\n| curl   | 7.75s    |\n| curl   | 6.36s    |\n| htcat  | 2.69s    |\n| htcat  | 2.50s    |\n| htcat  | 3.25s    |\n\nResults while performing a transfer of the same 13MB file from S3 to\nEC2, but all within Virginia:\n\n| Tool       | TLS | Time     |\n|------------|-----|----------|\n| curl       | no  | 0.29s    |\n| curl       | no  | 0.75s    |\n| curl       | no  | 0.44s    |\n| htcat      | no  | 0.30s    |\n| htcat      | no  | 0.30s    |\n| htcat      | no  | 0.48s    |\n| curl       | yes | 2.69s    |\n| curl       | yes | 2.69s    |\n| curl       | yes | 2.62s    |\n| htcat      | yes | 1.37s    |\n| htcat      | yes | 0.45s    |\n| htcat      | yes | 0.59s    |\n\nResults while performing a 4.6MB transfer on a fast (same-region)\nlink.  This file is small enough that `htcat` disables multi-request\nparallelism.  Given that, it's unclear why `htcat` performs markedly\nbetter on the TLS tests than `curl`.\n\n| Tool       | TLS | Time     |\n|------------|-----|----------|\n| curl       | no  | 0.14s    |\n| curl       | no  | 0.13s    |\n| curl       | no  | 0.14s    |\n| htcat      | no  | 0.23s    |\n| htcat      | no  | 0.16s    |\n| htcat      | no  | 0.17s    |\n| curl       | yes | 0.95s    |\n| curl       | yes | 0.97s    |\n| curl       | yes | 0.99s    |\n| htcat      | yes | 0.38s    |\n| htcat      | yes | 0.34s    |\n| htcat      | yes | 0.24s    |\n","funding_links":[],"categories":["Go","Utilities","Weapons","公用事业公司","工具库`可以提升效率的通用代码库和工具`","工具库","Utility","实用工具","實用工具"],"sub_categories":["Utility/Miscellaneous","HTTP Clients","Tools","实用程序/Miscellaneous","Fail injection","Advanced Console UIs","查询语","交流","\u003cspan id=\"高级控制台用户界面-advanced-console-uis\"\u003e高级控制台用户界面 Advanced Console UIs\u003c/span\u003e","高级控制台界面","高級控制台界面"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhtcat%2Fhtcat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhtcat%2Fhtcat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhtcat%2Fhtcat/lists"}