{"id":20129774,"url":"https://github.com/ananthb/chonker","last_synced_at":"2025-04-09T16:12:26.926Z","repository":{"id":172661261,"uuid":"649580676","full_name":"ananthb/chonker","owner":"ananthb","description":"Download large files in parallel chunks in Go.","archived":false,"fork":false,"pushed_at":"2024-09-18T19:34:59.000Z","size":238,"stargazers_count":23,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-09T16:12:08.340Z","etag":null,"topics":["go","golang"],"latest_commit_sha":null,"homepage":"https://devhuman.net/posts/chonker","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"sudhirj/ranger","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ananthb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-05T07:34:11.000Z","updated_at":"2024-12-14T15:37:28.000Z","dependencies_parsed_at":"2024-06-15T17:50:21.414Z","dependency_job_id":"7f7c32dd-93e1-42aa-8bdc-58176a38d9be","html_url":"https://github.com/ananthb/chonker","commit_stats":null,"previous_names":["ananthb/ranger","ananthb/chonker"],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ananthb%2Fchonker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ananthb%2Fchonker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ananthb%2Fchonker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ananthb%2Fchonker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ananthb","download_url":"https://codeload.github.com/ananthb/chonker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248065283,"owners_count":21041872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang"],"created_at":"2024-11-13T20:36:03.426Z","updated_at":"2025-04-09T16:12:26.886Z","avatar_url":"https://github.com/ananthb.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Chonker\n\n[![Chonker](chonker.jpg)](https://www.freepik.com/free-vector/hand-drawn-fat-cat-cartoon-illustration_58564188.htm)\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/ananthb/chonker.svg)](https://pkg.go.dev/github.com/ananthb/chonker)\n\n*Download large files as parallel chunks using HTTP Range Requests in Go.*\n\n[![CI](https://github.com/ananthb/chonker/actions/workflows/ci.yml/badge.svg)](https://github.com/ananthb/chonker/actions/workflows/ci.yml)\n\nChonker works on Go 1.19, oldstable, and stable releases.\n\n## What does Chonker do?\n\nChonker speeds up downloads from cloud services like Amazon S3 \u0026 CloudFront.\nIt does this in two ways.\n\n1. Download small pieces of a file (a.k.a a chunk) using\n   [HTTP Range](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range)\n   requests.\n2. Download chunks in parallel.\n\n## Why?\n\nChonker allows CDN services to cache and serve files even if the entire\nfile is bigger than the individual object cache limit.\n\nIt also overcomes the per-connection limit that blob storage services often have\nby opening connections in parallel.\n\nThe Go standard library\n[HTTP Client](https://pkg.go.dev/net/http#hdr-Clients_and_Transports)\ndownloads files as buffered streams of bytes. The Client fetches bytes into a\nrequest buffer as fast as it can, and you read bytes from the buffer as fast as\nyou can.\n\n*Why is this a problem?*\n\nThis works great when one beefy connection to an origin server can use the entire\navailable network bandwidth between you and it. Blob file services like Amazon S3\nand caching CDNs like Amazon CloudFront impose per-connection limits, but\nsupport an almost unlimited number of connections from each client.\n\nIf you are downloading a large file from S3, its almost always better\nto download the file in chunks, using parallel connections.\n\nSee the [S3 Developer Guide](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html)\nand the [CloudFront Developer Guide](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RangeGETs.html)\nfor more information on cache sizes and parallel GETs.\n\n## Use Chonker\n\nUse `chonker.Do` to fetch a response for a request, or create a \"chonky\" `http.Transport`\nthat fetches requests using HTTP Range sub-requests.\n\nChonker integrates well with Go download libraries.\n[Grab](https://github.com/cavaliergopher/grab) and other download managers\ncan use a `http.Client` with a \"chonky\" `http.Transport`.\nIn turn, Chonker functions accept HTTP clients that could provide automatic retries\nor detailed logs. See\n[Heimdall](https://github.com/gojek/heimdall) or [go-retryablehttp](https://github.com/hashicorp/go-retryablehttp)\nfor more.\n\n### [Chonk](cmd/chonk)\n\nChonk is a Go program that uses the chonker library to download a URL into a local\nfile. Run `chonk -h` for usage details.\n\n```shell\ngo build -o chonk ./cmd/chonk\n\n./chonk https://example.com\n```\n\n### [`test.sh`](test.sh)\n\n`test.sh` is a BASH shell script that exercises the `chonk` program with\na list of files of varying sizes. Run `test.sh -h` for usage details.\n\n## License\n\nThe Chonker cat illustration is from [Freepik](https://www.freepik.com/free-vector/hand-drawn-fat-cat-cartoon-illustration_58564188.htm)\n\nChonker is available under the terms of the MIT license.\n\nSee [LICENSE](LICENSE) for the full license text.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fananthb%2Fchonker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fananthb%2Fchonker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fananthb%2Fchonker/lists"}