{"id":19815932,"url":"https://github.com/replicate/pget","last_synced_at":"2025-04-12T20:46:04.990Z","repository":{"id":179483252,"uuid":"639145336","full_name":"replicate/pget","owner":"replicate","description":"parallel fetch","archived":false,"fork":false,"pushed_at":"2025-04-11T20:46:29.000Z","size":496,"stargazers_count":123,"open_issues_count":24,"forks_count":7,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-04-11T21:38:58.481Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/replicate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-10T21:23:34.000Z","updated_at":"2025-04-04T02:25:12.000Z","dependencies_parsed_at":"2024-01-19T17:45:27.662Z","dependency_job_id":"c99e88b2-4f86-4d7b-9829-e0578dfea780","html_url":"https://github.com/replicate/pget","commit_stats":{"total_commits":246,"total_committers":11,"mean_commits":"22.363636363636363","dds":0.6097560975609756,"last_synced_commit":"a98b885a63e604933cfc6356e74ee53fd0a2cea1"},"previous_names":["replicate/pget"],"tags_count":55,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fpget","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fpget/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fpget/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fpget/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/replicate","download_url":"https://codeload.github.com/replicate/pget/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631701,"owners_count":21136559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T10:07:47.470Z","updated_at":"2025-04-12T20:46:04.968Z","avatar_url":"https://github.com/replicate.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PGet - Parallel File Downloader \u0026 Extractor\n\nPGet is a high performance, concurrent file downloader built in Go. It is designed to speed up and optimize file downloads from cloud storage services such as Amazon S3 and Google Cloud Storage.\n\nThe primary advantage of PGet is its ability to download files in parallel using multiple threads. By dividing the file into chunks and downloading multiple chunks simultaneously, PGet significantly reduces the total download time for large files.\n\nIf the downloaded file is a tar archive, PGet can automatically extract the contents of the archive in memory, thus removing the need for an additional extraction step.\n\nThe efficiency of PGet's tar extraction lies in its approach to handling data. Instead of writing the downloaded tar file to disk and then reading it back into memory for extraction, PGet conducts the extraction directly from the in-memory download buffer. This method avoids unnecessary memory copies and disk I/O, leading to an increase in performance, especially when dealing with large tar files. This makes PGet not just a parallel downloader, but also an efficient file extractor, providing a streamlined solution for fetching and unpacking files.\n\n\u003e [!NOTE]  \n\u003e This project is not related to [Code-Hex/pget](https://github.com/Code-Hex/pget). The two projects share the same name and similar goals, but are completely different codebases with different capabilities.\n\n## Install  \n\nYou can download and install the latest release of PGet directly from GitHub by running the following commands in a terminal:\n\n```console\nsudo curl -o /usr/local/bin/pget -L \"https://github.com/replicate/pget/releases/latest/download/pget_$(uname -s)_$(uname -m)\"\nsudo chmod +x /usr/local/bin/pget\n```\n\nIf you're using macOS, you can install PGet with Homebrew:\n\n```console\nbrew tap replicate/tap\nbrew install replicate/tap/pget\n```\n\nOr you can build from source and install it with these commands\n(requires Go 1.19 or later):\n\n```console\nmake\nsudo make install\n```\n\nThis builds a static binary that can work inside containers.\n\n## Usage\n\n### Default Mode\n    pget \u003curl\u003e \u003cdest\u003e [-c concurrency] [-x]\n\n#### Parameters\n\n- \\\u003curl\\\u003e: The URL of the file to download.\n- \\\u003cdest\\\u003e: The destination where the downloaded file will be stored.\n- -c concurrency: The number of concurrent downloads. Default is 4 times the number of cores.\n- -x: Extract the tar file after download. If not set, the downloaded file will be saved as is.\n\n#### Default-Mode Command-Line Options\n- `-x`, `--extract`\n  - Extract archive after download\n  - Type: `bool`\n  - Default: `false`\n\n#### Example\n\n    pget https://storage.googleapis.com/replicant-misc/sd15.tar ./sd15 -x\n\nThis command will download Stable Diffusion 1.5 weights to the path ./sd15 with high concurrency. After the file is downloaded, it will be automatically extracted.\n\n### Multi-File Mode\n    pget multifile \u003cmanifest-file\u003e\n\n#### Parameters\n- \\\u003cmanifest-file\\\u003e: A path to a manifest file containing (new line delimited) pairs of URLs and local destination file paths. The use of `-` allows for reading from STDIN\n\n#### Examples\n\nRead the manifest file from a path on disk:\n\n    pget multifile /path/to/manifest.txt\n\nRead the manifest file from STDIN:\n \n    pget multifile - \u003c manifest.txt\n\nPipe to multifile form from another command:\n\n    cat manifest.txt | pget multifile -\n\nAn example `manifest.txt` file might look like this:\n\n```txt\nhttps://example.com/image1.jpg /local/path/to/image1.jpg\nhttps://example.com/document.pdf /local/path/to/document.pdf\nhttps://example.com/music.mp3 /local/path/to/music.mp3\n```\n\n#### Multi-file specific options\n- `--max-concurrent-files`\n  - Maximum number of files to download concurrently\n  - Default: `40`\n  - Type `Integer`\n- `--max-conn-per-host`\n  - Maximum number of (global) concurrent connections per host\n  - Default: `40`\n  - Type `Integer`\n\n### Global Command-Line Options\n- `--concurrency`\n  - Maximum number of chunks to download in parallel for a given file\n  - Type: `Integer`\n  - Default: `4 * runtime.NumCPU()`\n- `--connect-timeout`\n  - Timeout for establishing a connection, format is \u003cnumber\u003e\u003cunit\u003e, e.g. 10s\n  - Type: `Duration`\n  - Default: `5s`\n- `-f`, `--force`\n  - Force download, overwriting existing file\n  - Type: `bool`\n  - Default: `false`\n- `--log-level`\n  - Log level (debug, info, warn, error)\n  - Type: `string`\n  - Default: `info`\n- `-m`, `--chunk-size string`\n  - Chunk size (in bytes) to use when downloading a file (e.g. 10M)\n  - Type: `string`\n  - Default: `125M`\n- `--resolve`\n  - Resolve hostnames to specific IPs, can be specified multiple times, format \u003chostname\u003e:\u003cport\u003e:\u003cip\u003e (e.g. example.com:443:127.0.0.1)\n  - Type: `string\n- `-r`, `--retries`\n  - Number of retries when attempting to retrieve a file\n  - Type: `Integer`\n  - Default: `5`\n- `-v`, `--verbose`\n  - Verbose mode (equivalent to `--log-level debug`)\n  - Type: `bool`\n  - Default: `false`\n\n#### Deprecated\n- `--max-chunks` (deprecated, use `--concurrency` instead)\n  - Maximum number of chunks for downloading a given file\n  - Type: `Integer`\n  - Default: `4 * runtime.NumCPU()`\n- `-m`, `--minimum-chunk-size string` (deprecated, use `--chunk-size` instead)\n  - Minimum chunk size (in bytes) to use when downloading a file (e.g. 10M)\n  - Type: `string`\n  - Default: `16M`\n\n## Error Handling\n\nPGet includes some error handling:\n\n1. If a download any chunks fails, it will automatically retry up to 5 times before giving up.\n2. If the downloaded file size does not match the expected size, it will also retry the download.\n\n## Future Improvements\n\n- as chunks are downloaded, start either writing to disk or extracting\n- can we check the content hash of the file in the background?\n- support for zip files?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplicate%2Fpget","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freplicate%2Fpget","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplicate%2Fpget/lists"}