{"id":20253540,"url":"https://github.com/chanzuckerberg/s3parcp","last_synced_at":"2025-04-10T23:43:47.415Z","repository":{"id":41977829,"uuid":"221324532","full_name":"chanzuckerberg/s3parcp","owner":"chanzuckerberg","description":"Faster than s3cp","archived":false,"fork":false,"pushed_at":"2022-04-20T21:56:58.000Z","size":3734,"stargazers_count":39,"open_issues_count":4,"forks_count":3,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-04-10T23:43:41.799Z","etag":null,"topics":["aws","checksum","go","mmap","s3","s3cp"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chanzuckerberg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-12T22:30:54.000Z","updated_at":"2025-03-26T16:51:08.000Z","dependencies_parsed_at":"2022-08-12T01:11:07.941Z","dependency_job_id":null,"html_url":"https://github.com/chanzuckerberg/s3parcp","commit_stats":null,"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chanzuckerberg%2Fs3parcp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chanzuckerberg%2Fs3parcp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chanzuckerberg%2Fs3parcp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chanzuckerberg%2Fs3parcp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chanzuckerberg","download_url":"https://codeload.github.com/chanzuckerberg/s3parcp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248317726,"owners_count":21083527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","checksum","go","mmap","s3","s3cp"],"created_at":"2024-11-14T10:25:29.840Z","updated_at":"2025-04-10T23:43:47.399Z","avatar_url":"https://github.com/chanzuckerberg.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3parcp [![Latest Version](https://img.shields.io/github/release/chanzuckerberg/s3parcp.svg?style=flat?maxAge=86400)](https://github.com/chanzuckerberg/s3parcp/releases) ![Check](https://github.com/chanzuckerberg/s3parcp/workflows/Check/badge.svg) [![codecov](https://codecov.io/gh/chanzuckerberg/s3parcp/branch/main/graph/badge.svg)](https://codecov.io/gh/chanzuckerberg/s3parcp) [![GitHub license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://github.com/chanzuckerberg/idseq-web/blob/master/LICENSE) ![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)\n\ns3parcp is a CLI wrapper around [AWS's Go SDK's Downloader](https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#NewDownloader). This downloader provides a chunked parallel download implementation from s3 offering speeds faster than [s3cp](https://github.com/aboisvert/s3cp). The API is inspired by `cp`.\n\n## Installation\n\n### Linux\n\n#### Debian (Ubuntu/Mint)\n\nDownload and install the `.deb`:\n\n```bash\nRELEASES=chanzuckerberg/s3parcp/releases\nVERSION=$(curl https://api.github.com/repos/${RELEASES}/latest | jq -r .name | sed s/^v//)\nDOWNLOAD=s3parcp_${VERSION}_linux_amd64.deb\ncurl -L https://github.com/${RELEASES}/download/v${VERSION}/${DOWNLOAD} -o s3parcp.deb\nsudo dpkg -i s3parcp.deb\nrm s3parcp.deb\n```\n\n#### Fedora (RHEL/CentOS)\n\nDownload and install the `.rpm`:\n\n```bash\nRELEASES=chanzuckerberg/s3parcp/releases\nVERSION=$(curl https://api.github.com/repos/${RELEASES}/latest | jq -r .name | sed s/^v//)\nDOWNLOAD=s3parcp_${VERSION}_linux_amd64.rpm\ncurl -L https://github.com/${RELEASES}/download/v${VERSION}/${DOWNLOAD} -o s3parcp.rpm\nsudo rpm -i s3parcp.rpm\nrm s3parcp.rpm\n```\n\n### MacOS\n\nInstall via homebrew:\n\n```bash\nbrew tap chanzuckerberg/tap\nbrew install s3parcp\n```\n\n### Binary\n\nDownload the appropriate binary for your platform:\n\n```bash\nRELEASES=chanzuckerberg/s3parcp/releases\nPLATFORM=#linux,darwin,windows\nVERSION=$(curl https://api.github.com/repos/${RELEASES}/latest | jq -r .name | sed s/^v//)\nDOWNLOAD=s3parcp_${VERSION}_${PLATFORM}_amd64.tar.gz\ncurl -L https://github.com/${RELEASES}/download/v${VERSION}/${DOWNLOAD} | tar zx\n```\n\n### Windows\n\nDownload the appropriate binary for your platform:\n\n```bash\nRELEASES=chanzuckerberg/s3parcp/releases\nVERSION=$(curl https://api.github.com/repos/${RELEASES}/latest | jq -r .name | sed s/^v//)\nDOWNLOAD=s3parcp_${VERSION}_windows_amd64.tar.gz\ncurl -L https://github.com/${RELEASES}/download/v${VERSION}/${DOWNLOAD} | tar zx\n```\n\n## Usage\n\n```plain\nUsage:\n  s3parcp [OPTIONS] [Source] [Destination]\n\nApplication Options:\n  -p, --part-size=                  Part size in bytes of parts to be downloaded\n  -c, --concurrency=                Download concurrency\n  -b, --buffer-size=                Size of download buffer in bytes\n      --checksum                    Compare checksum if downloading or place checksum\n                                    in metadata if uploading\n  -r, --recursive                   Copy directories or folders recursively\n      --version                     Print the current version\n      --s3_url=                     A custom s3 API url (also available as an environment\n                                    variable 'S3PARCP_S3_URL', the flag takes precedence)\n      --max-retries=                Max per chunk retries (default: 3)\n      --disable-ssl                 Disable SSL\n      --disable-cached-credentials  Disable caching AWS credentials\n  -v, --verbose                     verbose logging\n\nHelp Options:\n  -h, --help                        Show this help message\n\nArguments:\n  Source:                           Source to copy from\n  Destination:                      Destination to copy to (Optional, defaults to source's base\n                                    name)\n```\n\n### Examples\n\n#### Uploading\n\n```bash\ns3parcp my/local/file s3://my-bucket/my-object\n```\n\n#### Downloading\n\n```bash\ns3parcp s3://my-bucket/my-object my/local/file\n```\n\n#### Tuning Chunk Parameters\n\n**Note**: These example parameters don't necessarily represent good parameters for your system. s3parcp uses sane defaults so it is recommended to use the default parameters unless you have reason to believe your values will work better.\n\n```bash\nPART_SIZE=1048576 # 1 MB\nBUFFER_SIZE=10485760 # 10 MB\nCONCURRENCY=8\ns3parcp \\\n  --part-size $PART_SIZE \\\n  --concurrency $CONCURRENCY \\\n  --buffer-size $BUFFER_SIZE \\\n  my/local/file s3://my-bucket/my-object\n```\n\n#### Using CRC32C Checksum\n\nYou must upload your file to s3 with s3parcp and the --checksum flag to use this feature for downloads.\n\nUpload your file:\n\n```bash\ns3parcp --checksum my/local/file s3://my-bucket/my-object\n```\n\nThe checksum should be stored in the s3 object's metadata with the key `x-amz-meta-crc32c-checksum`.\n\nDownload your file:\n\n```bash\ns3parcp --checksum s3://my-bucket/my-object my/new/local/file\n```\n\n## Features\n\n### checksum\n\nThis tool comes with a parallelized crc32c checksum validator. The AWS SDK does not support checksums for multipart downloads. If you include the `--checksum` flag when uploading a checksum of your file will be computed and stored in the object's metadata in s3 with the key `x-amz-meta-crc32c-checksum`. When downloading, the `--checksum` flag will compute an independent crc32c checksum of the downloaded file and compare it of the checksum in the object's metadata.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchanzuckerberg%2Fs3parcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchanzuckerberg%2Fs3parcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchanzuckerberg%2Fs3parcp/lists"}