{"id":25839267,"url":"https://github.com/sethkor/s3kor","last_synced_at":"2025-07-11T15:04:26.752Z","repository":{"id":37891850,"uuid":"160281763","full_name":"sethkor/s3kor","owner":"sethkor","description":"Fast S3 Tools built in GoLang Using Multiparts and Concurrency","archived":false,"fork":false,"pushed_at":"2022-06-14T23:07:56.000Z","size":174,"stargazers_count":30,"open_issues_count":3,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-06-20T07:55:19.936Z","etag":null,"topics":["aws","bucket","cli","copy","fast","free","go","golang","homebrew","linux","mac","multi","multipart","parts","remove","s3","sync","thread","tools","versions"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sethkor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-04T02:05:37.000Z","updated_at":"2023-12-15T20:23:53.000Z","dependencies_parsed_at":"2022-08-20T01:40:58.920Z","dependency_job_id":null,"html_url":"https://github.com/sethkor/s3kor","commit_stats":null,"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sethkor%2Fs3kor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sethkor%2Fs3kor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sethkor%2Fs3kor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sethkor%2Fs3kor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sethkor","download_url":"https://codeload.github.com/sethkor/s3kor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241317004,"owners_count":19943189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","bucket","cli","copy","fast","free","go","golang","homebrew","linux","mac","multi","multipart","parts","remove","s3","sync","thread","tools","versions"],"created_at":"2025-03-01T04:26:15.182Z","updated_at":"2025-03-01T04:26:15.801Z","avatar_url":"https://github.com/sethkor.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3kor\nFast AWS S3 command line tools built in [Go](https://golang.org/) using multiparts and multiple threads for fast parallel actions like copy, list and remove to AWS S3.  Also supported is copying or syncing to a destination bucket with a seperate set of credentials which is not a feature of the aws cli.  It's intended as a drop in replacement for the [aws cli s3](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html) set of commands so all flags, values and args should be the same with the exception of a few new ones.\n\nEasiest way to install if you're on a Mac or Linux (amd64 or arm64)  is to use [Homebrew](https://brew.sh/)\n\nType:\n\n```\nbrew tap sethkor/tap\nbrew install s3kor\n```\n\nFor other platforms take a look at the releases in Github.  I build binaries for:\n\n|OS            | Architecture                           |\n|:------------ |:-------------------------------------- |\n|Mac (Darwin)  | amd64 (aka x86_64)                     |\n|Linux         | amd64, arm64, 386 (32 bit) |\n|Windows       | amd64                   |\n\nLet me know if you would like a particular os/arch binary regularly built.\n\nOr build it yourself by go getting it:\n```\ngo get github.com/sethkor/s3kor\n```\n\n\nThe cli emulates the [aws cli s3](https://aws.amazon.com/cli/) commands as close as possible so as to be a drop in replacement.  Supported s3 operations:\n\n- [X] Copy - cp\n- [X] Remove - rm\n- [X] List - ls\n- [X] Syncronize - sync\n\nUse `--help` on the command line to help you along the way.\n\n```cassandraql\nusage: s3kor [\u003cflags\u003e] \u003ccommand\u003e [\u003cargs\u003e ...]\n\ns3 tools using golang concurency\n\nFlags:\n  --help                                    Show context-sensitive help (also try --help-long and --help-man).\n  --custom-endpoint-url=CUSTOM-ENDPOINT-URL AWS S3 Custom Endpoint URL\n  --profile=PROFILE                         AWS credentials/config file profile to use\n  --region=REGION                           AWS region\n  --verbose                                 Verbose Logging\n  --version                                 Show application version.\n\nCommands:\n  help [\u003ccommand\u003e...]\n    Show help.\n\n  rm [\u003cflags\u003e] \u003cS3Uri\u003e\n    remove\n\n  ls [\u003cflags\u003e] \u003cS3Uri\u003e\n    list\n\n  cp [\u003cflags\u003e] \u003csource\u003e \u003cdestination\u003e\n    copy\n\n  sync [\u003cflags\u003e] \u003csource\u003e \u003cdestination\u003e\n    sync\n\n```\n\n`--profile` will always look in the usual place for your aws `credentials` or `config` file.\n\n### Automatic region detection for your buckets\nAll commands can take the `--auto-region` flag to automatically detect the right region for your bucket operation, rather than you passing the specific region with `--region`.\n\n### Custom Endpoints\n\nAll commands can take the `--custom-endpoint-url=CUSTOM-ENDPOINT-URL` to use a custom endpoint. This has been tested with an AWS SnowBall ( https://aws.amazon.com/snowball/ ) and also AWS VPC S3 Endpoint ( https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html ).\n\nExample with a Snowball:\n\n`s3kor --custom-endpoint-url=http://10.20.30.40:8080 --profile=snowball ls s3://name-of-the-snowball-bucket/`\n\n\n\n## Copy - cp\nThis is WIP, some further features to come.  Features available:\n\n- [X] Copy to S3\n- [X] Copy from S3\n- [X] Copy S3 to S3\n- [X] Copy S3 to S3 in another account with seperate credentials\n\nPlease raise an issue if theres a specific feature you. would like considered or prioritised.\n\n```\n  -q, --quiet                   Does not display the operations performed from the specified command.\n  -r, --recursive               Recursively copy\n  -c, --concurrent=50           Maximum number of concurrent uploads to S3.\n      --sse=AES256              Specifies server-side encryption of the object in S3. Valid values are AES256 and aws:kms.\n      --sse-kms-key-id=SSE-KMS-KEY-ID\n                                The AWS KMS key ID that should be used to server-side encrypt the object in S3.\n      --acl=private             Object ACL\n      --storage-class=STANDARD  Storage Class\n      --dest-profile=DEST-PROFILE\n                                Destination bucket AWS credentials/config file profile to use if different from --profile\n      --accelerate              Use S3 Acceleration\n\nArgs:\n  \u003csource\u003e       file or s3 location\n  \u003cdestination\u003e  file or s3 location\n```\n\nThe maximum concurrent uploads (`--concurrent` or `-c`) is dependent not only on your upload bandwidth but also the maximum open file limits per process on your system and the performance of the soucre drive.\n\nYou can check your file limits in linux, macos and other flavour of OS with `ulimit -n`.  Changing this limit in the os is possible and not always dangerous.  Instructions on how to change it vary between OS so they are not described here.  `s3kor` impacts these limits both in walking the file system and uploading the file so there is not a 1 to 1 correlation between the max limit ond the value you pass to `--concurrent`.  Try to pass `s3kor` a max value that is about 20% less than the systems max limit value.\n\nCurrently if you hit a file limit, the error is not reported.\n\nFor optimal throughput consider using a S3 VPC Gateway endpoint if you are executing s3kor from within an AWS VPC.\n\nAnd remember the performance of the source storage device is important, you don't want to choke it reading lots of data at once.  Use an optimized iops device or SAN.\n\nMeta data is copied when copying from S3 to S3.\n\nAs well as the usualy S3 storage classes, you can specify some additional classes:\n* `S3KOR_OPT_IA`: optimize to `STANDARD_IA` for files greater than 128K\n* `S3KOR_OPT_GLACIER`: optimize to `GLACIER` for files greater than 12K\n* `S3KOR_OPT_DEEPARCHIVE`: optimise to `GLACIER_DEEPARCHIVE` for files greater than 4K\n\nThese additional storage classes instruct s3kor to optimise the storage class based on the source object/file size.  Doing\nthis when copying the object/file removes the need for a lifecycle transition event to do the transition for you which can become\nvery expensive when you have a bucket with a lot of objects.\n\n### Supports different AWS account credentials for source and destination buckets\n\nIf you ever need to copy objects to an account which you don't own and need a seperate set of AWS credentials to access it, s3kor is perfect for the job.  For S3 to S3 with different AWS credentials, objects must be downloaded from the source first and then uploaded.  To cater for large objects and to limit memory usage this is done utilizing multi parts of 5MB in size and will attempt to limit in memory storage.  This relies on GO garbage collector to tidy things up promptly which doesn't always happen.  Some further optimizations are WIP but expect this operation to consume a bit of memory.\n\n### Multiparts\nMultipart chunks size for Upload or Download is set to 5MB.  Any objects greater than 5MB in size shall be sent in multiparts. `s3kor` will send up to 5 parts concurrently per object.\n\n### Progress Bar Behaviour\nThe progress bar is not aware of multipart operations for now.  It will only update once a large object operation is complete.\n\n### S3 to S3 Large objects over 5GB\nLarge objects S3 to S3 are supported.  Since this happens in the AWS backend, the part size for these operations is set to 5GB for maximum performance.  `s3kor` will copy up to 5 parts concurrently per object.\n\n### ACL\nSets the ACL for the object when the command is performed. If you use this parameter you must have the \"s3:PutObjectAcl\" permission included in the list of actions for your IAM policy.  Only accepts values of `private`, `public-read`, `public-read-write`, `authenticated-read`, `aws-exec-read`, `bucket-owner-read`, `bucket-owner-full-control` and `log-delivery-write`\n\nDefaults to `private`\n\n### Storage Class\nThe type of storage to use for the object. Valid choices are: `STANDARD`, `EDUCED_REDUNDANCY`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `GLACIER`, `DEEP_ARCHIVE`.\n\nDefaults to `STANDARD`\n\n## Remove - rm\n```\n  -q, --quiet            Does not display the operations performed from the specified command.\n  -r, --recursive        Recurisvley delete\n      --all-versions     Delete all versions\n      --multi-part       Abort all inprogress multipart uploads\n```\n\nIt is possible to remove all versions of an object recursivley in a single command which is not possible with the AWS cli.\n\nRemember when using `--all-versions` to delete all versions of an object at once, the eventual consistency model of S3 applies.\n\nWhen deleting a large number of objects, the final outcome may not be reflected by `ls` immediately due to eventual consistency.\n\n## List - ls\nNothing special here.  Just remember S3 has prefixes, not directory paths.\n\nYou can list all versions if you like, the version id is outputted first:\n\n```\n      --all-versions     List all versions\n```\n\n## Sync\nFeatures available:\n\n- [X] Sync to S3\n- [X] Sync from S3\n- [X] Sync S3 to S3\n- [X] Copy S3 to S3 in another account with seperate credentials\n\nSync only compares mod timestamps and sizes as the only true way to get a MD5 or some other checksum of an S3 object is to download it comletley first.  An objects ETag is not the MD5 of the object.\n\n```\n  -q, --quiet                   Does not display the operations performed from the specified command.\n  -c, --concurrent=50           Maximum number of concurrent uploads to S3.\n      --sse=AES256              Specifies server-side encryption of the object in S3. Valid values are AES256 and aws:kms.\n      --sse-kms-key-id=SSE-KMS-KEY-ID\n                                The AWS KMS key ID that should be used to server-side encrypt the object in S3.\n      --acl=private             Object ACL\n      --storage-class=STANDARD  Storage Class\n      --dest-profile=DEST-PROFILE\n                                Destination bucket AWS credentials/config file profile to use if different from --profile\n      --accelerate              Use S3 Acceleration\n\nArgs:\n  \u003csource\u003e       file or s3 location\n  \u003cdestination\u003e  file or s3 location\n```\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsethkor%2Fs3kor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsethkor%2Fs3kor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsethkor%2Fs3kor/lists"}