{"id":23740134,"url":"https://github.com/cobbzilla/s3s3mirror","last_synced_at":"2025-04-09T11:09:27.440Z","repository":{"id":8503617,"uuid":"10113304","full_name":"cobbzilla/s3s3mirror","owner":"cobbzilla","description":"Mirror one S3 bucket to another S3 bucket, or to/from the local filesystem.","archived":false,"fork":false,"pushed_at":"2024-10-03T18:50:50.000Z","size":245795,"stargazers_count":428,"open_issues_count":28,"forks_count":92,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-04-02T09:58:56.603Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cobbzilla.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-05-17T00:34:28.000Z","updated_at":"2025-03-06T16:43:47.000Z","dependencies_parsed_at":"2025-01-28T13:15:10.006Z","dependency_job_id":"5b1e4209-aa4f-41e9-b7c5-2eb53afffaa4","html_url":"https://github.com/cobbzilla/s3s3mirror","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cobbzilla%2Fs3s3mirror","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cobbzilla%2Fs3s3mirror/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cobbzilla%2Fs3s3mirror/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cobbzilla%2Fs3s3mirror/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cobbzilla","download_url":"https://codeload.github.com/cobbzilla/s3s3mirror/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248027406,"owners_count":21035594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-31T09:47:16.253Z","updated_at":"2025-04-09T11:09:27.411Z","avatar_url":"https://github.com/cobbzilla.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"s3s3mirror\n==========\n\nA utility for mirroring content from one S3 bucket to another.\n\nDesigned to be lightning-fast and highly concurrent, with modest CPU and memory requirements.\n\nAn object will be copied if and only if at least one of the following holds true:\n\n* The object does not exist in the destination bucket.\n* The \"sync strategy\" triggers (by default uses the Etag sync strategy)\n    * Etag Strategy (Default): If the size or Etags don't match between the source and destination bucket.\n    * Size Strategy: If the sizes don't match between the source and destination bucket.\n    * Size and Last Modified Strategy: If the source and destination objects have a different size, or the source bucket object has a more recent last modified date. \n\nWhen copying, the source metadata and ACL lists are also copied to the destination object.\n\nNote: [the 2.1-stable branch](https://github.com/cobbzilla/s3s3mirror/tree/2.1-stable) supports copying to/from local directories.\n\n### Motivation\n\nI started with \"s3cmd sync\" but found that with buckets containing many thousands of objects, it was incredibly slow\nto start and consumed *massive* amounts of memory. So I designed s3s3mirror to start copying immediately with an intelligently\nchosen \"chunk size\" and to operate in a highly-threaded, streaming fashion, so memory requirements are much lower.\n\nRunning with 100 threads, I found the gating factor to be *how fast I could list items from the source bucket* (!?!)\nWhich makes me wonder if there is any way to do this faster. I'm sure there must be, but this is pretty damn fast.\n\n### AWS Credentials\n\n* s3s3mirror will first look for credentials in your system environment. If variables named AWS\\_ACCESS\\_KEY\\_ID and AWS\\_SECRET\\_ACCESS\\_KEY are defined, then these will be used.\n* Next, it checks for a ~/.s3cfg file (which you might have for using s3cmd). If present, the access key and secret key are read from there.\n* IAM Roles can be used on EC2 instances by specifying the --iam flag\n* If none of the above is found, it will error out and refuse to run.\n\n### System Requirements\n\n* Java 8 or higher\n\n### Building\n\n    mvn package\n\nNote that s3s3mirror now has a prebuilt jar checked in to github, so you'll only need to do this if you've been playing with the source code.\nThe above command requires that Maven 3 is installed.\n\n### License\n\ns3s3mirror is available under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).\n\n### Usage\n\n    s3s3mirror.sh [options] \u003csource-bucket\u003e[/src-prefix/path/...] \u003cdestination-bucket\u003e[/dest-prefix/path/...]\n\n### Versions\n\nThe 1.x branch (currently master) has been in use by the most number of people and is the most battle tested.\n\nThe 2.x branch supports copying between S3 and any local filesystem. It has seen heavy use and performs well, but is not as widely used as the 1.x branch.\n\n**In the near future, the 1.x branch will offshoot from master, and the 2.x branch will be merged into master.** There are a handful of features\non the 1.x branch that have not yet been ported to 2.x. If you can live without them, I encourage you to use the 2.x branch. If you really need them,\nI encourage you to port them to the 2.x branch, if you have the ability.\n\n### Options\n\n    -c (--ctime) N           : Only copy objects whose Last-Modified date is younger than this many days\n                               For other time units, use these suffixes: y (years), M (months), d (days), w (weeks),\n                                                                         h (hours), m (minutes), s (seconds)\n    -i (--iam) : Attempt to use IAM Role if invoked on an EC2 instance\n    -P (--profile) VAL        : Use a specific profile from your credential file (~/.aws/config)\n    -m (--max-connections) N  : Maximum number of connections to S3 (default 100)\n    -n (--dry-run)            : Do not actually do anything, but show what would be done (default false)\n    -r (--max-retries) N      : Maximum number of retries for S3 requests (default 5)\n    -p (--prefix) VAL         : Only copy objects whose keys start with this prefix\n    -d (--dest-prefix) VAL    : Destination prefix (replacing the one specified in --prefix, if any)\n    -e (--endpoint) VAL       : AWS endpoint to use (or set AWS_ENDPOINT in your environment)\n    -X (--delete-removed)     : Delete objects from the destination bucket if they do not exist in the source bucket\n    -t (--max-threads) N      : Maximum number of threads (default 100)\n    -v (--verbose)            : Verbose output (default false)\n    -z (--proxy) VAL          : host:port of proxy server to use.\n                                Defaults to proxy_host and proxy_port defined in ~/.s3cfg,\n                                or no proxy if these values are not found in ~/.s3cfg\n    -u (--upload-part-size) N : The upload size (in bytes) of each part uploaded as part of a multipart request\n                                for files that are greater than the max allowed file size of 5368709120 bytes (5 GB)\n                                Defaults to 4294967296 bytes (4 GB)\n    -C (--cross-account-copy) : Copy across AWS accounts. Only Resource-based policies are supported (as\n                                specified by AWS documentation) for cross account copying\n                                Default is false (copying within same account, preserving ACLs across copies)\n                                If this option is active, the owner of the destination bucket will receive full control\n                                \n    -s (--ssl)                    : Use SSL for all S3 api operations (default false)\n    -E (--server-side-encryption) : Enable AWS managed server-side encryption (default false)\n    -l (--storage-class)\t\t  : S3 storage class \"Standard\" or \"ReducedRedundancy\" (default Standard)\n    -S (--size-only)              : Only takes size of objects in consideration when determining if a copy is required.\n    -L (--size-and-last-modified) : Uses size and last modified to determine if files have change like the AWS CLI and ignores etags. If -S (--size-only) is also specified that strategy is selected over this strategy.\n\n\n### Examples\n\nCopy everything from a bucket named \"source\" to another bucket named \"dest\"\n\n    s3s3mirror.sh source dest\n\nCopy everything from \"source\" to \"dest\", but only copy objects created or modified within the past week\n\n    s3s3mirror.sh -c 7 source dest\n    s3s3mirror.sh -c 7d source dest\n    s3s3mirror.sh -c 1w source dest\n    s3s3mirror.sh --ctime 1w source dest\n\nCopy everything from \"source/foo\" to \"dest/bar\"\n\n    s3s3mirror.sh source/foo dest/bar\n    s3s3mirror.sh -p foo -d bar source dest\n\nCopy everything from \"source/foo\" to \"dest/bar\" and delete anything in \"dest/bar\" that does not exist in \"source/foo\"\n\n    s3s3mirror.sh -X source/foo dest/bar\n    s3s3mirror.sh --delete-removed source/foo dest/bar\n    s3s3mirror.sh -p foo -d bar -X source dest\n    s3s3mirror.sh -p foo -d bar --delete-removed source dest\n\nCopy within a single bucket -- copy everything from \"source/foo\" to \"source/bar\"\n\n    s3s3mirror.sh source/foo source/bar\n    s3s3mirror.sh -p foo -d bar source source\n\nBAD IDEA: If copying within a single bucket, do *not* put the destination below the source\n\n    s3s3mirror.sh source/foo source/foo/subfolder\n    s3s3mirror.sh -p foo -d foo/subfolder source source\n*This might cause recursion and raise your AWS bill unnecessarily*\n\n###### If you've enjoyed using s3s3mirror and are looking for a warm-fuzzy feeling, consider dropping a little somethin' into my [tip jar](https://cobbzilla.org/tipjar.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcobbzilla%2Fs3s3mirror","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcobbzilla%2Fs3s3mirror","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcobbzilla%2Fs3s3mirror/lists"}