{"id":16677560,"url":"https://github.com/jamesmcm/s3rename","last_synced_at":"2025-03-21T18:32:05.260Z","repository":{"id":56616217,"uuid":"260439924","full_name":"jamesmcm/s3rename","owner":"jamesmcm","description":"Tool to mass-rename S3 keys","archived":false,"fork":false,"pushed_at":"2020-10-28T17:09:12.000Z","size":97,"stargazers_count":19,"open_issues_count":9,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-13T13:26:52.644Z","etag":null,"topics":["hacktoberfest","rust","rust-lang","s3"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jamesmcm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-01T11:09:59.000Z","updated_at":"2024-06-06T21:13:21.000Z","dependencies_parsed_at":"2022-08-15T22:00:50.633Z","dependency_job_id":null,"html_url":"https://github.com/jamesmcm/s3rename","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesmcm%2Fs3rename","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesmcm%2Fs3rename/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesmcm%2Fs3rename/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesmcm%2Fs3rename/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jamesmcm","download_url":"https://codeload.github.com/jamesmcm/s3rename/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221817508,"owners_count":16885554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","rust","rust-lang","s3"],"created_at":"2024-10-12T13:26:49.698Z","updated_at":"2024-10-28T10:34:58.794Z","avatar_url":"https://github.com/jamesmcm.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3rename\n\ns3rename is a tool to mass-rename keys within an S3 bucket.\n\nThe interface is designed to mimic the Perl [rename](https://www.unix.com/man-page/linux/1/prename/) utility on\nGNU/Linux (also known as `prename` and `perl-rename`).\n\ns3rename uses asynchronous requests to rename the keys in parallel, as\nfast as possible.\n\nThe expression provided is applied to the entire key, allowing you to\nrename parent \"directories\".\n\nObject properties are preserved, unless the `--no-preserve-properties` \nflag is used.\n\nObject ACL (Access Control List) settings will also be preserved, unless\nthe `--no-preserve-acl` flag is used.\n\nIt is highly recommended to use the `--dry-run` flag at first to ensure the\nchanges reflect what you intend.\n\n## Usage\n\nNote that regardless of the prefix used for filtering in the S3 URL\nprovided, the regex is applied to the __whole key__. This is necessary\nto allow for full changes of the directory structure.\n\n```\nUSAGE:\n    s3rename [FLAGS] [OPTIONS] \u003cexpr\u003e \u003cs3-url\u003e\n\nFLAGS:\n    -n, --dry-run                   Do not carry out modifications (only print)\n    -h, --help                      Prints help information\n        --no-anonymous-groups       Do not allow anonymous capture groups i.e. \\1, \\2 - may be useful when dealing with\n                                    keys containing backslashes\n        --no-preserve-acl           Do not preserve Object ACL settings (all will be set to private)\n        --no-preserve-properties    Do not preserve object properties (saves retrieving per-object details) - using this\n                                    flag will remove any encryption\n        --no-overwrite              Do not overwrite existing keys\n    -q, --quiet                     Do not print key modifications\n    -V, --version                   Prints version information\n    -v, --verbose                   Print debug messages\n\nOPTIONS:\n        --aws-region \u003caws-region\u003e    AWS Region (will be taken from bucket region if not overridden here)\n        --canned-acl \u003ccanned-acl\u003e    Canned access_control_list override - sets this ACL for all renamed keys [possible\n                                     values: private, public-read, public-read-write, aws-exec-read, authenticated-read,\n                                     bucket-owner-read, bucket-owner-full-control]\n\nARGS:\n    \u003cexpr\u003e      Perl RegEx Replace Expression (only s/target/replacement/flags form supported)\n    \u003cs3-url\u003e    S3 URL: s3://bucket-name/optional-key-prefix\n```\n\n### Examples\n\ns3rename uses the Perl regular expression format (like sed) to rename\nfiles:\n\n```\n$ aws s3 ls s3://s3rename-test-bucket --recursive\n2020-05-01 12:30:25         16 testnewfile.txt\n\n$ ./s3rename \"s/new/old\" s3://s3rename-test-bucket/test\nRenaming testnewfile.txt to testoldfile.txt\n\n$ aws s3 ls s3://s3rename-test-bucket --recursive\n2020-05-01 12:33:48         16 testoldfile.txt\n```\n\nThe `--dry-run` flag will print changes to be made without carrying them\nout. This is __highly__ recommended before running changes.\n\nBy default ACL settings for objects will be preserved (unless\n`--no-preserve-acl` is passed), however this does\nnot apply to ACL settings which depend on the bucket ACL (i.e. public\nwrite access).\n\nThe `--canned-acl \u003ccanned-acl\u003e` option can be used to set the ACL of all\nrenamed objects to the provided [canned ACL](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl).\nNote that some canned ACLs are affected by bucket settings (such as\n`public-read-write`).\n\n### Renaming flat files to a nested directory structure for AWS Glue\n\nThis program was originally inspired by the need to rename the keys of \nthousands of files which were stored in a flat structure, so that they\ncould be correctly parsed by AWS Glue which requires a nested structure \nwith the \"directory\" names corresponding to the partitions.\n\n```\n$ aws s3 ls s3://s3rename-test-bucket/datatest --recursive\n2020-05-01 12:38:33          0 datatest/\n2020-05-01 12:38:43          0 datatest/data_2020-04-01.txt\n2020-05-01 12:38:43          0 datatest/data_2020-04-02.txt\n2020-05-01 12:38:43          0 datatest/data_2020-04-03.txt\n2020-05-01 12:38:43          0 datatest/data_2020-04-04.txt\n2020-05-01 12:38:43          0 datatest/data_2020-04-05.txt\n2020-05-01 12:38:43          0 datatest/data_2020-05-01.txt\n2020-05-01 12:38:43          0 datatest/data_2020-05-02.txt\n2020-05-01 12:38:43          0 datatest/data_2020-06-01.txt\n\n$ ./s3rename 's/data_(?P\u003cyear\u003e[0-9]{4})-(?P\u003cmonth\u003e[0-9]{2})-(?P\u003cday\u003e[0-9]{2}).txt/year=$year\\/month=$month\\/day=$day\\/data_$year-$month-$day.txt/g' s3://s3rename-test-bucket/datatest\nRenaming datatest/ to datatest/\nRenaming datatest/data_2020-04-01.txt to datatest/year=2020/month=04/day=01/data_2020-04-01.txt\nRenaming datatest/data_2020-04-02.txt to datatest/year=2020/month=04/day=02/data_2020-04-02.txt\nRenaming datatest/data_2020-04-03.txt to datatest/year=2020/month=04/day=03/data_2020-04-03.txt\nRenaming datatest/data_2020-04-04.txt to datatest/year=2020/month=04/day=04/data_2020-04-04.txt\nRenaming datatest/data_2020-04-05.txt to datatest/year=2020/month=04/day=05/data_2020-04-05.txt\nRenaming datatest/data_2020-05-01.txt to datatest/year=2020/month=05/day=01/data_2020-05-01.txt\nRenaming datatest/data_2020-05-02.txt to datatest/year=2020/month=05/day=02/data_2020-05-02.txt\nRenaming datatest/data_2020-06-01.txt to datatest/year=2020/month=06/day=01/data_2020-06-01.txt\n\n$ aws s3 ls s3://s3rename-test-bucket/datatest --recursive\n2020-05-01 12:38:33          0 datatest/\n2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=01/data_2020-04-01.txt\n2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=02/data_2020-04-02.txt\n2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=03/data_2020-04-03.txt\n2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=04/data_2020-04-04.txt\n2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=05/data_2020-04-05.txt\n2020-05-01 12:39:38          0 datatest/year=2020/month=05/day=01/data_2020-05-01.txt\n2020-05-01 12:39:38          0 datatest/year=2020/month=05/day=02/data_2020-05-02.txt\n2020-05-01 12:39:38          0 datatest/year=2020/month=06/day=01/data_2020-06-01.txt\n```\n\nNote the use of single quotes for the sed regex string to avoid issues\nwith the $ symbols in the shell.\n\nYou can also use anonymous capture groups, with the replacement parts\nmarked either by $ or \\, i.e.:\n\n```\n's/data_([0-9]{4})-([0-9]{2})-([0-9]{2}).txt/year=\\1\\/month=\\2\\/day=\\3\\/data_\\1-\\2-\\3.txt/g'\n```\n\nis equivalent to the above, and equivalent to:\n\n\n```\n's/data_([0-9]{4})-([0-9]{2})-([0-9]{2}).txt/year=$1\\/month=$2\\/day=$3\\/data_$1-$2-$3.txt/g'\n```\n\nUse multiple dollar symbols to escape the dollars (for literal dollar\nsymbols).\n\n## Installation\n\ns3rename depends on OpenSSL at runtime.\n\nBuilding from source requires a Rust toolchain and Cargo.\n\nIf you use this tool please consider starring [the Github repo](https://github.com/jamesmcm/s3rename) and voting\nfor the package on [the AUR](https://aur.archlinux.org/packages/s3rename/).\n\n### Arch Linux (via [the Arch User Repository (AUR)](https://aur.archlinux.org/packages/s3rename/))\n\nUsing the yay AUR helper:\n\n```bash\n$ yay -S s3rename\n```\n\nAlternatively you can [manually install the package from the AUR](https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages).\n\n### Cargo (via [crates.io](https://crates.io/crates/s3rename))\n\n```bash\n$ cargo install s3rename\n```\n\nThe `s3rename` binary will then be in your Cargo binaries directory (and\nthis should already be on your `$PATH`.\n\n### Cargo (from this repository)\n\ns3rename can be installed via Cargo from this cloned repository:\n\n```bash\n$ git clone git@github.com:jamesmcm/s3rename.git\n$ cd s3rename\n$ cargo install --path .\n```\n\nThe `s3rename` binary will then be in your Cargo binaries directory (and\nthis should already be on your `$PATH`.\n\n### Linux x86_64 binary\n\nStatic binaries compiled for Linux x86_64 are available in the [Github\nreleases](https://github.com/jamesmcm/s3rename/releases).\n\n## Known Issues\n\n* Buckets and objects using [S3 Object\n  Lock](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html)\n  are currently unsupported.\n* Expiry rules set with prefixes in the bucket properties will not be \n  updated (so any keys moved out of the scope of these rules will no\n  longer have the expiry rules applied). In the future a specific\n  command to update expiry rules may be added.\n* s3rename does not support custom encryption keys for encrypted buckets (i.e. if\n  your encryption key is not generated and stored by AWS). This could be added in a\n  future version.\n* The rename operation is not fully atomic (since it involves\n  separate CopyObject and DeleteObject requests) - this means that if\n  s3rename is terminated suddenly during operation, the bucket could be left with\n  copied files where the originals have not been renamed (re-running\n  s3rename with the same arguments would fix this). \n\n## S3 Billing\n\ns3rename operates on keys within the same bucket and so should trigger\nno [data transfer costs](https://aws.amazon.com/s3/pricing/).\n\nWhilst it does use CopyObjectRequests to carry out the renaming, the\nadditional data does not exist for long and should trigger no costs for\ndata usage:\n\nRegarding billing for data storage, the [S3 Billing documentation](https://aws.amazon.com/s3/faqs/#Billing) states:\n\n\u003e The volume of storage billed in a month is based on the average storage used throughout the month.\n\u003e This includes all object data and metadata stored in buckets that you created under your AWS account.\n\u003e We measure your storage usage in “TimedStorage-ByteHrs,” which are added up at the end of the month to generate your monthly charges.\n\n## License\n\ns3rename is licensed under either of:\n\n* Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)\n* MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)\n\nat your option.\n\n## Contribution\n\nUnless you explicitly state otherwise, any contribution intentionally submitted\nfor inclusion in the work by you, as defined in the Apache-2.0 license,\nshall be dual licensed as above, without any additional terms or conditions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesmcm%2Fs3rename","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjamesmcm%2Fs3rename","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesmcm%2Fs3rename/lists"}