{"id":23053992,"url":"https://github.com/emcecs/objectscale-rereplication-tool","last_synced_at":"2025-04-03T04:40:42.430Z","repository":{"id":145106112,"uuid":"416519055","full_name":"EMCECS/objectscale-rereplication-tool","owner":"EMCECS","description":"A tool to inventory replication status or re-trigger replication for object versions in a bucket with an associated CRR policy.","archived":false,"fork":false,"pushed_at":"2021-10-12T22:57:09.000Z","size":150,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-02-08T18:45:41.220Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EMCECS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-12T22:41:11.000Z","updated_at":"2021-10-12T22:56:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"1b93baaf-64e0-421c-8c5b-a17af0a7081a","html_url":"https://github.com/EMCECS/objectscale-rereplication-tool","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EMCECS%2Fobjectscale-rereplication-tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EMCECS%2Fobjectscale-rereplication-tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EMCECS%2Fobjectscale-rereplication-tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EMCECS%2Fobjectscale-rereplication-tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EMCECS","download_url":"https://codeload.github.com/EMCECS/objectscale-rereplication-tool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246939177,"owners_count":20857916,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-16T00:32:25.374Z","updated_at":"2025-04-03T04:40:42.412Z","avatar_url":"https://github.com/EMCECS.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ObjectScale CRR Inventory and Re-Replication Tool\n\nA tool to inventory replication status or re-trigger replication for object versions in a bucket with an associated CRR policy.\n\n## Usage\n\n1. Install Java\n1. Download the latest [released version](https://github.com/EMCECS/objectscale-rereplication-tool/releases) of the tool.\n1. Run the tool:  \n```text\njava -jar rereplication-tool-1.0.jar -e \u003cendpoint\u003e -b \u003cbucket\u003e (-i|-r) -f \u003cinventory-file\u003e [options]\n```  \n\n### Running an Inventory (`-i`)\n\nTo run an inventory, specify the `-i` option.  An inventory will list all the versions in the bucket,\nfilter based on optional criteria, and generate a CSV report.\n\n#### Inventory Options\n\nOption | Flag | Description\n--|--|--\nFilter: Current Versions Only | `--current-version` | Only inventories the current object versions. WARNING: this could produce a very large file\nFilter: Failed Current Versions | `--failed-current-version` | Only inventories current object versions that have failed replication (this is the default)\nFilter: All Versions | `--all-versions` | Inventories all object versions in the bucket. WARNING: this could produce a massive file\nFilter: Bucket Prefix | `--prefix` | Only inventories object keys that are under the given prefix\n\n### Re-triggering Replication (`-r`)\n\nTo re-trigger replication for failed current versions, specify the `-r` option, and provide an inventory\nwith the failed object versions to re-trigger.  The tool will read the inventory file and re-trigger replication\nby issueing PUT+COPY operations for any failed current versions.\n\nNote that the tool will also accept a flat list of object keys, or a 2-column CSV with object keys in the first column\nand version IDs in the second column.  You can use this to explicitly trigger replication for those keys and versions.  Be aware that\ntriggering replication means writing a new current version of each object by copying a previous version.  Please\nbe cautious when using an explicit list of keys or version IDs that does not include is-latest or replication-status\ndetails.\n\n#### Re-Replication Options\n\nOption | Flag | Description\n--|--|--\nSupport Custom ACLs | `re-replicate-custom-acls` | Adds support for custom ACLs during re-replication. Disabled by default. Most users should not need to worry about custom ACLs, but if you know your application is using per-object ACLs, you will need to enable this to maintain them\n\n## Report Fields\n\nThe inventory report will generate a CSV with the following fields (in this order):\n\nField|Data Type|Description\n--|--|--\nKey|string|The object key (name)\nVersionId|string|The object version ID\nIsDeleteMarker|boolean|True if the version is a delete-marker\nIsLatest|boolean|True if the version is the current (latest) version of the key\nLastModified|datetime|The time the version was written to the bucket\nETag|string|The S3 ETag (almost always the MD5-hex of the data)\nSize|number|The size of the object data in bytes\nOwnerId|string|The object owner (S3 user)\nReplicationStatus|string|Replication status of the object (`PENDING`, `COMPLETE`, or `FAILED`)\n\nNote: The CSV file will have a header row with these field names in it.\n\n## Full CLI Syntax\n```text\nusage: java -jar rereplication-tool-1.2.jar -e \u003cendpoint\u003e -b \u003cbucket\u003e\n            (-i|-r) -f \u003cinventory-file\u003e [options]\noptions:\n -a,--access-key \u003caccess-key\u003e         The AWS Access Key ID to access the\n                                      bucket (if not using an AWS profile)\n    --all-versions                    Inventory all object versions\n -b,--bucket \u003cbucket-name\u003e            The bucket to inventory\n -c,--current-version                 Only inventory the current object\n                                      versions (do not include\n                                      previous/non-current versions)\n -d,--debug                           Debug logging\n -e,--endpoint \u003cendpoint-uri\u003e         ObjectScale S3 endpoint URL. This\n                                      includes scheme and port (i.e.\n                                      https://10.1.4.5:9021)\n -f,--file \u003cinventory-file\u003e           The file to read when triggering\n                                      re-replication, or write when\n                                      generating an inventory (in CSV\n                                      format)\n    --failed-current-version          Only inventory the current object\n                                      versions that failed replication (do\n                                      not include previous/non-current\n                                      versions or replicated versions) -\n                                      this is the default\n    --force-overwrite                 When performing inventory, if the\n                                      inventory file already exists,\n                                      overwrite it\n -h,--help                            Print this help text\n -i,--inventory                       Perform an inventory of the bucket\n                                      and output to CSV\n -p,--profile \u003cprofile-name\u003e          The AWS CLI profile to use for\n                                      credentials, if other than default\n                                      (configuration must be set for this\n                                      profile)\n    --prefix \u003cbucket-prefix\u003e          Only inventory objects in the bucket\n                                      that are under this prefix\n -r,--re-replicate                    Trigger re-replication of a list of\n                                      objects from a provided file.\n                                      Re-replication is triggered by\n                                      COPYing the object to itself to\n                                      create a new version, which will\n                                      trigger CRR policy replication for\n                                      that new version\n    --re-replicate-custom-acls        Adds support for custom ACLs during\n                                      re-replication. WARNING: this will\n                                      triple the API calls to S3 and take\n                                      longer to complete\n -s,--secret-key \u003csecret-key\u003e         The AWS Secret Key to access the\n                                      bucket (if not using an AWS profile)\n -t,--threads \u003cthread-count\u003e          The size of the thread pool used to\n                                      HEAD and COPY objects for inventory\n                                      or re-replication\n    --unsafe-disable-ssl-validation   Disables SSL/TLS certificate\n                                      validation - this is NOT safe!\n -v,--verbose                         Verbose logging\n```\n\n# Dependency Licenses\n\nTo generate a dependency license report, simply execute the following build task:\n\n```shell\n./gradlew generateLicenseReport\n```\n\n# API Docs\n\nJavadoc is available here: https://emcecs.github.io/objectscale-rereplication-tool/latest/javadoc/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femcecs%2Fobjectscale-rereplication-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femcecs%2Fobjectscale-rereplication-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femcecs%2Fobjectscale-rereplication-tool/lists"}