{"id":13908127,"url":"https://github.com/scality/s3utils","last_synced_at":"2026-02-10T14:04:17.006Z","repository":{"id":37710659,"uuid":"151485124","full_name":"scality/s3utils","owner":"scality","description":"S3 Connector and Zenko Utilities","archived":false,"fork":false,"pushed_at":"2026-01-20T16:47:48.000Z","size":2007,"stargazers_count":5,"open_issues_count":5,"forks_count":2,"subscribers_count":47,"default_branch":"development/1","last_synced_at":"2026-01-21T01:39:36.620Z","etag":null,"topics":["artesca","cloud","object","ring","s3","storage","zenko"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scality.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2018-10-03T21:51:24.000Z","updated_at":"2026-01-12T20:55:37.000Z","dependencies_parsed_at":"2023-10-20T18:29:35.176Z","dependency_job_id":"eedc5eae-9edb-4cd5-86c2-928586e521af","html_url":"https://github.com/scality/s3utils","commit_stats":null,"previous_names":[],"tags_count":123,"template":false,"template_full_name":null,"purl":"pkg:github/scality/s3utils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scality","download_url":"https://codeload.github.com/scality/s3utils/tar.gz/refs/heads/development/1","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fs3utils/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28844937,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-28T10:53:21.605Z","status":"ssl_error","status_checked_at":"2026-01-28T10:53:20.789Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artesca","cloud","object","ring","s3","storage","zenko"],"created_at":"2024-08-06T23:02:29.184Z","updated_at":"2026-01-28T11:18:33.008Z","avatar_url":"https://github.com/scality.png","language":"JavaScript","funding_links":[],"categories":["HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"# s3utils\nS3 Connector and Zenko Utilities\n\nRun the Docker container as (replace `\u003ctag\u003e` with the tag of s3utils image to be used)\n```\ndocker run --net=host -e 'ACCESS_KEY=accessKey' -e 'SECRET_KEY=secretKey' -e 'ENDPOINT=http://127.0.0.1:8000' -e 'REPLICATION_GROUP_ID=RG001'\nghcr.io/scality/s3utils:\u003ctag\u003e node scriptName bucket1[,bucket2...]\n```\n\n## Trigger CRR on existing objects\n\n1. Enable versioning and set up replication on the bucket.\n2. Run script as\n```\nnode crrExistingObjects.js testbucket1,testbucket2\n```\n\n### Mandatory environment variables,\n\n* **REPLICATION_GROUP_ID**\n\n### Optional environment variables,\n\n* **DEBUG**: set to 1 to output debug level information.\n\n### Extra environment variables\n\nAdditionally, the following extra environment variables can be passed\nto the script to modify its behavior:\n\n#### TARGET_REPLICATION_STATUS\n\nComma-separated list of replication statuses to target for CRR\nrequeueing. The recognized statuses are:\n\n* **NEW**: No replication status is attached to the object. This is\n   the state of objects written without any CRR policy attached to\n   the bucket that would have triggered CRR on them.\n\n* **PENDING**: The object replication status is PENDING.\n\n* **COMPLETED**: The object replication status is COMPLETED.\n\n* **FAILED**: The object replication status is FAILED.\n\n* **REPLICA**: The object replication status is REPLICA (objects that\n   were put to a target site via CRR have this status).\n\nThe default script behavior is to affect objects that have no\nreplication status attached (so equivalent to\n`TARGET_REPLICATION_STATUS=NEW`).\n\nExamples:\n\n`TARGET_REPLICATION_STATUS=PENDING,COMPLETED`\n\nRequeue objects that either have a replication status of PENDING or\nCOMPLETED for CRR, do not requeue the others.\n\n`TARGET_REPLICATION_STATUS=NEW,PENDING,COMPLETED,FAILED`\n\nTrigger CRR on all original source objects (not replicas) in a bucket.\n\n`TARGET_REPLICATION_STATUS=REPLICA`\n\nFor disaster recovery notably, it may be useful to reprocess REPLICA\nobjects to re-sync a backup bucket to the primary site.\n\n#### TARGET_PREFIX\n\nOptional prefix of keys to scan\n\nExample:\n\n`TARGET_PREFIX=foo/`\n\nOnly scan keys beginning with \"foo/\"\n\n\n#### STORAGE_TYPE\n\nComma-separated list of the destination storage location types. This is used\nonly if a replication destination is a public cloud.\n\nThe recognized statuses are:\n\n* **aws_s3**: The destination storage location type is Amazon S3.\n\n* **azure**: The destination storage location type is Microsoft Azure Blob\n   Storage.\n\n* **gcp**: The destination storage location type is Google Cloud Storage.\n\nExamples:\n\n`STORAGE_TYPE=aws_s3`\n\nThe destination storage location type is AWS.\n\n`STORAGE_TYPE=aws_s3,azure,gcp`\n\nThe destination storage type is a one-to-many configuration replicating to AWS,\nAzure, and GCP destination storage locations.\n\n#### WORKERS\n\nSpecify how many parallel workers should run to update object\nmetadata. The default is 10 parallel workers.\n\nExample:\n\n`WORKERS=50`\n\n#### MAX_UPDATES, MAX_SCANNED\n\nSpecify a maximum number of metadata updates (MAX_UPDATES) or scanned\nentries (MAX_SCANNED) to execute before stopping the script.\n\nIf the script reaches this limit, it outputs a log line containing\nthe KeyMarker and VersionIdMarker to pass to the next invocation (as\nenvironment variables `KEY_MARKER` and `VERSION_ID_MARKER`) and the\nupdated bucket list without the already completed buckets. At the next\ninvocation of the script, those two environment variables must be\nset and the updated bucket list passed on the command line to resume\nwhere the script stopped.\n\nThe default is unlimited (will process the complete listing of buckets\npassed on the command line).\n\n**If the script queues too many objects and Backbeat cannot\n process them quickly enough, Kafka may drop the oldest entries**,\n and the associated objects will stay in the **PENDING** state\n permanently without being replicated. When the number of objects\n is large, it is a good idea to limit the batch size and wait\n for CRR to complete between invocations.\n\nExamples:\n\n`MAX_UPDATES=10000`\n\nThis limits the number of updates to 10,000 objects, which requeues\na maximum of 10,000 objects to replicate before the script stops.\n\n`MAX_SCANNED=10000`\n\nThis limits the number of scanned objects to 10,000 objects before\nthe script stops.\n\n#### KEY_MARKER\n\nSet to resume from where an earlier invocation stopped (see\n[MAX_UPDATES, MAX_SCANNED](#MAX_UPDATES)).\n\nExample:\n\n`KEY_MARKER=\"some/key\"`\n\n#### VERSION_ID_MARKER\n\nSet to resume from where an earlier invocation stopped (see\n[MAX_UPDATES](#MAX_UPDATES)).\n\nExample:\n\n`VERSION_ID_MARKER=\"123456789\"`\n\n\n### Example use cases\n\n#### CRR existing objects after setting a replication policy for the first time\n\nFor this use case, it's not necessary to pass any extra environment\nvariable, because the default behavior is to process objects without a\nreplication status attached.\n\nTo avoid requeuing too many entries at once, pass this value:\n\n```\nexport MAX_UPDATES=10000\n```\n\n#### Re-queue objects stuck in PENDING state\n\nIf Kafka has dropped replication entries, leaving objects stuck in a\nPENDING state without being replicated, pass the following extra\nenvironment variables to reprocess them:\n\n```\nexport TARGET_REPLICATION_STATUS=PENDING\nexport MAX_UPDATES=10000\n```\n\n**Warning**: This may cause replication of objects already in the\nKafka queue to repeat. To avoid this, set the backbeat consumer\noffsets of \"backbeat-replication\" Kafka topic to the latest topic\noffsets before launching the script, to skip over the existing\nconsumer log.\n\n#### Replicate entries that failed a previous replication\n\nIf entries have permanently failed to replicate with a FAILED\nreplication status and were lost in the failed CRR API, it's still\npossible to re-attempt replication later with the following\nextra environment variables:\n\n```\nexport TARGET_REPLICATION_STATUS=FAILED\nexport MAX_UPDATES=10000\n```\n\n#### Re-sync a primary site completely to a new DR site\n\nTo re-sync objects to a new DR site (for example, when the original\nDR site is lost) force a new replication of all original objects\nwith the following environment variables (after setting the proper\nreplication configuration to the DR site bucket):\n\n```\nexport TARGET_REPLICATION_STATUS=NEW,PENDING,COMPLETED,FAILED\nexport MAX_UPDATES=10000\n```\n\n#### Re-sync a DR site back to the primary site\n\nWhen objects have been lost from the primary site you can re-sync\nobjects from the DR site to the primary site by re-syncing the\nobjects that have a REPLICA status with the following environment\nvariables (after setting the proper replication configuration\nfrom the DR bucket to the primary bucket):\n\n```\nexport TARGET_REPLICATION_STATUS=REPLICA\nexport MAX_UPDATES=10000\n```\n\n# Empty a versioned bucket\n\nThis script deletes all versions of objects in the bucket, including delete markers,\nand aborts any ongoing multipart uploads to prepare the bucket for deletion.\n\n**Note: This deletes the data associated with objects and is not recoverable**\n```\nnode cleanupBuckets.js testbucket1,testbucket2\n```\n\n# List objects by replication status\n\nTwo scripts are available for listing objects by replication status:\n\n## List objects with FAILED replication status\n\nTo list only objects that have failed replication:\n\n```\nnode listFailedObjects.js testbucket1,testbucket2\n```\n\n## List objects by configurable replication status\n\nTo list objects with a specific or multiple replication statuses:\n\n```\nnode listObjectsByReplicationStatus.js testbucket1,testbucket2\n```\n\n## Required environment variables\n\nThe following environment variable is **required** for `listObjectsByReplicationStatus.js`:\n\n### REPLICATION_STATUS\n\nComma-separated list of replication statuses to list. The recognized statuses are:\n\n* **NEW**: No replication status is attached to the object.\n* **PENDING**: The object replication status is PENDING.\n* **COMPLETED**: The object replication status is COMPLETED.\n* **FAILED**: The object replication status is FAILED.\n* **REPLICA**: The object replication status is REPLICA.\n\nExamples:\n\n`REPLICATION_STATUS=PENDING`\n\nList objects that have a replication status of PENDING.\n\n`REPLICATION_STATUS=PENDING,FAILED`\n\nList objects that either have a replication status of PENDING or FAILED.\n\n# Verify existence of sproxyd keys\n\nThis script verifies that :\n\n1. all sproxyd keys referenced by objects in S3 buckets exist on the RING\n2. sproxyd keys are unique across versions of the same object\n3. object metadata is not an empty JSON object '{}'\n\nIt can help to identify objects affected by the S3C-1959 bug (1), or\none of S3C-2731, S3C-3778 (2), S3C-5987 (3).\n\nThe script can also be used to generate block digests from the listing\nresults as seen by the leader, for the purpose of finding\ndiscrepancies between raft members, that can be caused by bugs like\nS3C-5739.\n\n## Usage\n\n```\nnode verifyBucketSproxydKeys.js\n```\n\n## Mandatory environment variables:\n\n* **BUCKETD_HOSTPORT**: ip:port of bucketd endpoint\n\n* **SPROXYD_HOSTPORT**: ip:port of sproxyd endpoint\n\n* One of:\n\n  * **BUCKETS**: comma-separated list of buckets to scan\n  * Example : `BUCKETS=bucket1,bucket2,bucket3`\n\n* or:\n\n  * **RAFT_SESSIONS**: comma-separated list of raft sessions to scan\n  * Example :\n    ```\n    docker run --net=host \\\n    -e 'BUCKETD_HOSTPORT=127.0.0.1:9000' \\\n    -e 'SPROXYD_HOSTPORT=127.0.0.1:8181' \\\n    -e 'RAFT_SESSIONS=1' \\\n    ghcr.io/scality/s3utils:1.12.5 \\\n    node verifyBucketSproxydKeys.js\n    ```\n\n* or:\n\n  * **KEYS_FROM_STDIN**: reads objects to scan from stdin if this\n    environment variable is set, where the input is a stream of JSON\n    objects, each of the form:\n    `{\"bucket\":\"bucketname\",\"key\":\"objectkey\\u0000objectversion\"}` -\n    **note**: if `\\u0000objectversion` is not present, it checks the master key\n  * Example :\n    ```\n    cat raft_session_1_output.txt | docker run -i --net=host --rm \\\n    -e 'SPROXYD_HOSTPORT=127.0.0.1:8181' \\\n    -e 'BUCKETD_HOSTPORT=127.0.0.1:9000' \\\n    -e 'KEYS_FROM_STDIN=1' \\\n    ghcr.io/scality/s3utils:1.12.5 \\\n    node verifyBucketSproxydKeys.js | tee ring_scan_raft_session_1.txt\n    ```\n\n## Optional environment variables:\n\n* **WORKERS**: concurrency value for sproxyd requests (default 100)\n\n* **FROM_URL**: URL from which to resume scanning (\"s3://bucket[/key]\")\n\n* **VERBOSE**: set to 1 for more verbose output (shows one line for every sproxyd key)\n\n* **LOG_PROGRESS_INTERVAL**: interval in seconds between progress update log lines (default 10)\n\n* **LISTING_LIMIT**: number of keys to list per listing request (default 1000)\n\n* **MPU_ONLY**: only scan objects uploaded with multipart upload method\n\n* **NO_MISSING_KEY_CHECK**: do not check for existence of sproxyd\n    keys, for a performance benefit - other checks like duplicate keys\n    and missing metadata are still done\n\n* **LISTING_DIGESTS_OUTPUT_DIR**: output listing digests into the\n    specified directory (in the LevelDB format)\n\n* **LISTING_DIGESTS_BLOCK_SIZE**: number of keys in each listing\n    digest block (default 1000)\n\n## Output\n\nThe output of the script consists of JSON log lines. The most\nimportant ones are described below.\n\n### Info\n\n```\nprogress update\n```\n\nThis log message is reported at regular intervals, every 10 seconds by\ndefault unless LOG_PROGRESS_INTERVAL environment variable is\ndefined. It shows statistics about the scan in progress.\n\nLogged fields:\n\n* **scanned**: number of objects scanned\n\n* **skipped**: number of objects skipped (in case MPU_ONLY is set)\n\n* **haveMissingKeys**: number of object versions found with at least\n    one missing sproxyd key\n\n* **haveDupKeys**: number of object versions found with at least one\n    sproxyd key shared with another object version\n\n* **haveEmptyMetadata**: number of objects found with an empty metadata\n    blob i.e. `{}`\n\n* **url**: current URL `s3://bucket[/object]` for the current listing\n    iteration, can be used to resume a scan from this point passing\n    FROM_URL environment variable.\n\nExample:\n\n```\n{\"name\":\"s3utils:verifyBucketSproxydKeys\",\"time\":1585700528238,\"skipped\":0,\"scanned\":115833,\n\"haveMissingKeys\":2,\"haveDupKeys\":0,\"url\":\"s3://some-bucket/object1234\",\"level\":\"info\",\n\"message\":\"progress update\",\"hostname\":\"e923ec732b42\",\"pid\":67}\n```\n\n### Issue reports\n\n```\nsproxyd check reported missing key\n```\n\nThis log message is reported when a missing sproxyd key has been found\nin an object. The message may appear more than once for the same\nobject in case multiple sproxyd keys are missing.\n\nLogged fields:\n\n* **objectUrl**: URL of the affected object version:\n    `s3://bucket/object[%00UrlEncodedVersionId]`\n\n* **sproxydKey**: sproxyd key that is missing\n\n```\nduplicate sproxyd key found\n```\n\nThis message is reported when two versions (possibly the same) are\nsharing an identical, duplicated sproxyd key.\n\nIt represents a data loss risk, since data loss would occur when\ndeleting one of the versions with a duplicate key, causing the other\nversion to refer to deleted data in its location array.\n\nLogged fields:\n\n* **objectUrl**: URL of the first affected object version with an\n    sproxyd key shared with the second affected object:\n    `s3://bucket/object[%00UrlEncodedVersionId]`\n\n* **objectUrl2**: URL of the second affected object version with an\n    sproxyd key shared with the first affected object:\n    `s3://bucket/object[%00UrlEncodedVersionId]`\n\n* **sproxydKey**: sproxyd key that is duplicated between the two\n    object versions\n\n```\nobject with empty metadata found\n```\n\nThis message is reported when an object that has an empty metadata\nblob as its value is found.\n\nThose objects may have been created during an upgrade scenario where\nconnectors were upgraded to a S3C version \u003e= 7.4.10.2, but metadata\nstateful nodes were not yet upgraded, resulting in the bug described\nby S3C-5987.\n\nSuch objects can normally be safely deleted because they are generated\nonly when an \"AbortMultipartUpload\" operation is executed on the\nobject, meaning that the object was the target of an MPU in progress\nthat was aborted, and it was the last state of this object (otherwise\nit would have been updated by non-empty metadata).\n\n**Note**: even on versioned buckets, the affected keys are always\n  non-versioned.\n\nLogged fields:\n\n* **objectUrl**: URL of the affected object: `s3://bucket/object`\n\n**How to delete all such objects**:\n\nAssuming `verifyBucketSproxydKeys.log` contains the output of the\nprevious run of the verify script, this shell command can be used to\ndelete the objects with empty metadata:\n\n```\nJQ_SCRIPT='\nselect(.message == \"object with empty metadata found\") |\n.objectUrl |\nsub(\"s3://\";\"http://localhost:9000/default/bucket/\")'\njq -r \"${JQ_SCRIPT}\" verifyBucketSproxydKeys.log | while read url; do\n    [ \"$(curl -s \"${url}\")\" = '{}' ] \u0026\u0026 {\n        echo \"deleting ${url}\";\n        curl -XDELETE \"${url}\" || echo \"failed to delete ${url}\";\n    } || true\ndone\n```\n\n\n# Compare follower's databases against leader view\n\nThe **CompareRaftMembers/followerDiff** tool compares Metadata leveldb\ndatabases on the repd follower on which it is run against the leader's\nview, and outputs the differences to the file path given as the\nDIFF_OUTPUT_FILE environment variable.\n\nIn this file, it outputs each key that differs as line-separated JSON\nentries, where each entry can be one of:\n\n- `[{ key, value }, null]`: this key is present on this follower but\n  not on the leader\n\n- `[null, { key, value }]`: this key is not present on this follower\n  but is present on the leader\n\n- `[{ key, value: \"{value1}\" }, { key, value: \"{value2}\" }]`: this key\n  has a different value between this follower and the leader: \"value1\"\n  is the value seen on the follower and \"value2\" the value seen on the\n  leader.\n\nIt is possible and recommended to speed-up the comparison by providing\na pre-computed digests database via the LISTING_DIGESTS_INPUT_DIR\nenvironment variable, so that ranges of keys that match the digests\ndatabase do not have to be checked by querying the leader. The\npre-computed digests database can be generated via a run of\n\"verifyBucketSproxydKeys\" script, providing it the\nLISTING_DIGESTS_OUTPUT_DIR environment variable.\n\n## Usage\n\n```\nnode followerDiff.js\n```\n\n## Mandatory environment variables\n\n* **BUCKETD_HOSTPORT**: ip:port of bucketd endpoint\n\n* **DATABASES**: space-separated list of databases to scan\n\n* **DIFF_OUTPUT_FILE**: file path where diff output will be stored\n\n## Optional environment variables\n\n* **LISTING_DIGESTS_INPUT_DIR**: read listing digests from the\n  specified LevelDB database\n\n* **PARALLEL_SCANS**: number of databases to scan in parallel (default 4)\n\n* **EXCLUDE_FROM_CSEQS**: mapping of raft sessions to filter on, where\n  keys are raft session IDs and values are the cseq value for that\n  raft session. Filtering will be based on all oplog records more\n  recent than the given \"cseq\" for the raft session. Input diff\n  entries not belonging to one of the declared raft sessions are\n  discarded from the output. The value must be in the following JSON\n  format:\n\n  * `{\"rsId\":cseq[,\"rsId\":cseq...]}`\n\n  * Example: `{\"1\":1234,\"4\":4567,\"6\":6789}`\n\n  * This configuration would cause diff entries of which bucket/key\n    appear in one of the following to be discarded from the output:\n\n    * oplog of raft session 1 after cseq=1234\n\n    * or oplog of raft session 4 after cseq=4567\n\n    * or oplog of raft session 6 after cseq=6789\n\n    * or any other raft session's oplog at any cseq\n\n## Scan Procedure with Stable Leader\n\n**Note**: you may use this procedure when leaders are stable,\n  i.e. leader changes are rare in normal circumstances. When leaders\n  are unstable, use the [unstable\n  leader](#scan-procedure-with-unstable-leader) procedure instead.\n\nIn order to scan a set of raft sessions to detect discrepancies\nbetween raft members, you may use the following procedure involving\nthe two scan tools\n[verifyBucketSproxydKeys](#verify-existence-of-sproxyd-keys) and\n[followerDiff](#compare-followers-databases-against-leader-view).\n\nMake sure that no leader change occurs from the beginning to the end\nof running this procedure.\n\nThe general idea is to compare each follower's view in turn against\nthe current leader's view, without disturbing the leader in order to\navoid downtime.\n\nTo speed up the comparisons, a digests database is first generated\nfrom the current leader's view of all raft sessions, which is then\nused by individual comparisons on followers to skim quickly over what\nhasn't changed without sending requests to the leader.\n\n- **Step Pre-check** is to be run on the supervisor\n\n- **Steps Prep-1 to Prep-3** are to be run once, on a host having\n  access to bucketd as well as having direct SSH access to stateful\n  hosts (e.g. the supervisor)\n\n- **Step Scan-1 to Scan-4** are to be run repeatedly for each active\n  stateful host, so five times on most systems\n\n- **Step Results-1** is to be run once at the end of the scan, to\n  gather results\n\nNote: If running on RH8 or Rocky Linux 8, you may adapt the given\ndocker commands with `crictl` instead.\n\n### Pre-check: followers must be up-to-date\n\nOn each raft session, check that all followers are up-to-date with the leader.\n\nYou may use the following command on the supervisor to check this:\n\n```\n./ansible-playbook -i env/s3config/inventory tooling-playbooks/check-status-metadata.yml\n```\n\nResults are gathered in `/tmp/results.txt`: for all raft sessions,\nmake sure that all `committed` values across active `md[x]-cluster[y]`\nmembers are within less than 100 entries to each other. Ignore values\nof `wsb[x]-cluster[y]` members.\n\n### Step Prep-1: Generate the digests database on one metadata server\n\nStart by scanning all raft sessions using `verifyBucketSproxydKeys`,\nto generate a digests database in order to have a more efficient scan\nof followers later on, that does not require an entire listing of the\nleader again.\n\nProvide the environment variables **LISTING_DIGESTS_OUTPUT_DIR** and\nenable **NO_MISSING_KEY_CHECK** to speed up the listing.\n\nIf the command is restarted, e.g. after an error or manual abortion,\nmake sure to delete the `${DIGESTS_PATH}` directory before running the\ncommand again to generate a clean and efficient database.\n\nExample:\n\n```\nDIGESTS_PATH=~/followerDiff-digests-$(date -I);\ndocker run \\\n--net=host \\\n--rm \\\n-e 'BUCKETD_HOSTPORT=127.0.0.1:9000' \\\n-e 'RAFT_SESSIONS=1,2,3,4,5,6,7,8' \\\n-e 'LISTING_DIGESTS_OUTPUT_DIR=/digests' \\\n-v \"${DIGESTS_PATH}:/digests\" \\\n-e 'NO_MISSING_KEY_CHECK=1' \\\nghcr.io/scality/s3utils:1.13.23 \\\nnode verifyBucketSproxydKeys.js \\\n| tee -a verifyBucketSproxydKeys.log\n```\n\n### Step Prep-2: Copy the digests database to each active metadata stateful server\n\nExample:\n\n```\nUSER=root\nfor HOST in storage-1 storage-2 storage-3 storage-4 storage-5; do\n    scp -r ${DIGESTS_PATH} ${USER}@${HOST}:followerDiff-digests\ndone\n```\n\n### Step Prep-3: Save the list of raft sessions to scan on each stateful server\n\nThe following script computes the list of raft sessions to scan for\neach stateful server, and outputs the result as a series of ssh\ncommands to execute to save the list on each active stateful. It\nrequires the 'jq' command to be installed where it is executed.\n\nIt starts with all listed raft sessions for each active stateful, but\nremoves the raft sessions on each active stateful on which it is the\nleader.\n\nYou may save the following script to a file before executing it with \"sh\".\n\n```\n#!/bin/sh\n\n### Config\n\nBUCKETD_ENDPOINT=localhost:9000\nUSER=root\nRS_LIST=\"1 2 3 4 5 6 7 8\"\n\n\n### Script\n\nJQ_SCRIPT_EACH='\n.leader.host as $leader\n| .connected[]\n| select(.host != $leader)\n| { host, $rs }'\n\nJQ_SCRIPT_ALL='\ngroup_by(.host)\n| .[]\n| .[0].host as $host\n| \"echo \\(map(.rs) | join(\" \")) \u003e /tmp/rs-to-scan\"\n| @sh \"ssh -n \\($user)@\\($host) \\(.)\"'\n\nfor RS in $RS_LIST; do\n    curl -s http://${BUCKETD_ENDPOINT}/_/raft_sessions/${RS}/info \\\n    | jq --arg rs ${RS} \"${JQ_SCRIPT_EACH}\";\ndone | jq -rs --arg user ${USER} \"${JQ_SCRIPT_ALL}\"\n```\n\nExecuting this script gives on the standard output, the list of ssh\ncommands to run to save the list of raft sessions to scan in a\ntemporary file on each active stateful member, for example:\n\n```\nssh -n 'root'@'storage-1' 'echo 3 4 5 7 \u003e /tmp/rs-to-scan'\nssh -n 'root'@'storage-2' 'echo 1 2 3 5 6 7 8 \u003e /tmp/rs-to-scan'\nssh -n 'root'@'storage-3' 'echo 1 2 3 4 5 6 8 \u003e /tmp/rs-to-scan'\nssh -n 'root'@'storage-4' 'echo 1 2 3 4 5 6 7 8 \u003e /tmp/rs-to-scan'\nssh -n 'root'@'storage-5' 'echo 1 2 4 6 7 8 \u003e /tmp/rs-to-scan'\n```\n\nCheck that what you get looks like the above, then execute those\ncommands on the shell.\n\nExample, assuming you saved the above script as `output-rs-to-scan-commands.sh`:\n\n```\n# sh output-rs-to-scan-commands.sh\nssh -n 'root'@'storage-1' 'echo 3 4 5 7 \u003e /tmp/rs-to-scan'\nssh -n 'root'@'storage-2' 'echo 1 2 3 5 6 7 8 \u003e /tmp/rs-to-scan'\nssh -n 'root'@'storage-3' 'echo 1 2 3 4 5 6 8 \u003e /tmp/rs-to-scan'\nssh -n 'root'@'storage-4' 'echo 1 2 3 4 5 6 7 8 \u003e /tmp/rs-to-scan'\nssh -n 'root'@'storage-5' 'echo 1 2 4 6 7 8 \u003e /tmp/rs-to-scan'\n# sh output-rs-to-scan-commands.sh | bash -x\n```\n\n**The following steps are to be run for each active metadata stateful node.**\n\n### Step Scan-1: SSH to next stateful host to scan\n\nExample:\n\n```\nssh root@storage-1\n```\n\n### Step Scan-2: Gather current cseq of each raft session to scan\n\nThis step gathers the current cseq value of the follower repds for all\nraft sessions to be scanned. The result will be passed on to the\nfollowerDiff command in order to remove false positives due to live\nchanges while the script is running.\n\n```\nfor RS in $(cat /tmp/rs-to-scan); do\n    for REPD_CONF in ${ENV_HOST_DATA}/scality-metadata-bucket/conf/repd_*.json; do\n        REPD_RS=$(jq .raftSessionId ${REPD_CONF})\n        if [ $RS = $REPD_RS ]; then\n            REPD_ADMIN_PORT=$(jq .adminPort ${REPD_CONF})\n            echo -n '{\"'${RS}'\":'\n            curl -s \"http://localhost:${REPD_ADMIN_PORT}/_/raft/state\"\n            echo '}'\n        fi\n    done\ndone \\\n| jq -s 'reduce .[] as $item ({}; . + ($item | map_values(.committed)))' \\\n| tee /tmp/rs-cseqs.json\n```\n\nThis command should show a result resembling this, mapping raft\nsession numbers to latest cseq values, and storing it in\n`/tmp/rs-cseqs.json`:\n\n```\n{\n  \"3\": 17074,\n  \"4\": 11121,\n  \"5\": 15666,\n  \"7\": 169677\n}\n```\n\n### Step Scan-3: Stop all follower repd processes\n\nIn order to be able to scan the databases, follower repd processes\nmust be stopped with the following command:\n\n```\nfor RS in $(cat /tmp/rs-to-scan); do\n    for REPD_CONF in ${ENV_HOST_DATA}/scality-metadata-bucket/conf/repd_*.json; do\n        REPD_RS=$(jq .raftSessionId ${REPD_CONF})\n        if [ $RS = $REPD_RS ]; then\n            REPD_CONF_FILE=$(basename ${REPD_CONF})\n            REPD_NAME=${REPD_CONF_FILE%%.json}\n            docker exec -u root scality-metadata-bucket-repd \\\n            supervisorctl -c /conf/supervisor/supervisord-repd.conf stop repd:${REPD_NAME}\n        fi\n    done\ndone\n```\n\nIn case errors occur, restart the command.\n\n### Step Scan-4: Scan local databases\n\nRun the \"CompareRaftMembers/followerDiff\" tool to generate a\nline-separated JSON output file showing all differences found between\nthe leader and the local metadata databases, for raft sessions where\nthe repd process is a follower.\n\nNote that the container must mount all metadata databases mountpoints\nin order to have access to all databases.\n\nExample command:\n```\nDATABASE_VOLUME_MOUNTS=$(docker inspect scality-metadata-bucket-repd \\\n| jq -r '.[0].Mounts | map(select(.Source | contains(\"scality-metadata-databases-bucket\")) | \"-v \\(.Source):\\(.Destination)\") | .[]')\nDATABASE_MASTER_MOUNTPOINT=$(docker inspect scality-metadata-bucket-repd \\\n| jq -r '.[0].Mounts | map(select(.Source | contains(\"scality-metadata-databases-bucket\") and (contains(\"/ssd01/\") or contains(\"/ssd1/\"))) | .Destination) | .[]')\n\nmkdir -p scan-results\ndocker run --net=host --rm \\\n  -e 'BUCKETD_HOSTPORT=localhost:9000' \\\n  ${DATABASE_VOLUME_MOUNTS} \\\n  -e \"DATABASES_GLOB=$(cat /tmp/rs-to-scan | tr -d '\\n' | xargs -d' ' -IRS echo ${DATABASE_MASTER_MOUNTPOINT}/RS/0/*)\" \\\n  -v \"${PWD}/followerDiff-digests:/digests\" \\\n  -e \"LISTING_DIGESTS_INPUT_DIR=/digests\" \\\n  -v \"${PWD}/scan-results:/scan-results\" \\\n  -e \"DIFF_OUTPUT_FILE=/scan-results/scan-results.json\" \\\n  -e \"EXCLUDE_FROM_CSEQS=$(cat /tmp/rs-cseqs.json)\" \\\n  ghcr.io/scality/s3utils:1.13.23 \\\n  bash -c 'DATABASES=$(echo $DATABASES_GLOB) node CompareRaftMembers/followerDiff' \\\n| tee -a followerDiff.log\n\n```\n\n\n**Note**: the tool will refuse to override an existing diff output\n  file. If you need to re-run the command, first delete the output\n  file or give another path as DIFF_OUTPUT_FILE.\n\n### Step Scan-5: Restart all stopped repd processes\n\n```\nfor RS in $(cat /tmp/rs-to-scan); do\n    for REPD_CONF in ${ENV_HOST_DATA}/scality-metadata-bucket/conf/repd_*.json; do\n        REPD_RS=$(jq .raftSessionId ${REPD_CONF})\n        if [ $RS = $REPD_RS ]; then\n            REPD_CONF_FILE=$(basename ${REPD_CONF})\n            REPD_NAME=${REPD_CONF_FILE%%.json}\n            docker exec -u root scality-metadata-bucket-repd \\\n            supervisorctl -c /conf/supervisor/supervisord-repd.conf start repd:${REPD_NAME}\n        fi\n    done\ndone\n```\n\nIn case errors occur, restart the command.\n\n**If there are more active stateful hosts to scan, continue with\n[Step Scan-1](#step-scan-1-ssh-to-next-stateful-host-to-scan) on the next\nactive stateful host.**\n\n### Step Results-1: Gather results\n\nFinally, once all scans have been done on all active stateful hosts,\ngather all diff results for later analysis or [repair](#repair-procedure).\n\nFor example:\n\n```\nfor HOST in storage-1 storage-2 storage-3 storage-4 storage-5; do\n    scp -r ${USER}@${HOST}:scan-results ./scan-results.${HOST}\ndone\n```\n\nAt this point, each results file contains a newline-separated set of\nJSON entries describing each difference found on each of the servers,\nfor the databases used by repd processes acting as followers in the\nRaft protocol. This output is meant to be passed on to the [repair\nscript](#repair-objects-affected-by-raft-divergence) to actually\nrepair the divergences. More details on the output format can be found\nin the [tool's description\nsummary](#compare-followers-databases-against-leader-view).\n\n### Caveats\n\n- The more up-to-date the digests database is, the more efficient the\n  subsequent stateful host scans are. This means that it's better not\n  to wait too long between the digests database generation and each\n  host scan, in order to reduce the number of updates that occurred\n  in-between, therefore limiting the number of new listings that have\n  to be done on the leader.\n\n- It is possible to generate the digests database in multiple steps if\n  need be, however in such case it's best not to re-scan the same\n  range of keys multiple times outputting to the same digests\n  database, the risk being to render that range of digests much less\n  efficient to optimize the comparisons, or even useless. If an update\n  of the digests database is wanted, it's best to re-create a new one\n  from scratch (e.g. by deleting the old digests directory first).\n\n- The current version of the script does not work with Metadata bucket\n  format v1, only with v0. If the need arises, it could be made to\n  work with v1 with some more work.\n\n- The current version of the script does not support internal TLS\n  feature\n\n\n# Compare two followers database sets\n\nThe **CompareRaftMembers/compareFollowerDbs** tool compares two sets\nof Metadata leveldb databases, belonging to two different Metadata\nnodes, and outputs the differences to the file path given as the\nDIFF_OUTPUT_FILE environment variable.\n\nIn this file, it outputs each key that differs as line-separated JSON\nentries, where each entry can be one of:\n\n- `[{ key, value }, null]`: this key is present on the follower #1 but\n  not on follower #2\n\n- `[null, { key, value }]`: this key is not present on follower #1\n  but is present on follower #2\n\n- `[{ key, value: \"{value1}\" }, { key, value: \"{value2}\" }]`: this key\n  has a different value between follower #1 and follower #2: \"value1\"\n  is the value seen on follower #1 and \"value2\" the value seen on\n  follower #2.\n\n## Usage\n\n```\nnode compareFollowerDbs.js\n```\n\n## Mandatory environment variables\n\n* **DATABASES1**: space-separated list of databases of follower #1 to\n  compare against follower #2\n\n* **DATABASES2**: space-separated list of databases of follower #2 to\n  compare against follower #1\n\n* **DIFF_OUTPUT_FILE**: file path where diff output will be stored\n\n## Optional environment variables\n\n* **PARALLEL_SCANS**: number of databases to scan in parallel (default 4)\n\n* **EXCLUDE_FROM_CSEQS**: mapping of raft sessions to filter on, where\n  keys are raft session IDs and values are the cseq value for that\n  raft session. Filtering will be based on all oplog records more\n  recent than the given \"cseq\" for the raft session. Input diff\n  entries not belonging to one of the declared raft sessions are\n  discarded from the output. The value must be in the following JSON\n  format:\n\n  * `{\"rsId\":cseq[,\"rsId\":cseq...]}`\n\n  * Example: `{\"1\":1234,\"4\":4567,\"6\":6789}`\n\n  * This configuration would cause diff entries of which bucket/key\n    appear in one of the following to be discarded from the output:\n\n    * oplog of raft session 1 after cseq=1234\n\n    * or oplog of raft session 4 after cseq=4567\n\n    * or oplog of raft session 6 after cseq=6789\n\n    * or any other raft session's oplog at any cseq\n\n* **BUCKETD_HOSTPORT**: ip:port of bucketd endpoint, needed when\n  EXCLUDE_FROM_CSEQS is set (in order to read the Raft oplog)\n\n## Scan Procedure with Unstable Leader\n\n**Note**: you may use this procedure when leaders are unstable,\ni.e. leader changes are frequent or susceptible to happen during the\nduration of the scan procedure. When leaders are stable, it can be\nmore practical and efficient to use the [stable\nleader](#scan-procedure-with-stable-leader) procedure instead.\n\nIn order to scan a set of raft sessions to detect discrepancies\nbetween raft members, you may use the following procedure involving\nthe scan tool [compareFollowerDbs](#compare-two-followers-database-sets).\n\nThe general idea is to compare sets of databases belonging to two\ndifferent followers at a time.\n\n**Note**: Each repd node (including the current leader) has to be\nstopped at some point to copy or read its databases set, then\nrestarted. Stopping the leader can have a small temporary impact on\nthe traffic.\n\n\n### Define global configuration values\n\n- Choose one node on the system that will be the source from which we\n  will copy the set of leveldb databases to a filesystem location that\n  can be shared with other nodes later on. It can be any one of the\n  Metadata active nodes.\n\n  We will call this node **SourceNode** in the rest of this procedure,\n  and its hostname stored in the environment variable ``SOURCE_NODE``\n  in the provided commands.\n\n- Choose a list of Raft Sessions to scan\n\n  Decide on which raft sessions will be scanned during this run of the\n  procedure. It may be all data Raft Sessions (usually 8 with IDs 1\n  through 8), or a subset.\n\n  We will store this set as a space-separated list of raft session\n  IDs, for example `1 2 3 4 5 6 7 8`, in the environment variable\n  `RAFT_SESSIONS_TO_SCAN` in the provided commands.\n\n- Choose a unique shared filesystem location that:\n\n  - has enough storage space to host Metadata databases of the raft\n    sessions being scanned (from a single node)\n\n  - will be network-accessible from each Metadata node (i.e. contents\n    can be `rsync` from/to Metadata hosts)\n\n  We will save this remote rsync location as\n  `REMOTE_DATABASES_HOST_PATH` environment variable in the provided\n  commands, for example `root@node-with-storage:/path/to/source-node-dbs`\n\n- Choose a filesystem location on each stateful node that has enough\n  storage space to host Metadata databases of the raft sessions being\n  scanned (from a single node) to receive a copy of the remote\n  databases from **SourceNode**. It can be for example a location on\n  the RING spinner disks. We will save this path to the environment\n  variable `LOCAL_DATABASES_PATH`, for example\n  `/scality/disk1/source-node-dbs`.\n\n- `env_host_data` Ansible variable value will be provided as\n  `ENV_HOST_DATA` environment variable in the provided commands.\n\nThe following steps will be run as follows:\n\n- **Step Pre-check** is to be run on the supervisor\n\n- **Steps Prep-1 to Prep-5** are to be run on **SourceNode** only\n\n- **Step Scan-1 to Scan-5** are to be run repeatedly for each active\n  stateful host other than **SourceNode**, so four times on most\n  systems\n\n- **Step Results-1** is to be run once at the end of the scan, to\n  gather results\n\n### Pre-check: followers must be up-to-date\n\nOn each raft session, check that all followers are up-to-date with the leader.\n\nYou may use the following command on the supervisor to check this:\n\n```\n./ansible-playbook -i env/s3config/inventory tooling-playbooks/check-status-metadata.yml\n```\n\nResults are gathered in `/tmp/results.txt`: for all raft sessions,\nmake sure that all `committed` values across active `md[x]-cluster[y]`\nmembers are within less than 100 entries to each other. Ignore values\nof `wsb[x]-cluster[y]` members.\n\n### Step Prep-1: Gather current cseq of each raft session to scan\n\nThis step gathers the current cseq value of the **SourceNode** repd\nfor all raft sessions to be scanned. The result will be passed on to\nthe `compareFollowerDbs` command in order to remove false positives\ndue to live changes while the procedure is executed.\n\n```\nfor RS in ${RAFT_SESSIONS_TO_SCAN}; do\n    for REPD_CONF in ${ENV_HOST_DATA}/scality-metadata-bucket/conf/repd_*.json; do\n        REPD_RS=$(jq .raftSessionId ${REPD_CONF})\n        if [ $RS = $REPD_RS ]; then\n            REPD_ADMIN_PORT=$(jq .adminPort ${REPD_CONF})\n            echo -n '{\"'${RS}'\":'\n            curl -s \"http://localhost:${REPD_ADMIN_PORT}/_/raft/state\"\n            echo '}'\n        fi\n    done\ndone \\\n| jq -s 'reduce .[] as $item ({}; . + ($item | map_values(.committed)))' \\\n| tee /tmp/rs-cseqs.json\n```\n\nThis command should show a result resembling this, mapping raft\nsession numbers to latest cseq values, and storing it in\n`/tmp/rs-cseqs.json`:\n\n```\n{\n  \"1\": 17074,\n  \"2\": 11121,\n  \"3\": 15666,\n  \"4\": 169677\n}\n```\n\n### Step Prep-2: Copy cseqs file to other nodes\n\nCopy the file just created on **SourceNode** to each other node that\nwill be scanned.\n\nAssuming **SourceNode** is `storage-1` and there are 5 nodes to scan,\nthis command would copy the file to each of the other 4 nodes:\n\n```\nfor NODE in storage-2 storage-3 storage-4 storage-5; do\n    scp /tmp/rs-cseqs.json root@${NODE}:/tmp/\ndone\n```\n\n### Step Prep-3: Stop repd processes\n\nIn order to be able to copy the databases, repd processes for the\ncorresponding raft sessions to scan must be stopped.\n\nIt can be done with the following script:\n\n```\nfor RS in ${RAFT_SESSIONS_TO_SCAN}; do\n    for REPD_CONF in ${ENV_HOST_DATA}/scality-metadata-bucket/conf/repd_*.json; do\n        REPD_RS=$(jq .raftSessionId ${REPD_CONF})\n        if [ $RS = $REPD_RS ]; then\n            REPD_CONF_FILE=$(basename ${REPD_CONF})\n            REPD_NAME=${REPD_CONF_FILE%%.json}\n            docker exec -u root scality-metadata-bucket-repd \\\n            supervisorctl -c /conf/supervisor/supervisord-repd.conf stop repd:${REPD_NAME}\n        fi\n    done\ndone\n```\n\nIn case errors occur, restart the command.\n\n### Step Prep-4: Copy databases to the remote location\n\nCopy the set of LevelDB databases hosting Metadata for the raft\nsessions in `RAFT_SESSIONS_TO_SCAN`, to the remote filesystem location\ndefined as `REMOTE_DATABASES_HOST_PATH`.\n\nAssuming **SourceNode** has SSH access to the remote database host, it\ncan be done with the following script:\n\n```\nfor RS in ${RAFT_SESSIONS_TO_SCAN}; do\n    echo \"copying databases for raft session ${RS} to ${REMOTE_DATABASES_HOST_PATH}\"\n    rsync -a ${ENV_HOST_DATA}/scality-metadata-databases-bucket/${RS} ${REMOTE_DATABASES_HOST_PATH}/\ndone\n```\n\nIn case errors occur, restart the command.\n\n\n### Step Prep-5: Restart repd processes\n\nIt can be done with the following script:\n\n```\nfor RS in ${RAFT_SESSIONS_TO_SCAN}; do\n    for REPD_CONF in ${ENV_HOST_DATA}/scality-metadata-bucket/conf/repd_*.json; do\n        REPD_RS=$(jq .raftSessionId ${REPD_CONF})\n        if [ $RS = $REPD_RS ]; then\n            REPD_CONF_FILE=$(basename ${REPD_CONF})\n            REPD_NAME=${REPD_CONF_FILE%%.json}\n            docker exec -u root scality-metadata-bucket-repd \\\n            supervisorctl -c /conf/supervisor/supervisord-repd.conf start repd:${REPD_NAME}\n        fi\n    done\ndone\n```\n\nIn case errors occur, restart the command.\n\n**Important**: Run the following Scan steps on each stateful node\n  except **SourceNode**.\n\n### Step Scan-1: SSH to next stateful host to scan\n\nSSH to the next stateful host (excluding **SourceNode**) where the\nnext scan will run to compare against **SourceNode**.\n\nExample:\n\n```\nssh root@storage-2\n```\n\n### Step Scan-2: Copy remote databases locally\n\nCopy the set of LevelDB databases previously saved from **SourceNode**\nto the remote location locally.\n\nIt can be done with the following script:\n\n```\nrsync -a ${REMOTE_DATABASES_HOST_PATH}/ ${LOCAL_DATABASES_PATH}/\n```\n\n### Step Scan-3: Stop repd processes\n\nIn order to be able to scan the databases, repd processes must be\nstopped with the following command:\n\n```\nfor RS in ${RAFT_SESSIONS_TO_SCAN}; do\n    for REPD_CONF in ${ENV_HOST_DATA}/scality-metadata-bucket/conf/repd_*.json; do\n        REPD_RS=$(jq .raftSessionId ${REPD_CONF})\n        if [ $RS = $REPD_RS ]; then\n            REPD_CONF_FILE=$(basename ${REPD_CONF})\n            REPD_NAME=${REPD_CONF_FILE%%.json}\n            docker exec -u root scality-metadata-bucket-repd \\\n            supervisorctl -c /conf/supervisor/supervisord-repd.conf stop repd:${REPD_NAME}\n        fi\n    done\ndone\n```\n\nIn case errors occur, restart the command.\n\n\n### Step Scan-4: Scan local databases\n\nRun the \"CompareRaftMembers/compareFollowerDbs\" tool to generate a\nline-separated JSON output file showing all differences found between\nthis node's databases and the SourceNode's databases, for the chosen\nlist of raft sessions to scan.\n\nNote that the container must mount all metadata databases mountpoints\nin order to have access to all local databases, as well as mount\n`LOCAL_DATABASES_PATH`.\n\nExample command:\n```\nMY_DATABASE_VOLUME_MOUNTS=$(docker inspect scality-metadata-bucket-repd \\\n| jq -r '.[0].Mounts | map(select(.Source | contains(\"scality-metadata-databases-bucket\")) | \"-v \\(.Source):\\(.Destination)\") | .[]')\nDATABASE_MASTER_MOUNTPOINT=$(docker inspect scality-metadata-bucket-repd \\\n| jq -r '.[0].Mounts | map(select(.Source | contains(\"scality-metadata-databases-bucket\") and (contains(\"/ssd01/\") or contains(\"/ssd1/\"))) | .Destination) | .[]')\n\nmkdir -p scan-results\ndocker run --net=host --rm \\\n  -e 'BUCKETD_HOSTPORT=localhost:9000' \\\n  ${MY_DATABASE_VOLUME_MOUNTS} \\\n  -v ${LOCAL_DATABASES_PATH}:/databases2 \\\n  -e \"DATABASES1_GLOB=$(echo ${RAFT_SESSIONS_TO_SCAN} | tr -d '\\n' | xargs -d' ' -IRS echo ${DATABASE_MASTER_MOUNTPOINT}/RS/0/*)\" \\\n  -e \"DATABASES2_GLOB=$(echo ${RAFT_SESSIONS_TO_SCAN} | tr -d '\\n' | xargs -d' ' -IRS echo /databases2/RS/0/*)\" \\\n  -v \"${PWD}/scan-results:/scan-results\" \\\n  -e \"DIFF_OUTPUT_FILE=/scan-results/scan-results.json\" \\\n  -e \"EXCLUDE_FROM_CSEQS=$(cat /tmp/rs-cseqs.json)\" \\\n  ghcr.io/scality/s3utils:1.13.23 \\\n  bash -c 'DATABASES1=$(echo $DATABASES1_GLOB) DATABASES2=$(echo $DATABASES2_GLOB) node CompareRaftMembers/compareFollowerDbs' \\\n| tee -a compareFollowerDbs.log\n\n```\n\n**Note**: the tool will refuse to override an existing diff output\n  file. If you need to re-run the command, first delete the output\n  file or give another path as DIFF_OUTPUT_FILE.\n\n### Step Scan-5: Restart stopped repd processes\n\n```\nfor RS in ${RAFT_SESSIONS_TO_SCAN}; do\n    for REPD_CONF in ${ENV_HOST_DATA}/scality-metadata-bucket/conf/repd_*.json; do\n        REPD_RS=$(jq .raftSessionId ${REPD_CONF})\n        if [ $RS = $REPD_RS ]; then\n            REPD_CONF_FILE=$(basename ${REPD_CONF})\n            REPD_NAME=${REPD_CONF_FILE%%.json}\n            docker exec -u root scality-metadata-bucket-repd \\\n            supervisorctl -c /conf/supervisor/supervisord-repd.conf start repd:${REPD_NAME}\n        fi\n    done\ndone\n```\n\nIn case errors occur, restart the command.\n\n**If there are more active stateful hosts to scan, continue with\n[Step Scan-1](#step-scan-1-ssh-to-next-stateful-host-to-scan-1) on the next\nactive stateful host.**\n\n### Step Results-1: Gather results\n\nFinally, once all scans have been done on all active stateful hosts\nexcept **SourceNode**, gather all diff results for later analysis\nor [repair](#repair-procedure).\n\nFor example:\n\n```\nfor HOST in storage-2 storage-3 storage-4 storage-5; do\n    scp -r ${USER}@${HOST}:scan-results ./scan-results.${HOST}\ndone\n```\n\nAt this point, each results file contains a newline-separated set of\nJSON entries describing each difference found on each of the servers\nagainst **SourceNode**, for the databases used by each repd process on\neach node. This output is meant to be passed on to the [repair\nscript](#repair-objects-affected-by-raft-divergence) to actually\nrepair the divergences. More details on the output format can be found\nin the [tool's description\nsummary](#compare-followers-databases-against-leader-view).\n\n### Caveats\n\n- The current version of the script does not work with Metadata bucket\n  format v1, only with v0. If the need arises, it could be made to\n  work with v1 with some more work.\n\n- The current version of the script does not support internal TLS\n  feature\n\n\n# Repair objects affected by raft divergence\n\nThe repair script provided here attempts to automatically repair the\nobject metadata affected by divergences between leaders and followers,\nto fix consistency while making sure the repaired object is\nreadable. It does not necessarily mean that the repaired version will\nbe the legitimate version from the application point of view, but the\ntool strives to be non-destructive and best-effort.\n\nThis script checks each entry from stdin, corresponding to a line from\nthe output of followerDiff.js, and repairs the object on metadata if\ndeemed safe to do so with a readable version, and if there's no\nambiguity on what's the best way to repair.\n\n## Usage\n\n```\n    node CompareRaftMembers/repairObjects.js\n```\n\n## Mandatory environment variables\n\n* **BUCKETD_HOSTPORT**: ip:port of bucketd endpoint\n\n* **SPROXYD_HOSTPORT**: ip:port of sproxyd endpoint\n\n## Optional environment variables\n\n* **VERBOSE**: set to 1 for more verbose output (shows one line for\nevery sproxyd key checked)\n\n* **DRY_RUN**: set to 1 to log statuses without attempting to repair anything\n\n## Logs\n\nThe script outputs logs in JSON. Each object repair analysis or\nattempt outputs a single log line, for example:\n\n```\n{\"name\":\"s3utils:CompareRaftMembers:repairObjects\",\"time\":1658796563031,\"bucket\":\"s3c-5862-nv-0072\",\"key\":\"test-2022-07-05T21-06-16-960Z/190\",\"repairStatus\":\"AutoRepair\",\"repairSource\":\"Leader\",\"repairMaster\":false,\"level\":\"info\",\"message\":\"repaired object metadata successfully\",\"hostname\":\"jonathan-storage-1\",\"pid\":1}\n```\n\nIn the above log, the repair status is `AutoRepair`, which means the\nscript attempted to repair automatically the object metadata, and it\nsucceeded.\n\nThe status can be one of:\n\n* **AutoRepair**: the object can be repaired automatically\n  (`repairSource` described the source metadata for the repair, either\n  `Leader` or `Follower`).\n\n* **ManualRepair**: the object can be repaired but an operator needs\n  to decide on which version to repair from. This is normally when\n  both versions are readable and the script is unable to take an\n  automatic decision. There is currently no support to use\n  `objectRepair` to help with those cases, but it can be done via the\n  bucketd API.\n\n* **NotRepairable**: the object metadata is either absent or\n  unreadable from both leader and follower (the message gives extra\n  detail), so the script did not attempt any repair.\n\n* **UpdatedByClient**: this status is given when the repair script\n  detects that the current metadata does not match one of the leader\n  or follower's entry in the current diff entry being processed. This\n  normally means that the application has overwritten the object with\n  new contents since the scan ran. It can then safely be ignored, and\n  the repair script did not attempt to repair anything here.\n\nAdditionally, logs may contain such lines, when a check failed on a\nparticular sproxyd key that belongs to one instance of the metadata\n(whether on leader or follower):\n\n```\n{\"name\":\"s3utils:CompareRaftMembers:repairObjects\",\"time\":1658796563000,\"bucketdUrl\":\"http://localhost:9000/default/bucket/s3c-5862-nv-0072/test-2022-07-05T21-06-16-960Z%2F190\",\"sproxydKey\":\"A553B230F05B0B55C2D090233B280559E730D320\",\"level\":\"error\",\"message\":\"sproxyd check reported missing key\",\"hostname\":\"jonathan-storage-1\",\"pid\":1}\n```\n\nThose are expected when metadata divergence exists, they can give some\ncomplementary information for diagnosis.\n\nAt the end of the run, a single line shows the final count per status\n(which is also shown during the run every 10 seconds). For example\nhere, we know that there were 2 automatic repairs performed, and one\nrequiring a manual check and repair (which entry can be found from the\nlogs, looking for \"ManualRepair\" status):\n\n```\n{\"name\":\"s3utils:CompareRaftMembers:repairObjects\",\"time\":1658796563064,\"countByStatus\":{\"AutoRepair\":2,\"AutoRepairError\":0,\"ManualRepair\":1,\"NotRepairable\":0,\"UpdatedByClient\":0},\"level\":\"info\",\"message\":\"completed repair\",\"hostname\":\"jonathan-storage-1\",\"pid\":1}\n```\n\n## Repair Procedure\n\nTo repair a set of objects with inconsistencies, first make sure that\nyou ran one of the scan procedures (with [stable\nleader](#scan-procedure-with-stable-leader) or [unstable\nleader](#scan-procedure-with-unstable-leader)) and got results ready\nfrom `scan-results.[host]` directories.\n\n### Example Repair Command\n\nAssuming the hosts are named storage-1 through storage-5, this command\ncan be used to launch the repair in dry-run mode first (it's a good\npractice to get a sense of what the script will attempt to do before\nexecuting it for real, looking at the logs first):\n\n```\ncat scan-results.storage-{1..5}/scan-results.jsonl | \\\ndocker run -i --net=host --rm \\\n  -e \"BUCKETD_HOSTPORT=localhost:9000\" \\\n  -e \"SPROXYD_HOSTPORT=localhost:8181\" \\\n  -e \"DRY_RUN=1\" \\\n  ghcr.io/scality/s3utils:1.13.23 \\\n  node CompareRaftMembers/repairObjects | tee -a repairObjects.log\n```\n\nIf results make sense, it can be executed without the \"DRY_RUN\"\noption to execute the possible automatic repairs, if any:\n\n```\ncat scan-results.storage-{1..5}/scan-results.jsonl | \\\ndocker run -i --net=host --rm \\\n  -e \"BUCKETD_HOSTPORT=localhost:9000\" \\\n  -e \"SPROXYD_HOSTPORT=localhost:8181\" \\\n  ghcr.io/scality/s3utils:1.13.23 \\\n  node CompareRaftMembers/repairObjects | tee -a repairObjects.log\n```\n\n# Remove delete markers\n\nThe removeDeleteMarkers.js script removes delete markers from one or\nmore versioning-suspended bucket(s).\n\n## Usage\n\n```\n    node removeDeleteMarkers.js bucket1[,bucket2...]\n```\n\n## Mandatory environment variables\n\n* **ENDPOINT**: S3 endpoint\n\n* **ACCESS_KEY**: S3 account access key\n\n* **SECRET_KEY**: S3 account secret key\n\n## Optional environment variables:\n\n* **TARGET_PREFIX**: only process a specific prefix in the bucket(s)\n\n* **WORKERS**: concurrency value for listing / batch delete requests (default 10)\n\n* **LOG_PROGRESS_INTERVAL**: interval in seconds between progress update log lines (default 10)\n\n* **LISTING_LIMIT**: number of keys to list per listing request (default 1000)\n\n* **KEY_MARKER**: resume processing from a specific key\n\n* **VERSION_ID_MARKER**: resume processing from a specific version ID\n\n## Output\n\nThe output of the script consists of JSON log lines.\n\n### Info\n\nOne log line is output for each delete marker deleted, e.g.:\n\n```\n{\"name\":\"s3utils::removeDeleteMarkers\",\"time\":1586304708269,\"bucket\":\"some-bucket\",\"objectKey\":\"some-key\",\"versionId\":\"3938343133363935343035383633393939393936524730303120203533342e3432363436322e393035353030\",\"level\":\"info\",\"message\":\"delete marker deleted\",\"hostname\":\"c3f84aa473a2\",\"pid\":88}\n```\n\nThe script also logs a progress update, every 10 seconds by default:\n\n```\n{\"name\":\"s3utils::removeDeleteMarkers\",\"time\":1586304694304,\"objectsListed\":257000,\"deleteMarkersDeleted\":0,\"deleteMarkersErrors\":0,\"bucketInProgress\":\"some-bucket\",\"keyMarker\":\"some-key-marker\",\"versionIdMarker\":\"3938343133373030373134393734393939393938524730303120203533342e3430323230362e373139333039\",\"level\":\"info\",\"message\":\"progress update\",\"hostname\":\"c3f84aa473a2\",\"pid\":88}\n```\n\n# Repair duplicate sproxyd keys\n\nThis script repairs object versions that share sproxyd keys with\nanother version, particularly due to bug S3C-2731.\n\nThe repair action consists of copying the data location that is\nduplicated to a new sproxyd key (or set of keys for MPU), and updating\nthe metadata to reflect the new location, resulting in two valid\nversions with distinct data, though identical in content.\n\nThe script does not remove any version even if the duplicate was due\nto an internal retry in the metadata layer, because either version\nmight be referenced by S3 clients in some cases.\n\n## Usage\n\n```\nnode repairDuplicateVersions.js\n```\n\n## Standard Input\n\nThe standard input must be fed with the JSON logs output by the\nverifyBucketSproxydKeys.js s3utils script. This script only processes\nthe log entries containing the message \"duplicate sproxyd key found\"\nand ignores other entries.\n\n## Mandatory environment variables\n\n* **OBJECT_REPAIR_BUCKETD_HOSTPORT**: ip:port of bucketd endpoint\n\n* **OBJECT_REPAIR_SPROXYD_HOSTPORT**: ip:port of sproxyd endpoint\n\n## Example\n\n```\ncat /tmp/verifyBucketSproxydKeys.log | docker run -i zenko/s3utils:latest bash -c 'OBJECT_REPAIR_BUCKETD_HOSTPORT=127.0.0.1:9000 OBJECT_REPAIR_SPROXYD_HOSTPORT=127.0.0.1:8181 node repairDuplicateVersions.js' \u003e /tmp/repairDuplicateVersions.log\n```\n\n## Caveat\n\nWhile the script is running, there is a possibility, although slim,\nthat it updates metadata for an object that is being updated at the\nexact same time by a client application, either through \"PutObjectAcl\"\nor \"PutObjectTagging\" operations which can modify existing\nversions. In such case, there is a risk that either:\n* the application update will be lost\n* or the script will not repair the item properly\n\nIf this looks like a potential risk to a customer running the script,\nwe suggest to disable those operations on the clients during the time\nthe script is running (or alternatively, disable all writes), to avoid\nany such risk.\n\n\n# Repair object metadata with duplicate 'versionId' field\n\nThis script repairs object versions that have a duplicate versionId\nfield in the master key and a missing version key, in particular due\nto bug S3C-7861 (CRR with incompatible S3C versions between source and\ntarget).\n\nThe repair action consists of fetching the source metadata from the\nmaster key, fixing the JSON versionId field to be the first versionId\npresent in the binary blob (assumed to be the original one, hence\ncorrect), then copying it to both the master key and a new version\nkey.\n\n## Usage\n\n```\nnode repairDuplicateVersionIds.js\n```\n\n## Standard Input\n\nThe standard input must be fed with the JSON logs output by the\nverifyBucketSproxydKeys.js s3utils script. This script only processes\nthe log entries containing the message 'object master metadata with\nduplicate \"versionId\" field found' and ignores other entries.\n\n## Mandatory environment variables\n\n* **OBJECT_REPAIR_BUCKETD_HOSTPORT**: ip:port of bucketd endpoint\n\n## Example\n\n```\ncat /tmp/verifyBucketSproxydKeys.log | docker run -i ghcr.io/scality/s3utils:1.13.24 bash -c 'OBJECT_REPAIR_BUCKETD_HOSTPORT=127.0.0.1:9000 node repairDuplicateVersionIds.js' \u003e /tmp/repairDuplicateVersionIds.log\n```\n\n# Cleanup Noncurrent Versions\n\nThis script removes noncurrent versions and current/noncurrent delete\nmarkers, either all such objects or older than a specified\nlast-modified date.\n\n## Usage\n\n```\nnode cleanupNoncurrentVersions.js bucket1[,bucket2...]\n```\n\n## Mandatory environment variables\n\n* **S3_ENDPOINT**: S3 endpoint URL\n\n* **ACCESS_KEY**: S3 account/user access key\n\n* **SECRET_KEY**: S3 account/user secret key\n\n## Optional environment variables\n\n* **TARGET_PREFIX**: cleanup only inside this key prefix in each bucket\n\n* **MAX_LISTED**: maximum number of keys listed before exiting (default unlimited)\n\n* **MAX_DELETES**: maximum number of keys to delete before exiting (default unlimited)\n\n* **MARKER**: marker from which to resume the cleanup, logged at the end\nof a previous invocation of the script, uses the format:\n\n  ```\n  MARKER := encodeURI(bucketName)\n    \"|\" encodeURI(key)\n    \"|\" encodeURI(versionId)\n  ```\n\n* **OLDER_THAN**: cleanup only objects which last modified date is older\nthan this, as an ISO date or a number of days, e.g.:\n\n  * setting to \"2021-01-09T00:00:00Z\" limits the cleanup to objects\n  created or modified before Jan 9th 2021\n\n  * setting to \"30 days\" limits the cleanup to objects created more\n  than 30 days ago\n\n* **ONLY_DELETED**: if set to \"1\" or \"true\" or \"yes\", only remove\notherwise eligible noncurrent versions if the object's current version\nis a delete marker (also removes the delete marker)\n\n* **DELETED_BEFORE**: cleanup only objects whose current version is a delete\n    marker older than this date, e.g. setting to \"2021-01-09T00:00:00Z\"\n    limits the cleanup to objects deleted before Jan 9th 2021.\n    Implies `ONLY_DELETED = true`\n\n* **HTTPS_CA_PATH**: path to a CA certificate bundle used to\nauthentify the S3 endpoint\n\n* **HTTPS_NO_VERIFY**: set to 1 to disable S3 endpoint certificate check\n\n* **EXCLUDE_REPLICATING_VERSIONS**: if is set to '1,' 'true,' or 'yes,'\n    prevent the deletion of replicating versions\n\n## Example\n\n```\ndocker run --net=host -ti zenko/s3utils:latest bash -c 'S3_ENDPOINT=https://s3.customer.com ACCESS_KEY=123456 SECRET_KEY=789ABC OLDER_THAN=\"Jan 14 2021\" node cleanupNoncurrentVersions.js target-bucket-1,target-bucket-2' \u003e /tmp/cleanupNoncurrentVersions.log\n```\n\n# Count number and total size of current and noncurrent versions in a bucket\n\nThe bucketVersionsStats.js script counts the total number of objects\nand their cumulative size, and shows them separately for current and\nnoncurrent versions.\n\n## Usage\n\n```\n    node bucketVersionsStats.js\n```\n\n## Mandatory environment variables\n\n* **ENDPOINT**: S3 endpoint\n\n* **ACCESS_KEY**: S3 account access key\n\n* **SECRET_KEY**: S3 account secret key\n\n* **BUCKET**: S3 bucket name\n\n## Optional environment variables:\n\n* **TARGET_PREFIX**: only process a specific prefix in the bucket\n\n* **LISTING_LIMIT**: number of keys to list per listing request (default 1000)\n\n* **LOG_PROGRESS_INTERVAL**: interval in seconds between progress update log lines (default 10)\n\n* **KEY_MARKER**: start counting from a specific key\n\n* **VERSION_ID_MARKER**: start counting from a specific version ID\n\n## Output\n\nThe output of the script consists of JSON log lines.\n\nThe script logs a progress update, every 10 seconds by default, and a\nfinal summary at the end of execution with the total counts for the\nbucket.\n\nThe total counts may be possibly limited by the **TARGET_PREFIX**,\n**KEY_MARKER** and **VERSION_ID_MARKER** optional environment variables.\n\n- **stats** contains the output statistics:\n\n  - **current** refers to current versions of objects\n\n  - **noncurrent** refers to non-current versions of objects\n\n  - **total** is the sum of **current** and **noncurrent**\n\n- in **stats.current** and **stats.noncurrent**:\n\n  - **count** is the number of objects pertaining to the section\n\n  - **size** is the cumulative size in bytes of the objects pertaining\n  to the section\n\n### Example Progress Update\n\nNote: the JSON output is prettified here for readability, but the\nscript outputs this status on a single line.\n\n```\n{\n  \"name\": \"s3utils::bucketVersionsStats\",\n  \"time\": 1638823524419,\n  \"bucket\": \"example-bucket\",\n  \"stats\": {\n    \"total\": {\n      \"count\": 16000,\n      \"size\": 27205000\n    },\n    \"current\": {\n      \"count\": 14996,\n      \"size\": 26165000\n    },\n    \"noncurrent\": {\n      \"count\": 1004,\n      \"size\": 1040000\n    }\n  },\n  \"keyMarker\": \"test-2021-12-06T20-09-39-712Z/1732\",\n  \"versionIdMarker\": \"3938333631313738363132353838393939393939524730303120203131302e353533312e3135373439\",\n  \"level\": \"info\",\n  \"message\": \"progress update\",\n  \"hostname\": \"lab-store-0\",\n  \"pid\": 717\n}\n```\n\n### Example Final Summary\n\nNote: the JSON output is prettified here for readability, but the\nscript outputs this status on a single line.\n\n```\n{\n  \"name\": \"s3utils::bucketVersionsStats\",\n  \"time\": 1638823531629,\n  \"bucket\": \"example-bucket\",\n  \"stats\": {\n    \"total\": {\n      \"count\": 54267,\n      \"size\": 65472000\n    },\n    \"current\": {\n      \"count\": 53263,\n      \"size\": 64432000\n    },\n    \"noncurrent\": {\n      \"count\": 1004,\n      \"size\": 1040000\n    }\n  },\n  \"level\": \"info\",\n  \"message\": \"final summary\",\n  \"hostname\": \"lab-store-0\",\n  \"pid\": 717\n}\n```\n\n# Verify Replication\n\nThe verify replication script checks/compares the replication source with the replication destination. The script produces logs about missing objects on destination and destination objects whose size doesnt match the source size. The script verifies only current version of objects. It relies on standard s3 api's for the source and the destination.\n\n## Usage\n\n```shell\nnode VerifyReplication/index.js\n```\n\n## Mandatory environment variables\n\n* **SRC_ENDPOINT**: replication source s3 endpoint\n\n* **SRC_BUCKET**: replication source bucket\n\n* **SRC_ACCESS_KEY**: replication source s3 account access key\n\n* **SRC_SECRET_KEY**: replication source s3 account secret key\n\n* **DST_BUCKET**: replication destination bucket\n\n* **DST_ACCESS_KEY**: replication destination s3 account access key\n\n* **DST_SECRET_KEY**: replication destination s3 account secret key\n\n## Optional environment variables:\n\n* **SRC_BUCKET_PREFIXES**: comma separated list of prefixes, listing will be limited to these prefixes\n\n* **SRC_DELIMITER**: delimiter used with prefixes in source (default '/')\n\n* **LISTING_LIMIT**: number of keys to list per listing request (default 1000)\n\n* **LISTING_WORKERS**: number of concurrent workers to handle listing (default 10)\n\n* **LOG_PROGRESS_INTERVAL**: interval in seconds between progress update log lines (default 10)\n\n* **BUCKET_MATCH**: set to 1 if bucket match is enabled with replication, when not enabled objects will be replicated with keyname pattern `sourceBucket/key`\n\n* **COMPARE_OBJECT_SIZE**: set to 1 to compare source and destination object sizes\n\n* **DST_ENDPOINT**: replication destination s3 endpoint (only for storage type `aws_s3`)\n\n* **DST_STORAGE_TYPE**: destination storage type, currently supports only `aws_s3`\n\n* **DST_REGION**: destination s3 region (only for storage type `aws_s3`, default `us-east-1`)\n\n* **DST_MD_REQUEST_WORKERS**: number of concurrent workers to handle destination metadata requests (default 50)\n\n* **HTTPS_CA_PATH**: path to a CA certificate bundle used to authentify the source S3 endpoint\n\n* **HTTPS_NO_VERIFY**: set to 1 to disable source S3 endpoint certificate check\n\n* **SHOW_CLIENT_LOGS_IF_AVAILABLE**: set to 1 to show storage client logs if available (enable only for debugging as it may produce a lot of log entries), this may be disregarded if the client (depending on storage type) does not support it\n\n* **SKIP_OLDER_THAN**: skip replication verification of objects whose last modified date is older than this, set this as an ISO date or a number of days e.g.,\n\n  * setting to \"2022-11-30T00:00:00Z\" skips the verification of objects created or modified before Nov 30th 2022\n  * setting to \"30 days\" skips the verification of objects created or modified more than 30 days ago\n\n\n## Output\n\nThe output of the script consists of JSON log lines.\n\n### Info\n\nOne log line is output for each missing object,\n\n```json\n{\n  \"name\":\"s3utils:verifyReplication\",\n  \"time\":1670279729106,\n  \"key\":\"rna-4/rna33\",\n  \"size\":0,\"srcLastModified\":\"2022-12-01T12:27:11.469Z\",\n  \"level\":\"info\",\n  \"message\":\"object missing in destination\",\n  \"hostname\":\"scality.local\",\n  \"pid\":24601\n}\n```\n\nand for each size mismatched object,\n\n```json\n{\n  \"name\":\"s3utils:verifyReplication\",\n  \"time\":1670279723474,\n  \"key\":\"rna-2/34\",\n  \"srcSize\":1,\n  \"dstSize\":0,\n  \"srcLastModified\":\"2022-12-01T11:31:27.528Z\",\n  \"dstLastModified\":\"2022-12-01T11:33:42.000Z\",\n  \"level\":\"info\",\n  \"message\":\"object size does not match in destination\",\n  \"hostname\":\"scality.local\",\n  \"pid\":24601\n}\n```\n\nand for each failed metadata retrieval (after retries),\n\n```json\n{\n  \"name\":\"s3utils:verifyReplication\",\n  \"time\":1670887351694,\n  \"error\":{\n    \"message\":\"socket hang up\",\n    \"code\":\"TimeoutError\",\n    \"time\":\"2022-12-12T23:22:31.694Z\",\n    \"region\":\"us-east-1\",\n    \"hostname\":\"localhost\",\n    \"retryable\":true\n    },\n  \"bucket\":\"src-bucket\",\n  \"key\":\"pref1/key404\",\n  \"level\":\"error\",\n  \"message\":\"error getting metadata\",\n  \"hostname\":\"scality.local\",\n  \"pid\":11398\n}\n```\n\nThe script also logs a progress update (a summary), every 10 seconds by default,\n\n```json\n{\n  \"name\":\"s3utils:verifyReplication\",\n  \"time\":1670280035601,\n  \"srcListedCount\":492,\n  \"dstProcessedCount\":492,\n  \"skippedByDate\": 123,\n  \"missingInDstCount\":123,\n  \"sizeMismatchCount\":123,\n  \"replicatedCount\":120,\n  \"dstFailedMdRetrievalsCount\":3,\n  \"dstBucket\":\"dst-bucket\",\n  \"srcBucket\":\"src-bucket\",\n  \"prefixFilters\":[\"pref1\",\"pref2\"],\n  \"skipOlderThan\": \"2022-11-30T00:00:00Z\",\n  \"level\":\"info\",\n  \"message\":\"completed replication verification\",\n  \"hostname\":\"scality.local\",\n  \"pid\":24780\n}\n```\n\nDescription of the properties in the summary/progress update log,\n\n- **srcListedCount** - total number of objects listed from source\n- **dstProcessedCount** - total number of objects checked in destination (includes missing, mismatched \u0026 failed)\n- **skippedByDate** - total number of objects skipped verification because of a date filter `SKIP_OLDER_THAN`\n- **missingInDstCount** - total number of objects missing in destination\n- **sizeMismatchCount** - total number of objects with mismtached size in destination\n- **dstFailedMdRetrievalsCount** - total number of failed object metadata retrievals (after retries) from destination\n- **replicatedCount** - total number of successful replications\n- **srcBucket** - source bucket\n- **dstBucket** - destination bucket\n\n## Example\n\n```shell\ndocker run \\\n  --net=host \\\n  -e 'SRC_ENDPOINT=http://source.s3.com:8000' \\\n  -e 'SRC_BUCKET=src-bucket' \\\n  -e 'SRC_ACCESS_KEY=src_access_key' \\\n  -e 'SRC_SECRET_KEY=src_secret_key' \\\n  -e 'DST_ENDPOINT=http://destination.s3.com:8000' \\\n  -e 'DST_ACCESS_KEY=dst_access_key' \\\n  -e 'DST_SECRET_KEY=dst_secret_key' \\\n  -e 'DST_BUCKET=dst-bucket' \\\n  -e 'BUCKET_MATCH=1' \\\n  -e 'COMPARE_OBJECT_SIZE=1' \\\n  -e 'SRC_BUCKET_PREFIXES=pref1,pref2' \\\n  -e 'SKIP_OLDER_THAN=\"2022-11-30T00:00:00Z\"' \\\n  ghcr.io/scality/s3utils:latest \\\n  node VerifyReplication/index.js\n```\n\n# UtapiV2 Service Report\n\nGenerates a service level usage report from UtapiV2 in HTML or JSON format.\n\n## Usage\n\nCommands must be executed on a stateful server that hosts the target UtapiV2 deployment.\n\n```\ndocker run --net=host --entrypoint python3 -v /scality/ssd01/s3/scality-utapi/conf:/conf:ro -v $PWD:/output zenko/s3utils utapi/service-report.py --output /output\n```\n\n## Flags\n\n```\n  -c CONFIG, --config CONFIG\n                        Specify an alternate config file (default: /conf/config.json)\n  -r MAX_RETRIES, --max-retries MAX_RETRIES\n                        Max retries before failing a request to an external service (default: 2)\n  -p PARALLEL_QUERIES, --parallel-queries PARALLEL_QUERIES\n                        Max number of parallel queries to warp 10 (default: 5)\n  -j, --json            Output raw reports in json format (default: False)\n  -o OUTPUT, --output OUTPUT\n                        Write report to this directory (default: Current Directory)\n  --debug               Enable debug level logging (default: False)\n  --dry-run             Don't do any computation. Only validate and print the configuration. (default: False)\n```\n\n# UtapiV2 Service level metrics sidecar\n\nREST API to provide service level reports for UtapiV2\n\n## Usage\n\n\n**Using Warp 10 backend**\n\n```shell\ndocker run -d \\\n  --network=host \\\n  -e SIDECAR_API_KEY=dev_key_change_me \\\n  -e SIDECAR_SCALE_FACTOR=1.4 \\\n  -e SIDECAR_WARP10_NODE=\"md1-cluster1:4802@127.0.0.1:4802\" \\\n  scality/s3utils service-level-sidecar/index.js\n```\n\n**Using Scuba backend**\n```shell\ndocker run -d \\\n  --network=host \\\n  -e SIDECAR_API_KEY=dev_key_change_me \\\n  -e SIDECAR_SCALE_FACTOR=1.4 \\\n  -e SIDECAR_ENABLE_SCUBA=true \\\n  -e SIDECAR_SCUBA_BUCKETD_BOOTSTRAP=127.0.0.1:19000 \\\n  scality/s3utils service-level-sidecar/index.js\n```\n\n**Example output**\n```shell\ncurl -X POST -H \"Authorization: Bearer dev_key_change_me\" localhost:24742/api/report | jq\n{\n  \"account\": [\n    {\n      \"arn\": \"arn:aws:iam::081797933446:/test_1669749866/\",\n      \"name\": \"test_1669749866\",\n      \"obj_count\": 25,\n      \"bytes_stored\": 25,\n      \"bytes_stored_total\": 35\n    },\n    {\n      \"arn\": \"arn:aws:iam::024022147664:/test_1669748782/\",\n      \"name\": \"test_1669748782\",\n      \"obj_count\": 25,\n      \"bytes_stored\": 25,\n      \"bytes_stored_total\": 35\n    }\n  ],\n  \"bucket\": {\n    \"arn:aws:iam::081797933446:/test_1669749866/\": [\n      {\n        \"name\": \"test3\",\n        \"obj_count\": 25,\n        \"bytes_stored\": 25,\n        \"bytes_stored_total\": 35\n      }\n    ],\n    \"arn:aws:iam::024022147664:/test_1669748782/\": [\n      {\n        \"name\": \"test\",\n        \"obj_count\": 15,\n        \"bytes_stored\": 15,\n        \"bytes_stored_total\": 21\n      },\n      {\n        \"name\": \"test2\",\n        \"obj_count\": 10,\n        \"bytes_stored\": 10,\n        \"bytes_stored_total\": 14\n      }\n    ]\n  },\n  \"service\": {\n    \"obj_count\": 50,\n    \"bytes_stored\": 50,\n    \"bytes_stored_total\": 70\n  }\n}\n```\n\n### Authentication\n\nBy default a random API key is generated and logged to stdout at startup.\n\n```json\n{\"name\":\"s3utils::utapi-service-sidecar\",\"time\":1669677797644,\"key\":\"c489cc0f5bec1c757be7aecd7cdd61dc9db5bfbc4e4f7bad\",\"level\":\"info\",\"message\":\"using random API key\",\"hostname\":\"f2988d47e450\",\"pid\":1}\n```\n\nA custom static key can be set using the `SIDECAR_API_KEY` environment variable.\n\n```shell\ndocker run -d \\\n  --network=host \\\n  -e SIDECAR_API_KEY=dev_key_change_me \\\n  scality/s3utils service-level-sidecar/index.js\n```\n\nAPI requests to `/api/report` require this API token to be present in the `Authorization` header.\n\n```shell\ncurl -X POST -H \"Authorization: Bearer dev_key_change_me\" localhost:24742/api/report\n```\n\n### Scale Factor\n\nScale factor is provided as a floating point number that represents the percentage to scale reported byte values.\nIt is used to account for the overhead of erasure coding on the storage backend.\nFor example, a value of `1.4` will output byte numbers 140% of the calculated value ie 100 bytes will be reported as 140 bytes.\nIf this produces a non-integer result it is rounded up to the next whole number.\n\nIt can be set using the `SIDECAR_SCALE_FACTOR` environment variable.\n\n```shell\ndocker run -d \\\n  --network=host \\\n  -e SIDECAR_SCALE_FACTOR=1.4 \\\n  scality/s3utils service-level-sidecar/index.js\n```\n\n### TLS\n\nTLS can be enabled for the api server as well as internal connections to vault and bucketd.\nTo enable TLS for the API server specify the path to the TLS certificate and key using\nthe `SIDECAR_TLS_KEY_PATH` and `SIDECAR_TLS_CERT_PATH` environment variables.\nA CA can optionally be set using `SIDECAR_TLS_CA_PATH`.\nCertificates should be mounted into the container using a docker volume.\n\n```shell\ndocker run -d \\\n  --network=host \\\n  -v $PWD/certs:/certs \\\n  -e SIDECAR_TLS_CERT_PATH=/certs/cert.pem \\\n  -e SIDECAR_TLS_KEY_PATH=/certs/key.pem \\\n  -e SIDECAR_TLS_CA_PATH=/certs/ca.pem \\ // CA path is optional\n  scality/s3utils service-level-sidecar/index.js\n```\n\nTo enable TLS for internal connections to vault or bucketd set `SIDECAR_VAULT_ENABLE_TLS=true` or `SIDECAR_BUCKETD_ENABLE_TLS=true` respectively.\n\n```shell\ndocker run -d \\\n  --network=host \\\n  -e SIDECAR_VAULT_ENABLE_TLS=true \\\n  -e SIDECAR_BUCKETD_ENABLE_TLS=true \\\n  scality/s3utils service-level-sidecar/index.js\n```\n\n### Backend addresses\n#### Vault\n\nVault is configured using `SIDECAR_VAULT_ENDPOINT`\n\n```shell\ndocker run -d \\\n  --network=host \\\n  -e SIDECAR_VAULT_ENDPOINT=127.0.0.1:8500 \\\n  scality/s3utils service-level-sidecar/index.js\n```\n\n#### Bucketd\n\nBucketd is configured using `SIDECAR_BUCKETD_BOOTSTRAP`\n\n```shell\ndocker run -d \\\n  --network=host \\\n  -e SIDECAR_BUCKETD_BOOTSTRAP=127.0.0.1:9000 \\\n  scality/s3utils service-level-sidecar/index.js\n```\n\n#### Warp 10\n\nThe Warp 10 address is configured using `SIDECAR_WARP10_NODE`.\nThe Warp 10 `nodeId` must be included (normally matches ansible inventory name plus port ie `md1-cluster1:4802`).\nThe format is `\u003cnodeId\u003e@\u003chost\u003e:\u003cport\u003e`.\n\n```shell\ndocker run -d \\\n  --network=host \\\n  -e SIDECAR_WARP10_NODE=\"md1-cluster1:4802@127.0.0.1:4802\" \\\n  scality/s3utils service-level-sidecar/index.js\n```\n\n#### Scuba\n\nThe scuba backend can be enabled by setting `SIDECAR_ENABLE_SCUBA`.\nA bucketd address can be provided using `SIDECAR_SCUBA_BUCKETD_BOOTSTRAP`.\nIf a bucketd address is not provided `127.0.0.1:19000` will be used.\nInternal TLS support can be enabled using `SIDECAR_SCUBA_BUCKETD_ENABLE_TLS`.\n\n```shell\ndocker run -d \\\n  --network=host \\\n  -e SIDECAR_ENABLE_SCUBA=true \\\n  -e SIDECAR_SCUBA_BUCKETD_BOOTSTRAP=127.0.0.1:19000 \\\n  scality/s3utils service-level-sidecar/index.js\n```\n\n### Other Settings\n\n- Log level can be set using `SIDECAR_LOG_LEVEL` (defaults to `info`)\n- The concurrency used to query backend APIs can be set using `SIDECAR_CONCURRENCY_LIMIT` (defaults to 10)\n- The maximum retries allowed when querying backend APIs can be set using `SIDECAR_RETRY_LIMIT` (defaults to 5)\n- The IP address to bind can be set using `SIDECAR_HOST` (default `0.0.0.0`)\n- The port to bind can be set using `SIDECAR_PORT` (default `24742`)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Fs3utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscality%2Fs3utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Fs3utils/lists"}