{"id":13615923,"url":"https://github.com/scality/backbeat","last_synced_at":"2026-05-11T23:07:56.289Z","repository":{"id":37271246,"uuid":"81391071","full_name":"scality/backbeat","owner":"scality","description":"Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.","archived":false,"fork":false,"pushed_at":"2026-02-12T18:44:55.000Z","size":25275,"stargazers_count":58,"open_issues_count":15,"forks_count":23,"subscribers_count":53,"default_branch":"development/9.2","last_synced_at":"2026-02-13T02:18:55.858Z","etag":null,"topics":["artesca","async","cross-region-replication","disaster-recovery","javascript","nodejs","replication","s3","zenko"],"latest_commit_sha":null,"homepage":"https://www.zenko.io","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scality.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2017-02-09T00:31:22.000Z","updated_at":"2026-02-10T00:57:45.000Z","dependencies_parsed_at":"2023-11-12T09:25:56.446Z","dependency_job_id":"233c1dcb-513f-47c5-878b-53fa8803416e","html_url":"https://github.com/scality/backbeat","commit_stats":null,"previous_names":[],"tags_count":580,"template":false,"template_full_name":null,"purl":"pkg:github/scality/backbeat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fbackbeat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fbackbeat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fbackbeat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fbackbeat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scality","download_url":"https://codeload.github.com/scality/backbeat/tar.gz/refs/heads/development/9.2","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fbackbeat/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29636050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T22:32:43.237Z","status":"ssl_error","status_checked_at":"2026-02-19T22:32:38.330Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artesca","async","cross-region-replication","disaster-recovery","javascript","nodejs","replication","s3","zenko"],"created_at":"2024-08-01T20:01:20.594Z","updated_at":"2026-02-19T23:08:14.644Z","avatar_url":"https://github.com/scality.png","language":"JavaScript","funding_links":[],"categories":["HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"# Zenko Backbeat\n\n![backbeat logo](res/backbeat-logo.png)\n![Build Status](https://github.com/scality/backbeat/actions/workflows/release.yaml/badge.svg)\n\n## OVERVIEW\n\nBackbeat is an engine with a messaging system at its heart.\nIt's part of Zenko, [Scality](http://www.scality.com/)’s\nOpen Source Multi-Cloud Data Controller.\nLearn more about Zenko at [Zenko.io](http://www.zenko.io/)\n\nBackbeat is optimized for queuing metadata updates and dispatching work\nto long-running tasks in the background.\nThe core engine can be extended for many use cases,\nwhich are called extensions, as listed below.\n\n## EXTENSIONS\n\n### Asynchronous Replication\n\nThis feature replicates objects from one S3 bucket to another S3\nbucket in a different geographical region. The extension uses the\nlocal Metadata journal as the source of truth and replicates object\nupdates in a FIFO order.\n\n## DESIGN\n\n- [Backbeat core design](/DESIGN.md)\n- [CRR to AWS S3 workflow](/docs/crr-to-aws-s3.md)\n- [Object Lifecycle management](/docs/lifecycle.md)\n- [Metrics](/docs/metrics.md)\n- [CRR Pause and Resume](/docs/pause-resume.md)\n- [Healthcheck](/docs/healthcheck.md)\n- [Site Level CRR](/docs/site-level-crr.md)\n- [Transient Source](/docs/transient-crr-source.md)\n- [Out of Band updates from RING](/docs/oob-s3-ring.md)\n\n## QUICKSTART\n\nThis guide assumes the following:\n\n* Using MacOS\n* `brew` is installed (get it [here](https://brew.sh/))\n* `node` is installed (version 6.9.5)\n* `yarn` is installed (version 3.10.10)\n* `aws` is installed (version 1.11.1)\n\n### Kafka and Zookeeper\n\n#### Install and start kafka and zookeeper servers\n\n```\nmkdir ~/kafka \u0026\u0026 \\\ncd ~/kafka \u0026\u0026 \\\ncurl https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz | \\\ntar xvz \u0026\u0026 \\\nsed 's/zookeeper.connect=.*/zookeeper.connect=localhost:2181\\/backbeat/' \\\nkafka_2.11-0.11.0.0/config/server.properties \u003e \\\nkafka_2.11-0.11.0.0/config/server.properties.backbeat\n```\n\nStart the zookeeper server:\n\n```\nzookeeper-server-start kafka_2.11-0.11.0.0/config/zookeeper.properties\n```\n\nIn a different Shell start the kafka server:\n\n```\nkafka-server-start kafka_2.11-0.11.0.0/config/server.properties.backbeat\n```\n\n#### Install and run Redis on port 6379 (default port)\n\nThe \"Failed Entry Processor\" (started by Queue Populator) stores\nthe \"failed replication entries\" to the Redis sorted set.\n\n#### Create log offset Zookeeper node\n\n```\nzkCli -server localhost:2181/backbeat\n\ncreate /queue-populator\n```\n\n#### Create the kafka topics\n\n##### backbeat-replication\n\n```\nkafka-topics --create \\\n--zookeeper localhost:2181/backbeat \\\n--replication-factor 1 \\\n--partitions 1 \\\n--topic backbeat-replication\n```\n\n##### backbeat-replication-status\n\n```\nkafka-topics --create \\\n--zookeeper localhost:2181/backbeat \\\n--replication-factor 1 \\\n--partitions 1 \\\n--topic backbeat-replication-status\n```\n\n##### backbeat-replication-failed\n\n```\nkafka-topics --create \\\n--zookeeper localhost:2181/backbeat \\\n--replication-factor 1 \\\n--partitions 1 \\\n--topic backbeat-replication-failed\n```\n\n### Vault\n\n```\ngit clone https://github.com/scality/Vault ~/replication/vault \u0026\u0026 \\\ncd ~/replication/vault \u0026\u0026 \\\nyarn i\n```\n\nRun vault with \"in-memory backend\"\n\n```\nchmod 400 ./tests/utils/keyfile\n\nVAULT_DB_BACKEND=\"MEMORY\" yarn start\n```\n\n\u003e :warning: **with \"in-memory backend\"**, all data is lost after you stop the process.\n\nor with a \"mongodb backend\"\n\n```\nchmod 400 ./tests/utils/keyfile\n\nVAULT_DB_BACKEND=\"MONGODB\" yarn start\n```\n\n\u003e mongodb can be installed follwing these [steps](https://github.com/scality/backbeat/blob/development/8.3/docs/run-oob-locally.md#mongo-db-deploy-replica-set)\n\n### CloudServer\n\n```\ngit clone https://github.com/scality/cloudserver ~/replication/cloudserver \u0026\u0026 \\\ncd ~/replication/cloudserver \u0026\u0026 \\\nyarn i\n```\n\nGenerate AWS access/secret key with full IAM and S3 access.\n\nAdd your keys `aws configure --profile aws-account`\n\nCreate AWS destination versioning-enabled bucket.\n\n```sh\naws s3api create-bucket --bucket \u003cDESTINATION_BUCKET_NAME\u003e --profile aws-account\n\naws s3api put-bucket-versioning \\\n--bucket \u003cDESTINATION_BUCKET_NAME\u003e \\\n--versioning-configuration Status=Enabled \\\n--profile aws-account\n```\n\nReplace existing `./locationConfig.json` with:\n\n```json\n{\n    \"us-east-1\": {\n      \"details\": {\n        \"supportsVersioning\": true\n      },\n      \"isTransient\": false,\n      \"legacyAwsBehavior\": false,\n      \"objectId\": \"0b1d9226-a694-11eb-bc21-baec55d199cd\",\n      \"type\": \"file\"\n    },\n    \"aws-location\": {\n        \"type\": \"aws_s3\",\n        \"legacyAwsBehavior\": true,\n        \"details\": {\n            \"awsEndpoint\": \"s3.amazonaws.com\",\n            \"bucketName\": \"\u003cDESTINATION_BUCKET_NAME\u003e\",\n            \"bucketMatch\": false,\n            \"credentialsProfile\": \"aws-account\",\n            \"serverSideEncryption\": true\n        }\n    }\n}\n```\n\nUpdate `./config.json` with\n\n```json\n\"replicationEndpoints\": [{ \"site\": \"aws-location\", \"type\": \"aws_s3\" }],\n```\n\nIn `./config.json`, make sure `recordLog.enabled` is set to `true`\n\n```\n\"recordLog\": {\n        \"enabled\": true,\n        ...\n}\n```\n\nRun Cloudserver\n\n```\nS3DATA=multiple S3METADATA=mongodb REMOTE_MANAGEMENT_DISABLE=true \\\nS3VAULT=scality yarn start\n```\n\n### Vaultclient\n\nCreate a \"Zenko\" account and generate keys\n\n```\nbin/vaultclient create-account --name bart --email dev@backbeat --port 8600\nbin/vaultclient generate-account-access-key --name bart --port 8600\naws configure --profile bart\n```\n\n### Backbeat\n\n```\ngit clone https://github.com/scality/backbeat ~/replication/backbeat \u0026\u0026 \\\ncd ~/replication/backbeat \u0026\u0026 \\\nyarn i\n```\n\nUpdate `conf/authdata.json` with bart informations and keys.\n\n```json\n{\n    \"accounts\": [{\n        \"name\": \"bart\",\n        \"arn\": \"aws:iam::331457510670:/bart\",\n        \"canonicalID\": \"2083781e15384e30f48c651a948ec2dc1e1801c4af24c2750a166823e28ca570\",\n        \"displayName\": \"bart\",\n        \"keys\": {\n            \"access\": \"20TNCD06HOCSLQSABFZP\",\n            \"secret\": \"1P43SL0ekJjXnQvliV0KgMibZ=N2lKZO4dpnWzbF\"\n        }\n    }\n    ]\n}\n```\n\nUpdate `conf/config.json` section `extensions.replication.source.auth`\n\n```json\n\"auth\": {\n    \"type\": \"account\",\n    \"account\": \"bart\",\n    \"vault\": {\n        \"host\": \"127.0.0.1\",\n        \"port\": 8500,\n        \"adminPort\": 8600\n    }\n}\n```\n\nMake sure `conf/config.json` section\n`extensions.replication.destination.bootstrapList` includes:\n\n```json\n{ \"site\": \"aws-location\", \"type\": \"aws_s3\" }\n```\n\n#### Queue populator\n\n```\nS3_REPLICATION_METRICS_PROBE=true REMOTE_MANAGEMENT_DISABLE=true \\\nyarn run queue_populator\n```\n\n#### Queue processor\n\n```\nS3_REPLICATION_METRICS_PROBE=true REMOTE_MANAGEMENT_DISABLE=true \\\nyarn run queue_processor aws-location\n```\n\n#### Replication status processor\n\n```\nS3_REPLICATION_METRICS_PROBE=true REMOTE_MANAGEMENT_DISABLE=true \\\nyarn run replication_status_processor\n```\n\n### AWS S3 CLI\n\nCreate a source bucket with versioning enabled:\n\n```\naws s3api create-bucket \\\n--bucket sourcebucket \\\n--endpoint-url http://127.0.0.1:8000 \\\n--profile bart\n```\n\n```\naws s3api put-bucket-versioning \\\n--bucket sourcebucket \\\n--versioning-configuration Status=Enabled \\\n--endpoint-url=http://127.0.0.1:8000 \\\n--profile bart\n```\n\n#### Set up replication\n\nCreate `replication.json`\n\n```json\n{\n    \"Role\": \"arn:aws:iam::root:role/s3-replication-role\",\n    \"Rules\": [\n        {\n            \"Status\": \"Enabled\",\n            \"Prefix\": \"\",\n            \"Destination\": {\n                \"Bucket\": \"arn:aws:s3:::sourcebucket\",\n                \"StorageClass\": \"aws-location\"\n            }\n        }\n    ]\n}\n```\n\n```\naws s3api put-bucket-replication \\\n--bucket sourcebucket \\\n--replication-configuration file://replication.json \\\n--endpoint-url=http://127.0.0.1:8000 \\\n--profile bart\n```\n\n#### Put object to be replicated\n\n```\naws s3api put-object \\\n--key key0 \\\n--body file \\\n--bucket sourcebucket \\\n--endpoint-url=http://127.0.0.1:8000 \\\n--profile bart\n```\n\n#### Check that object has been replicated\n\n```\naws s3api head-object \\\n--key bucketsource/key0 \\\n--bucket \u003cDESTINATION_BUCKET_NAME\u003e \\\n--profile aws-account\n```\n\n#### Structure\n\nIn our `$HOME` directory, we now have the following directories:\n\n```\n$HOME\n├── kafka\n│   └── kafka_2.11-0.11.0.0\n├── replication\n    ├── backbeat\n    ├── cloudserver\n    └── vault\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Fbackbeat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscality%2Fbackbeat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Fbackbeat/lists"}