{"id":17570848,"url":"https://github.com/uber/storagetapper","last_synced_at":"2025-06-11T17:39:45.249Z","repository":{"id":41443356,"uuid":"90499311","full_name":"uber/storagetapper","owner":"uber","description":"StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service","archived":false,"fork":false,"pushed_at":"2023-03-19T22:39:47.000Z","size":813,"stargazers_count":340,"open_issues_count":24,"forks_count":66,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-10-21T22:22:08.246Z","etag":null,"topics":["avro","cdc","clickhouse","etl","hdfs","json","kafka","msgpack","mysql","postgresql","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uber.png","metadata":{"files":{"readme":"README.md","changelog":"changelog/changelog.go","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-07T00:31:41.000Z","updated_at":"2024-10-14T08:20:06.000Z","dependencies_parsed_at":"2024-06-18T20:10:35.366Z","dependency_job_id":"ccc40967-22ab-4afd-b6f7-9929b410d4fc","html_url":"https://github.com/uber/storagetapper","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fstoragetapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fstoragetapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fstoragetapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fstoragetapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uber","download_url":"https://codeload.github.com/uber/storagetapper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242472674,"owners_count":20134006,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","cdc","clickhouse","etl","hdfs","json","kafka","msgpack","mysql","postgresql","s3"],"created_at":"2024-10-21T18:01:18.413Z","updated_at":"2025-03-07T22:30:24.695Z","avatar_url":"https://github.com/uber.png","language":"Go","readme":"StorageTapper\n-------------\n\nOverview\n--------\n\n[![Build Status](https://github.com/uber/storagetapper/workflows/Go/badge.svg)]()\n[![Go Report Card](https://goreportcard.com/badge/github.com/uber/storagetapper)](https://goreportcard.com/report/github.com/uber/storagetapper)\n[![codecov](https://codecov.io/gh/uber/storagetapper/branch/master/graph/badge.svg)](https://codecov.io/gh/uber/storagetapper)\n\nStorageTapper is a scalable realtime MySQL change data streaming, logical backup\nand logical replication service.\n\nStoragetapper is deployed in production at Uber and used to produce snapshot and\nrealtime changed data of thousands of MySQL tables across multiple datacenters.\n\nIt is also used as a backup service to snapshot hundreds of terrabytes\nof Schemaless data to HDFS and S3 with optional asymmetric encryption and\ncompression.\n\nIt reads data from source transforms according to the specified event\nformat and produces data to destination.\n\nSupported event sources:\n* MySQL\n* Schemaless\n\nSupported event destinations:\n* Kafka\n* HDFS\n* S3\n* Local file\n* MySQL (experimental)\n* Postgres (experimental)\n* Clickhouse (experimental)\n\nSupported event formats:\n* Avro\n* [JSON](./doc/commonformat.md)\n* MsgPack\n* SQL\n\nFeatures:\n* Fields filtering\n* Snapshot row filtering\n* Producing realtime changes only, i.e. omitting initial snapshot\n* Snapshot and changelog events ordering\n* Periodic snapshots\n* Snapshot index hinting\n\nStoragetapper keeps its jobs state in MySQL database and automatically distributes jobs\nbetween configured number of workers.\n\nIt is also aware of node roles and takes snapshot from the slave nodes in order \nto reduce load on master nodes. StorageTapper can also optionally further throttle its reads.\nBinlogs are streamed from master nodes for better SLAs.\n\nService is dynamically configurable through [RESTful API](./doc/endpoints.md) or\nbuilt-in UI.\n\nBuild \u0026 Install\n---------------\n\n## Debian \u0026 Ubuntu\n```sh\ncd storagetapper\nmake deb \u0026\u0026 dpkg -i ../storagetapper_1.0_amd64.deb\n```\n\n## Others\n```sh\ncd storagetapper\nmake \u0026\u0026 make install\n```\n\n## Development\n\n### Linux\n\n```sh\n/bin/bash scripts/install_deps.sh # install all dependencies: MySQL, Kafka, HDFS, S3, ...\nmake test # run all tests\nGO111MODULE=on TEST_PARAM=\"-test.run=TestLocalBasic\" /bin/bash scripts/run_tests.sh ./pipe # individual test\n```\n\n### Non Linux\n```sh\nmake test-env\n$ make test\n```\n\nConfiguration\n-------------\n\nStoragetapper loads configuration from the following files and location in the\ngiven order:\n```sh\n    /etc/storagetapper/base.yaml\n    /etc/storagetapper/production.yaml\n    $(HOME)/base.yaml\n    $(HOME)/production.yaml\n    $(STORAGETAPPER_CONFIG_DIR)/base.yaml\n    $(STORAGETAPPER_CONFIG_DIR)/production.yaml\n```\n\nAvailable options described in [Configuration](./doc/options.md) section\n\nLicense\n-------\nThis software is licensed under the [MIT License](LICENSE).\n\n","funding_links":[],"categories":["Integrations"],"sub_categories":["Data Transfer and Synchronization"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Fstoragetapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuber%2Fstoragetapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Fstoragetapper/lists"}