{"id":14956789,"url":"https://github.com/altinity/clickhouse-sink-connector","last_synced_at":"2025-05-16T09:05:43.314Z","repository":{"id":37208191,"uuid":"472249049","full_name":"Altinity/clickhouse-sink-connector","owner":"Altinity","description":"Replicate data from MySQL, Postgres and MongoDB to ClickHouse®","archived":false,"fork":false,"pushed_at":"2025-05-13T18:18:14.000Z","size":269593,"stargazers_count":258,"open_issues_count":196,"forks_count":62,"subscribers_count":9,"default_branch":"develop","last_synced_at":"2025-05-16T09:05:37.919Z","etag":null,"topics":["avro","cdc","clickhouse","debezium","kafka","kafka-connect","mongo","mysql","postgresql","replication","sqlserver"],"latest_commit_sha":null,"homepage":"https://www.altinity.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Altinity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-03-21T08:32:44.000Z","updated_at":"2025-05-15T08:43:28.000Z","dependencies_parsed_at":"2025-04-12T09:43:38.362Z","dependency_job_id":"60771b0f-49a6-40d4-b153-25c6d620d8ae","html_url":"https://github.com/Altinity/clickhouse-sink-connector","commit_stats":{"total_commits":1965,"total_committers":24,"mean_commits":81.875,"dds":0.4854961832061069,"last_synced_commit":"c882fcac89d77f8ce3440c3bb67792617902048c"},"previous_names":[],"tags_count":37,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-sink-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-sink-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-sink-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-sink-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Altinity","download_url":"https://codeload.github.com/Altinity/clickhouse-sink-connector/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254501557,"owners_count":22081528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","cdc","clickhouse","debezium","kafka","kafka-connect","mongo","mysql","postgresql","replication","sqlserver"],"created_at":"2024-09-24T13:13:31.759Z","updated_at":"2025-05-16T09:05:38.304Z","avatar_url":"https://github.com/Altinity.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License](http://img.shields.io/:license-apache%202.0-brightgreen.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)\n[![Sink Connector(Kafka version) tests](https://github.com/Altinity/clickhouse-sink-connector/actions/workflows/sink-connector-kafka-tests.yml/badge.svg)](https://github.com/Altinity/clickhouse-sink-connector/actions/workflows/sink-connector-kafka-tests.yml)\n[![Sink Connector(Light-weight) Tests](https://github.com/Altinity/clickhouse-sink-connector/actions/workflows/sink-connector-lightweight-tests.yml/badge.svg)](https://github.com/Altinity/clickhouse-sink-connector/actions/workflows/sink-connector-lightweight-tests.yml)\n\u003ca href=\"https://altinity.com/slack\"\u003e\n  \u003cimg src=\"https://img.shields.io/static/v1?logo=slack\u0026logoColor=959DA5\u0026label=Slack\u0026labelColor=333a41\u0026message=join%20conversation\u0026color=3AC358\" alt=\"AltinityDB Slack\" /\u003e\n\u003c/a\u003e\n\u003cimg alt=\"Docker Pulls\" src=\"https://img.shields.io/docker/pulls/altinityinfra/clickhouse-sink-connector\"\u003e\n### Latest Releases\nhttps://github.com/Altinity/clickhouse-sink-connector/releases\n\n# Altinity Sink Connector for ClickHouse®\n\nThe Altinity Sink Connector moves data automatically from \ntransactional database tables in MySQL and PostgreSQL to ClickHouse\nfor analysis. \n\n## Features\nRefer [Feature Matrix](doc/feature_matrix.md) for detailed features.\n\n* [Initial data dump and load(MySQL)](sink-connector/python/README.md) \n* Change data capture of new transactions using [Debezium](https://debezium.io/)\n* Automatic loading into ClickHouse\n* Sources: Support for MySQL, PostgreSQL (other databases experimental)\n* Target: Support for ClickHouse ReplacingMergeTree\n* Able to recover/restart from failures on source or target\n* Handle upstream schema changes automatically\n* Checksum-based table comparisons\n* Scalable to 1000s of tables\n* Multiple deployment models\n  * Lightweight: single process that transfers from source to target.\n  * Kafka: separate source and target processes using Kafka as transport.\n* Distribution as [Docker](https://hub.docker.com/layers/altinityinfra/clickhouse-sink-connector/408-97b1d3d83ef93c1b76a2b1c4d9c544dc67fbbec3-lt/images/sha256-d134bc05e50df7f63025e776ab6e3216c6622cd159eb0f2d459ea2ce8975f396?context=explore)\n container\n\n## Getting Started\n\n[QuickStart Guide: Lightweight (MySQL)](doc/quickstart.md)\\\n[QuickStart Guide: Lightweight (PostgreSQL)](doc/quickstart_postgres.md)\\\n[QuickStart Guide: Kafka](doc/quickstart_kafka.md)\n\n## Blog Articles\n\nFirst two are good tutorials on MySQL and PostgreSQL respectively. \n\n- [Altinity Sink Connector (MySQL)](https://altinity.com/blog/fast-mysql-to-clickhouse-replication-announcing-the-altinity-sink-connector-for-clickhouse)\n- [Altinity Sink Connector (PostgreSQL)](https://altinity.com/blog/replicating-data-from-postgresql-to-clickhouse-with-the-altinity-sink-connector)\n- [ClickHouse as an analytic extension for MySQL](https://altinity.com/blog/using-clickhouse-as-an-analytic-extension-for-mysql?utm_campaign=Brand\u0026utm_content=224583767\u0026utm_medium=social\u0026utm_source=linkedin\u0026hss_channel=lcp-10955938)\n\n## Reference Documentation\n\n### General \n\n* [Architecture Overview](doc/architecture.md)\n* [Lightweight Sink Connect CLI](doc/sink_connector_cli.md)\n* [Connection Pool](doc/connection_pool.md)\n* [Mutable Data Handling](doc/mutable_data.md)\n* [ClickHouse Table Engine Types](doc/clickhouse_engines.md)\n* [Troubleshooting](doc/Troubleshooting.md)\n* [TimeZone and DATETIME/TIMESTAMP](doc/timezone.md)\n* [Replication Start Position](doc/replication_start_position.md)\n* [Logging](doc/logging.md)\n* [Production Setup](doc/production_setup.md)\n* [Adding new tables(Incremental Snapshot)](doc/incremental_snapshot.md)\n* [Multiple Connectors](doc/multiple_connectors.md)\n* [Configuration](doc/configuration.md)\n* [State Storage](doc/state_storage.md)\n* [Data Type Mapping](doc/data_types.md)\n\n### Operations\n\n* [Monitoring](doc/Monitoring.md)\n* [Load Testing with Sysbench](doc/Performance.md)\n\n### Development\n\n* [Development](doc/development.md)\n* [Testing](doc/TESTING.md)\n\n## Comparison with other technologies\n- [Comparison](doc/comparison.md)\n\n## Roadmap \n\n[2025 Roadmap](https://github.com/Altinity/clickhouse-sink-connector/issues/401)\n\n## Help\n\nFile an issue or contact us on the Altinity public Slack workspace. Use \nthe link on the Slack badge at the top of this page. \n\n## Contributing\n\nContributions to the project are welcome in any form. \n\n* Submit issues documenting feature requests and bugs\n* Submit PRs to make changes\n* Talk about the project, write blog articles, or give presentations\n\nWe recommend that you file an issue before implementing feature additions \nor major fixes. We are happy to provide guidance and encouragement!\n\n## Commercial Support\n\nAltinity is the primary maintainer of the Sink Connector. It is used\ntogether with Altinity.Cloud as well as self-managed ClickHouse\ninstallations.  Altinity.Cloud and is also used in self-managed\ninstallations. Altinity offers a range of software and services related\nto ClickHouse and analytic applications built on ClickHouse. \n\n- [Official website](https://altinity.com/) - Get a high level overview of Altinity and our offerings.\n- [Altinity.Cloud](https://altinity.com/cloud-database/) - Run ClickHouse in our cloud or yours.\n- [Altinity Support](https://altinity.com/support/) - Get Enterprise-class support for ClickHouse and Sink Connector.\n- [Slack](https://altinity.com/slack) - Talk directly with ClickHouse users and Altinity devs.\n- [Contact us](https://hubs.la/Q020sH3Z0) - Contact Altinity with your questions or issues.\n- [Free consultation](https://hubs.la/Q020sHkv0) - Get a free consultation with a ClickHouse expert today.\n\n### Credits\nYourKit supports the Altinity Sink Connector and other open source projects with its full-featured Java Profiler.  Click the YourKit logo below to learn more.\u003cbr/\u003e\n[![](https://www.yourkit.com/images/yklogo.png)](http://www.yourkit.com/java/profiler/index.jsp)\u003cbr/\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltinity%2Fclickhouse-sink-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faltinity%2Fclickhouse-sink-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltinity%2Fclickhouse-sink-connector/lists"}