https://github.com/cqllum/db2ch
⚡Replicate your Database to Clickhouse in real time without Debezium, Kafka, or Clickhouse Materialized Engines
https://github.com/cqllum/db2ch
cdc clickhouse go golang mssql mysql postgresql realtime realtime-framework replication
Last synced: 10 months ago
JSON representation
⚡Replicate your Database to Clickhouse in real time without Debezium, Kafka, or Clickhouse Materialized Engines
- Host: GitHub
- URL: https://github.com/cqllum/db2ch
- Owner: cqllum
- License: apache-2.0
- Created: 2024-05-30T11:03:58.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-01T21:28:01.000Z (about 2 years ago)
- Last Synced: 2025-07-05T22:42:12.593Z (12 months ago)
- Topics: cdc, clickhouse, go, golang, mssql, mysql, postgresql, realtime, realtime-framework, replication
- Language: Go
- Homepage:
- Size: 42 KB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
### VERSION 0.0 - Not available for Production - POC Development
## ⚠️Warning
`db2ch` is still under heavy development. We reccommend that you not try to use it for production use until development has progressed more and bugs are less likely to occur.
# db2ch
`db2ch` is an open-source framework designed to simplify Change Data Capture (CDC) to ClickHouse without the need for complex setups like Debezium, Kafka, or ClickHouse Materialized Views. This project supports real-time data replication from PostgreSQL, MySQL, and MSSQL to ClickHouse.
## Features
- **Real-time Replication**: Efficiently stream data changes from PostgreSQL, MySQL, and MSSQL to ClickHouse in real-time.
- **Simplicity**: No need for additional tools like Debezium, Kafka, or ClickHouse Materialized Engines.
- **High Throughput**: Built with Go, ensuring high performance and low latency.
- **Control and Monitor**: HTTP APIs for controlling and monitoring replication processes.
- **Automatic Table Creation**: `db2ch` will automatically map source data types to Clickhouse data types and create tables.
- **Primary Key Based Backfill Capability**: `db2ch` also includes a capability to perform a backfil to bring across historic data.
- **Column Control**: `db2ch` replication allows you to control which columns you bring across.
## Project Structure
```
db2ch/
├── main.go
├── config/
│ ├── config.go
│ ├── config.json
├── api/
│ ├── handler.go
│ └── routes.go
├── replication/
│ ├── replication.go
│ ├── clickhouse.go
│ ├── mysql.go
│ └── mssql.go
└── go.mod
```
### Folder Descriptions
- `config/` - The purpose of this directory is define configuration details (connection details, data mapping, http port, etc).
- `api/` - Contains the `routes` for HTTP API endpoints and a handler for core api functionality.
- `replication/` - The framework behind listening to data and producing data.
## Getting Started
### Prerequisites
- Go 1.16+ installed
- PostgreSQL, MySQL, and MSSQL databases set up
- ClickHouse set up
### Installation
1. Clone the repository:
```sh
git clone https://github.com/cqllum/db2ch.git
cd db2ch
```