https://github.com/getdozer/dozer

Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.
https://github.com/getdozer/dozer

api apis clickhouse data datawarehouse debe etl low-code postgres realtime rust snowflake sql streaming

Last synced: about 8 hours ago
JSON representation

Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.

• Host: GitHub
• URL: https://github.com/getdozer/dozer
• Owner: getdozer
• License: agpl-3.0
• Created: 2022-08-31T11:20:38.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-06-18T22:23:39.000Z (6 months ago)
• Last Synced: 2024-11-27T23:02:40.518Z (14 days ago)
• Topics: api, apis, clickhouse, data, datawarehouse, debe, etl, low-code, postgres, realtime, rust, snowflake, sql, streaming
• Language: Rust
• Homepage: https://getdozer.io
• Size: 59.3 MB
• Stars: 1,514
• Watchers: 14
• Forks: 124
• Open Issues: 144
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE.txt
  • Code of conduct: CODE_OF_CONDUCT.md
  • Security: SECURITY.md

Awesome Lists containing this project

• stars - getdozer/dozer - time data movement tool that leverages CDC from various sources and moves data into various sinks. (HarmonyOS / Windows Manager)
• awesome-clickhouse - getdozer/dozer - Dozer is a real-time data movement tool that leverages Change Data Capture (CDC) from various sources to move data into various sinks. (Integrations / ETL and Data Processing)

README

        
## Overview

Dozer is a **real time data movement tool leveraging CDC from various sources to multiple sinks.**

Dozer is magnitudes of times faster than Debezium+Kafka and natively supports stateless transformations. 

Primarily used for moving data into warehouses. In our own application, we move data to **Clickhouse** and build data APIs and integration with LLMs. 

## How to use it

Dozer runs with a single configuration file like the following:

```yaml

app_name: dozer-bench

version: 1

connections:

  - name: pg_1

    config: !Postgres

      user: user

      password: postgres

      host: localhost

      port: 5432

      database: customers

sinks:

  - name: customers

    config: !Dummy

      table_name: customers

```

Full documentation can be found [here](https://github.com/getdozer/dozer/blob/main/dozer-types/src/models/config.rs#L15)

## Supported Sources

| Connector            | Extraction | Resuming | Enterprise          |

| -------------------- | ---------- | -------- | ------------------- |

| Postgres             | ✅          | ✅        | ✅                   |

| MySQL                | ✅          | ✅        | ✅                   |

| Snowflake            | ✅          | ✅        | ✅                   |

| Kafka                | ✅          | 🚧        | ✅                   |

| MongoDB              | ✅          | 🎯        | ✅                   |

| Amazon S3            | ✅          | 🎯        | ✅                   |

| Google Cloud Storage | ✅          | 🎯        | ✅                   |

| **Oracle             | ✅          | ✅        | **Enterprise Only** |

| **Aerospike          | ✅          | ✅        | **Enterprise Only** |

## Supported Sinks

| Database   | Connectivity | Enterprise          |

| ---------- | ------------ | ------------------- |

| Clickhouse | ✅            |                     |

| Postgres   | ✅            |                     |

| MySQL      | ✅            |                     |

| Big Query  | ✅            |                     |

| Oracle     | ✅            | **Enterprise Only** |

| Aerospike  | ✅            | **Enterprise Only** |