https://github.com/artie-labs/reader
Perform historical snapshots without database locks and read change data capture logs from databases. Artie Reader is compatible with Debezium and is written in Go.
https://github.com/artie-labs/reader
apache-kafka cdc change-data-capture data-integration database debezium golang kafka
Last synced: 5 months ago
JSON representation
Perform historical snapshots without database locks and read change data capture logs from databases. Artie Reader is compatible with Debezium and is written in Go.
- Host: GitHub
- URL: https://github.com/artie-labs/reader
- Owner: artie-labs
- License: other
- Created: 2023-08-20T16:30:46.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2024-11-19T00:03:24.000Z (11 months ago)
- Last Synced: 2024-11-19T01:17:45.503Z (11 months ago)
- Topics: apache-kafka, cdc, change-data-capture, data-integration, database, debezium, golang, kafka
- Language: Go
- Homepage: https://artie.com
- Size: 1.34 MB
- Stars: 34
- Watchers: 4
- Forks: 3
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
Artie Reader reads from databases to perform historical snapshots and also reads change data capture (CDC) logs for continuous streaming. The generated messages are Debezium capable.
## Benefits
* Historical table snapshots do not require database locks, which means Artie Reader minimizes impact to database performance and avoids situations like replication slot overflow.
* Debezium compatible. The generated messages are consistent with Debezium’s message format.
* Portable and easy to operate. Shipped as a standalone binary with no external dependencies.
## Architecture
## Supports:
| | Snapshot | Streaming |
|------------|----------|-----------|
| DynamoDB | ✅ | ✅ |
| MongoDB | ✅ | ✅ |
| MySQL | ✅ | 🚧 |
| PostgreSQL | ✅ | ❌ |
| SQL Server | ✅ | ❌ |
## Running
To get started, you'll need a `config.yaml` file, you can see examples of this in the [examples](https://github.com/artie-labs/reader/tree/master/examples) folder.
```bash
go run main.go --config config.yaml
```
