https://github.com/zourzouvillys/laredo
Real-time data sync from PostgreSQL to in-memory targets. Baseline snapshots, change streaming, pluggable targets, replication fan-out.
https://github.com/zourzouvillys/laredo
cdc change-data-capture golang grpc in-memory logical-replication postgresql real-time
Last synced: 2 months ago
JSON representation
Real-time data sync from PostgreSQL to in-memory targets. Baseline snapshots, change streaming, pluggable targets, replication fan-out.
- Host: GitHub
- URL: https://github.com/zourzouvillys/laredo
- Owner: zourzouvillys
- License: apache-2.0
- Created: 2026-03-20T22:40:13.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-04-14T01:22:59.000Z (2 months ago)
- Last Synced: 2026-04-14T02:27:21.876Z (2 months ago)
- Topics: cdc, change-data-capture, golang, grpc, in-memory, logical-replication, postgresql, real-time
- Language: Go
- Homepage: https://zrz.io/laredo/
- Size: 1.6 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Laredo
A Go library and service for capturing baseline snapshots from data sources and streaming real-time changes through pluggable targets.
## Architecture
Laredo is organized in three layers:
1. **Core Library** — engine, interfaces, types, pipeline orchestration. Zero opinions on config format, transport, or logging.
2. **Optional Modules** — gRPC OAM/Query service, HOCON config loader, metrics bridges, source/target/snapshot implementations.
3. **Pre-built Service (`laredo-server`)** — a ready-to-run binary with HOCON config, gRPC, metrics, structured logging, and signal handling. Ships with the `laredo` CLI tool.
### Sources
- **PostgreSQL** — logical replication (ephemeral or stateful mode)
- **S3 + Kinesis** — S3 baseline snapshots with Kinesis change streams
### Targets
- **HTTP Sync** — forward changes as batched HTTP requests
- **Compiled In-Memory** — domain objects via pluggable compiler functions
- **Indexed In-Memory** — raw rows with configurable secondary indexes
- **Replication Fan-Out** — multiplex one source to N gRPC clients (snapshot + journal + live stream)
### Pipeline
Each pipeline binds a source table to a target, with optional filters and transforms. The engine manages baseline loading, change streaming, ACK coordination, backpressure, error isolation, TTL, and snapshots.
## Building
```bash
# Build all packages
make build
# Run tests
make test
# Lint
make lint
# Build Docker image
make docker
```
## Project Layout
```
laredo.go, types.go, source.go, ... Core interfaces (root package)
internal/engine/ Engine implementation
source/{pg,kinesis,testsource}/ Source implementations
target/{httpsync,memory,fanout}/ Target implementations
snapshot/{local,s3,jsonl}/ Snapshot store/serializer implementations
filter/, transform/ Built-in pipeline filters and transforms
service/ gRPC services (OAM, Query, Replication)
config/ HOCON config loading
metrics/{prometheus,otel}/ Metrics bridges
client/fanout/ Go fan-out client library
cmd/laredo-server/ Pre-built service binary
cmd/laredo/ CLI tool
```
## Library Usage
```go
engine, errs := laredo.NewEngine(
laredo.WithSource("pg_main", pgSource),
laredo.WithPipeline("pg_main", laredo.Table("public", "config_document"), memTarget),
laredo.WithObserver(myObserver),
)
if len(errs) > 0 {
log.Fatalf("config errors: %v", errs)
}
engine.Start(ctx)
engine.AwaitReady(30 * time.Second)
defer engine.Stop(ctx)
```
See [docs/spec.md](docs/spec.md) for the full design specification.