https://github.com/gustapinto/from-to
Moving CDC data from PostgreSQL to Apache Kafka with optional mapping using Lua functions
https://github.com/gustapinto/from-to
cdc go kafka lua postgres
Last synced: about 2 months ago
JSON representation
Moving CDC data from PostgreSQL to Apache Kafka with optional mapping using Lua functions
- Host: GitHub
- URL: https://github.com/gustapinto/from-to
- Owner: gustapinto
- License: mit
- Created: 2025-03-11T21:20:24.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-04-08T12:48:23.000Z (about 1 year ago)
- Last Synced: 2025-04-08T13:29:39.563Z (about 1 year ago)
- Topics: cdc, go, kafka, lua, postgres
- Language: Go
- Homepage:
- Size: 87.9 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# From To
## Installation
The pre-built binaries for **FromTo** are available in the [releases page](https://github.com/gustapinto/from-to/releases), they are the recommended installation option, but you can also compile the code yourself
### Compiling as binaries
1. To compile **FromTo** first clone this repository in your machine
2. Then open the cloned folder on a shell and run `make build` to build the binaries for all supported platforms, or run `make build/` to build for an specific platform. The currently supported platforms are:
- linux
- windows
- macos
#### Building Pre Requisites
1. [Go 1.24+](https://go.dev/)
2. [GNU Make](https://www.gnu.org/software/make/) (Optional)
## Example usage
```bash
./from_to_linux_amd64 -manifest=./from_to.yaml
```
## Example config
```yaml
input:
connector: "postgres"
postgresConfig:
dsn: "postgres://from-to-user:from-to-passw@localhost:5432/from-to-db?sslmode=disable"
tables:
- "sales"
outputs:
salesKafkaOutput:
connector: "kafka"
kafkaConfig:
bootstrapServers:
- "localhost:9094"
topics:
- name: "publicSales"
mappers:
salesMapper:
type: "lua"
luaConfig:
filePath: "./example/mappers.lua"
function: "map_sales_event_with_http"
channels:
salesChannel:
from: "sales"
to: "publicSales"
output: "salesKafkaOutput"
mapper: "salesMapper"
```
Additional options and specific documentation on inputs, outputs and mappers can be found at the [examples folder](https://github.com/gustapinto/from-to/blob/main/example)
## Connector support
The currently supported connectors and mappers are:
- **PostgreSQL (postgres):** Input connector
- **Kafka (kafka):** Output connector
- **Webhook (webhook):** Output connector
- **Lua (lua):** Mapper
## Lua support
**FromTo** supports [Lua](https://www.lua.org/) scripting to create row mappers, an example mapper can be found at [example/mappers.lua](https://github.com/gustapinto/from-to/blob/main/example/mappers.lua). It uses the [yuin/gopher-lua](https://github.com/yuin/gopher-lua) VM and preloads some of its libraries for improved DX.
### Preloaded libraries
- [cjoudrey/gluahttp](https://github.com/cjoudrey/gluahttp)
- [layeh.com/gopher-json](https://github.com/layeh/gopher-json)
## Development
**FormTo** development follows the [Uber Go Style Guide](https://github.com/uber-go/guide/blob/master/style.md)
### Development Pre Requisites
1. [Go 1.24+](https://go.dev/)
2. [GNU Make](https://www.gnu.org/software/make/) (Optional)
3. [Docker](https://www.docker.com/) or an compatible alternative
4. [Docker Compose](https://docs.docker.com/compose/) or an compatible alternative