Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/hellobike/amazonriver

amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务
https://github.com/hellobike/amazonriver

elasticsearch es go kafka pg postgres postgresql realtime replication stream sync wal

Last synced: 3 months ago
JSON representation

amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务

Host: GitHub
URL: https://github.com/hellobike/amazonriver
Owner: hellobike
License: apache-2.0
Created: 2018-12-25T07:28:04.000Z (about 6 years ago)
Default Branch: master
Last Pushed: 2019-12-03T05:49:41.000Z (about 5 years ago)
Last Synced: 2024-06-19T05:46:24.849Z (8 months ago)
Topics: elasticsearch, es, go, kafka, pg, postgres, postgresql, realtime, replication, stream, sync, wal
Language: Go
Homepage:
Size: 329 KB
Stars: 220
Watchers: 19
Forks: 58
Open Issues: 5
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        # amazonriver [![CircleCI](https://circleci.com/gh/hellobike/amazonriver.svg?style=svg)](https://circleci.com/gh/hellobike/amazonriver) [![Actions Status](https://github.com/hellobike/amazonriver/workflows/Go/badge.svg)](https://github.com/hellobike/amazonriver/actions)

[English doc](./README_EN.md)

[![Go Report Card](https://goreportcard.com/badge/github.com/hellobike/amazonriver)](https://goreportcard.com/report/github.com/hellobike/amazonriver)

[![golang](https://img.shields.io/badge/Language-Go-green.svg?style=flat)](https://golang.org)

[![GoDoc](https://godoc.org/github.com/hellobike/amazonriver?status.svg)](https://godoc.org/github.com/hellobike/amazonriver)

[![GitHub release](https://img.shields.io/github/release/hellobike/amazonriver.svg)](https://github.com/hellobike/amazonriver/releases)

amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务

## 版本支持

- Postgresql 9.4 or later

- Kafka 0.8 or later

- ElasticSearch 5.x

## 架构图

![架构图](./doc/arch.png)

## 原理

amazonriver 利用pg内部的逻辑复制功能,通过在pg创建逻辑复制槽,接收数据库的逻辑变更,通过解析test_decoding特定格式的消息,得到逻辑数据

## 安装使用

### 安装

```shell

$git clone https://github.com/hellobike/amazonriver

$cd amazonriver

$go install

```

### 使用

    amazonriver -config config.json

## PG 配置

PG数据库需要预先开启逻辑复制[pg配置](./doc/pg.md)

## amazonriver 配置

### 监控

amazonriver支持使用prometheus来监控同步数据状态,[配置Grafana监控](./doc/prometheus.md)

### 同步到 elasticsearch

[同步到elasticsearch](./doc/es.md)

### 同步到 kafka

[同步到kafka](./doc/kafka.md)

## 性能测试

[性能测试](./doc/test.md)

## 许可

amazonriver 使用 Apache License 2 许可