{"id":19989898,"url":"https://github.com/hellobike/amazonriver","last_synced_at":"2025-05-04T09:33:44.443Z","repository":{"id":57485214,"uuid":"163057959","full_name":"hellobike/amazonriver","owner":"hellobike","description":"amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务","archived":false,"fork":false,"pushed_at":"2019-12-03T05:49:41.000Z","size":337,"stargazers_count":220,"open_issues_count":5,"forks_count":58,"subscribers_count":19,"default_branch":"master","last_synced_at":"2024-06-19T05:46:24.849Z","etag":null,"topics":["elasticsearch","es","go","kafka","pg","postgres","postgresql","realtime","replication","stream","sync","wal"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hellobike.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-25T07:28:04.000Z","updated_at":"2024-05-04T10:28:00.000Z","dependencies_parsed_at":"2022-08-26T13:41:10.303Z","dependency_job_id":null,"html_url":"https://github.com/hellobike/amazonriver","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hellobike%2Famazonriver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hellobike%2Famazonriver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hellobike%2Famazonriver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hellobike%2Famazonriver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hellobike","download_url":"https://codeload.github.com/hellobike/amazonriver/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224391390,"owners_count":17303609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","es","go","kafka","pg","postgres","postgresql","realtime","replication","stream","sync","wal"],"created_at":"2024-11-13T04:50:45.988Z","updated_at":"2024-11-13T04:50:48.103Z","avatar_url":"https://github.com/hellobike.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# amazonriver [![CircleCI](https://circleci.com/gh/hellobike/amazonriver.svg?style=svg)](https://circleci.com/gh/hellobike/amazonriver) [![Actions Status](https://github.com/hellobike/amazonriver/workflows/Go/badge.svg)](https://github.com/hellobike/amazonriver/actions)\n\n[English doc](./README_EN.md)\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/hellobike/amazonriver)](https://goreportcard.com/report/github.com/hellobike/amazonriver)\n[![golang](https://img.shields.io/badge/Language-Go-green.svg?style=flat)](https://golang.org)\n[![GoDoc](https://godoc.org/github.com/hellobike/amazonriver?status.svg)](https://godoc.org/github.com/hellobike/amazonriver)\n[![GitHub release](https://img.shields.io/github/release/hellobike/amazonriver.svg)](https://github.com/hellobike/amazonriver/releases)\n\namazonriver 是一个将postgresql的实时数据同步到es或kafka的服务\n\n## 版本支持\n\n- Postgresql 9.4 or later\n- Kafka 0.8 or later\n- ElasticSearch 5.x\n\n## 架构图\n\n![架构图](./doc/arch.png)\n\n## 原理\n\namazonriver 利用pg内部的逻辑复制功能,通过在pg创建逻辑复制槽,接收数据库的逻辑变更,通过解析test_decoding特定格式的消息,得到逻辑数据\n\n## 安装使用\n\n### 安装\n\n```shell\n$git clone https://github.com/hellobike/amazonriver\n$cd amazonriver\n$go install\n```\n\n### 使用\n\n    amazonriver -config config.json\n\n## PG 配置\n\nPG数据库需要预先开启逻辑复制[pg配置](./doc/pg.md)\n\n## amazonriver 配置\n\n### 监控\n\namazonriver支持使用prometheus来监控同步数据状态,[配置Grafana监控](./doc/prometheus.md)\n\n### 同步到 elasticsearch\n\n[同步到elasticsearch](./doc/es.md)\n\n### 同步到 kafka\n\n[同步到kafka](./doc/kafka.md)\n\n## 性能测试\n\n[性能测试](./doc/test.md)\n\n## 许可\n\namazonriver 使用 Apache License 2 许可\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhellobike%2Famazonriver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhellobike%2Famazonriver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhellobike%2Famazonriver/lists"}