{"id":18006596,"url":"https://github.com/aikuyun/flinkx","last_synced_at":"2025-04-04T11:41:04.882Z","repository":{"id":39934470,"uuid":"272433962","full_name":"aikuyun/flinkx","owner":"aikuyun","description":"flinkx 一些修改","archived":false,"fork":false,"pushed_at":"2022-11-16T09:26:04.000Z","size":12840,"stargazers_count":1,"open_issues_count":7,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-09T21:31:06.033Z","etag":null,"topics":["data","flink"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aikuyun.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-15T12:34:02.000Z","updated_at":"2021-08-24T09:16:29.000Z","dependencies_parsed_at":"2022-09-13T20:41:12.939Z","dependency_job_id":null,"html_url":"https://github.com/aikuyun/flinkx","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aikuyun%2Fflinkx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aikuyun%2Fflinkx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aikuyun%2Fflinkx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aikuyun%2Fflinkx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aikuyun","download_url":"https://codeload.github.com/aikuyun/flinkx/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247174374,"owners_count":20896074,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","flink"],"created_at":"2024-10-30T01:08:59.212Z","updated_at":"2025-04-04T11:41:04.858Z","avatar_url":"https://github.com/aikuyun.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"FlinkX\n============\n\n[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)\n\nEnglish | [中文](README_CH.md)\n\n# Communication\n\n- We are recruiting **Big data platform development engineers**.If you want more information about the position, please add WeChat ID [**ysqwhiletrue**] or email your resume to [sishu@dtstack.com](mailto:sishu@dtstack.com).\n\n- We use [DingTalk](https://www.dingtalk.com/) to communicate,You can search the group number [**30537511**] or scan the QR code below to join the communication group\n  \n  \u003cdiv align=center\u003e\n     \u003cimg src=docs/images/ding.jpg width=300 /\u003e\n   \u003c/div\u003e\n\n# Introduction\n\nFlinkX is a data synchronization tool based on Flink. FlinkX can collect static data, such as MySQL, HDFS, etc, as well as real-time changing data, such as MySQL binlog, Kafka, etc. FlinkX currently includes the following features:\n\n- Most plugins support concurrent reading and writing of data, which can greatly improve the speed of reading and writing;\n\n- Some plug-ins support the function of failure recovery, which can restore tasks from the failed location and save running time; [Failure Recovery](docs/restore.md)\n\n- The Reader plugin for relational databases supports interval polling. It can continuously collect changing data; [Interval Polling](docs/offline/reader/mysqlreader.md)\n\n- Some databases support opening Kerberos security authentication;  [Kerberos](docs/kerberos.md)\n\n- Limit the reading speed of Reader plugins and reduce the impact on business databases;\n\n- Save the dirty data when writing data;\n\n- Limit the maximum number of dirty data;\n\n- Multiple running modes: Local,Standalone,Yarn Session,Yarn Per;\n\nThe following databases are currently supported:\n\n|                        | Database Type  | Reader                          | Writer                          |\n|:----------------------:|:--------------:|:-------------------------------:|:-------------------------------:|\n| Batch Synchronization  | MySQL          | [doc](docs/offline/reader/mysqlreader.md)        | [doc](docs/offline/writer/mysqlwriter.md)      |\n|                        | Oracle         | [doc](docs/offline/reader/oraclereader.md)       | [doc](docs/offline/writer/oraclewriter.md)     |\n|                        | SqlServer      | [doc](docs/offline/reader/sqlserverreader.md)    | [doc](docs/offline/writer/sqlserverwriter.md)  |\n|                        | PostgreSQL     | [doc](docs/offline/reader/postgresqlreader.md)   | [doc](docs/offline/writer/postgresqlwriter.md) |\n|                        | DB2            | [doc](docs/offline/reader/db2reader.md)          | [doc](docs/offline/writer/db2writer.md)        |\n|                        | GBase          | [doc](docs/offline/reader/gbasereader.md)        | [doc](docs/offline/writer/gbasewriter.md)      |\n|                        | ClickHouse     | [doc](docs/offline/reader/clickhousereader.md)   | [doc](docs/offline/writer/clickhousewriter.md) |\n|                        | PolarDB        | [doc](docs/offline/reader/polardbreader.md)      | [doc](docs/offline/writer/polardbwriter.md)    |\n|                        | SAP Hana       | [doc](docs/offline/reader/saphanareader.md)      | [doc](docs/offline/writer/saphanawriter.md)    |\n|                        | Teradata       | [doc](docs/offline/reader/teradatareader.md)     | [doc](docs/offline/writer/teradatawriter.md)   |\n|                        | Phoenix        | [doc](docs/offline/reader/phoenixreader.md)      | [doc](docs/offline/writer/phoenixwriter.md)    |\n|                        | 达梦            | [doc](docs/offline/reader/dmreader.md)           | [doc](docs/offline/writer/dmwriter.md)        |\n|                        | Cassandra      | [doc](docs/offline/reader/cassandrareader.md)    | [doc](docs/offline/writer/cassandrawriter.md)  |\n|                        | ODPS           | [doc](docs/offline/reader/odpsreader.md)         | [doc](docs/offline/writer/odpswriter.md)       |\n|                        | HBase          | [doc](docs/offline/reader/hbasereader.md)        | [doc](docs/offline/writer/hbasewriter.md)      |\n|                        | MongoDB        | [doc](docs/offline/reader/mongodbreader.md)      | [doc](docs/offline/writer/mongodbwriter.md)    |\n|                        | Kudu           | [doc](docs/offline/reader/kudureader.md)         | [doc](docs/offline/writer/kuduwriter.md)       |\n|                        | ElasticSearch  | [doc](docs/offline/reader/esreader.md)           | [doc](docs/offline/writer/eswriter.md)         |\n|                        | FTP            | [doc](docs/offline/reader/ftpreader.md)          | [doc](docs/offline/writer/ftpwriter.md)        |\n|                        | HDFS           | [doc](docs/offline/reader/hdfsreader.md)         | [doc](docs/offline/writer/hdfswriter.md)       |\n|                        | Carbondata     | [doc](docs/offline/reader/carbondatareader.md)   | [doc](docs/offline/writer/carbondatawriter.md) |\n|                        | Stream         | [doc](docs/offline/reader/streamreader.md)       | [doc](docs/offline/writer/carbondatawriter.md) |\n|                        | Redis          |                                                  | [doc](docs/offline/writer/rediswriter.md)      |\n|                        | Hive           |                                                  | [doc](docs/offline/writer/hivewriter.md)       |\n| Stream Synchronization | Kafka          | [doc](docs/realTime/reader/kafkareader.md)       | [doc](docs/realTime/writer/kafkawriter.md)     |\n|                        | EMQX           | [doc](docs/realTime/reader/emqxreader.md)        | [doc](docs/realTime/writer/emqxwriter.md)      |\n|                        | MySQL Binlog   | [doc](docs/realTime/reader/binlogreader.md)      |                                                |\n|                        | MongoDB Oplog  | [doc](docs/realTime/reader/mongodboplogreader.md)|                                                |\n|                        | PostgreSQL WAL | [doc](docs/realTime/reader/pgwalreader.md)       |                                                |\n|                        | Oracle Logminer| Coming Soon|                                              |\n|                        | SqlServer CDC  | Coming Soon |                                               |\n\n# Quick Start\n\nPlease click [Quick Start](docs/quickstart.md)\n\n# General Configuration\n\nPlease click [General Configuration](docs/generalconfig.md)\n\n# Statistics Metric\n\nPlease click [Statistics Metric](docs/statistics.md)\n\n# Kerberos\n\nPlease click [Kerberos](docs/kerberos.md)\n\n# How to contribute FlinkX\n\nPlease click [Contribution](docs/contribution.md)\n\n# License\n\nFlinkX is under the Apache 2.0 license. See the [LICENSE](http://www.apache.org/licenses/LICENSE-2.0) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faikuyun%2Fflinkx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faikuyun%2Fflinkx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faikuyun%2Fflinkx/lists"}