{"id":16599190,"url":"https://github.com/collabh/flink-connector-kudu","last_synced_at":"2025-03-21T13:32:28.899Z","repository":{"id":45577485,"uuid":"344154158","full_name":"collabH/flink-connector-kudu","owner":"collabH","description":"基于Apache-bahir-kudu-connector的flink-connector-kudu，支持Flink1.11.x DynamicTableSource/Sink，支持Range分区等","archived":false,"fork":false,"pushed_at":"2023-05-30T06:40:51.000Z","size":145,"stargazers_count":46,"open_issues_count":3,"forks_count":24,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-01T06:51:12.767Z","etag":null,"topics":["datastream","flink","flink-dynamic-source","flink-sql","kudu","kudu-connector"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/collabH.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-03T14:27:43.000Z","updated_at":"2024-11-12T08:38:05.000Z","dependencies_parsed_at":"2024-10-28T10:28:49.738Z","dependency_job_id":"da01cb77-b7b3-470c-9178-59c2eb0956c8","html_url":"https://github.com/collabH/flink-connector-kudu","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabH%2Fflink-connector-kudu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabH%2Fflink-connector-kudu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabH%2Fflink-connector-kudu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabH%2Fflink-connector-kudu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/collabH","download_url":"https://codeload.github.com/collabH/flink-connector-kudu/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244141580,"owners_count":20404835,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datastream","flink","flink-dynamic-source","flink-sql","kudu","kudu-connector"],"created_at":"2024-10-12T00:10:38.625Z","updated_at":"2025-03-21T13:32:28.627Z","avatar_url":"https://github.com/collabH.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![flink-connector-kudu](https://github.com/collabH/flink-connector-kudu/actions/workflows/maven.yml/badge.svg?branch=master)](https://github.com/collabH/flink-connector-kudu/actions/workflows/maven.yml)\n# Kudu Connector\n\n* 基于Apache-Bahir-Kudu-Connector改造而来的满足公司内部使用的Kudu Connector，支持特性Range分区、定义Hash分桶数、支持Flink1.11.x动态数据源等，改造后已贡献部分功能给社区。\n\n# Tag版本\n\n* v1.0.0使用kudu-client:1.10.0\n  * 注意点:目前使用kudu1.13之前的版本，kudu不支持delete ignore,因此在数据delete的时候该条数据一定要存在否则会出现`not primary \n  key`异常，目前的connector中解决方法为如果判断是Delete，则根据主键查询，查询不到数据则不进行删除(这样存在的问题是Delete操作需要一次查询IO，个人建议升级Kudu版本至1.14,\n  改造RowDataUpsertOperationMapper将newDelete改成newDeleteIgnore即可。)\n* v1.1.0使用kudu-client:1.14.0\n  * 不存在v1.0.0问题\n  \n# Version\n\n* branch:feature_support_with_flink113x\n  * 兼容Flink1.13.x主要改造点为KuduCatalogFactory，1.13.x过期的`TableSchema`类并未修改(不影响使用)\n\n# 使用姿势\n\n* clone代码后，改造pom项目坐标后上传公司私服使用\n\n\n## Kudu Catalog使用\n\n### 创建Catalog\n\n```java\nStreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();\ncatalog = new KuduCatalog(\"cdh01:7051,cdh02:7051,cdh03:7051\");\ntableEnv = KuduTableTestUtils.createTableEnvWithBlinkPlannerStreamingMode(env);\ntableEnv.registerCatalog(\"kudu\", catalog);\ntableEnv.useCatalog(\"kudu\");\n```\n\n### Catalog API\n\n```java\n// dropTable\n catalog.dropTable(new ObjectPath(\"default_database\", \"test_Replice_kudu\"), true);\n // 通过catalog操作表\ntableEnv.sqlQuery(\"select * from test\");\ntableEnv.executeSql(\"drop table test\");\ntableEnv.executeSql(\"insert into testRange values(1,'hsm')\");\n\n```\n\n## FlinkSQL\n\n### KuduTable Properties\n\n* 通过`connector.type`和`connector`区分使用`TableSourceFactory`还是`KuduDynamicTableSource`\n\n```properties\nkudu.table=指定映射的kudu表\nkudu.masters=指定的kudu master地址\nkudu.hash-columns=指定的表的hash分区键,多个使用\",\"分割\nkudu.replicas=kudu tablet副本数，默认为3\nkudu.hash-partition-nums=hash分区的桶个数，默认为2 * replicas\nkudu.range-partition-rule=range分区规则，rangeKey#leftValue,RightValue:rangeKey#leftValue1,RightValue1，rangeKey必须为主键\nkudu.primary-key-columns=kudu表主键，多个实用\",\"分割，主键定义必须有序\nkudu.lookup.cache.max-rows=kudu时态表缓存最大缓存行，默认为不开启\nkudu.lookup.cache.ttl=kudu时态表cache过期时间\nkudu.lookup.max-retries=时态表join时报错重试次数，默认为3\n```\n\n#### Flink1.10.x版本\n\n```java\nCREATE TABLE TestTableTableSourceFactory (\n  first STRING,\n  second STRING,\n  third INT NOT NULL\n) WITH (\n  'connector.type' = 'kudu',\n  'kudu.masters' = '...',\n  'kudu.table' = 'TestTable',\n  'kudu.hash-columns' = 'first',\n  'kudu.primary-key-columns' = 'first,second'\n)\n```\n\n#### Flink1.11.x版本\n\n```sql\nCREATE TABLE TestTableKuduDynamicTableSource (\n  first STRING,\n  second STRING,\n  third INT NOT NULL\n) WITH (\n  'connector' = 'kudu',\n  'kudu.masters' = '...',\n  'kudu.table' = 'TestTable',\n  'kudu.hash-columns' = 'first',\n  'kudu.primary-key-columns' = 'first,second'\n)\n```\n\n## DataStream使用\n\n* DataStream使用方式具体查看`bahir-flink`官方，目前对于数仓工程师使用场景偏少。\n\n# 版本迭代\n\n## 1.1版本Feature\n\n* 增加Hash分区bucket属性配置,通过`kudu.hash-partition-nums`配置\n* 增加Range分区规则,支持Hash和Range分区同时使用,通过参数`kudu.range-partition-rule`\n  配置,规则格式如:`range分区规则，rangeKey#leftValue,RightValue:rangeKey#leftValue1,RightValue1`\n* 增加Kudu时态表支持,通过`kudu.lookup.*`相关函数控制内存数据的大小和TTL\n\n```java\n /**\n     * lookup缓存最大行数\n     */\n  public static final String KUDU_LOOKUP_CACHE_MAX_ROWS = \"kudu.lookup.cache.max-rows\";\n    /**\n     * lookup缓存过期时间\n     */\n    public static final String KUDU_LOOKUP_CACHE_TTL = \"kudu.lookup.cache.ttl\";\n    /**\n     * kudu连接重试次数\n     */\n    public static final String KUDU_LOOKUP_MAX_RETRIES = \"kudu.lookup.max-retries\";\n```\n\n## 实现机制\n\n* 自定义`KuduLookupFunction`,使得KuduTableSource实现`LookupableTableSource`接口将自定义`LookupFunction`\n  返回已提供时态表的功能,底层缓存没有使用`Flink JDBC`的`Guava Cache`而是使用效率更高的`Caffeine Cache`使得其缓存效率更高,同时也减轻了因大量请求为Kudu带来的压力\n\n## 未来展望\n\n### 当前问题\n\n1. SQL语句主键无法自动推断\n\n\u003e 目前基于`Apache Bahir Kudu Connector`增强的功能主要是为了服务公司业务,在使用该版本的connector也遇到了问题,SQL的主键无法自动推断导致数据无法直接传递到下游,内部通过天宫引擎通过`Flink Table API`的`sqlQuery`方法将结果集查询为一个`Table`对象,然后将`Table`转换为`DataStream\u003cTuple2\u003cBoolean,Row\u003e\u003e`撤回流,最终通过`Kudu Connector`提供的`KuduSink`的`UpsertOperationMapper`对象将撤回流输出到`Kudu`中。\n\n### 后续计划\n\n* 计划提供动态数据源来解决这一问题,将`Flink 1.11.x`之前的`KuduTableSource/KuduTableSink`改造为`DynamicSource/Sink`接口实现`Source/Sink`,以此解决主键推断问题。\n\n## 1.2版本Feature\n\n* 改造支持`Flink 1.11.x`之后的`DynamicSource/Sink`，以此解决SQL语句主键无法推断问题，支持流批JOIN功能的SQL语句方式，无需在通过转换成DataStream的方式进行多表Join操作。\n* 内嵌Metrics上报机制，通过对`Flink`动态工厂入口处对操作的kudu表进行指标埋点，从而更加可视化的监控kudu表数据上报问题。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollabh%2Fflink-connector-kudu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcollabh%2Fflink-connector-kudu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollabh%2Fflink-connector-kudu/lists"}