https://github.com/ft20082/flink-connector-redis

A simple redis sql connector for Flink
https://github.com/ft20082/flink-connector-redis

Last synced: 7 days ago
JSON representation

A simple redis sql connector for Flink

• Host: GitHub
• URL: https://github.com/ft20082/flink-connector-redis
• Owner: ft20082
• Created: 2021-11-18T01:57:14.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2021-11-18T02:19:14.000Z (almost 3 years ago)
• Last Synced: 2024-08-02T14:07:41.849Z (3 months ago)
• Language: Java
• Size: 30.3 KB
• Stars: 5
• Watchers: 1
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# flink-connector-redis

flink connector redis for flink sql

maven dependency

```xml

    org.apache.flink

    flink-connector-redis

    1.14.0-SNAPSHOT

```

# how to use

```sql

CREATE TABLE redis_table (

    `key` STRING,

    `map_key_sid` INT,

    `map_key_install_date` STRING,

    PRIMARY KEY (`key`) NOT ENFORCED 

) WITH (

    'connector' = 'redis',

    'host' = '127.0.0.1'

);

-- insert 

INSERT INTO  redis_table

SELECT `key`, `map_key_sid`, `map_key_install_date` FROM T;

-- query lookup join

INSERT INTO console_output

SELECT

    s.ouid,

    dim.map_key_sid,

    dim.map_key_install_date

FROM kafka_source as s

left join redis_table FOR SYSTEM_TIME AS OF s.proctime AS dim

on s.ouid = dim.key

where s.ouid is not null

```

# connector options

| 字段 | Required | 默认值  | 类型 | 说明 |

| ---- | ---- | ---- | ---- | ---- |

| connector | required  | (none) | String | `redis` |

| host | required  | (none) | String | Redis IP |

| port | optional | 6379 | Integer | Redis 端口 |

| password | optional | null | String | 如果没有设置，则为 null  |

| db | optional | 0 | Integer | 默认使用 db0 |

| pool-size | optional | 10 | Integer | 连接池大小 |

| timeout | optional | 3000  | Integer | 连接超时时间，单位 ms，默认 1s |

| test-on-borrow | optional | false | Boolean | 测试连接是否有效 |

| mode | optional | hash | String | redis 操作模式，使用 `hash` 或 `string` |

| sink.parallelism | optional | (none) | Integer | 默认并行度 |

| sink.key-ttl | optional | 0s | Integer | 设置的 key 的超时时间，单位秒 |

| sink.max-retry | optional | 3 | Integer | 设置写入的重试次数 |

| lookup.cache.max-rows | optional | 10000 | Integer | lookup 缓存大小 |

| lookup.cache.ttl | optional | 0 | Integer | 缓存过期时间 |

| lookup.max-retries | optional | 3 | Integer | lookup 失败重试次数 |

> mode 对应的存储模式，如果是 `hash` 则按照字段存储，如果是 `string` 则使用 k1=v1,k2=v3,... 存储

# Data Type Mapping

| Flink Sql Type | Redis conversion |

| ---- | ---- |

| CHAR | Store as string |

| VARCHAR | Store as string |

| STRING | Store as string |

| BOOLEAN | String toString(boolean val) 
 boolean toBoolean(String str) |

| BINARY | String toString(byte[] val) 
 byte[] toBytes(String str), Store as Base64 encoder string |

| VARBINARY | String toString(byte[] val) 
 byte[] toBytes(String str), Store as Base64 encoder string |

| DECIMAL | String toString(BigDecimal val) 
 BigDecimal toDecimal(String str) |

| TINYINT | String toString(byte val) 
 byte toByte(String str), Store as Base64 encoder string |

| SMALLINT | String toString(short val) 
 short toShort(String str) |

| INT | String toString(int val) 
 int toInt(String str) |

| BIGINT | String toString(long val) 
 long toLong(String str) |

| FLOAT | String toString(float val) 
 float toFloat(String str) |

| DOUBLE | String toString(double val) 
 double toDouble(String str) |

| DATE | Store the string type: '2021-10-10' |

| TIME | Store the string type: '10:10' |

| TIMESTAMP | Store the milliseconds since epoch as long value |

| ARRAY | Not supported  |

| MAP | Not supported |

| MULTISET | Not supported |

| ROW | Not supported |

# bug reporter

issue | [email protected]