https://github.com/xiyuan-fengyu/flumenettysourceandsink
https://github.com/xiyuan-fengyu/flumenettysourceandsink
flume flume-sink-plugin flume-source-plugin netty
Last synced: 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/xiyuan-fengyu/flumenettysourceandsink
- Owner: xiyuan-fengyu
- Created: 2017-12-13T09:19:55.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2017-12-26T03:56:32.000Z (over 7 years ago)
- Last Synced: 2024-12-29T00:53:51.610Z (4 months ago)
- Topics: flume, flume-sink-plugin, flume-source-plugin, netty
- Language: Java
- Size: 31.3 KB
- Stars: 1
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## FlumeNettySourceAndSink
[TOC]
### 简介
通过 com.xiyuan.flume.sink.NettyServerSink 和 com.xiyuan.flume.source.NettyClientSource， 将公网服务器的数据采集到内网服务器
### 使用
将 release/flume-netty-source-sink-1.0.0.jar 放到 $FLUME_HOME/lib 目录下
### 配置示例
公网服务器 flume 配置
```
# 定义agent名， source、channel、sink的名称
producer.sources = r1
producer.channels = c1
producer.sinks = k1
# 具体定义source
producer.sources.r1.type = TAILDIR
producer.sources.r1.positionFile = /usr/env/apache-flume-1.8.0/logs/taildir_position.json
producer.sources.r1.filegroups = f1
producer.sources.r1.filegroups.f1 = /user/local/nginx/logs/.*log
producer.sources.r1.fileHeader = true
producer.sources.r1.deserializer.maxLineLength = 10240
producer.sources.r1.channels = c1
# 具体定义channel
producer.channels.c1.type = memory
producer.channels.c1.capacity = 10000
producer.channels.c1.transactionCapacity = 128
#具体定义sink
producer.sinks.k1.type =com.xiyuan.flume.sink.NettyServerSink
#producer.sinks.k1.transactionTime=5000
producer.sinks.k1.host = 0.0.0.0
producer.sinks.k1.port = 9090
#producer.sinks.k1.users = user_default,user_test
producer.sinks.k1.batchSize = 32
producer.sinks.k1.channel = c1
```
内网服务器 flume 配置
```
# 定义agent名， source、channel、sink的名称
consumer.sources = r1
consumer.channels = c1
consumer.sinks = k1
# 具体定义source
consumer.sources.r1.type =com.xiyuan.flume.source.NettyClientSource
consumer.sources.r1.host = 公网服务器IP
consumer.sources.r1.port = 9090
#consumer.sources.r1.user = user_default
consumer.sources.r1.deserializer.maxLineLength = 10240
consumer.sources.r1.channels = c1
# 具体定义channel
consumer.channels.c1.type = memory
consumer.channels.c1.capacity = 10000
consumer.channels.c1.transactionCapacity = 1000
#具体定义sink
consumer.sinks.k1.type = logger
#consumer.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
#consumer.sinks.k1.hostname = 0.0.0.0
#consumer.sinks.k1.port = 9090
consumer.sinks.k1.channel = c1
```