{"id":13811509,"url":"https://github.com/streamnative/pulsar-flume-ng-sink","last_synced_at":"2025-07-24T22:32:30.331Z","repository":{"id":51270851,"uuid":"172894114","full_name":"streamnative/pulsar-flume-ng-sink","owner":"streamnative","description":"An Apache Flume Sink implementation to publish data to Apache pulsar","archived":false,"fork":false,"pushed_at":"2022-10-05T03:19:01.000Z","size":34,"stargazers_count":20,"open_issues_count":3,"forks_count":14,"subscribers_count":25,"default_branch":"master","last_synced_at":"2024-11-19T06:53:05.206Z","etag":null,"topics":["apache-flume","apache-pulsar","logcollector","messaging","pubsub"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamnative.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2019-02-27T10:27:27.000Z","updated_at":"2024-06-14T01:25:13.000Z","dependencies_parsed_at":"2023-01-19T06:15:26.800Z","dependency_job_id":null,"html_url":"https://github.com/streamnative/pulsar-flume-ng-sink","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-flume-ng-sink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-flume-ng-sink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-flume-ng-sink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-flume-ng-sink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamnative","download_url":"https://codeload.github.com/streamnative/pulsar-flume-ng-sink/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227482464,"owners_count":17779968,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-flume","apache-pulsar","logcollector","messaging","pubsub"],"created_at":"2024-08-04T04:00:22.473Z","updated_at":"2024-12-01T04:12:49.026Z","avatar_url":"https://github.com/streamnative.png","language":"Java","funding_links":[],"categories":["Logging"],"sub_categories":[],"readme":"# Flume Ng Pulsar Sink\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)\n[![Build Status](https://travis-ci.org/streamnative/flume-ng-pulsar-sink.svg?branch=master)](https://travis-ci.org/streamnative/flume-ng-pulsar-sink)\n\n\nThis is a [Flume](https://github.com/apache/flume) Sink implementation that can publish data to a [Pulsar](https://github.com/apache/pulsar) topic\n\n## Compatibility\n\nThis sink is developed and tested using Apache Flume NG 1.9.0 and Apache Pulsar Client 2.3.0.\n\n### Requirements\n\n- [Docker](https://docs.docker.com/docker-for-mac/install/)\n\n### Clone the project\n\n```bash\n$ git clone https://github.com/streamnative/flume-ng-pulsar-sink.git\n```\n\n### Start Pulsar Standalone\n\n```$xslt\ndocker pull apachepulsar/pulsar:2.3.0\ndocker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-flume-standalone apachepulsar/pulsar:2.3.0 bin/pulsar standalone\n```\n\n### Start Pulsar Consumer\n\nStart a [consumer](src/test/python/pulsar-flume.py) to consume messages from topic `flume-test-topic`.\n\n```$xslt\ndocker cp src/test/python/pulsar-flume.py pulsar-flume-standalone:/pulsar\ndocker exec -it pulsar-flume-standalone /bin/bash\npython pulsar-flume.py\n```\n\n### Setup up Flume\n\n#### Prepare Build Environment\n\nOpen a new terminal to start a docker instance `flume` of `maven:3.6-jdk-8` in the same network as `pulsar-flume-standalone`\nwe started at previous step. We will use this `flume` docker instace to install Flume and Flume-Ng-Pulsar-Sink.\n\n```$xslt\ndocker pull maven:3.6-jdk-8\ndocker run -d -it --link pulsar-flume-standalone -p 44445:44445 --name flume maven:3.6-jdk-8 /bin/bash\n```\n\n#### Install Flume\n\nGo to the docker instance `flume`\n\n```$xslt\ndocker exec -it flume /bin/bash\n```\n\nAt `flume` instance:\n\n```\nwget http://apache.01link.hk/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz\ntar -zxvf apache-flume-1.9.0-bin.tar.gz\n```\n\n#### Install Pulsar Sink\n\nAt `flume` instance:\n\n```$xslt\ngit clone https://github.com/streamnative/flume-ng-pulsar-sink\ncd flume-ng-pulsar-sink\nmvn clean package\ncd ..\ncp flume-ng-pulsar-sink/target/flume-ng-pulsar-sink-1.9.0.jar apache-flume-1.9.0-bin/lib/\nexit\n```\n\n#### Configure Flume\n\nCopy the example configurations to `flume`:\n\n- [flume-example.conf](src/test/resources/flume-example.conf)\n- [flume-env.sh](src/test/resources/flume-env.sh)\n\n```$xslt\ndocker cp src/test/resources/flume-example.conf flume:/apache-flume-1.9.0-bin/conf/\ndocker cp src/test/resources/flume-env.sh flume:/apache-flume-1.9.0-bin/conf/\n```\n\n#### Start Flume Ng Agent\n\n```$xslt\ndocker exec -it flume /bin/bash\n```\n\nAt `flume` instance:\n\n```$xslt\napache-flume-1.9.0-bin/bin/flume-ng agent --conf apache-flume-1.9.0-bin/conf/ -f apache-flume-1.9.0-bin/conf/flume-example.conf -n a1\n```\n\n### Send Data\n\nOpen another terminal, send data to port 44445 of flume\n\n```$xslt\n➜  ~ telnet localhost 44445\nTrying ::1...\nConnected to localhost.\nEscape character is '^]'.\nhello\nOK\nworld\nOK\n```\n\nAt the terminal running `pulsar-consumer.py`, you will see following output:\n\n```$xslt\n'eceived message: 'hello\n'eceived message: 'world\n``` \n\n### Cleanup \n\n`flume` and `pulsar-flume-standalone` are running at background. Please remember to kill them at the end of this tutorial.\n\n```bash\n$ docker ps | grep pulsar-flume-standalone | awk '{ print $1 }' | xargs docker kill\n$ docker ps | grep flume | awk '{ print $1 }' | xargs docker kill\n```\n\n## Installation\n\n### Requirements\n\n- JDK 1.8+\n- Apache Maven 3.x\n\n### Build from Source\n\nClone the project from Github:\n\n```bash\n$ git clone https://github.com/streamnative/flume-ng-pulsar-sink.git\n```\n\nBuilding the Flume Ng Sink using maven:\n\n```bash\n$ cd flume-ng-pulsar-sink\n$ mvn clean package\n```\n\nOnce it is built successfully, you will find a jar `flume-ng-pulsar-sink-\u003cversion\u003e.jar` generated under `target` directory.\nYou can drop the built jar at your flume installation under `lib` directory.\n\n## Usage\n\n### Configurations\n\n|Name|Description|Default|\n|---|---|---|\n|useAvroEventFormat|  Whether use avro format for event |false|\n|syncMode|  Mode of send data to pulsar |true|\n\n#### Client \n\n|Name|Description|Default|\n|---|---|---|\n|serviceUrl| Whether non-persistent topics are enabled on the broker |localhost:6650|\n|authPluginClassName| name of the Authentication-Plugin you want to use |\"\"|\n|authParamsString| string which represents parameters for the Authentication-Plugin, e.g., \"key1:val1,key2:val2\" |\"\"|\n|tlsCertFile| path of tls cert file |\"\"|\n|tlsKeyFile| path of tls key file |\"\"|\n|useTLS| Whether to turn on TLS, if to start, use protocol pulsar+ssl |false|\n|operationTimeout| Set the operation timeout (default: 30 seconds) |30s|\n|numIoThreads| Set the number of threads to be used for handling connections to brokers |1|\n|numListenerThreads| Set the number of threads to be used for message listeners |1|\n|connectionsPerBroker| Sets the max number of connection that the client library will open to a single broker. |1|\n|enableTcpNoDelay| Configure whether to use TCP no-delay flag on the connection, to disable Nagle algorithm. |false|\n|tlsTrustCertsFilePath| Set the path to the trusted TLS certificate file |false|\n|allowTlsInsecureConnection| Configure whether the Pulsar client accept untrusted TLS certificate from broker |false|\n|enableTlsHostnameVerification| whether to enable TLS hostname verification |false|\n|statsInterval| the interval between each stat info |60|\n|maxConcurrentLookupRequests| Number of concurrent lookup-requests allowed to send on each broker-connection to prevent overload on broker. |60|\n|maxLookupRequests| Number of max lookup-requests allowed on each broker-connection to prevent overload on broker. |60|\n|maxNumberOfRejectedRequestPerConnection| Set max number of broker-rejected requests in a certain time-frame (30 seconds) after which current connection will be closed and client creates a new connection that give chance to connect a different broker |50|\n|keepAliveIntervalSeconds| Set keep alive interval in seconds for each client-broker-connection. |30|\n|connectionTimeout| Set the duration of time to wait for a connection to a broker to be established. |30|\n\n#### Producer\n|Name|Description|Default|\n|---|---|---|\n|topicName| Specify the topic this producer will be publishing on. |\"\"|\n|producerName| Specify a name for the producer |\"\"|\n|sendTimeout| Set the send timeout |30s|\n|blockIfQueueFull| Set whether the send and sendAsync operations should block when the outgoing message queue is full. |false|\n|enableBatching| Control whether automatic batching of messages is enabled for the producer |true|\n|batchMessagesMaxMessagesPerBatch| maximum number of messages in a batch |1000|\n|batchDelay| the batch delay |1ms|\n|messageRoutingMode| the message routing mode, SinglePartition,RoundRobinPartition, CustomPartition(0,1,2) |1|\n|hashingSchema| JavaStringHash,Murmur3_32Hash(0,1) |0|\n|compressionType| NONE,LZ4,ZLIB,ZSTD(0,1,2,3) |0|\n\n# License\n\nThis project is licensed under the [Apache License 2.0](LICENSE).\n\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fstreamnative%2Fflume-ng-pulsar-sink.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fstreamnative%2Fflume-ng-pulsar-sink?ref=badge_large)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fpulsar-flume-ng-sink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamnative%2Fpulsar-flume-ng-sink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fpulsar-flume-ng-sink/lists"}