Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/siddhi-io/siddhi-io-kafka
Extension that can be used to receive events from a Kafka cluster and to publish events to a Kafka cluster
https://github.com/siddhi-io/siddhi-io-kafka
extension io kafka message-queue siddhi
Last synced: 6 days ago
JSON representation
Extension that can be used to receive events from a Kafka cluster and to publish events to a Kafka cluster
- Host: GitHub
- URL: https://github.com/siddhi-io/siddhi-io-kafka
- Owner: siddhi-io
- License: apache-2.0
- Created: 2017-05-17T07:59:15.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2024-10-03T18:05:20.000Z (4 months ago)
- Last Synced: 2025-01-11T17:09:46.577Z (13 days ago)
- Topics: extension, io, kafka, message-queue, siddhi
- Language: Java
- Homepage: https://siddhi-io.github.io/siddhi-io-kafka/
- Size: 4.84 MB
- Stars: 18
- Watchers: 30
- Forks: 50
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Siddhi IO Kafka
======================================
[](https://wso2.org/jenkins/job/siddhi/job/siddhi-io-kafka/)
[](https://github.com/siddhi-io/siddhi-io-kafka/releases)
[](https://github.com/siddhi-io/siddhi-io-kafka/releases)
[](https://github.com/siddhi-io/siddhi-io-kafka/issues)
[](https://github.com/siddhi-io/siddhi-io-kafka/commits/master)
[](https://opensource.org/licenses/Apache-2.0)
The **siddhi-io-kafka extension** is an extension to Siddhi that receives and publishes events from and to Kafka.
For information on Siddhi and it's features refer Siddhi Documentation.
## Download
* Versions 5.x and above with group id `io.siddhi.extension.*` from here.
* Versions 4.x and lower with group id `org.wso2.extension.siddhi.*` from here.
## Latest API Docs
Latest API Docs is 5.0.18.
## Features
A Kafka sink publishes events processed by WSO2 SP to a topic with a partition for a Kafka cluster. The events can be published in the TEXT XML JSON or Binary format.
If the topic is not already created in the Kafka cluster, the Kafka sink creates the default partition for the given topic. The publishing topic and partition can be a dynamic value taken from the Siddhi event.
To configure a sink to use the Kafka transport, the type parameter should have kafka as its value.
* kafka-replay-request *(Sink)*
This sink is used to request replay of specific range of events on a specified partition of a topic.
* kafkaMultiDC *(Sink)*
A Kafka sink publishes events processed by WSO2 SP to a topic with a partition for a Kafka cluster. The events can be published in the TEXT XML JSON or Binary format.
If the topic is not already created in the Kafka cluster, the Kafka sink creates the default partition for the given topic. The publishing topic and partition can be a dynamic value taken from the Siddhi event.
To configure a sink to publish events via the Kafka transport, and using two Kafka brokers to publish events to the same topic, the type parameter must have kafkaMultiDC as its value.
* kafka *(Source)*
A Kafka source receives events to be processed by WSO2 SP from a topic with a partition for a Kafka cluster. The events received can be in the TEXT XML JSON or Binary format.
If the topic is not already created in the Kafka cluster, the Kafka sink creates the default partition for the given topic.
* kafka-replay-response *(Source)*
This source is used to listen to replayed events requested from kafka-replay-request sink
* kafkaMultiDC *(Source)*
The Kafka Multi-Datacenter(DC) source receives records from the same topic in brokers deployed in two different kafka clusters. It filters out all the duplicate messages and ensuresthat the events are received in the correct order using sequential numbering. It receives events in formats such as TEXT, XML JSON and Binary`.The Kafka Source creates the default partition '0' for a given topic, if the topic has not yet been created in the Kafka cluster.
## Installation
For installing this extension in the Streaming Integrator Server, and to add the dependent jars, refer Streaming Integrator documentation section on downloading and installing siddhi extensions.\
For installing this extension in the Streaming Integrator Tooling, and to add the dependent jars, refer Streaming Integrator documentation section on installing siddhi extensions.
## Dependencies
Following JARs will be converted to osgi and copied to `WSO2SI_HOME/lib` and `WSO2SI_HOME/samples/sample-clients/lib` which are in `/libs` directory.
- kafka_2.11-*.jar
- kafka-clients-*.jar
- metrics-core-*.jar
- scala-library-2.11.*.jar
- scala-parser-combinators_2.11.*.jar (if exists)
- zkclient-*.jar
- zookeeper-*.jar
#### Setup Kafka
As a prerequisite, you have to start the Kafka message broker. Please follow better steps.
1. Download the Kafka [distribution](https://kafka.apache.org/downloads)
2. Unzip the above distribution and go to the ‘bin’ directory
3. Start the zookeeper by executing below command,
```bash
zookeeper-server-start.sh config/zookeeper.properties
```
4. Start the Kafka broker by executing below command,
```bash
kafka-server-start.sh config/server.properties
```
Refer the Kafka documentation for more details, https://kafka.apache.org/quickstart
## Support and Contribution
* We encourage users to ask questions and get support via StackOverflow, make sure to add the `siddhi` tag to the issue for better response.
* If you find any issues related to the extension please report them on the issue tracker.
* For production support and other contribution related information refer Siddhi Community documentation.