Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/streamnative/kop
Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar
https://github.com/streamnative/kop
apache-kafka apache-pulsar dotnet event-streaming kafka kafka-on-pulsar kop pub-sub pubsub pulsar
Last synced: 4 days ago
JSON representation
Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar
- Host: GitHub
- URL: https://github.com/streamnative/kop
- Owner: streamnative
- License: apache-2.0
- Archived: true
- Created: 2019-07-10T07:55:16.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2024-01-24T08:00:07.000Z (9 months ago)
- Last Synced: 2024-08-01T18:28:37.645Z (3 months ago)
- Topics: apache-kafka, apache-pulsar, dotnet, event-streaming, kafka, kafka-on-pulsar, kop, pub-sub, pubsub, pulsar
- Language: Java
- Homepage: https://streamnative.io/docs/kop
- Size: 7.44 MB
- Stars: 447
- Watchers: 44
- Forks: 132
- Open Issues: 176
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
- Security: docs/security.md
Awesome Lists containing this project
README
# Kafka-on-Pulsar (KoP)
> **Note**:
>
> KoP is now archived. It's recommended to try KSN (Kafka on StreamNative), see https://docs.streamnative.io/docs/kafka-on-cloud
KoP (Kafka on Pulsar) brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the KoP protocol handler to your existing Pulsar cluster, you can migrate your existing Kafka applications and services to Pulsar without modifying the code. This enables Kafka applications to leverage Pulsar’s powerful features, such as:
- Streamlined operations with enterprise-grade multi-tenancy
- Simplified operations with a rebalance-free architecture
- Infinite event stream retention with Apache BookKeeper and tiered storage
- Serverless event processing with Pulsar Functions
KoP, implemented as a Pulsar [protocol handler](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/protocol/ProtocolHandler.java) plugin with the protocol name "kafka", is loaded when Pulsar broker starts. It helps reduce the barriers for people adopting Pulsar to achieve their business success by providing a native Kafka protocol support on Apache Pulsar. By integrating the two popular event streaming ecosystems, KoP unlocks new use cases. You can leverage advantages from each ecosystem and build a truly unified event streaming platform with Apache Pulsar to accelerate the development of real-time applications and services.
KoP implements the Kafka wire protocol on Pulsar by leveraging the existing components (such as topic discovery, the distributed log library - ManagedLedger, cursors and so on) that Pulsar already has.
The following figure illustrates how the Kafka-on-Pulsar protocol handler is implemented within Pulsar.
## Version compatibility
The version of KoP `x.y.z.m` conforms to Pulsar `x.y.z`, while `m` is the patch version number. KoP might also be compatible with older patched versions, but it's not guaranteed. See [upgrade.md](./docs/upgrade.md) for details.
KoP is compatible with Kafka clients 0.9 or higher. For Kafka clients 3.2.0 or higher, you have to add the following configurations in KoP because of [KIP-679](https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default).
```properties
kafkaTransactionCoordinatorEnabled=true
brokerDeduplicationEnabled=true
```
## How to use KoP
You can configure and manage KoP based on your requirements. Check the following guides for more details.
- [Quick Start](docs/kop.md)
- [Configure KoP](docs/configuration.md)
- [Monitor KoP](docs/reference-metrics.md)
- [Upgrade](docs/upgrade.md)
- [Secure KoP](docs/security.md)
- [Schema Registry](docs/schema.md)
- [Implementation: How to converse Pulsar and Kafka](docs/implementation.md)
## Project Maintainers
- [@aloyszhang](https://github.com/aloyszhang)
- [@BewareMyPower](https://github.com/BewareMyPower)
- [@Demogorgon314](https://github.com/Demogorgon314)
- [@dockerzhang](https://github.com/dockerzhang)
- [@hangc0276](https://github.com/hangc0276)
- [@jiazhai](https://github.com/jiazhai)
- [@PierreZ](https://github.com/PierreZ)
- [@wenbingshen](https://github.com/wenbingshen)
- [@wuzhanpeng](https://github.com/wuzhanpeng)
## License
This library is licensed under the terms of the [Apache License 2.0](LICENSE) and may include packages written by third parties which carry their own copyright notices and license terms.
## About StreamNative
Founded in 2019 by the original creators of Apache Pulsar, [StreamNative](https://streamnative.io/) is one of the leading contributors to the open-source Apache Pulsar project. We have helped engineering teams worldwide make the move to Pulsar with [StreamNative Cloud](https://streamnative.io/product), a fully managed service to help teams accelerate time-to-production.