{"id":13783402,"url":"https://github.com/streamnative/pulsar-io-kafka","last_synced_at":"2025-05-11T19:30:42.823Z","repository":{"id":35445063,"uuid":"198404008","full_name":"streamnative/pulsar-io-kafka","owner":"streamnative","description":"Pulsar IO Kafka Connector","archived":true,"fork":false,"pushed_at":"2023-03-17T16:24:40.000Z","size":205,"stargazers_count":24,"open_issues_count":7,"forks_count":12,"subscribers_count":30,"default_branch":"master","last_synced_at":"2024-11-17T20:47:08.026Z","etag":null,"topics":["apache-kafka","apache-pulsar","event-streaming","kafka","pubsub","pulsar","pulsar-io","schema","schema-registry"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamnative.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-07-23T10:02:59.000Z","updated_at":"2023-03-17T16:25:07.000Z","dependencies_parsed_at":"2024-01-18T15:51:47.525Z","dependency_job_id":"1a59659e-46c3-4f22-a311-26da5748315b","html_url":"https://github.com/streamnative/pulsar-io-kafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":"streamnative/pulsar-io-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-io-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-io-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-io-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-io-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamnative","download_url":"https://codeload.github.com/streamnative/pulsar-io-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253620939,"owners_count":21937445,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","apache-pulsar","event-streaming","kafka","pubsub","pulsar","pulsar-io","schema","schema-registry"],"created_at":"2024-08-03T19:00:20.697Z","updated_at":"2025-05-11T19:30:42.112Z","avatar_url":"https://github.com/streamnative.png","language":"Java","funding_links":[],"categories":["Connectors","Development","Message Queues"],"sub_categories":["Connectors","Apache Pulsar"],"readme":"# Archived repository\n\nThis repository has been archived. \n\nThe Pulsar IO Kafka Connector code can now be found at [apache/pulsar/pulsar-io/kafka](https://github.com/apache/pulsar/tree/master/pulsar-io/kafka). \n\nSource documentation: https://pulsar.apache.org/docs/io-kafka-source/\n\nSink documentation: https://pulsar.apache.org/docs/io-kafka-sink/\n\n---\n\n## Pulsar IO Kafka Connector\n\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fstreamnative%2Fpulsar-io-kafka.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fstreamnative%2Fpulsar-io-kafka?ref=badge_shield)\n\n\npulsar-io-kafka is a [Pulsar IO Connector](http://pulsar.apache.org/docs/en/io-overview/) for copying data between\nKafka and Pulsar.\n\n\u003e NOTE: This connector is an enhanced version of `pulsar-io-kafka` connector to support schema.\n\n### Get started\n\nThis provides a step-by-step example how to use this Kafka source connector to copy *json* data from a Kafka topic\nto a Pulsar topic and save the data in *AVRO* format.\n\n#### Build pulsar-io-kafka connector\n\n1. Git clone `pulsar-io-kafka`. Assume *PULSAR_IO_KAFKA_HOME* is the home directory for your\n   cloned `pulsar-io-kafka` repo for the remaining steps.\n   ```\n   $ git clone https://github.com/streamnative/pulsar-io-kafka\n   ```\n\n2. Build the connector in `${PULSAR_IO_KAFKA_HOME}` directory.\n   ```\n   mvn clean install -DskipTests\n   ```\n   After successfully built the connector, a *NAR* package is generated under *target* directory. The *NAR* package\n   is the one you used for \n   ```\n   $ ls target/pulsar-io-kafka-2.5.0-SNAPSHOT.nar\n   target/pulsar-io-kafka-2.5.0-SNAPSHOT.nar\n   ```\n\n#### Prepare a config for running pulsar-io-kafka connector\n\nAn example yaml config is available [here](https://github.com/streamnative/pulsar-io-kafka/blob/master/conf/pulsar-io-kafka.yaml)\n\nThis example yaml config is used for copying json data from Kafka topic *input-topic* to Pulsar topic *output-topic* and\nsave the messages in AVRO format.\n\n#### Run pulsar-io-kafka connector\n\n1. Download Pulsar 2.4.0 release from [Pulsar website](http://pulsar.apache.org/en/download/) and follow\n   the [instructions](http://pulsar.apache.org/docs/en/standalone/) to start a standalone Pulsar.\n   Assume *PULSAR_HOME* is the home directory for your Pulsar installation for the remaining steps.\n\n   Example command to start a standalone Pulsar.\n   ```\n   cd ${PULSAR_HOME}\n   bin/pulsar standalone\n   ```\n\n2. Download Kafka release from [Kafka website](http://kafka.apache.org/downloads) and follow\n   the [instructions](http://kafka.apache.org/quickstart) to start a Kafka server.\n   Assume *KAFKA_HOME* is the home directory for your Kafka installation for the remaining steps.\n\n   Example commands to start a Kafka server.\n   ```\n   cd ${KAFKA_HOME}\n   bin/zookeeper-server-start.sh config/zookeeper.properties\n   bin/kafka-server-start.sh config/server.properties\n   ```\n\n3. Create a Kafka topic.\n   \n   \u003e Make sure the Kafka topic name is the one you configured in `pulsar-io-kafka.yaml`.\n\n   ```\n   bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --create --topic input-topic --replication-factor 1 --partitions 4\n   ```\n\n4. Copy the pulsar-io-kafka conenctor to `${PULSAR_HOME}/connectors` directory.\n\n   ```\n   cd ${PULSAR_HOME}\n   mkdir -p connectors\n   cp ${PULSAR_IO_KAFKA_HOME}/target/pulsar-io-kafka-2.5.0-SNAPSHOT.nar connectors/\n   ```\n\n5. Localrun the pulsar-io-kafka connector.\n\n   \u003e NOTE: `--destination-topic-name` is used by pulsar io runtime but not by this `pulsar-io-kafka` connector. We can not omit this \n   \u003e       setting at this momement. So you can given any *unused* topic name for now.\n\n   ```\n   cd ${PULSAR_HOME}\n   bin/pulsar-admin sources localrun -a connectors/pulsar-io-kafka-2.5.0-SNAPSHOT.nar --tenant public --namespace default --name test-kafka-source --source-config-file ${PULSAR_IO_KAFKA_HOME}/conf/pulsar-io-kafka.yaml --destination-topic-name test-kafka-source-topic\n   ```\n   Once the connector is running, keep this terminal open during the remaining steps.\n\n\n6. Now we can use a *json* kafka producer and an *avro* pulsar consumer to verify if the connector is working as expected.\n\n   Start a *json* Kafka producer to produce 100 messages.\n   ```\n   cd ${PULSAR_IO_KAFKA_HOME}\n   bin/kafka-json-producer.sh localhost:9092 input-topic 100\n   ```\n\n   Start an *avro* Pulsar consumer to consume the 100 messages (in avro format).\n   ```\n   cd ${PULSAR_IO_KAFKA_HOME}\n   bin/pulsar-avro-consumer.sh pulsar://localhost:6650 output-topic sub\n   ```\n\n   You will see similar output in the terminal you run `pulsar-avro-consumer.sh`:\n   ```\n   Receive message : key = user-99, value = User(name=user-99, age=990, address=address-99)\n   ```\n\n\n## License\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fstreamnative%2Fpulsar-io-kafka.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fstreamnative%2Fpulsar-io-kafka?ref=badge_large)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fpulsar-io-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamnative%2Fpulsar-io-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fpulsar-io-kafka/lists"}