{"id":20332281,"url":"https://github.com/wavemaker/kafka-connector","last_synced_at":"2025-08-21T07:14:51.217Z","repository":{"id":72173378,"uuid":"276607534","full_name":"wavemaker/kafka-connector","owner":"wavemaker","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-21T06:46:31.000Z","size":39,"stargazers_count":2,"open_issues_count":1,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-11T21:37:59.267Z","etag":null,"topics":["wavemaker-connector"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wavemaker.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-02T09:41:29.000Z","updated_at":"2024-10-21T06:45:46.000Z","dependencies_parsed_at":"2024-11-14T20:27:16.170Z","dependency_job_id":"97c58f0f-316b-4afe-a570-885dbb690b56","html_url":"https://github.com/wavemaker/kafka-connector","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/wavemaker/kafka-connector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wavemaker%2Fkafka-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wavemaker%2Fkafka-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wavemaker%2Fkafka-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wavemaker%2Fkafka-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wavemaker","download_url":"https://codeload.github.com/wavemaker/kafka-connector/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wavemaker%2Fkafka-connector/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271442056,"owners_count":24760353,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["wavemaker-connector"],"created_at":"2024-11-14T20:25:49.918Z","updated_at":"2025-08-21T07:14:51.176Z","avatar_url":"https://github.com/wavemaker.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Connector Introduction\n\nConnector is a Java based backend extension for WaveMaker applications. Connectors are built as Java modules \u0026 exposes java based SDK to interact with the connector implementation.\nEach connector is built for a specific purpose and can be integrated with one of the external services. Connectors are imported \u0026 used in the WaveMaker application. Each connector runs on its own container thereby providing the ability to have it’s own version of the third party dependencies.\n\n## Features of Connectors\n\n1. Connector is a java based extension which can be integrated with external services and reused in many Wavemaker applications.\n1. Each connector can work as an SDK for an external system.\n1. Connectors can be imported once in a WaveMaker application and used many times in the applications by creating multiple instances.\n1. Connectors are executed in its own container in the WaveMaker application, as a result there are no dependency version conflict issues between connectors.\n\n## About kafka-connector\n\nThis Connector exposes APIs to intracts with Apacahe Kafka Cluster which can be integrated into WaveMaker Application.\n\n## Kafka Introduction\nApache Kafka is a distributed publish-subscribe messaging system.Kafka is fast, scalable, durable, fault-tolerant and distributed messaging system by design.\nIt was originally developed by Linked later it became of part of Apache project.\n\n## Kafka Fundamentals\n\n### Topic\nMessages belongs to a particular category is known as topic.\n\n### Partition\nTopics are split into partitions, for each topic Kafka will keep at least one partition.\n\n### Broker\nA Kafka cluster consists of one or more servers (Kafka brokers) running Kafka. A Kafka server, a Kafka broker and a Kafka node all refer to the same concept in cluster.\n\n### Kafka Controller\nWithin a Kafka cluster, a single broker serves as the active controller which is responsible for state management of partitions and replicas. So in your case, if you have a cluster with 100 brokers, one of them will act as the controller.\n\n### Zoo Keeper\nZookeeper is a top-level software developed by Apache that acts as a centralized service and is used to maintain naming and configuration data.\nZookeeper keeps track of status of the Kafka cluster nodes and it also keeps track of Kafka topics, partitions etc.\n\n### Consumer\nConsumers read data from brokers. Consumers subscribes to one or more topics and consume published messages by pulling data from the brokers.\n\n### Consumer Group\nWhen multiple consumers are subscribed to a topic and belong to the same consumer group.\nEach Consumer group has unique Id and each group can register for one or more topics.\n\n## Build \nYou can build this connector using following command\n```\nmvn clean install \n```\n\n## Deploy \nYou can import connector dist/kafka.zip artifact in WaveMaker studio application using file upload option.\nOn after deploying Kafka-Connector in the WaveMaker studio application, make sure to update connector properties in the profile properties.Such as \nzookeeper IP, Kafka brokers IPs...etc\n\n## Using Connector in WaveMaker\nThis connector has three connector interfaces which are used for different operations in Kafka cluster.\n\n### TopicConnector\nThis connector Interface apis are used to manages topics in the Kafka cluster.Such as creating, Listing, Deleting, Updating Topics in the cluster.\n```\n@Autowired\nprivate KafkaTopicConnector kafkaTopicConnector;\n\nString topicName = \"demoTopic\";\nkafkaTopicConnector.createTopic(topicName, 2, 1, new Properties());\n```\n### ProducerConnector\nThis connector interface apis are used to send massages to specific topic in the kafka cluster\n```\n@Autowired\nprivate KafkaProducerConnector kafkaProducerConnector;\n\nKafkaProducer\u003cInteger, String\u003e kafkaProducer = kafkaProducerConnector.getKafkaProducer(topicName, Integer.class, String.class);\nkafkaProducer.send(1,\"Hello, Welcome to kafka connector\");\n```\n### ConsumerConnector\nThis connector interface apis are used to consume message from a specific topic using consumer groups in the kafka cluster.\n```\n@Autowired\nprivate KafkaConsumerConnector kafkaConsumerConnector;\n\nKafkaConsumer\u003cString, String\u003e kafkaConsumer = kafkaConsumerConnector.getKafkaConsumer(\"group1\", 2, Integer.class, String.class);\nkafkaConsumer.start(topicName, new KafkaConsumerCallback\u003cInteger, String\u003e() {\n   @Override\n   public void onMessage(ConsumerRecord\u003cInteger, String\u003e record, Acknowledgment acknowledgment) {\n      logger.info(\"Consumer receiving messages : \" + record.key() + \":\" + record.value() + \" in partition [ \" + record.partition() + \" ]\" + \" with offset no [ \" + record.offset() + \" ]\");\n   }\n});\n```\nApart from above apis, there are other apis in each connector interface, please visit connector interfaces in the kafka-api module.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwavemaker%2Fkafka-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwavemaker%2Fkafka-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwavemaker%2Fkafka-connector/lists"}