{"id":20049156,"url":"https://github.com/sanjuthomas/kafka-connect-marklogic","last_synced_at":"2025-05-05T11:30:44.286Z","repository":{"id":45702639,"uuid":"85435138","full_name":"sanjuthomas/kafka-connect-marklogic","owner":"sanjuthomas","description":"Kafka Sink Connect MarkLogic","archived":false,"fork":false,"pushed_at":"2024-06-30T23:58:18.000Z","size":9288,"stargazers_count":19,"open_issues_count":5,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-05-03T21:11:53.990Z","etag":null,"topics":["apache","java","kafka","kafka-connect","kafka-connector","marklogic","rest"],"latest_commit_sha":null,"homepage":"http://sanjuthomas.com","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sanjuthomas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-18T22:08:45.000Z","updated_at":"2025-03-31T12:14:34.000Z","dependencies_parsed_at":"2023-02-15T13:16:02.252Z","dependency_job_id":null,"html_url":"https://github.com/sanjuthomas/kafka-connect-marklogic","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanjuthomas%2Fkafka-connect-marklogic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanjuthomas%2Fkafka-connect-marklogic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanjuthomas%2Fkafka-connect-marklogic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanjuthomas%2Fkafka-connect-marklogic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sanjuthomas","download_url":"https://codeload.github.com/sanjuthomas/kafka-connect-marklogic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252488878,"owners_count":21756226,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache","java","kafka","kafka-connect","kafka-connector","marklogic","rest"],"created_at":"2024-11-13T11:48:46.750Z","updated_at":"2025-05-05T11:30:42.901Z","avatar_url":"https://github.com/sanjuthomas.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overview\n\nKafka Connect MarkLogic is a sink-only connector that pulls messages from Kafka and stores them in MarkLogic as JSON documents. Consider using the Kafka MarkLogic Sync Connector for the following use cases.\n\n1. Near real-time ingestion requirements.\n2. Regulate the traffic towards MarkLogic.\n3. Requirement to maintain the order of the messages. \n4. A Kafka ecosystem exists.\n5. Replay messages using an offset.\n\n## High-Level Architecture Diagram\n\n![Kafka Connect MarkLogic](KafkaConnectMarkLogic.png)\n\nThis project is about the component marked in green. Refer [here](http://confluent.io/product/connectors) to see the available list of source connectors.\n\n## Prerequisites\n\n[Apache ZooKeeper](https://zookeeper.apache.org) and [Apache Kafka](https://kafka.apache.org) are installed and running on your machine. Please refer to respective sites to download and start ZooKeeper and Kafka. \n\n## What is MarkLogic?\nThe MarkLogic is a multi-model schemaless NoSQL database to store, manage, and search JSON, XML, and RDF triples. For more details, please refer to MarkLogic's official [website](marklogic.com).\n\n## What is Apache Kafka?\nApache Kafka is an open-source stream processing platform written in Scala and Java by the Apache Software Foundation. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. For more details, please refer to the [kafka home page](https://kafka.apache.org/).\n\n## Implementation details \n\nTo send data to MarkLogic, this connector uses MarkLogic REST API. By default, the /v1/documents endpoint at port 8000 is used. You may change that in the marklogic-sink.properties file. You may use your own REST/Service extension instead of the out-of-the-box document API to transform anything on the way in.\n\nTo listen to multiple topics, please add the topic name in the marklogic-sink.properties file. The current implementation will take the topic name and use it as the MarkLogic document collection name.\n\n## Data Mapping\n\nMarkLogic is a schemaless document store/NoSQL database. Since we are working with plain JSON data, we don't need a schema to serialize and deserialize the messages. \n\n**For stand-alone mode**, please copy ```kafka_home/config/connect-standalone.properties``` to create ```kafka_home/config/marklogic-connect-standalone.properties``` file. Open ```kafka_home/config/marklogic-connect-standalone.properties``` and set the following properties to false.\n\n```\nkey.converter.schemas.enable=false\nvalue.converter.schemas.enable=false\n```\n\n**For distributed mode**, please copy ```kafka_home/config/connect-distributed.properties``` to create ```kafka_home/config/marklogic-connect-distributed.properties``` file. Open ```kafka_home/config/marklogic-connect-distributed.properties``` and set the following properties to false.\n\n```\nkey.converter.schemas.enable=false\nvalue.converter.schemas.enable=false\n```\n\nIn distributed mode, if you run more than one worker per host, the ```rest.port``` settings must have different values for each instance. By default REST interface is available at 8083.\n\n## How do you deploy the connector in Kafka?\n\nThis is a maven project. To create an [uber](https://maven.apache.org/plugins/maven-shade-plugin/index.html) jar, execute the following maven goals.\n\n```mvn clean compile package shade:shade install```\n\nCopy the artifact ```kafka-connect-marklogic-1.0.jar``` to kakfa_home/lib folder.\n\nCopy the [marklogic-sink.properties](https://github.com/sanjuthomas/kafka-connect-marklogic/blob/master/config/marklogic-sink.properties) file into kafka_home/config folder. Update the content of the property file according to your environment.\n\nAlternatively, you may keep the ```kafka-connect-marklogic-1.0.jar``` in another directory and export that directory into Kafka class path before starting the connector.\n\n## How to start the connector in stand-alone mode?\n\nOpen a shell prompt, move to kafka_home, and execute the following.\n\n```\nbin/connect-standalone.sh config/marklogic-connect-standalone.properties config/marklogic-sink.properties\n```\n\n## How to start the connector in distributed mode?\n\nOpen a shell prompt, move to kafka_home, and execute the following.\n\n```\nbin/connect-distributed.sh config/marklogic-connect-distributed.properties config/marklogic-sink.properties\n```\n\n## How to produce some messages?\n\nSome examples of message producers are available [here](https://github.com/sanjuthomas/marklogic-kafka-sample-client).\n\n## Contact\n\nCreate an issue in the GitHub.\n\n## Contribute\n\nPlease refer to [CONTRIBUTING.md](https://github.com/sanjuthomas/kafka-connect-marklogic/blob/master/CONTRIBUTING.md)\n\n## License\nThe project is licensed under the MIT license.\n\n## Looking for a connector that supports XML messages?\n\nDrop me a line - ml@sanju.org\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanjuthomas%2Fkafka-connect-marklogic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsanjuthomas%2Fkafka-connect-marklogic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanjuthomas%2Fkafka-connect-marklogic/lists"}