{"id":18554489,"url":"https://github.com/oracle/coherence-kafka","last_synced_at":"2025-04-09T23:31:31.840Z","repository":{"id":45315237,"uuid":"421244292","full_name":"oracle/coherence-kafka","owner":"oracle","description":"Coherence Kafka Integration","archived":false,"fork":false,"pushed_at":"2024-11-27T23:46:38.000Z","size":153,"stargazers_count":5,"open_issues_count":2,"forks_count":3,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-03-24T13:37:53.768Z","etag":null,"topics":["coherence","kafka"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"upl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oracle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-26T01:51:07.000Z","updated_at":"2024-11-27T23:46:36.000Z","dependencies_parsed_at":"2024-11-06T21:24:54.313Z","dependency_job_id":"e584b2f4-7d58-4c4e-a89e-376d506b74ab","html_url":"https://github.com/oracle/coherence-kafka","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Fcoherence-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Fcoherence-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Fcoherence-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle%2Fcoherence-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oracle","download_url":"https://codeload.github.com/oracle/coherence-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248129739,"owners_count":21052632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coherence","kafka"],"created_at":"2024-11-06T21:22:18.264Z","updated_at":"2025-04-09T23:31:26.830Z","avatar_url":"https://github.com/oracle.png","language":"Java","funding_links":[],"categories":["Enterprise Systems"],"sub_categories":["Oracle Coherence"],"readme":"# Coherence integration with Apache Kafka\n\nThis project provides component libraries that support moving data from a Coherence map to a Kafka topic and vice versa.\n\nSupported on Coherence CE 21.06 and later.\n\n## Getting Started\n\nThe Coherence Kafka integration project publishes a set of jar dependencies that you use in your application.\nThere is nothing to install, you use the Maven modules just as you would any other Java dependencies.\n\n## The following components are provided\n\n- **Kafka Entry Store** - a CacheStore type of extension that copies data from a Coherence map to a Kafka topic upon modification. This is contained in the `coherence-kafka-core` project.\n- **Kafka Sink Connector** - an Apache Kafka Connect plugin of type \"sink\", which copies data posted to a Kafka topic to a Coherence map. This is contained in the `coherence-kafka-sink` project\n\nSerializers and deserializers can be configured to handle data to and from Kafka, or an optimized pass-through mode will handle serialized entries which are already in a Coherence binary form.\n\n## Kafka Entry Store\n\nWhen configuring a cachestore-scheme as follows, Coherence will send entries to the specified Kafka broker:\n\n\u003cdetails\u003e\n\u003csummary\u003eClick to see config\u003c/summary\u003e\n\u003cp\u003e\n\n```xml\n\u003c?xml version=\"1.0\"?\u003e\n\n\u003ccache-config xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n              xmlns=\"http://xmlns.oracle.com/coherence/coherence-cache-config\"\n              xmlns:kafka=\"class://com.oracle.coherence.kafka.KafkaNamespaceHandler\"\n              xsi:schemaLocation=\"http://xmlns.oracle.com/coherence/coherence-cache-config coherence-cache-config.xsd\"\u003e\n  \u003ccaching-scheme-mapping\u003e\n    \u003ccache-mapping\u003e\n      \u003ccache-name\u003efoo\u003c/cache-name\u003e\n      \u003cscheme-name\u003epartitioned-rwbm-kafka\u003c/scheme-name\u003e\n    \u003c/cache-mapping\u003e\n  \u003c/caching-scheme-mapping\u003e\n\n  \u003ccaching-schemes\u003e\n    \u003cdistributed-scheme\u003e\n      \u003cscheme-name\u003epartitioned-rwbm-kafka\u003c/scheme-name\u003e\n\n      \u003cbacking-map-scheme\u003e\n        \u003cpartitioned\u003etrue\u003c/partitioned\u003e\n        \u003cread-write-backing-map-scheme\u003e\n          \u003cinternal-cache-scheme\u003e\n            \u003clocal-scheme\u003e\n            \u003c/local-scheme\u003e\n          \u003c/internal-cache-scheme\u003e\n          \u003ccachestore-scheme\u003e\n            \u003cclass-scheme\u003e\n              \u003ckafka:producer\u003e\n                \u003c!-- use default class --\u003e\n                \u003ckafka:topic-name\u003e{cache-name}\u003c/kafka:topic-name\u003e\n                \u003c!-- default for passthrough is false --\u003e\n                \u003ckafka:bootstrap-servers\u003elocalhost:9092\u003c/kafka:bootstrap-servers\u003e\n                \u003ckafka:key-serializer\u003eorg.apache.kafka.common.serialization.StringSerializer\u003c/kafka:key-serializer\u003e\n                \u003ckafka:value-serializer\u003eorg.apache.kafka.common.serialization.StringSerializer\u003c/kafka:value-serializer\u003e\n                \u003ckafka:max-block-ms\u003e5000\u003c/kafka:max-block-ms\u003e\n              \u003c/kafka:producer\u003e\n            \u003c/class-scheme\u003e\n          \u003c/cachestore-scheme\u003e\n        \u003c/read-write-backing-map-scheme\u003e\n      \u003c/backing-map-scheme\u003e\n\n      \u003cautostart\u003etrue\u003c/autostart\u003e\n    \u003c/distributed-scheme\u003e\n\n  \u003c/caching-schemes\u003e\n\u003c/cache-config\u003e\n```\n\n\u003c/p\u003e\n\u003c/details\u003e\n\nThe `kafka:producer` portion takes properties that are intended for a Kafka producer and uses them (the dash is turned into a dot, for example `bootstrap-servers` to `bootstrap.servers`)\n\n### Supported operations are:\n- `load()` and `loadAll()`: not supported; see Kafka Sink Connector for this\n- `store()`: Corresponds to a `put()` operation on a `NamedMap`, and performs a `producer.send()` on the specified Kafka topic\n- `storeAll()`: Corresponds to a `putAll()` operation on a `NamedMap`, where each element in the map will be sent to the specified Kafka topic\n- `erase()`: Corresponds to a `remove()` operation on a `NamedMap`; when a topic is configured with a `compact` policy, all records with the same key will be deleted (aka `tombstone`)\n- `eraseAll()`: Corresponds to a `removeAll()` operation on a `NamedMap`, and is equivalent to calling `erase()` on each element of the passed map\n\nFor more details see the `coherence-kafka-core` project.\n\n## Kafka Sink Connector\n\nThe sink connector is built as a zip file intended for being deployed in a Connect process' environment.\n\n### Supported operations are:\n- Coherence Entry creation/update: records published on a topic are forwarded to a Coherence NamedMap and corresponding entries are created\n- Coherence Entry deletion: when Kafka records have a null value, the corresponding entry in Coherence is removed if it existed\n\n**Note:** Kafka records with a null key cannot be represented in Coherence and are discarded with a logged message such as:\n\n```text\nconnect\u003e [2021-10-20 19:21:35,680] ERROR The key is null for record: SinkRecord{kafkaOffset=1, timestampType=CreateTime} ConnectRecord{topic='MyTopic', kafkaPartition=0, key=null, keySchema=Schema{STRING}, value=User(NoKey, 20), valueSchema=Schema{STRUCT}, timestamp=1634757687699, headers=ConnectHeaders(headers=)} (com.oracle.coherence.kafka.connect.sink.CoherenceSinkTask)\n```\n\nConfiguration is done using a curl command such as the following (Confluent Kafka)\n\n\u003cdetails\u003e\u003csummary\u003eClick to see command\u003c/summary\u003e\n\u003cp\u003e\n\n```\ncurl -X POST -H \"Content-Type: application/json\" \\\n    http://localhost:8083/connectors \\\n    -d '{\"name\":\"coh-sink\",\n             \"config\":\n                 {\n                     \"connector.class\":\"com.oracle.coherence.kafka.connect.CoherenceSinkConnector\",\n                     \"topics\":\"MyTopic\",\n                     \"coherence.cache.mappings\":\"MyTopic-\u003eMyCache\"\n                 }\n         }'\n```\n\n\u003c/p\u003e\n\u003c/details\u003e\n\nor a properties file for Kafka OSS\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\u003cp\u003e\n\n```properties\nname=coh-sink\ntopics=MyTopic\ntasks.max=1\nconnector.class=com.oracle.coherence.kafka.connect.CoherenceSinkConnector\ncoherence.cache.mappings=MyTopic-\u003eMyCache\nkey.converter=com.oracle.coherence.kafka.connect.util.CustomConverter\nvalue.converter=com.oracle.coherence.kafka.connect.util.CustomConverter\nvalue.converter.serializer=org.apache.kafka.common.serialization.StringSerializer\nvalue.converter.deserializer=org.apache.kafka.common.serialization.StringDeserializer\nkey.converter.serializer=org.apache.kafka.common.serialization.StringSerializer\nkey.converter.deserializer=org.apache.kafka.common.serialization.StringDeserializer\n```\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n## Contributing\n\nThis project welcomes contributions from the community. Before submitting a pull request, please [review our contribution guide](./CONTRIBUTING.md)\n\n## Security\n\nPlease consult the [security guide](./SECURITY.md) for our responsible security vulnerability disclosure process\n\n## License\n\nCopyright (c) 2021, 2023 Oracle and/or its affiliates.\n\nReleased under the Universal Permissive License v1.0 as shown at\n\u003chttps://oss.oracle.com/licenses/upl/\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle%2Fcoherence-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foracle%2Fcoherence-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle%2Fcoherence-kafka/lists"}