{"id":20908984,"url":"https://github.com/jaredpetersen/kafka-connect-arangodb","last_synced_at":"2025-05-13T06:34:08.083Z","repository":{"id":57733600,"uuid":"167766869","full_name":"jaredpetersen/kafka-connect-arangodb","owner":"jaredpetersen","description":"🥑 Kafka connect sink connector for ArangoDB","archived":false,"fork":false,"pushed_at":"2020-07-20T06:15:15.000Z","size":389,"stargazers_count":29,"open_issues_count":7,"forks_count":8,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-01T19:36:33.245Z","etag":null,"topics":["arango","arangodb","kafka","kafka-connect"],"latest_commit_sha":null,"homepage":"https://www.confluent.io/connector/kafka-connect-arangodb/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaredpetersen.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null},"funding":{"github":"jaredpetersen","custom":"https://paypal.me/jaredtpetersen"}},"created_at":"2019-01-27T03:51:19.000Z","updated_at":"2024-12-28T00:49:50.000Z","dependencies_parsed_at":"2022-09-26T22:11:28.237Z","dependency_job_id":null,"html_url":"https://github.com/jaredpetersen/kafka-connect-arangodb","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaredpetersen%2Fkafka-connect-arangodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaredpetersen%2Fkafka-connect-arangodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaredpetersen%2Fkafka-connect-arangodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaredpetersen%2Fkafka-connect-arangodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaredpetersen","download_url":"https://codeload.github.com/jaredpetersen/kafka-connect-arangodb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253889877,"owners_count":21979715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arango","arangodb","kafka","kafka-connect"],"created_at":"2024-11-18T14:09:46.870Z","updated_at":"2025-05-13T06:34:07.689Z","avatar_url":"https://github.com/jaredpetersen.png","language":"Java","funding_links":["https://github.com/sponsors/jaredpetersen","https://paypal.me/jaredtpetersen"],"categories":["Databases"],"sub_categories":["ArangoDB"],"readme":"# Kafka Connect ArangoDB Connector\n[![Build Status](https://github.com/jaredpetersen/kafka-connect-arangodb/workflows/Release/badge.svg)](https://github.com/jaredpetersen/kafka-connect-arangodb/actions)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/io.github.jaredpetersen/kafka-connect-arangodb/badge.svg)](https://maven-badges.herokuapp.com/maven-central/io.github.jaredpetersen/kafka-connect-arangodb)\n\nKafka Connect Sink Connector for ArangoDB\n\n## Usage\nKafka Connect ArangoDB is a Kafka Connector that translates record data into `REPSERT` and `DELETE` queries that are performed against ArangoDB. Only sinking data is supported at this time.\n\nRequires ArangoDB 3.4 or higher.\n\nA full example of how Kafka Connect ArangoDB can be integrated into a Kafka cluster is available in the [development documentation](/docs/development/).\n\n### Record Formats and Structures\nThe following record formats are supported:\n* Avro (Recommended)\n* JSON with Schema\n* Plain JSON\n\nWith each of these formats, the record value can be structured in one of the following ways:\n* Simple (Default)\n* Change Data Capture\n\n#### Simple\nThe Simple format is a slim record value structure that only provides the information necessary for writing to the database.\n\nWhen written as plain JSON, the record value looks something like:\n```json\n{\n  \"someKey\": \"changed value\",\n  \"otherKey\": true\n}\n```\n\nWhen the Kafka Connect ArangoDB Connector receives records adhering to this format, it will translate it into the following ArangoDB database changes and perform them:\n* Records with a `null` value are \"tombstone\" records and will result in the deletion of the document from the database\n* Records with a non-`null` value will be repserted (replace the full document if it exists already, insert the full document if it does not)\n\nThis format is the default, so no extra configuration is required.\n\n#### CDC\nThe CDC format is a record value structure that is designed to handle records produced by Change Data Capture systems like [Debezium](https://debezium.io/). Each record value should be an object with properties `before` and `after` that store the \"before\" and \"after\" state of the document, respectively.\n\nWhen written as plain JSON, the record value looks something like:\n```json\n{\n  \"before\": {\n    \"someKey\": \"some value\",\n    \"otherKey\": false\n  },\n  \"after\": {\n    \"someKey\": \"changed value\",\n    \"otherKey\": true\n  }\n}\n```\n\nWhen the Kafka Connect ArangoDB Connector receives this data, it will translate it into the following ArangoDB database changes and perform them:\n* Records with a `null` value are \"tombstone\" records and are are ignored\n* Record with a non-`null` value and a `null` value for `after` will result in the deletion of the document\n* Records with a non-`null` value and a non-`null` value for `after` will be repserted (replace the full document if it exists already, insert the full document if it does not)\n\nTo use this record format, configure it as a Kafka Connect Single Message Transformation in the connector's config:\n```json\n{\n  . . .\n  \"transforms\": \"cdc\",\n  \"transforms.cdc.type\": \"io.github.jaredpetersen.kafkaconnectarangodb.sink.transforms.Cdc\"\n}\n```\n\n### Topics\nThe name of the topic determines the name of the collection the record will be written to.\n\nRecord with topics that are just a plain string like `products` will go into a collection with the name `products`. If the record's topic name is period-separated like `dbserver1.mydatabase.customers`, the last period-separated value will be the collection's name (`customers` in this case). Each configured Kafka Connect ArangoDB Connector will only output data into a single database instance.\n\n### Foreign Keys and Edge Collections\nIn most situations, the record values that you will want to sink into ArangoDB is not in a format that ArangoDB can use effectively. ArangoDB has it's own format for foreign keys (`{ \"foreignKey\": \"MyCollection/1234\" }`) and edges between vertices (`{ \"_from\": \"MyCollection/1234\", \"_to\": \"MyCollection/5678\" }`) that your input data likely doesn't implement by default. It is recommended that you build your own custom [Kafka Streams application](https://kafka.apache.org/documentation/streams/) to perform these mappings.\n\n## Configuration\n### Connector Properties\n| Name                     | Description                         | Type     | Default | Importance |\n| ------------------------ | ----------------------------------- | -------- | ------- | ---------- |\n| `arangodb.host`          | ArangoDB server host.               | string   |         | high       |\n| `arangodb.port`          | ArangoDB server host port number.   | int      |         | high       |\n| `arangodb.user`          | ArangoDB connection username.       | string   |         | high       |\n| `arangodb.password`      | ArangoDB connection password.       | password | \"\"      | high       |\n| `arangodb.useSsl`        | ArangoDB use SSL connection.        | boolean  | false   | high       |\n| `arangodb.database.name` | ArangoDB database name.             | string   |         | high       |\n\n### Single Message Transformations\n| Type                                                               | Description                                        |\n| ------------------------------------------------------------------ | -------------------------------------------------- |\n| `io.github.jaredpetersen.kafkaconnectarangodb.sink.transforms.Cdc` | Converts records from CDC format to Simple format. |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaredpetersen%2Fkafka-connect-arangodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaredpetersen%2Fkafka-connect-arangodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaredpetersen%2Fkafka-connect-arangodb/lists"}