{"id":18481491,"url":"https://github.com/joker1007/embulk-output-kafka","last_synced_at":"2025-04-08T17:30:32.934Z","repository":{"id":35111074,"uuid":"207844313","full_name":"joker1007/embulk-output-kafka","owner":"joker1007","description":"Embulk output plugin for Apache Kafka","archived":false,"fork":false,"pushed_at":"2024-08-05T08:48:01.000Z","size":240,"stargazers_count":2,"open_issues_count":6,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-23T16:39:25.731Z","etag":null,"topics":["embulk","embulk-output-plugin","kafka","kafka-producer"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joker1007.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-11T15:21:24.000Z","updated_at":"2024-05-13T14:09:33.000Z","dependencies_parsed_at":"2024-05-03T17:24:55.326Z","dependency_job_id":"f8597fbf-aa2b-41c1-84dc-9c44ef394dd8","html_url":"https://github.com/joker1007/embulk-output-kafka","commit_stats":{"total_commits":62,"total_committers":2,"mean_commits":31.0,"dds":0.08064516129032262,"last_synced_commit":"5bfb8264e148296514cc8d4b6b19710f64c71ae9"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joker1007%2Fembulk-output-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joker1007%2Fembulk-output-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joker1007%2Fembulk-output-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joker1007%2Fembulk-output-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joker1007","download_url":"https://codeload.github.com/joker1007/embulk-output-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247892205,"owners_count":21013658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embulk","embulk-output-plugin","kafka","kafka-producer"],"created_at":"2024-11-06T12:23:59.016Z","updated_at":"2025-04-08T17:30:32.573Z","avatar_url":"https://github.com/joker1007.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka output plugin for Embulk\n[![Java CI](https://github.com/joker1007/embulk-output-kafka/actions/workflows/gradle.yml/badge.svg)](https://github.com/joker1007/embulk-output-kafka/actions/workflows/gradle.yml)\n\n## Compatibility\n\n| embulk-output-kafka   | embulk          |\n| --------------------- | ------          |\n| 0.4.x                 | 0.11.x or later |\n| 0.3.x                 | 0.9.x or later  |\n\n## Overview\n\n* **Plugin type**: output\n* **Load all or nothing**: no\n* **Resume supported**: no\n* **Cleanup supported**: yes\n\n## Configuration\n\n- **broker**: kafka broker host and port (array(string), required)\n- **topic**: target topic name (string, required)\n- **topic_column**: use column value as target topic (string, default: `null`)\n- **schema_registry_url**: Schema Registy URL that is needed for avro format (string, default: `null`)\n- **serialize_format**: use column value as target topic (enum, required, `json` or `avro_with_schema_registry`)\n- **avsc_file**: avro schema file path (string, default: `null`)\n- **avsc**: inline avro schema config (json, default: `null`)\n- **subject_name**: subject name for schema_registry (string, default: `null`)\n- **ignore_columns**: remove columns from output  (array(string), default: `[]`)\n- **key_column_name**: use column value as record key (string, default: `null`, it can use columns in `ignore_columns`)\n- **partition_column_name**: use column value as partition id (string, default: `null`, this value is prefer to `key_column_name`, and if partition_column value is null, use key_column for partitioning)\n- **column_for_deletion**: Determine to delete (string, default: `null`, `column_for_deletion` column must be boolean. If the value of the column is `true`, KafkaProducer sends `null` value to a Kafka Broker.)\n- **record_batch_size**: kafka producer record batch size (integer, default: `1000`)\n- **acks**: kafka producer require acks (string, default: `\"1\"`)\n- **retries**: kafka producer max retry count (integer, default: `1`)\n- **other_producer_configs**: other producer configs (json, default: `{}`)\n- **value_subject_name_strategy**: Set SchemaRegistry subject name strategy (string, default: `null`, ex. `io.confluent.kafka.serializers.subject.RecordNameStrategy`)\n\nIf use `avro_with_schema_registry` format, following configs are required.\n\n- **schema_registry_url**\n\nIf avsc and avsc_file are null, embulk-output-kafka fetch a schema from schema registry.\nBut currently, embulk-output-kafka supports only TopicNameStrategy.\nIf you want to use another subject name, use `subject_name` parameter.\n\n## Example\n\n```yaml\nin:\n  type: file\n  path_prefix: ./src/test/resources/in1\n  parser:\n    charset: UTF-8\n    newline: CRLF\n    type: csv\n    delimiter: ','\n    quote: '\"'\n    escape: '\"'\n    null_string: 'NULL'\n    skip_header_lines: 1\n    columns:\n    - {name: 'id', type: string}\n    - {name: 'int_item', type: long}\n    - {name: 'varchar_item', type: string}\n\nout:\n  type: kafka\n  topic: \"json-topic\"\n  serialize_format: json\n  brokers:\n    - \"localhost:9092\"\n```\n\n```yaml\nin:\n  type: file\n  path_prefix: ./src/test/resources/in_complex\n  parser:\n    charset: UTF-8\n    newline: CRLF\n    type: csv\n    delimiter: \"\\t\"\n    quote: \"\\0\"\n    escape: \"\\0\"\n    null_string: 'NULL'\n    skip_header_lines: 1\n    columns:\n    - {name: 'id', type: string}\n    - {name: 'int_item', type: long}\n    - {name: 'time', type: timestamp, format: \"%Y-%m-%dT%H:%M:%S\"}\n    - {name: 'array', type: json}\n    - {name: 'data', type: json}\n\nout:\n  type: kafka\n  topic: \"avro-complex-topic\"\n  acks: all\n  retries: 3\n  brokers:\n    - \"localhost:9092\"\n  schema_registry_url: \"http://localhost:48081/\"\n  serialize_format: avro_with_schema_registry\n  other_producer_configs:\n    buffer.memory: \"67108864\"\n  avsc:\n    type: record\n    name: ComplexRecord\n    fields: [\n      {name: \"id\", type: \"string\"},\n      {name: \"int_item\", type: \"long\"},\n      {name: \"time\", type: \"long\", logicalType: \"timestamp-milli\"},\n      {name: \"array\", type: {type: \"array\", items: \"long\"}},\n      {name: \"data\", type: {type: \"record\", name: \"InnerData\", fields: [\n        {name: \"hoge\", type: \"string\"},\n        {name: \"aaa\", type: [\"null\", \"string\"]},\n        {name: \"array\", type: {type: \"array\", items: \"long\"}},\n      ]}},\n    ]\n```\n\n\n## Build\n\n```\n$ ./gradlew gem  # -t to watch change of files and rebuild continuously\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoker1007%2Fembulk-output-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoker1007%2Fembulk-output-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoker1007%2Fembulk-output-kafka/lists"}