{"id":27627235,"url":"https://github.com/treasure-data/kafka-fluentd-consumer","last_synced_at":"2025-07-13T19:34:15.166Z","repository":{"id":55610675,"uuid":"42180926","full_name":"treasure-data/kafka-fluentd-consumer","owner":"treasure-data","description":"Kafka Consumer for Fluentd","archived":false,"fork":false,"pushed_at":"2019-03-15T14:16:28.000Z","size":157,"stargazers_count":32,"open_issues_count":5,"forks_count":18,"subscribers_count":95,"default_branch":"master","last_synced_at":"2025-04-23T13:59:19.210Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/treasure-data.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-09T13:44:07.000Z","updated_at":"2024-05-30T04:39:23.000Z","dependencies_parsed_at":"2022-08-15T04:20:41.494Z","dependency_job_id":null,"html_url":"https://github.com/treasure-data/kafka-fluentd-consumer","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/treasure-data/kafka-fluentd-consumer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fkafka-fluentd-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fkafka-fluentd-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fkafka-fluentd-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fkafka-fluentd-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/treasure-data","download_url":"https://codeload.github.com/treasure-data/kafka-fluentd-consumer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fkafka-fluentd-consumer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265194965,"owners_count":23726002,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-23T13:53:43.732Z","updated_at":"2025-07-13T19:34:15.117Z","avatar_url":"https://github.com/treasure-data.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Consumer for Fluentd\n\nThis integration is a simple Java application that you can use to consume data from Kafka to Fluentd. You can download the application from this page and then complete the following instructions.\n\n## Build\n\nUse gradle 2.1 or later.\n\n    $ gradle shadowJar\n\n## Run\n\n### Run Kafka\n\nYou need to be running Kafka for the consumer to work. To test Kafka locally, follow the steps described in [Kafka's Quickstart](http://kafka.apache.org/documentation.html#quickstart).\n\n    # start zookeeper\n    $ bin/zookeeper-server-start.sh config/zookeeper.properties\n    \n    # start kafka\n    $ bin/kafka-server-start.sh config/server.properties\n\nThen create a topic called `test`\n\n    # create test topic\n    $ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test\n\nWhen the 'test' topic is created, add a few messages in it. Make sure message is valid JSON.\n\n    # send multiple messages\n    $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test \n    {\"a\": 1}\n    {\"a\": 1, \"b\": 2}\n\nYou can confirm messages were submitted correctly with this command.\n\n    $ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning\n    {\"a\": 1}\n    {\"a\": 1, \"b\": 2}\n\n### Run Kafka Consumer for Fluentd\n\nModify `config/fluentd-consumer.properties` with an appropriate configuration. Remember to change to `fluentd.consumer.topics=test`. Then, launch the process like this.\n\n    $ java -Dlog4j.configuration=file:///path/to/log4j.properties -jar build/libs/kafka-fluentd-consumer-0.3.2-all.jar config/fluentd-consumer.properties\n\nThis will forward logs to Fluentd (localhost:24224). This consumer uses log4j so you can change logging configurations via `-Dlog4j.configuration` argument.\n\n## Run Kafka Consumer for Fluentd via in_exec\n\nTo host a consumer as a child process of Fluentd, use the following code: \n\n    \u003csource\u003e\n      type forward\n    \u003c/source\u003e\n    \n    \u003csource\u003e\n      type exec\n      command java -Dlog4j.configuration=file:///path/to/log4j.properties -jar /path/to/kafka-fluentd-consumer-0.3.2-all.jar /path/to/config/fluentd-consumer.properties\n      tag dummy\n      format json\n    \u003c/source\u003e\n\n## TODO\n\n- Support more format, e.g. msgpack.\n- Add metrics features\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Fkafka-fluentd-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftreasure-data%2Fkafka-fluentd-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Fkafka-fluentd-consumer/lists"}