{"id":13530435,"url":"https://github.com/mre/kafka-influxdb","last_synced_at":"2025-10-08T02:15:49.922Z","repository":{"id":20343818,"uuid":"23618466","full_name":"mre/kafka-influxdb","owner":"mre","description":"High performance Kafka consumer for InfluxDB. Supports collectd message formats.","archived":false,"fork":false,"pushed_at":"2022-12-08T10:40:57.000Z","size":1633,"stargazers_count":217,"open_issues_count":15,"forks_count":53,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-09-28T08:29:21.396Z","etag":null,"topics":["consumer","influxdb","kafka-consumer","kafka-influxdb","logstash","performance","producer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mre.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"mre","patreon":"hellorust"}},"created_at":"2014-09-03T11:59:02.000Z","updated_at":"2025-06-17T10:09:47.000Z","dependencies_parsed_at":"2023-01-14T00:00:23.907Z","dependency_job_id":null,"html_url":"https://github.com/mre/kafka-influxdb","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/mre/kafka-influxdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mre%2Fkafka-influxdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mre%2Fkafka-influxdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mre%2Fkafka-influxdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mre%2Fkafka-influxdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mre","download_url":"https://codeload.github.com/mre/kafka-influxdb/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mre%2Fkafka-influxdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278877092,"owners_count":26061381,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consumer","influxdb","kafka-consumer","kafka-influxdb","logstash","performance","producer"],"created_at":"2024-08-01T07:00:49.804Z","updated_at":"2025-10-08T02:15:49.907Z","avatar_url":"https://github.com/mre.png","language":"Python","funding_links":["https://github.com/sponsors/mre","https://patreon.com/hellorust"],"categories":["Collecting data into InfluxDB","Python"],"sub_categories":["Plugins"],"readme":"# Kafka-InfluxDB\n\n[![Build Status](https://travis-ci.org/mre/kafka-influxdb.svg?branch=master)](https://travis-ci.org/mre/kafka-influxdb)\n[![Coverage Status](https://codecov.io/gh/mre/kafka-influxdb/branch/master/graph/badge.svg)](https://codecov.io/gh/mre/kafka-influxdb)\n[![Code Climate](https://codeclimate.com/github/mre/kafka-influxdb/badges/gpa.svg)](https://codeclimate.com/github/mre/kafka-influxdb)\n[![PyPi Version](https://badge.fury.io/py/kafka_influxdb.svg)](https://badge.fury.io/py/kafka\\_influxdb)\n[![Scrutinizer](https://scrutinizer-ci.com/g/mre/kafka-influxdb/badges/quality-score.png?b=master)](https://scrutinizer-ci.com/g/mre/kafka-influxdb/?branch=master)\n\nA Kafka consumer for InfluxDB written in Python.  \nSupports InfluxDB 0.9.x and up. For InfluxDB 0.8.x support, check out the [0.3.0 tag](https://github.com/mre/kafka-influxdb/tree/v0.3.0).\n\n:warning: The project should work as expected and bug fixes are very welcome, but activity on new functionality is quite low. \nFor newer projects I recommend [vector](https://vector.dev/) instead, which is both faster and more versatile.\n\n\n## Use cases\n\nKafka will serve as a buffer for your metric data during high load.  \nAlso it's useful for sending metrics from offshore data centers with unreliable connections to your monitoring backend.\n\n![](https://raw.githubusercontent.com/mre/kafka-influxdb/master/assets/schema-small.png)\n\n## Quickstart\n\nFor a quick test, run kafka-influxdb inside a container alongside Kafka and InfluxDB. Some sample messages are generated automatically on startup (using kafkacat).\n\n#### Python 2:\n\n```\nmake\ndocker exec -it kafkainfluxdb\npython -m kafka_influxdb -c config_example.yaml -s\n```\n\n#### Python 3:\n\n```\nmake RUNTIME=py3\ndocker exec -it kafkainfluxdb\npython -m kafka_influxdb -c config_example.yaml -s\n```\n\n#### PyPy 5.x\n\n```\nmake RUNTIME=pypy\ndocker exec -it kafkainfluxdb\npypy3 -m kafka_influxdb -c config_example.yaml -s --kafka_reader=kafka_influxdb.reader.kafka_python\n```\n\n(Note that one additional flag is given: `--kafka_reader=kafka_influxdb.reader.kafka_python`. This is because PyPy is incompatible with the confluent kafka consumer which is a C-extension to librdkafka. Therefore we use the kafka\\_python library here, which is compatible with PyPy but a bit slower.)\n\n#### Docker:\n\n```\ndocker run mre0/kafka-influxdb\n```\n\nor simply\n\n```\nmake run\n```\n\n## Installation\n\n```\npip install kafka_influxdb\nkafka_influxdb -c config_example.yaml\n```\n\n## Contributing\n\nIf you like to contribute, please create a pull request with your change.  \nPlease run the tests before you submit the pull request `make test`.  \nIf you're unsure, whether a change will be accepted, you can also create an issue first, to discuss.  \nOr look at the already existing issues for inspiration.  \n\nThanks for contributing!\n\n## Performance\n\nThe following graph shows the number of messages/s read from Kafka for various Python versions and Kafka consumer plugins.  \nThis is testing against a Kafka topic with 10 partitions and five message brokers.\nAs you can see the best performance is achieved on Python 3 using the `-O` flag for bytecode optimization in combination with the `confluent-kafka` reader (default setup). Note that encoding and sending the data to InfluxDB might lower this maximum performance although you should still see a significant performance boost compared to logstash.\n\n![Benchmark results](assets/benchmark.png)\n\n## Benchmark\n\nFor a quick benchmark, you can start a complete `kafkacat -\u003e Kafka -\u003e kafka_influxdb -\u003e Influxdb` setup with the following command:\n\n    make\n\nThis will immediately start reading messages from Kafka and write them into InfluxDB. To see the output, you can use the InfluxDB cli.\n\n    docker exec -it docker_influxdb_1 bash # Double check your container name\n    influx\n    use metrics\n    show measurements\n\n## Supported formats\n\nYou can write a custom encoder to support any input and output format (even fancy things like Protobuf). Look at the examples inside the [`encoder`](./kafka_influxdb/encoder) directory to get started. The following formats are officially supported:\n\n#### Input formats\n\n- [Collectd Graphite ASCII format](https://collectd.org/wiki/index.php/Graphite): :\n\n```\nmydatacenter.myhost.load.load.shortterm 0.45 1436357630\n```\n\n- [Collectd JSON format](https://collectd.org/wiki/index.php/JSON):\n\n```json\n[{\n    \"values\":[\n       0.6\n    ],\n    \"dstypes\":[\n       \"gauge\"\n    ],\n    \"dsnames\":[\n       \"value\"\n    ],\n    \"time\":1444745144.824,\n    \"interval\":10.000,\n    \"host\":\"xx.example.internal\",\n    \"plugin\":\"cpu\",\n    \"plugin_instance\":\"1\",\n    \"type\":\"percent\",\n    \"type_instance\":\"system\"\n }]\n```\n\n-   [Raw InfluxDB line protocol (e.g. for Telegraf support)](https://github.com/mre/kafka-influxdb/issues/40):\n\n#### Output formats\n\n- [InfluxDB 0.9.2+ line protocol format](https://influxdb.com/docs/v0.9/write_protocols/line.html): :\n\n```\nload_load_shortterm,datacenter=mydatacenter,host=myhost value=\"0.45\" 1436357630\n```\n\n-   [InfluxDB 0.8.x JSON format](https://influxdb.com/docs/v0.8/api/reading_and_writing_data.html#writing-data-through-http) (*deprecated*)\n\n#### Custom encoders\n\nIf you are writing your custom encoder and you want to run it using the official docker image, you can simply mount it in the container:\n\n```\ndocker run -v `pwd`/config.yaml:/usr/src/app/config.yaml -v `pwd`/myencoder.py:/usr/src/app/myencoder.py mre0/kafka-influxdb --encoder=myencoder\n```\n\nAnother possibility is to create a custom Docker image that contains your encoder, for example:\n\n```\nFROM mre0/kafka-influxdb\n\nADD myencoder.py /usr/src/app/myencoder.py\nADD config.yaml /usr/src/app/\n\nCMD python -m kafka_influxdb -c config.yaml -v --encoder=myencoder\n```\n\n## Configuration\n\nTake a look at the `config-example.yaml` to find out how to create a config file.\nYou can overwrite the settings from the commandline. The following parameters are allowed:\n\n| Option                                                  | Description                                                                                           |\n|---------------------------------------------------------|-------------------------------------------------------------------------------------------------------|\n| `-h`, `--help`                                          | Show help message and exit                                                                            |\n| `--kafka_host KAFKA_HOST`                               | Hostname or IP of Kafka message broker (default: localhost)                                           |\n| `--kafka_port KAFKA_PORT`                               | Port of Kafka message broker (default: 9092)                                                          |\n| `--kafka_topic KAFKA_TOPIC`                             | Topic for metrics (default: my\\_topic)                                                                |\n| `--kafka_group KAFKA_GROUP`                             | Kafka consumer group (default: my\\_group)                                                             |\n| `--kafka_reader KAFKA_READER`                           | Kafka client library to use (kafka_python or confluent) (default: kafka_influxdb.reader.confluent)    |\n| `--influxdb_host INFLUXDB_HOST`                         | InfluxDB hostname or IP (default: localhost)                                                          |\n| `--influxdb_port INFLUXDB_PORT`                         | InfluxDB API port (default: 8086)                                                                     |\n| `--influxdb_user INFLUXDB_USER`                         | InfluxDB username (default: root)                                                                     |\n| `--influxdb_password INFLUXDB_PASSWORD`                 | InfluxDB password (default: root)                                                                     |\n| `--influxdb_dbname INFLUXDB_DBNAME`                     | InfluxDB database to write metrics into (default: metrics)                                            |\n| `--influxdb_use_ssl`                                    | Use SSL connection for InfluxDB (default: False)                                                      |\n| `--influxdb_verify_ssl`                                 | Verify the SSL certificate before connecting (default: False)                                         |\n| `--influxdb_timeout INFLUXDB_TIMEOUT`                   | Max number of seconds to establish a connection to InfluxDB (default: 5)                              |\n| `--influxdb_use_udp`                                    | Use UDP connection for InfluxDB (default: False)                                                      |\n| `--influxdb_retention_policy INFLUXDB_RETENTION_POLICY` | Retention policy for incoming metrics (default: autogen)                                              |\n| `--influxdb_time_precision INFLUXDB_TIME_PRECISION`     | Precision of incoming metrics. Can be one of 's', 'm', 'ms', 'u' (default: s)                         |\n| `--encoder ENCODER`                                     | Input encoder which converts an incoming message to dictionary (default: collectd\\_graphite\\_encoder) |\n| `--buffer_size BUFFER_SIZE`                             | Maximum number of messages that will be collected before flushing to the backend (default: 1000)      |\n| `-c CONFIGFILE`, `--configfile CONFIGFILE`              | Configfile path (default: None)                                                                       |\n| `-s`, `--statistics`                                    | Show performance statistics (default: True)                                                           |\n| `-v`, `--verbose`                                       | Set verbosity level. Increase verbosity by adding a v: -v -vv -vvv (default: 0)                       |\n| `--version`                                             | Show version                                                                                          |\n\n## Comparison with other tools\n\nThere is a Kafka input plugin and an InfluxDB output plugin for **logstash**. It supports Influxdb 0.9+. We've achieved a message throughput of around **5000 messages/second** with that setup. Check out the configuration at docker/logstash/config.conf. You can run the benchmark yourself:\n\n```\nmake RUNTIME=logstash\ndocker exec -it logstash\nlogstash -f config.conf\n```\n\nPlease send a Pull Request if you know of other tools that can be mentioned here.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmre%2Fkafka-influxdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmre%2Fkafka-influxdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmre%2Fkafka-influxdb/lists"}