{"id":15583675,"url":"https://github.com/david-lor/mqtt2influxdb","last_synced_at":"2026-04-20T19:02:36.636Z","repository":{"id":51352599,"uuid":"230592834","full_name":"David-Lor/MQTT2InfluxDB","owner":"David-Lor","description":"Log 🦟MQTT messages on 🕓InfluxDB","archived":false,"fork":false,"pushed_at":"2021-05-13T20:38:24.000Z","size":20,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-03-29T08:44:08.735Z","etag":null,"topics":["influx","influxdb","iot","mosquitto","mqtt","mqtt-logger","mqtt-topic","python","python3","python36","python37","python38","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/David-Lor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-28T10:10:09.000Z","updated_at":"2021-03-08T08:07:00.000Z","dependencies_parsed_at":"2022-09-24T21:50:33.184Z","dependency_job_id":null,"html_url":"https://github.com/David-Lor/MQTT2InfluxDB","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/David-Lor/MQTT2InfluxDB","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/David-Lor%2FMQTT2InfluxDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/David-Lor%2FMQTT2InfluxDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/David-Lor%2FMQTT2InfluxDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/David-Lor%2FMQTT2InfluxDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/David-Lor","download_url":"https://codeload.github.com/David-Lor/MQTT2InfluxDB/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/David-Lor%2FMQTT2InfluxDB/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32061251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["influx","influxdb","iot","mosquitto","mqtt","mqtt-logger","mqtt-topic","python","python3","python36","python37","python38","redis"],"created_at":"2024-10-02T20:20:28.015Z","updated_at":"2026-04-20T19:02:36.621Z","avatar_url":"https://github.com/David-Lor.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MQTT2InfluxDB\n\nPython Service that subscribes to one or multiple MQTT topics and insert the received messages on InfluxDB,\nusing a Redis DB as a queue to bulk insert messages on Influx, and keep them during Influx server downtime.\n\n- [Requirements](#requirements)\n- [Changelog](#changelog)\n- [TODO](#todo)\n- [FAQ](#faq)\n- [Settings](#settings)\n- [Data structure](#data-structure)\n- [Installing \u0026 running](#installing--running)\n\n## Requirements\n\n- Python \u003e= 3.6\n- Requirements listed on [requirements.txt](requirements.txt)\n- A MQTT broker\n- An InfluxDB server\n- A Redis server\n\n## Changelog\n\n- 0.0.2 - Add support for authentication \u0026 SSL to MQTT\n- 0.0.1 - Initial version (functional code)\n\n## TODO\n\n- Fix insertion of JSON with booleans (crashes data writing into Influx)\n- Handle errors on all connectors (MQTT, Redis, Influx)\n- Add locks to Redis while reading/writing to the queue\n- Only insert string payloads and not arbitrary binary data\n- Support MQTT authentication \u0026 SSL\n- Shorter payload debug logging records?\n- Set Redis optional (Redis connector being a mocked class with a Python list to store messages)\n- Support Redis authentication\n- Support InfluxDB authentication\n- Unit \u0026 integration tests\n\n## FAQ\n\n- **Why is this even a thing?**\n\n  When using MQTT for IoT communication, I found useful to have all the messages registered on some persistent storage.\n  This allows me to search through topics and see their context. And most important, can create statistics for\n  monitorized data, and even analytics and graphs, using InfluxDB + Chronograf, Grafana or other similar tools.\n\n- **Why is Redis required?**\n\n  On my particular setup scenario, the server that will hold the data will not be always available, while MQTT messages\n  get sent 24/7, and this service is running on a 24/7 device. A simpler solution, as a Python list, could be used,\n  but preferred to externalize it on a Redis queue.\n  \n  This can be useful if this service is down, restarts or even if the device running it restarts,\n  since Redis can persist the data on disk (must be configured on Redis).\n  \n  Other advantage is to do bulk insertions on InfluxDB, instead of performing a request by each single message received\n  on MQTT.\n\n- **Why not using Telegraf?**\n\n  Telegraf is part of the Influx stack, as a counterpart of Logstash on the Elastic stack.\n  It [can subscribe to MQTT](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer) and insert the received messages on Influx, but is not very flexible when dealing with different types of variables.\n\n- **Why parsing \u0026 storing the payload?**\n\n  By default, payloads are read as strings, but they get converted to numbers, booleans or JSON if possible.\n  InfluxDB is not very friendly with string to number parsing during queries to create, for example, graphs in Chronograf.\n  That's why all the messages get their payload parsed to the closest type and stored on a different field.\n\n## Settings\n\nSettings can be defined through environment variables or using a `.env` file (located within the `__main__.py` file).\n\n### MQTT Settings\n\n- `MQTT_BROKER`: MQTT broker host (default: `localhost`)\n- `MQTT_PORT`: MQTT broker port (default: `1883`)\n- `MQTT_SUBSCRIBE`: MQTT topic/s to subscribe to (default: `#`); split multiple topics by the `MQTT_SUBSCRIBE_SEPARATOR` (default: `,`)\n- `MQTT_SUBSCRIBE_SEPARATOR`: character/sequence used to split multiple topics on the `MQTT_SUBSCRIBE` env var (default: `,`)\n- `MQTT_CLIENT_ID`: MQTT client ID (default: `MQTT2InfluxDB_{uuid1(node)}`)\n- `MQTT_KEEPALIVE`: keepalive time in seconds (default: `60`)\n- `MQTT_QOS`: MQTT QoS used when subscribing to topic/s (default: `0`)\n\n### InfluxDB Settings\n\n- `INFLUX_HOST`: InfluxDB server host (default: `localhost`)\n- `INFLUX_PORT`: InfluxDB server port (default: `8086`)\n\n### Redis Settings\n\n- `REDIS_HOST`: Redis server host (default: `localhost`)\n- `REDIS_PORT`: Redis server port (default: `6379`)\n- `REDIS_DB`: Redis database index (default: `0`)\n- `REDIS_QUEUE_NAME`: Name of the key used to store messages in queue (default: `mqtt2influxdb_{uuid1(node)}`)\n\n### General (system) Settings\n\n- `LOG_LEVEL`: level of the internal logger (default: `INFO`)\n\n## Data structure\n\n```json\n{\n    \"measurement\": \"mqtt\",\n    \"tags\": {\n        \"topic\": \"{topic}\",\n        \"qos\": {qos}\n    },\n    \"time\": \"2020-01-02T16:20:00Z\",\n    \"fields\": {\n        \"payload\": \"1.2345\",\n        \"payload_number\": 1.2345\n    }\n}\n```\n\n- `measurement` is the same for all MQTT messages (`mqtt` by default)\n- `fields.payload` stores the payload as string\n- the payload tries to be converted to:\n    - number -\u003e `fields.payload_number`\n    - json -\u003e `fields.payload_json`\n    - boolean -\u003e `fields.payload_bool`\n\n## Installing \u0026 Running\n\nThe recommended method to install is using the [Python-Autoclonable-App](https://hub.docker.com/r/davidlor/python-autoclonable-app/) image.\n\n### Run through Docker with Python-Autoclonable-App image\n\n```bash\n# TODO\n```\n\n### Run through Docker-Compose\n\n```bash\ngit clone https://github.com/David-Lor/MQTT2InfluxDB.git\ncd MQTT2InfluxDB/tools/deployment/mqtt2influxdb\ndocker-compose up -d\n```\n\n### Run locally\n\n```bash\ngit clone https://github.com/David-Lor/MQTT2InfluxDB.git\ncd MQTT2InfluxDB\n\n# Set your environment variables\ncp sample.env .env\nnano .env\n\n# Run\npython3 .\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavid-lor%2Fmqtt2influxdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavid-lor%2Fmqtt2influxdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavid-lor%2Fmqtt2influxdb/lists"}