{"id":20186618,"url":"https://github.com/zimbora/mqtt-devices-parser","last_synced_at":"2026-05-07T13:14:15.355Z","repository":{"id":191600274,"uuid":"684696065","full_name":"zimbora/mqtt-devices-parser","owner":"zimbora","description":"NPM MODULE - parsing mqtt messages coming from devices","archived":false,"fork":false,"pushed_at":"2025-08-18T17:03:25.000Z","size":168,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-18T19:08:28.714Z","etag":null,"topics":["json","mqtt","mysql2","parsing"],"latest_commit_sha":null,"homepage":"https://iot.inloc.cloud","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zimbora.png","metadata":{"files":{"readme":"Readme.md","changelog":"Changelog.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-29T17:02:43.000Z","updated_at":"2025-08-18T17:03:13.000Z","dependencies_parsed_at":"2024-01-11T23:41:01.232Z","dependency_job_id":"d473fdd9-dcdd-4872-9cf8-d745b21b5baa","html_url":"https://github.com/zimbora/mqtt-devices-parser","commit_stats":null,"previous_names":["zimbora/mqtt-devices-parser"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/zimbora/mqtt-devices-parser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zimbora%2Fmqtt-devices-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zimbora%2Fmqtt-devices-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zimbora%2Fmqtt-devices-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zimbora%2Fmqtt-devices-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zimbora","download_url":"https://codeload.github.com/zimbora/mqtt-devices-parser/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zimbora%2Fmqtt-devices-parser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271441743,"owners_count":24760348,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","mqtt","mysql2","parsing"],"created_at":"2024-11-14T03:17:58.795Z","updated_at":"2026-05-07T13:14:15.350Z","avatar_url":"https://github.com/zimbora.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# mqtt-devices-parser\n\n## NPM Module\n  - MQTT parser\n  - Parses topics coming from devices and updates data on dB\n  - **NEW**: Kafka consumer support for multi-process message parsing\n\n## Deploy\n  ```\n  parser.deploy(config,projectsPath);\n  ```\n\n## Run\n  ```\n  parser.init(config,projects);\n  ```\n\n## Examples\n  check example path to see how to use this module\n\n## UnitTest\n\n\n## Dependencies\n\n- mysql8.0\n- MQTT\n- **NEW**: Confluentic Kafka (optional)\n\n## Front end\n- Node service running the following [project](https://github.com/zimbora/mgmt-iot-web)\n\n## Configuration\n\nThe default configuration uses ./config/index.js file\\\nIn your program set the same struct and pass it as argument on the module init call.\n\nAn mqtt account will be created using the following settings\nmqtt.user - mqtt user login\nmqtt.pwd - mqtt pwd login\n\n### Kafka Consumer Configuration (NEW)\n\nThe module now supports consuming messages from Kafka topics as an alternative or complement to MQTT. This enables multi-process message parsing using Kafka's shared subscription mechanism.\n\n#### Environment Variables\n\n**Kafka Settings:**\n- `KAFKA_ENABLED` - Set to `'true'` to enable Kafka consumer (default: `false`)\n- `KAFKA_BROKERS` - Comma-separated list of Kafka brokers (default: `'localhost:9092'`)\n- `KAFKA_GROUP_ID` - Consumer group ID for shared subscriptions (default: `'mqtt-devices-parser-group'`)\n- `KAFKA_CLIENT_ID` - Kafka client identifier (default: `'mqtt-devices-parser'`)\n\n**Authentication (SASL):**\n- `KAFKA_SASL_MECHANISM` - SASL mechanism: `'plain'`, `'scram-sha-256'`, `'scram-sha-512'` (default: `'plain'`)\n- `KAFKA_SASL_USERNAME` - SASL username (default: empty)\n- `KAFKA_SASL_PASSWORD` - SASL password (default: empty)\n\n**Security:**\n- `KAFKA_SSL_ENABLED` - Set to `'true'` to enable SSL (default: `false`)\n\n**Connection Settings:**\n- `KAFKA_CONNECTION_TIMEOUT` - Connection timeout in ms (default: `3000`)\n- `KAFKA_REQUEST_TIMEOUT` - Request timeout in ms (default: `30000`)\n\n**MQTT Parsing Control:**\n- `MQTT_PARSE_MESSAGES` - Set to `'false'` to disable direct MQTT message parsing (default: `true`)\n\n#### Usage Examples\n\n**Example 1: Use Kafka with SSL and SCRAM-SHA-256 authentication - Most Secure**\n```bash\nexport KAFKA_SSL_ENABLED=true\nexport KAFKA_SSL_CERTFILE='./certs/ca.cert'\nexport KAFKA_SSL_REJECT_UNAUTHORIZED=true\nexport KAFKA_BROKERS=kafka1:9092,kafka2:9092\nexport KAFKA_SASL_USERNAME=myuser\nexport KAFKA_SASL_PASSWORD=mypassword\nexport KAFKA_SASL_MECHANISM=scram-sha-256\n```\n**Example 2: Use Kafka with SSL No trust and SCRAM-SHA-256 authentication.\nProne to MITM attacks**\n```bash\nexport KAFKA_SSL_ENABLED=true\nexport KAFKA_SSL_CERTFILE=''\nexport KAFKA_SSL_REJECT_UNAUTHORIZED=false\nexport KAFKA_BROKERS=kafka1:9092,kafka2:9092\nexport KAFKA_SASL_USERNAME=myuser\nexport KAFKA_SASL_PASSWORD=mypassword\nexport KAFKA_SASL_MECHANISM=scram-sha-256\n```\n**Example 3: Use Kafka without SSL and SCRAM-SHA-256 authentication. Basic authentication with no encryption. Credentials are not shared**\n```bash\nexport KAFKA_SSL_ENABLED=false\nexport KAFKA_SSL_CERTFILE=''\nexport KAFKA_SSL_REJECT_UNAUTHORIZED=true\nexport KAFKA_BROKERS=kafka1:9092,kafka2:9092\nexport KAFKA_SASL_USERNAME=myuser\nexport KAFKA_SASL_PASSWORD=mypassword\nexport KAFKA_SASL_MECHANISM=scram-sha-256\n```\n\n**Example 4: Use KAFKA without authentication. Not secure at all**\n```bash\nexport KAFKA_ENABLED=true\nexport KAFKA_SSL_ENABLED=false\nexport KAFKA_SASL_USERNAME=''\nexport KAFKA_SASL_PASSWORD=''\n```\n\n**Example 5: KAFKA Multi-process deployment**\n```bash\n# Start multiple instances with the same group ID for load balancing\nexport KAFKA_ENABLED=true\nexport KAFKA_GROUP_ID=my-parser-group\nexport KAFKA_BROKERS=kafka1:9092,kafka2:9092\n# Each process will consume different partitions automatically\n```\n\n**Example 6: Use MQTT only (disable KAFKA parsing)**\n```bash\nexport KAFKA_ENABLED=false\n```\n\n#### Kafka Topic Structure\n\nKafka topics should correspond to project names configured in your `projects` array. Messages are expected to have:\n\n- **Topic**: The project name (e.g., `demo`, `freeRTOS2`)\n- **Key**: Device path part (e.g., `uid:123/status`)\n- **Value**: Message payload (same format as MQTT payloads)\n\nThe consumer will automatically format messages as `{project}/{key}` to match the expected MQTT topic structure.\n\n#### Shared Subscriptions\n\nWhen multiple instances use the same `KAFKA_GROUP_ID`, Kafka automatically distributes messages across consumers, enabling:\n\n- **Load balancing**: Messages are distributed across multiple parser instances\n- **High availability**: If one instance fails, others continue processing\n- **Scalability**: Add more instances to handle increased load\n\n#### Message Flow\n\n1. **MQTT Mode** (traditional): MQTT messages → `$.device.parseMessage`\n2. **Kafka Mode**: Kafka messages → Topic formatting → `$.device.parseMessage`  \n3. **Hybrid Mode**: Both MQTT and Kafka messages → `$.device.parseMessage`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzimbora%2Fmqtt-devices-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzimbora%2Fmqtt-devices-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzimbora%2Fmqtt-devices-parser/lists"}