{"id":47737710,"url":"https://github.com/ecodev/ruuvi-iot-ingestion","last_synced_at":"2026-04-02T23:00:48.208Z","repository":{"id":344729311,"uuid":"1182857495","full_name":"Ecodev/ruuvi-iot-ingestion","owner":"Ecodev","description":"InfluxDB templates","archived":false,"fork":false,"pushed_at":"2026-03-23T22:53:53.000Z","size":284,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-24T13:57:05.542Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Ecodev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-16T02:56:01.000Z","updated_at":"2026-03-24T11:13:27.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Ecodev/ruuvi-iot-ingestion","commit_stats":null,"previous_names":["hazhargaleh/influxdb-templates","hazhargaleh/ruuvi-iot-ingestion"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Ecodev/ruuvi-iot-ingestion","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ecodev%2Fruuvi-iot-ingestion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ecodev%2Fruuvi-iot-ingestion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ecodev%2Fruuvi-iot-ingestion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ecodev%2Fruuvi-iot-ingestion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Ecodev","download_url":"https://codeload.github.com/Ecodev/ruuvi-iot-ingestion/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ecodev%2Fruuvi-iot-ingestion/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31318132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T21:35:00.834Z","status":"ssl_error","status_checked_at":"2026-04-02T21:34:59.806Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-02T23:00:33.068Z","updated_at":"2026-04-02T23:00:48.178Z","avatar_url":"https://github.com/Ecodev.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ruuvi-iot-ingestion\n\nA high-performance Node.js/TypeScript backend service that collects sensor data from [RuuviTag](https://ruuvi.com/) Bluetooth sensors via a Ruuvi Gateway and MQTT, computes derived metrics, and stores them in InfluxDB and/or MariaDB.\n\n---\n\n## Table of contents\n\n- [Architecture](#architecture)\n- [Features](#features)\n- [Requirements](#requirements)\n- [Installation](#installation)\n- [Configuration](#configuration)\n- [Running the service](#running-the-service)\n- [Testing](#testing)\n- [Data pipeline](#data-pipeline)\n- [Stored metrics](#stored-metrics)\n- [MariaDB retention \u0026 downsampling](#mariadb-retention--downsampling)\n- [HTTP endpoints](#http-endpoints)\n- [Gateway configuration endpoint](#gateway-configuration-endpoint)\n- [Monitoring metrics](#monitoring-metrics)\n- [Nginx reverse proxy](#nginx-reverse-proxy)\n- [Docker](#docker)\n- [Production deployment](#production-deployment)\n- [Project structure](#project-structure)\n- [Logging](#logging)\n- [Optimizations](#optimizations)\n- [Troubleshooting](#troubleshooting)\n- [License](#license)\n\n---\n\n## Architecture\n\n```\nRuuviTags (BLE)\n      │\n      ▼  Bluetooth\nRuuvi Gateway  ──── HTTPS ────► /ruuvi-gw-cfg  (automatic config download)\n      │\n      ▼  MQTT over TLS\n  Mosquitto (your-domain-or-ip-of-server:8883)\n      │\n      ▼\nruuvi-iot-ingestion (this service)\n      │   • Zod validation\n      │   • Derived metric calculation\n      │   • Dual message buffer\n      │\n      ├──► InfluxDB v2   (time-series, Grafana dashboards)\n      └──► MariaDB       (SQL, HACCP reports, exports)\n```\n\nThe gateway decodes BLE packets and publishes JSON payloads over MQTT. This service consumes those messages, enriches them with computed metrics (dew point, VPD, battery %, etc.), and writes them to one or both databases in configurable batches.\n\nIt also serves gateway configuration files over HTTPS, allowing each Ruuvi Gateway to download its own settings automatically on startup and at a configurable interval.\n\n---\n\n## Features\n\n- **MQTT ingestion** — subscribes to `ruuvi/#` topics with TLS support and automatic reconnection\n- **Zod validation** — strict schema validation on every incoming payload, including the config itself at startup\n- **Derived metrics** — dew point, frost point, absolute humidity, VPD, air density, acceleration angles, battery percentage, and more (see [Stored metrics](#stored-metrics))\n- **Dual storage** — write to InfluxDB, MariaDB, or both simultaneously via a single env variable\n- **Batch writes** — configurable buffer size and flush interval for both databases\n- **Device name mapping** — map gateway and tag MAC addresses to human-readable names via `.env`\n- **Auto schema init** — MariaDB tables and views are created automatically on first start\n- **Retention \u0026 downsampling** — automatic deletion of old raw data and hourly aggregation for MariaDB (configurable)\n- **Gateway config endpoint** — serves per-gateway JSON config files over HTTPS with Basic/Bearer auth, resolved by gateway name, MAC address, or default fallback\n- **Health \u0026 metrics** — Fastify HTTP server exposing `/health` and `/metrics` (Prometheus via `prom-client`)\n- **Rate limiting** — built-in protection against brute-force and DoS attacks on HTTP endpoints\n- **Structured logging** — `pino` with pretty-print in development\n- **Docker support** — multi-stage `Dockerfile` with non-root user and `dumb-init` for proper signal handling\n- **Unit tests** — Jest test suite covering calculations and buffer logic\n\n---\n\n## Requirements\n\n- Node.js \u003e= 24\n- npm \u003e= 9\n- One or more [Ruuvi Gateways](https://ruuvi.com/gateway/) with MQTT support\n- InfluxDB v2 (if using InfluxDB storage)\n- MariaDB \u003e= 10.6 (if using MariaDB storage)\n\n---\n\n## Installation\n\n1. Clone the project and install dependencies:\n\n```bash\ngit clone https://github.com/Ecodev/ruuvi-iot-ingestion.git\ncd ruuvi-iot-ingestion\nnpm install\n```\n\n2. Create the `.env` file from the example:\n\n```bash\ncd config\ncp .env.example .env\n```\n\n3. Create the gateway configuration directory from the example:\n\n```bash\ncp -r config/gw_cfg-example config/gw_cfg\n```\n\nAdd a JSON file for each gateway, named after its MAC address without separators (e.g. `F32DEFE72E78.json`) or after its human-readable name (e.g. `station-1.json`). See [Gateway configuration endpoint](#gateway-configuration-endpoint) for the full resolution logic and file format.\n\n4. Generate secure credentials and add them to `config/.env`:\n\n```bash\n# Linux / macOS\necho \"GW_CFG_BEARER_TOKEN=$(openssl rand -hex 32)\" \u003e\u003e config/.env\necho \"HTTP_API_KEY=$(openssl rand -hex 32)\" \u003e\u003e config/.env\necho \"INFLUX_TOKEN=$(openssl rand -hex 32)\" \u003e\u003e config/.env\n\n# PowerShell (Windows)\n[Convert]::ToBase64String((1..32 | ForEach-Object { Get-Random -Max 256 }))\n```\n\n5. Edit `config/.env` and fill in your MQTT, InfluxDB, and MariaDB connection details.\n\n---\n\n## Configuration\n\nAll configuration is done via environment variables in `config/.env`. The entire configuration is validated with Zod at startup — the service will refuse to start and print a clear error message if a required variable is missing or has an invalid format.\n\n### MQTT\n\n| Variable | Default | Description |\n|---|---|---|\n| `MQTT_HOST` | `localhost` | MQTT broker hostname |\n| `MQTT_PORT` | `8883` | MQTT broker port |\n| `MQTT_PROTOCOL` | `mqtt` | Protocol — `mqtt`, `mqtts`, `ws`, `wss` |\n| `MQTT_TOPIC` | `ruuvi/#` | Topic to subscribe to |\n| `MQTT_USERNAME` | — | Optional broker username |\n| `MQTT_PASSWORD` | — | Optional broker password |\n| `MQTT_CA` | — | Path to CA certificate (TLS) |\n| `MQTT_CERT` | — | Path to client certificate (TLS) |\n| `MQTT_KEY` | — | Path to client key (TLS) |\n| `MQTT_REJECT_UNAUTHORIZED` | `true` | Reject invalid TLS certificates |\n\n### InfluxDB\n\n| Variable | Default | Description |\n|---|---|---|\n| `INFLUX_URL` | — | InfluxDB URL — must include the scheme, e.g. `http://localhost:8086` |\n| `INFLUX_TOKEN` | — | InfluxDB API token |\n| `INFLUX_ORG` | — | InfluxDB organisation name |\n| `INFLUX_BUCKET` | — | InfluxDB bucket name |\n\n### MariaDB\n\n| Variable | Default | Description |\n|---|---|---|\n| `MARIA_HOST` | `localhost` | MariaDB hostname |\n| `MARIA_PORT` | `3306` | MariaDB port |\n| `MARIA_USER` | `ruuvi` | Database user |\n| `MARIA_PASSWORD` | — | Database password |\n| `MARIA_DATABASE` | `ruuvi` | Database name |\n\n### Storage backend\n\n| Variable | Default | Description |\n|---|---|---|\n| `STORAGE_BACKEND` | `both` | `influxdb` \\| `mariadb` \\| `both` |\n\n### Buffering\n\n| Variable | Default | Description |\n|---|---|---|\n| `BUFFER_SIZE` | `500` | Max points before an InfluxDB flush is triggered |\n| `MARIA_BUFFER_SIZE` | `100` | Max rows before a MariaDB flush is triggered |\n| `FLUSH_INTERVAL` | `5000` | Periodic flush interval in milliseconds |\n\n### HTTP \u0026 security\n\n| Variable | Default | Description |\n|---|---|---|\n| `HTTP_PORT` | `3002` | Port for the health/metrics HTTP server |\n| `HTTP_API_KEY` | **required** | API key for `/health` and `/metrics` endpoints |\n| `GW_CFG_USER` | `ruuvi-cfg` | Username for Basic auth on `/ruuvi-gw-cfg` |\n| `GW_CFG_PASSWORD` | — | Password for Basic auth on `/ruuvi-gw-cfg` |\n| `GW_CFG_BEARER_TOKEN` | — | Bearer token for `/ruuvi-gw-cfg` (alternative to Basic auth) |\n\n`HTTP_API_KEY`, `GW_CFG_PASSWORD`, and `INFLUX_TOKEN` are independent credentials. Each can be rotated separately without affecting the others.\n\n### Device name mapping\n\nProvide JSON-encoded objects mapping MAC addresses to human-readable labels:\n\n```env\nGATEWAY_NAMES={\"F3:2D:EF:E7:2E:78\":\"Station 1\",\"C8:25:2D:8E:9C:2C\":\"Station 2\"}\nTAG_NAMES={\"CE:52:DE:73:84:F2\":\"Fridge 1\",\"AB:CD:EF:12:34:56\":\"Freezer\"}\n```\nMAC addresses are normalised internally (uppercased, separators stripped), so `F3:2D:EF:E7:2E:78` and `F32DEFE72E78` are treated as identical.\n\nIf a MAC address is not listed, the raw MAC is used as a fallback.\n\n### MariaDB retention \u0026 downsampling\n\n| Variable | Default | Description |\n|---|---|---|\n| `MARIA_RETENTION_ENABLED` | `true` | Enable automatic deletion of old raw data |\n| `MARIA_RETENTION_DAYS` | `60` | Retain raw data for this many days |\n| `MARIA_DOWNSAMPLE_ENABLED` | `true` | Enable hourly aggregation |\n| `MARIA_DOWNSAMPLE_RETENTION_DAYS` | `365` | Retain hourly data for this many days (`0` = forever) |\n| `MARIA_DOWNSAMPLE_DELETE_RAW` | `true` | Delete raw rows once they have been aggregated |\n| `MARIA_MAINTENANCE_INTERVAL_HOURS` | `6` | How often to run maintenance tasks (hours) |\n\n### General\n\n| Variable | Default | Description |\n|---|---|---|\n| `NODE_ENV` | `development` | `development` enables pretty-print logging |\n| `COMPANY_CODE` | `1177` | Ruuvi manufacturer company code (`0x0499`) |\n\n---\n\n## Running the service\n\n### Development\n\nRun with automatic reload on file changes:\n\n```bash\nnpm run dev\n```\n\n### Production\n\nCompile TypeScript and start the service:\n\n```bash\nnpm run build\nnpm start\n```\n\n### Linting \u0026 formatting\n\n```bash\nnpm run lint       # ESLint\nnpm run format     # Prettier\nnpm run typecheck  # TypeScript (no emit)\n```\n\n---\n\n## Testing\n\nThe project uses [Jest](https://jestjs.io/) with `ts-jest`.\n\n```bash\n# Run all tests\nnpm test\n\n# Watch mode — re-run on file changes\nnpm run test:watch\n\n# With coverage report\nnpm run test:coverage\n```\n\n### Test coverage\n\n| Module | Tests |\n|---|---|\n| `ruuviCalculations.ts` | `equilibriumVaporPressure`, `dewPoint`, `frostPoint`, `absoluteHumidity`, `airDensity`, `accelerationTotal`, `vaporPressureDeficit`, `accelerationAngles`, `batteryPercentage` |\n| `messageBuffer.ts` | Buffer accumulation, size-triggered flush, manual flush, empty buffer guard |\n\nAll 24 tests pass. ✅\n\n---\n\n## Data pipeline\n\nEach MQTT message goes through the following stages before being written to the database(s):\n\n1. **Flood protection** — messages larger than 8 KB and non-Ruuvi topics are discarded immediately\n2. **JSON parsing** — raw payload is parsed and validated against the Zod schema\n3. **Device resolution** — gateway and tag MAC addresses are resolved to human-readable names\n4. **Metric mapping** — decoded fields from the gateway payload are mapped to a `RuuviData` object\n5. **Derived metric calculation** — computed fields are added once, shared by both storage backends\n6. **Buffering** — the enriched `RuuviData` object is pushed to the InfluxDB buffer, the MariaDB buffer, or both, depending on `STORAGE_BACKEND`\n7. **Batch write** — buffers are flushed either when they reach their size limit or on the periodic flush interval\n\n---\n\n## Stored metrics\n\n### Raw metrics (from the RuuviTag firmware)\n\n| Field | Unit | Description |\n|---|---|---|\n| `temperature` | °C | Ambient temperature |\n| `humidity` | % | Relative humidity |\n| `pressure` | Pa | Atmospheric pressure |\n| `accelerationX/Y/Z` | g | Acceleration on each axis |\n| `batteryVoltage` | V | Battery voltage |\n| `txPower` | dBm | Transmit power |\n| `movementCounter` | — | Cumulative movement count |\n| `measurementSequenceNumber` | — | Packet sequence counter |\n| `dataFormat` | — | RuuviTag data format version |\n| `rssi` | dBm | Received signal strength |\n\n### Derived metrics (calculated by this service)\n\n| Field | Unit | Description |\n|---|---|---|\n| `dewPoint` | °C | Temperature at which condensation forms |\n| `frostPoint` | °C | Freezing point — more accurate than dew point below 0 °C |\n| `absoluteHumidity` | g/m³ | Mass of water vapour per unit volume of air |\n| `equilibriumVaporPressure` | Pa | Saturation vapour pressure (Magnus formula) |\n| `vaporPressureDeficit` | kPa | Key indicator for greenhouse horticulture (ideal: 0.8–1.2 kPa) |\n| `airDensity` | kg/m³ | Density of humid air |\n| `accelerationTotal` | g | Magnitude of the acceleration vector |\n| `accelerationAngleFromX/Y/Z` | ° | Tilt angles from each axis |\n| `batteryPercentage` | % | Estimated battery level (CR2477 discharge curve) |\n\n---\n\n## MariaDB retention \u0026 downsampling\n\nWhen `STORAGE_BACKEND` is `mariadb` or `both`, two optional maintenance tasks run on a configurable schedule.\n\n### Retention\n\nDeletes rows from the `measurements` table older than `MARIA_RETENTION_DAYS`. Deletions are capped at 5 000 rows per run to avoid locking the table on large datasets.\n\n### Downsampling\n\nAggregates completed hours from `measurements` into the `measurements_hourly` table. Each hourly row contains:\n\n- `AVG` for all continuous metrics (temperature, humidity, pressure, etc.)\n- `MIN` / `MAX` for temperature and humidity — useful for cold-chain compliance\n- `movement_counter_delta` — number of movements recorded during the hour\n\nOnce a raw hour has been successfully aggregated, raw rows for that hour can optionally be deleted (`MARIA_DOWNSAMPLE_DELETE_RAW=true`). The current hour is never aggregated — only complete hours are processed.\n\n### Resulting data lifecycle example\n\n```\nDay 0  → 60    Raw data in measurements            (30s resolution)\nDay 1  → 365   Hourly averages in measurements_hourly\nDay 365+        Hourly data deleted if DOWNSAMPLE_RETENTION_DAYS=365\n```\n\n---\n\n## HTTP endpoints\n\nThe service exposes a Fastify HTTP server protected by an API key and rate limiting (100 requests/minute per IP).\n\n`/health` and `/metrics` require the header:\n\n```\nx-api-key: \u003cHTTP_API_KEY\u003e\n```\n\n| Method | Path | Auth | Description |\n|---|---|---|---|\n| `GET` | `/health` | `x-api-key` | Returns `{\"status\":\"ok\"}` |\n| `GET` | `/metrics` | `x-api-key` | Prometheus metrics |\n| `GET` | `/ruuvi-gw-cfg` | Basic / Bearer | Gateway config (dynamic) |\n| `GET` | `/ruuvi-gw-cfg/*` | Basic / Bearer | Gateway config (file-based path) |\n\n### Quick check\n\n```bash\ncurl -H \"x-api-key: your-api-key\" http://localhost:3002/health\n# {\"status\":\"ok\"}\n\ncurl -H \"x-api-key: your-api-key\" http://localhost:3002/metrics\n# Prometheus text format\n```\n\n---\n\n## Gateway configuration endpoint\n\nThe service exposes a `/ruuvi-gw-cfg` endpoint that allows each Ruuvi Gateway to download its own configuration automatically via the **Automatic Configuration Download** feature in the gateway web UI.\n\n### How it works\n\nWhen the gateway calls the endpoint, it sends its MAC address in the `ruuvi_gw_mac` header (format `XX:XX:XX:XX:XX:XX`). The service resolves which config file to serve using the following priority:\n\n1. **Name-based** — looks up the MAC in `GATEWAY_NAMES`, converts the label to a slug, and serves `config/gw_cfg/\u003cslug\u003e.json` (e.g. `station-1.json`)\n2. **MAC-based** — serves `config/gw_cfg/\u003cMAC\u003e.json` (e.g. `F32DEFE72E78.json`)\n3. **Default fallback** — serves `config/gw_cfg/gw_cfg.json`\n\nThe served file is validated against the official [Ruuvi Gateway JSON schema](https://docs.ruuvi.com/ruuvi-gateway-firmware/data-formats/gateway-configuration) before being returned.\n\n### Config file directory\n\n```\nconfig/\n└── gw_cfg/\n    ├── station-1.json       ← resolved when GATEWAY_NAMES maps MAC → \"Station 1\"\n    ├── F32DEFE72E78.json    ← resolved by MAC directly (alternative naming)\n    └── gw_cfg.json          ← default fallback for unrecognised gateways\n```\n\n### Authentication\n\nThe endpoint supports both Basic and Bearer authentication, configured via `GW_CFG_USER` / `GW_CFG_PASSWORD` and `GW_CFG_BEARER_TOKEN` in `config/.env`.\n\n### Configuring the gateway\n\nIn the gateway web UI, go to **Advanced settings → Automatic Configuration Download** and set:\n\n| Field | Value |\n|---|---|\n| URL | `https://ruuvi-station.ecodev.ch/ruuvi-gw-cfg` |\n| Auth type | `Basic` (or `Bearer`) |\n| Username | value of `GW_CFG_USER` |\n| Password | value of `GW_CFG_PASSWORD` |\n| Refresh interval | `60` minutes (or as needed) |\n\nThe gateway will first try `GET /ruuvi-gw-cfg/F32DEFE72E78.json` then fall back to `GET /ruuvi-gw-cfg/gw_cfg.json` — both paths are handled by the same endpoint.\n\n### The config file must include at minimum\n\n```json\n{\n  \"remote_cfg_use\": true,\n  \"remote_cfg_url\": \"https://your-cfg-url.com/ruuvi-gw-cfg\",\n  \"remote_cfg_refresh_interval_minutes\": 60,\n  \"remote_cfg_auth_type\": \"basic\",\n  \"remote_cfg_auth_basic_user\": \"\u003cGW_CFG_USER\u003e\",\n  \"remote_cfg_auth_basic_pass\": \"\u003cGW_CFG_PASSWORD\u003e\"\n}\n```\n\nAll other attributes not present in the file retain their previous value on the gateway.\n\n### Test the endpoint manually\n\n```bash\n# With Basic auth + MAC header\ncurl -u your-user:your-password \\\n  -H \"ruuvi_gw_mac: F3:2D:EF:E7:2E:78\" \\\n  https://your-cfg-url.com/ruuvi-gw-cfg\n\n# Simulating what the gateway firmware does\ncurl -u your-user:your-password \\\n  https://your-cfg-url.com/ruuvi-gw-cfg/F32DEFE72E78.json\n```\n\n### Adding a new gateway\n\n1. Add the MAC → name mapping in `config/.env`:\n   ```env\n   GATEWAY_NAMES={\"F3:2D:EF:E7:2E:78\":\"Station 1\",\"C8:25:2D:8E:9C:2C\":\"Station 2\"}\n   ```\n2. Create `config/gw_cfg/station-2.json` with the new gateway's settings\n3. In the new gateway's web UI, set the same URL: `https://ruuvi-station.ecodev.ch/ruuvi-gw-cfg`\n4. Restart the service to reload the config — no nginx changes needed\n---\n\n## Monitoring metrics\n\nThe `/metrics` endpoint exposes the following custom metrics in addition to the default Node.js process metrics:\n\n| Metric | Type | Description |\n|---|---|---|\n| `ruuvi_mqtt_connected` | Gauge | MQTT connection status (`1` = connected, `0` = disconnected) |\n| `ruuvi_mqtt_messages_processed_total` | Counter | Total MQTT messages processed successfully |\n| `ruuvi_mqtt_messages_invalid_total` | Counter | Total invalid MQTT messages rejected by Zod |\n| `ruuvi_buffer_size{type=\"influx\"}` | Gauge | Current InfluxDB buffer fill level |\n| `ruuvi_buffer_size{type=\"maria\"}` | Gauge | Current MariaDB buffer fill level |\n\n### Scraping with Prometheus\n\n```yaml\n# prometheus.yml\nglobal:\n  scrape_interval: 15s\n\nscrape_configs:\n  - job_name: 'ruuvi'\n    static_configs:\n      - targets: ['localhost:3002']\n    metrics_path: '/metrics'\n    params:\n      x-api-key: ['your-api-key']\n```\n\n### Useful PromQL queries\n\n```\n# MQTT connection status\nruuvi_mqtt_connected\n\n# Message throughput (per minute)\nrate(ruuvi_mqtt_messages_processed_total[1m])\n\n# Error rate\nrate(ruuvi_mqtt_messages_invalid_total[1m])\n\n# Buffer utilization\nruuvi_buffer_size\n```\n\n### Grafana panels\n\n| Panel | Query |\n|---|---|\n| MQTT status | `ruuvi_mqtt_connected` |\n| Messages/min | `rate(ruuvi_mqtt_messages_processed_total[1m])` |\n| Error rate | `rate(ruuvi_mqtt_messages_invalid_total[1m])` |\n| Buffer level | `ruuvi_buffer_size` |\n\n---\n\n## Nginx reverse proxy\n\nThe service is designed to run behind an nginx reverse proxy that exposes it publicly over HTTPS.\n\n### Reference configuration\n\n```nginx\n# Required for Grafana Live WebSocket connections\nmap $http_upgrade $connection_upgrade {\n  default upgrade;\n  '' close;\n}\n\nupstream grafana {\n  server your-server-ip:3000;\n}\n\nupstream ruuvi_ingestion {\n  server your-server-ip:3002;\n}\n\n# HTTP → HTTPS redirect\nserver {\n  listen 80;\n  server_name your-domain.com;\n\n  location /.well-known/acme-challenge/ {\n    root /var/www/html;\n  }\n\n  location / {\n    return 301 https://$host$request_uri;\n  }\n}\n\nserver {\n  listen 443 ssl;\n  server_name your-domain.com;\n\n  ssl_certificate     /etc/letsencrypt/live/your-domain.com/fullchain.pem;\n  ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;\n\n  # Required to forward headers containing underscores (ruuvi_gw_mac)\n  underscores_in_headers on;\n\n  # Grafana dashboard\n  location / {\n    proxy_http_version 1.1;\n    proxy_pass         http://grafana;\n    proxy_set_header   Host              $http_host;\n    proxy_set_header   X-Real-IP         $remote_addr;\n    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;\n    proxy_set_header   X-Forwarded-Proto $scheme;\n    proxy_set_header   Upgrade           $http_upgrade;\n    proxy_set_header   Connection        upgrade;\n  }\n\n  # Grafana Live WebSocket\n  location /grafana/api/live/ {\n    proxy_http_version 1.1;\n    proxy_pass         http://grafana;\n    proxy_set_header   Upgrade    $http_upgrade;\n    proxy_set_header   Connection $connection_upgrade;\n    proxy_set_header   Host       $host;\n  }\n\n  # ruuvi-iot-ingestion — health, metrics, and gateway config endpoint\n  # The regex captures /ruuvi-gw-cfg/* subpaths used by the gateway firmware\n  location ~ ^/(health|metrics|ruuvi-gw-cfg.*) {\n    proxy_pass         http://ruuvi_ingestion;\n    proxy_http_version 1.1;\n    proxy_set_header   Host              $host;\n    proxy_set_header   X-Real-IP         $remote_addr;\n    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;\n    proxy_set_header   X-Forwarded-Proto $scheme;\n    proxy_pass_header  Authorization;\n    proxy_set_header   ruuvi_gw_mac      $http_ruuvi_gw_mac;\n  }\n}\n```\n\n### Key points\n\n- `underscores_in_headers on` is required — nginx silently drops headers containing underscores by default, which would prevent the `ruuvi_gw_mac` header from reaching the service\n- The `ruuvi-gw-cfg.*` regex captures both `/ruuvi-gw-cfg` and `/ruuvi-gw-cfg/F32DEFE72E78.json` (the path the gateway firmware constructs automatically)\n- MQTT (port 8883) is handled directly by Mosquitto on the server and does not need to be proxied through nginx\n\n### MQTT over TLS\n\nThe Ruuvi Gateway connects directly to Mosquitto on `your-server-ip:8883` using mutual TLS. If you want the gateway to use a hostname instead of an IP address, add a `stream` block to nginx on the host machine:\n\n```nginx\nstream {\n  upstream mosquitto {\n    server your-server-ip:8883;\n  }\n\n  server {\n    listen      8883;\n    proxy_pass  mosquitto;\n    # TLS passthrough — Mosquitto handles mTLS directly\n    proxy_timeout         600s;\n    proxy_connect_timeout 5s;\n  }\n}\n```\n\nThen set `mqtt_server: your-domain.com` in the gateway config file.\n\n---\n\n## Docker\n\nThe project ships with a multi-stage `Dockerfile` and a `docker-compose.yml` that starts the full stack.\n\n### Full stack with Docker Compose (recommended)\n\n```bash\n# Start all services\ndocker compose up -d\n\n# Check status\ndocker compose ps\n\n# Follow logs\ndocker compose logs -f ruuvi-ingestion\n\n# Stop\ndocker compose down\n\n# Stop and remove volumes (deletes all data)\ndocker compose down -v\n```\n\n### Services exposed\n\n| Service | Port | Description |\n|---|---|---|\n| `ruuvi-iot-ingestion` | `3002` | Health, metrics, and gateway config HTTP server |\n| `mosquitto` | `1883`, `9001` | MQTT broker |\n| `influxdb` | `8086` | InfluxDB UI \u0026 API |\n| `mariadb` | `3306` | MariaDB |\n\n### Build and run the image standalone\n\n```bash\ndocker build -t ruuvi-iot-ingestion:latest .\n\ndocker run -d \\\n  --name ruuvi \\\n  -p 3002:3002 \\\n  -v $(pwd)/config:/app/config \\\n  -e MQTT_HOST=mosquitto \\\n  -e MQTT_PORT=1883 \\\n  -e STORAGE_BACKEND=both \\\n  -e INFLUX_URL=http://influxdb:8086 \\\n  -e INFLUX_ORG=myorg \\\n  -e INFLUX_BUCKET=ruuvi \\\n  -e INFLUX_TOKEN=mytoken \\\n  -e MARIA_HOST=mariadb \\\n  -e MARIA_USER=ruuvi \\\n  -e MARIA_PASSWORD=ruuvi_pass \\\n  -e HTTP_API_KEY=your-secure-key \\\n  -e GW_CFG_USER=ecoadmin \\\n  -e GW_CFG_PASSWORD=your-gw-password \\\n  --network ruuvi-network \\\n  ruuvi-iot-ingestion:latest\n```\n\n\u003e **Note:** Mount `config/` as a volume so the service can read the gateway config files. The `docker-compose.yml` uses example credentials — replace all passwords, tokens, and API keys before deploying to production.\n\n---\n\n## Production deployment\n\n### Checklist\n\n- [ ] All secrets stored in a secrets manager or environment injection (never committed to git)\n- [ ] `HTTP_API_KEY` generated with `openssl rand -hex 32`\n- [ ] `GW_CFG_BEARER_TOKEN` generated with `openssl rand -hex 32`\n- [ ] TLS configured for MQTT (`mqtts`) and InfluxDB (`https`)\n- [ ] `MQTT_REJECT_UNAUTHORIZED=true`\n- [ ] `config/gw_cfg/` contains a JSON file for each gateway\n- [ ] Database backups scheduled\n- [ ] Monitoring scraping and Grafana alerts configured\n- [ ] Log forwarding to a centralized service (Loki, Datadog, etc.)\n- [ ] Health check endpoint monitored by an uptime service\n- [ ] Rollback plan documented\n\n### Performance tuning for high load\n\n```env\nBUFFER_SIZE=2000\nMARIA_BUFFER_SIZE=1000\nFLUSH_INTERVAL=3000\n```\n\n### Kubernetes example\n\n```yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: ruuvi-iot-ingestion\nspec:\n  replicas: 2\n  selector:\n    matchLabels:\n      app: ruuvi-iot-ingestion\n  template:\n    metadata:\n      labels:\n        app: ruuvi-iot-ingestion\n    spec:\n      containers:\n        - name: ruuvi-iot-ingestion\n          image: your-registry/ruuvi-iot-ingestion:latest\n          imagePullPolicy: Always\n          ports:\n            - containerPort: 3002\n          volumeMounts:\n            - name: gw-cfg\n              mountPath: /app/config/gw_cfg\n              readOnly: true\n          env:\n            - name: MQTT_HOST\n              valueFrom:\n                configMapKeyRef:\n                  name: ruuvi-config\n                  key: mqtt_host\n            - name: HTTP_API_KEY\n              valueFrom:\n                secretKeyRef:\n                  name: ruuvi-secrets\n                  key: api_key\n            - name: GW_CFG_PASSWORD\n              valueFrom:\n                secretKeyRef:\n                  name: ruuvi-secrets\n                  key: gw_cfg_password\n          livenessProbe:\n            httpGet:\n              path: /health\n              port: 3002\n              httpHeaders:\n                - name: x-api-key\n                  value: $(HTTP_API_KEY)\n            initialDelaySeconds: 10\n            periodSeconds: 30\n          readinessProbe:\n            httpGet:\n              path: /health\n              port: 3002\n              httpHeaders:\n                - name: x-api-key\n                  value: $(HTTP_API_KEY)\n            initialDelaySeconds: 5\n            periodSeconds: 10\n          resources:\n            requests:\n              memory: \"256Mi\"\n              cpu: \"250m\"\n            limits:\n              memory: \"512Mi\"\n              cpu: \"500m\"\n      volumes:\n        - name: gw-cfg\n          configMap:\n            name: ruuvi-gw-configs\n```\n\n---\n\n## Project structure\n\n```\n.\n├── Dockerfile\n├── docker-compose.yml\n├── config/\n│   ├── .env                       Environment variables (gitignored)\n│   ├── .env.example               Template for .env\n│   ├── gw_cfg/                    Gateway config files (gitignored)\n│   │   ├── station-1.json         Config for gateway \"Station 1\"\n│   │   └── gw_cfg.json            Default fallback config\n│   └── gw_cfg-example/            Example gateway configs (committed)\n│       └── gw_cfg.json\n├── schema/\n│   ├── mariadb_init.sql           MariaDB schema (used by Docker entrypoint)\n│   └── ruuvi_mqtt_data_with_timestamps.schema.json\n└── src/\n    ├── index.ts                   Entry point\n    ├── config/\n    │   └── env.ts                 Environment variable parsing \u0026 Zod validation\n    ├── http/\n    │   └── healthServer.ts        Fastify server — /health, /metrics, /ruuvi-gw-cfg\n    ├── influx-db/\n    │   └── influxDbService.ts     InfluxDB write client\n    ├── logger/\n    │   └── logger.ts              Pino logger\n    ├── maria-db/\n    │   ├── mariaDbService.ts      MariaDB pool, schema init, batch writes\n    │   └── mariaDbRetention.ts    Retention \u0026 downsampling tasks\n    ├── mqtt/\n    │   └── mqttService.ts         MQTT client, message processing, Monitoring metrics\n    ├── pipeline/\n    │   └── messageBuffer.ts       Generic batch buffer with monitoring gauge\n    ├── ruuvi/\n    │   ├── ruuviCalculations.ts   Derived metric formulas\n    │   ├── ruuviData.ts           RuuviData model class\n    │   ├── ruuviDecoder.ts        BLE manufacturer data decoder\n    │   ├── ruuviMqttDataWithTimestampsSchema.ts  Zod schema for MQTT payloads\n    │   └── gatewayConfigurationSchema.ts         Zod schema for gateway config files\n    ├── tests/\n    │   ├── messageBuffer.test.ts\n    │   └── ruuviCalculations.test.ts\n    └── types/\n        └── advlib-ble-manufacturers.d.ts\n```\n\n---\n\n## Logging\n\n- Logs are generated with `pino`\n- Key events (MQTT connection, batch writes, maintenance tasks, gateway config requests) are logged at `info` level\n- Decode errors and invalid payloads are logged as `warn`\n- In development (`NODE_ENV=development`), logs are pretty-printed via `pino-pretty`\n- In production, logs are output as JSON and can be forwarded to centralized services (Loki, Datadog, ELK, etc.)\n\n```bash\n# Follow logs in Docker\ndocker compose logs -f ruuvi-ingestion\n\n# Filter gateway config requests\ndocker compose logs -f ruuvi-ingestion | grep \"GW cfg\"\n\n# Kubernetes\nkubectl logs -f deployment/ruuvi-iot-ingestion\n```\n\n---\n\n## Optimizations\n\n- Increase `BUFFER_SIZE` to reduce the number of InfluxDB write operations under high message volume\n- Adjust `FLUSH_INTERVAL` to match your tag reporting interval (e.g. `30000` for 30 s reporting)\n- Use a local MQTT broker (Mosquitto is included in `docker-compose.yml`) to reduce network latency\n- Set `STORAGE_BACKEND=influxdb` if you do not need SQL exports, to skip unnecessary MariaDB writes\n- Set `MARIA_DOWNSAMPLE_DELETE_RAW=true` to keep the `measurements` table small and fast\n\n---\n\n## Troubleshooting\n\n### `HTTP_API_KEY` not set\n\nThe service refuses to start if `HTTP_API_KEY` is missing. Generate one and add it to `config/.env`:\n\n```bash\nopenssl rand -hex 32\n```\n\n### `INFLUX_URL` invalid format\n\nThe URL must include the scheme:\n\n```env\n# correct\nINFLUX_URL=http://localhost:8086\n\n# incorrect — missing http://\nINFLUX_URL=localhost:8086\n```\n\n### Gateway config returns 502\n\nnginx is receiving the request but cannot reach the service. Check in order:\n\n```bash\n# 1. Verify the service is listening\ncurl http://localhost:3002/health -H \"x-api-key: your-key\"\n\n# 2. Check nginx error logs\nsudo tail -50 /var/log/nginx/error.log | grep ruuvi-gw-cfg\n\n# 3. Confirm underscores_in_headers is on (required for ruuvi_gw_mac header)\ngrep underscores_in_headers /etc/nginx/sites-enabled/*\n\n# 4. Verify the nginx regex captures subpaths\n# location ~ ^/(health|metrics|ruuvi-gw-cfg.*) — note the .* after ruuvi-gw-cfg\n```\n\n### Gateway config returns 404\n\nThe gateway firmware requests `/ruuvi-gw-cfg/F32DEFE72E78.json` — the `.*` in the nginx regex and the `fastify.get('/ruuvi-gw-cfg/*', ...)` route must both be present to handle this path.\n\n### MQTT connection fails\n\n```bash\n# Check the broker is running\ndocker compose logs mosquitto\n\n# For local dev, MQTT_HOST should be localhost\n# Inside Docker Compose, MQTT_HOST should be mosquitto\n```\n\n### No data in the databases\n\n```bash\n# Check service logs for errors\ndocker compose logs ruuvi-ingestion\n\n# Verify MQTT messages are arriving\ndocker compose logs mosquitto\n\n# Check buffer metrics\ncurl -H \"x-api-key: your-key\" http://localhost:3002/metrics | grep ruuvi_buffer\n```\n\n### Real-time metrics monitoring\n\n```bash\nwatch 'curl -s -H \"x-api-key: your-key\" http://localhost:3002/metrics | grep ruuvi'\n```\n\n---\n\n## License\n\nMIT License — open source project\n\n## Author\n\n[Hazhar Galeh](https://github.com/hazhargaleh)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fecodev%2Fruuvi-iot-ingestion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fecodev%2Fruuvi-iot-ingestion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fecodev%2Fruuvi-iot-ingestion/lists"}