{"id":41223469,"url":"https://github.com/solectrus/ingest","last_synced_at":"2026-01-23T00:26:03.142Z","repository":{"id":285351260,"uuid":"952923796","full_name":"solectrus/ingest","owner":"solectrus","description":"Lightweight InfluxDB ingestion proxy with buffering and house power calculation","archived":false,"fork":false,"pushed_at":"2026-01-10T03:48:14.000Z","size":655,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2026-01-11T01:06:31.422Z","etag":null,"topics":["influxdb"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/solectrus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-22T06:44:53.000Z","updated_at":"2026-01-10T03:48:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"917e8aa9-5d06-4d53-829c-6efb0d94e6be","html_url":"https://github.com/solectrus/ingest","commit_stats":null,"previous_names":["solectrus/ingest"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/solectrus/ingest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solectrus%2Fingest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solectrus%2Fingest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solectrus%2Fingest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solectrus%2Fingest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/solectrus","download_url":"https://codeload.github.com/solectrus/ingest/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solectrus%2Fingest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28676011,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T20:48:19.482Z","status":"ssl_error","status_checked_at":"2026-01-22T20:48:14.968Z","response_time":144,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["influxdb"],"created_at":"2026-01-23T00:26:02.470Z","updated_at":"2026-01-23T00:26:03.107Z","avatar_url":"https://github.com/solectrus.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Continuous integration](https://github.com/solectrus/ingest/actions/workflows/ci.yml/badge.svg)](https://github.com/solectrus/ingest/actions/workflows/ci.yml)\n[![Maintainability](https://qlty.sh/badges/240ebaa0-dce5-4849-9fd9-ea17d71fc316/maintainability.svg)](https://qlty.sh/gh/solectrus/projects/ingest)\n[![wakatime](https://wakatime.com/badge/user/697af4f5-617a-446d-ba58-407e7f3e0243/project/3d478bcc-754c-4d6b-9a2d-fe70bf9eea9f.svg)](https://wakatime.com/badge/user/697af4f5-617a-446d-ba58-407e7f3e0243/project/3d478bcc-754c-4d6b-9a2d-fe70bf9eea9f)\n[![Code Coverage](https://qlty.sh/badges/240ebaa0-dce5-4849-9fd9-ea17d71fc316/coverage.svg)](https://qlty.sh/gh/solectrus/projects/ingest)\n\n# SOLECTRUS Ingest\n\nLightweight InfluxDB ingestion proxy with **buffering** and **house power calculation**.\n\n## Features\n\n- Accepts InfluxDB v2 Line Protocol via HTTP\n- Forwards all incoming data to InfluxDB\n- Recalculates house power and replaces incoming value\n- Buffers all incoming and outgoing data to SQLite\n- Exposes a statistics endpoint\n\n## Architecture\n\n### Without Ingest\n\n```mermaid\ngraph LR\n  CollectorA[SENEC-Collector]\n  CollectorB[Shelly-Collector]\n  CollectorC[MQTT-Collector]\n  Influx[InfluxDB]\n  Dashboard[Dashboard]\n\n  CollectorA --\u003e|push| Influx\n  CollectorB --\u003e|push| Influx\n  CollectorC --\u003e|push| Influx\n  Influx --\u003e|pull| Dashboard\n```\n\n### With Ingest\n\n```mermaid\ngraph LR\n  CollectorA[SENEC-Collector]\n  CollectorB[Shelly-Collector]\n  CollectorC[MQTT-Collector]\n  Influx[InfluxDB]\n  Ingest[Ingest]\n  Dashboard[Dashboard]\n\n  CollectorA --\u003e|push| Ingest\n  CollectorB --\u003e|push| Ingest\n  CollectorC --\u003e|push| Ingest\n  Ingest --\u003e|push| Influx\n  Influx --\u003e|pull| Dashboard\n```\n\n## House Power Calculation\n\nWhen using a balcony inverter, your house power value might be incorrect. Ingest recalculates the value using this formula:\n\n```\nHOUSE_POWER = INVERTER_POWER (total, including balcony inverter)\n            + GRID_IMPORT_POWER\n            + BATTERY_DISCHARGING_POWER\n            - BATTERY_CHARGING_POWER\n            - GRID_EXPORT_POWER\n            - WALLBOX_POWER\n            - HEATPUMP_POWER\n```\n\nSince the relevant sensor values may not arrive simultaneously, Ingest uses **interpolation** to align them. Whenever one of the relevant sensors updates, Ingest recalculates the house power based on that timestamp.\n\nThe calculated value replaces the original one. If you prefer to store the original value separately, you can define `INFLUX_SENSOR_HOUSE_POWER_CALCULATED` to write the result to a different measurement and/or field.\n\n## Example Docker Compose\n\n```yaml\nservices:\n  ingest:\n    image: ghcr.io/solectrus/ingest:latest\n    environment:\n      - INFLUX_SENSOR_INVERTER_POWER\n      - INFLUX_SENSOR_INVERTER_POWER_1\n      - INFLUX_SENSOR_INVERTER_POWER_2\n      - INFLUX_SENSOR_INVERTER_POWER_3\n      - INFLUX_SENSOR_INVERTER_POWER_4\n      - INFLUX_SENSOR_INVERTER_POWER_5\n      - INFLUX_SENSOR_GRID_IMPORT_POWER\n      - INFLUX_SENSOR_GRID_EXPORT_POWER\n      - INFLUX_SENSOR_BATTERY_DISCHARGING_POWER\n      - INFLUX_SENSOR_BATTERY_CHARGING_POWER\n      - INFLUX_SENSOR_WALLBOX_POWER\n      - INFLUX_SENSOR_HEATPUMP_POWER\n      - INFLUX_SENSOR_HOUSE_POWER\n      - INFLUX_EXCLUDE_FROM_HOUSE_POWER\n      - INFLUX_SENSOR_HOUSE_POWER_CALCULATED\n      - INFLUX_HOST\n      - INFLUX_PORT\n      - INFLUX_SCHEMA\n      - STATS_PASSWORD\n    depends_on:\n      - influxdb\n    ports:\n      - 4567:4567\n    volumes:\n      - ./path/to/ingest-data:/app/data\n\n  influxdb:\n    image: influxdb:2.7-alpine\n    ports:\n      - 8086:8086\n    volumes:\n      - ./path/to/influx-data:/var/lib/influxdb2\n```\n\n## Environment Variables\n\n### Sensor Configuration\n\nDefine measurement and field names for each sensor. Format: `measurement:field`, e.g. `SENEC:inverter_power`. Leave empty if a sensor is not available.\n\n| Variable                                  | Description               |\n| ----------------------------------------- | ------------------------- |\n| `INFLUX_SENSOR_INVERTER_POWER`            | Inverter power (total)    |\n| `INFLUX_SENSOR_INVERTER_POWER_1`          | Inverter power (1)        |\n| `INFLUX_SENSOR_INVERTER_POWER_2`          | Inverter power (2)        |\n| `INFLUX_SENSOR_INVERTER_POWER_3`          | Inverter power (3)        |\n| `INFLUX_SENSOR_INVERTER_POWER_4`          | Inverter power (4)        |\n| `INFLUX_SENSOR_INVERTER_POWER_5`          | Inverter power (5)        |\n| `INFLUX_SENSOR_GRID_IMPORT_POWER`         | Grid import power         |\n| `INFLUX_SENSOR_GRID_EXPORT_POWER`         | Grid export power         |\n| `INFLUX_SENSOR_BATTERY_DISCHARGING_POWER` | Battery discharging power |\n| `INFLUX_SENSOR_BATTERY_CHARGING_POWER`    | Battery charging power    |\n| `INFLUX_SENSOR_WALLBOX_POWER`             | Wallbox power             |\n| `INFLUX_SENSOR_HEATPUMP_POWER`            | Heat pump power           |\n| `INFLUX_SENSOR_HOUSE_POWER`               | House power               |\n\nThe inverter power is calculated as follows:\n\n1. If `INFLUX_SENSOR_INVERTER_POWER` power is given:\n\n```\nTotal inverter power = INFLUX_SENSOR_INVERTER_POWER\n```\n\n2. Otherwise, if `INFLUX_SENSOR_INVERTER_POWER` is **not** given:\n\n```\nTotal inverter power = INFLUX_SENSOR_INVERTER_POWER_1 +\n                       INFLUX_SENSOR_INVERTER_POWER_2 +\n                       INFLUX_SENSOR_INVERTER_POWER_3 +\n                       INFLUX_SENSOR_INVERTER_POWER_4 +\n                       INFLUX_SENSOR_INVERTER_POWER_5\n```\n\n### Other Settings\n\n| Variable                               | Description                               | Note            |\n| -------------------------------------- | ----------------------------------------- | --------------- |\n| `INFLUX_EXCLUDE_FROM_HOUSE_POWER`      | Exclude specific sensors from house power | Optional        |\n| `INFLUX_SENSOR_HOUSE_POWER_CALCULATED` | Output for calculated house power         | Optional        |\n| `INFLUX_HOST`                          | InfluxDB host, e.g. `influxdb`            | Required        |\n| `INFLUX_PORT`                          | InfluxDB port                             | Default: `8086` |\n| `INFLUX_SCHEMA`                        | InfluxDB schema                           | Default: `http` |\n| `RETENTION_HOURS`                      | SQLite retention period in hours          | Default: `12`   |\n| `STATS_PASSWORD`                       | Password for stats endpoint               | Optional        |\n\n## Endpoints\n\n### POST `/api/v2/write`\n\nStores and forwards incoming Line Protocol data to InfluxDB. Triggers recalculation of house power if relevant.\n\n### GET `/`\n\nDisplays a basic stats page (requires password if configured), showing throughput, queue size, buffer status, etc.\n\n### GET `/health`\n\nReturns JSON with HTTP 200 if the service is running (useful for detailed health checks).\n\n### GET `/ping`\n\nReturns just HTTP 204 if the service is running (useful for health checks).\n\n## Example cURL\n\n```bash\ncurl -X POST \"http://localhost:4567/api/v2/write?bucket=my-bucket\u0026org=my-org\u0026precision=ns\" \\\n  -H \"Authorization: Token my-token\" \\\n  -H \"Content-Type: application/json\" \\\n  --data-raw \"test_measurement,location=office value=42i $(( $(date +%s) * 1000000000 ))\"\n```\n\n## How it works\n\n- Incoming data is **persisted** in SQLite (configurable retention, default: 12 hours)\n- A queue forwards data to InfluxDB in **batches**\n- House power is recalculated **as soon as** any relevant sensor updates\n- Old data is removed periodically by a background cleanup worker\n\nIf anything goes wrong, please look at the logs!\n\n## FAQ\n\n### Should I use Ingest?\n\nOnly if you use a balcony inverter **and** your house power values are inaccurate. If you're using a single inverter, Ingest won't add value (except heating your Raspberry Pi).\n\n### Should I use Ingest for all collectors?\n\nUse Ingest only for collectors that send **any of the eight relevant sensors**.\n\nAll other collectors (e.g., [Tibber-Collector](https://github.com/solectrus/tibber-collector), [Forecast-Collector](https://github.com/solectrus/forecast-collector)) should send data **directly** to InfluxDB for better performance and simplicity.\n\n### Why not use Telegraf?\n\n[Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) is an **agent** for data collection. Ingest is a **proxy** that processes and forwards data. This makes it easier to reuse existing collectors by simply changing the destination URL.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolectrus%2Fingest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsolectrus%2Fingest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolectrus%2Fingest/lists"}