{"id":17568740,"url":"https://github.com/alberanid/pve2otelcol","last_synced_at":"2025-08-22T08:31:03.868Z","repository":{"id":258779669,"uuid":"871601591","full_name":"alberanid/pve2otelcol","owner":"alberanid","description":"collect logs from a Proxmox node and its LXC containers and send them to a OpenTelementry collector like Grafana Alloy / Loki","archived":false,"fork":false,"pushed_at":"2024-12-19T15:16:33.000Z","size":288,"stargazers_count":2,"open_issues_count":5,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-19T16:26:17.929Z","etag":null,"topics":["journald","logging","logs","lxc","lxc-containers","opentelemetry","proxmox","pve","systemd"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alberanid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-12T12:21:00.000Z","updated_at":"2024-12-19T15:27:10.000Z","dependencies_parsed_at":"2024-11-20T19:32:20.547Z","dependency_job_id":"1bc1d181-0e64-446c-ac91-5d29bee6ed71","html_url":"https://github.com/alberanid/pve2otelcol","commit_stats":{"total_commits":40,"total_committers":3,"mean_commits":"13.333333333333334","dds":"0.050000000000000044","last_synced_commit":"4304163cb505d0456a7692310f5ea43f1f54c6d3"},"previous_names":["alberanid/pve2otelcol"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alberanid%2Fpve2otelcol","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alberanid%2Fpve2otelcol/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alberanid%2Fpve2otelcol/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alberanid%2Fpve2otelcol/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alberanid","download_url":"https://codeload.github.com/alberanid/pve2otelcol/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230575835,"owners_count":18247483,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["journald","logging","logs","lxc","lxc-containers","opentelemetry","proxmox","pve","systemd"],"created_at":"2024-10-21T17:05:47.845Z","updated_at":"2024-12-20T10:46:58.114Z","avatar_url":"https://github.com/alberanid.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pve2otelcol\n\n**pve2otelcol** is a program to monitor the VMs running on a [Proxmox Virtual Environment](https://www.proxmox.com/) (*PVE*) node, collecting their logs and sending them to an [OpenTelemetry collector](https://opentelemetry.io/), trying to be as little intrusive as possible.\n\nThis means that no agent is needed on individual VMs: logs are collected by a program running on the PVE node itself.\n\n**pve2otelcol** can monitor the [journald](https://www.freedesktop.org/software/systemd/man/latest/systemd-journald.service.html) logs of the PVE node and of each running VM, periodically monitoring them for start and stop events.\n\nThe logs, connected in JSON format, are parsed and sent to the OpenTelemetry collector where they can be easily routed, parsed, filtered, inspected and visualized directly in Grafana.\n\n## Disclaimer and limitations\n\nThis software is in alpha state; ideas for improvements can be [discussed on Github](https://github.com/alberanid/pve2otelcol/discussions); in the same way, any [bug report](https://github.com/alberanid/pve2otelcol/issues) and pull request is welcome.\n\nAt the moment **it can't monitor Qemu/KVM virtual machines**, since running `qm exec VMID -- journalctl --follow` produces no output to be parsed (it's not a stream like the `pct exec VMID -- journalctl --follow` that's used to monitor LXC containers).\n\n## Building it\n\nJust run:\n\n```sh\ngo build .\n```\n\n## Run it\n\nCopy it to a PVE node and run:\n\n```sh\n./pve2otelcol --verbose --otlp-grpc-url http://collector.address:4317\n```\n\nwhere *collector.address:4317* is the address and port of an [OpenTelemetry gRPC](https://opentelemetry.io/docs/specs/otlp/) collector.\n\nA popular collector is [Grafana Alloy](https://grafana.com/oss/alloy-opentelemetry-collector/), which is usually deployed along with [Grafana Loki](https://grafana.com/docs/loki/latest/) and the [Grafana visualizer](https://grafana.com/oss/grafana/).\n\n**pve2otelcol** has numerous other command line options, see `./pve2otelcol --help` for more information. The defaults should be reasonable values in most of the cases.\n\n### Systemd unit\n\nTo better integrate it with your PVE node, you can use the provided systemd unit file.\n\nA quick guide, to be run as root (do not forget to edit the pve2otelcol.service beforehand, to point it to your OpenTelemetry collector):\n\n```sh\ncp pve2otelcol /usr/local/bin/\nchmod 755 /usr/local/bin/pve2otelcol\ncp goodies/pve2otelcol.service /etc/systemd/system/\nsystemctl daemon-reload\nsystemctl enable pve2otelcol.service\nsystemctl start pve2otelcol.service\n```\n\n## Alloy and Loki configuration\n\nWhile the setup of Alloy and Loki is well outside the scope of this document, here you can find a skeleton configuration file for both of them.\n\n```text\n// Sample config for Alloy.\n//\n// For a full configuration reference, see https://grafana.com/docs/alloy\nlogging {\n  level = \"warn\"\n}\n\nprometheus.exporter.unix \"default\" {\n  include_exporter_metrics = true\n  disable_collectors       = [\"mdadm\"]\n}\n\nprometheus.scrape \"default\" {\n  targets = array.concat(\n    prometheus.exporter.unix.default.targets,\n    [{\n      // Self-collect metrics\n      job         = \"alloy\",\n      __address__ = \"127.0.0.1:12345\",\n    }],\n  )\n\n  forward_to = []\n}\n\notelcol.receiver.otlp \"default\" {\n  grpc {\n    endpoint = \"0.0.0.0:4317\"\n    keepalive {}\n  }\n\n  http {\n    endpoint = \"0.0.0.0:4318\"\n  }\n\n  output {\n    logs = [otelcol.processor.batch.default.input]\n  }\n}\n\notelcol.processor.batch \"default\" {\n  output {\n    logs = [otelcol.exporter.loki.default.input]\n  }\n}\n\notelcol.exporter.loki \"default\" {\n  forward_to = [loki.write.local.receiver]\n}\n\nloki.write \"local\" {\n  endpoint {\n    url = \"http://localhost:3100/loki/api/v1/push\"\n  }\n}\n```\n\n```yaml\n# Sample config for Loki 3.3.\n# For a full configuration reference, see https://grafana.com/docs/loki/latest/configure/\nauth_enabled: false\n\nserver:\n  http_listen_port: 3100\n  grpc_listen_port: 9096\n  log_level: warn\n  grpc_server_max_concurrent_streams: 1000\n\ncommon:\n  instance_addr: 127.0.0.1\n  path_prefix: /var/lib/loki\n  storage:\n    filesystem:\n      chunks_directory: /var/lib/loki/chunks\n      rules_directory: /var/lib/loki/rules\n  replication_factor: 1\n  ring:\n    kvstore:\n      store: inmemory\n\nquery_range:\n  results_cache:\n    cache:\n      embedded_cache:\n        max_size_mb: 100\n\nschema_config:\n  configs:\n    - from: 2020-10-20\n      store: tsdb\n      object_store: filesystem\n      schema: v13\n      index:\n        prefix: index_\n        period: 24h\n\npattern_ingester:\n  enabled: true\n  metric_aggregation:\n    enabled: true\n    loki_address: localhost:3100\n\nruler:\n  alertmanager_url: http://localhost:9093\n\nfrontend:\n  encoding: protobuf\n\nlimits_config:\n  ingestion_rate_mb: 24\n  ingestion_burst_size_mb: 36\n```\n\nAfter that, you can start **pve2otelcol** and point it to the Alloy collector and then add the Loki data source in Grafana and begin creating dashboards.\n\n## Grafana dashboards examples\n\n### view logs by severity level\n\nQuery: `count by(level) (count_over_time({exporter=\"OTLP\"} [1h]))`\n\n\u003cdetails\u003e\n\u003csummary\u003eClick for complete panel JSON\u003c/summary\u003e\n\n```json\n{\n  \"id\": 10,\n  \"type\": \"timeseries\",\n  \"title\": \"Logs per hour by level\",\n  \"gridPos\": {\n    \"x\": 0,\n    \"y\": 32,\n    \"h\": 8,\n    \"w\": 12\n  },\n  \"fieldConfig\": {\n    \"defaults\": {\n      \"custom\": {\n        \"drawStyle\": \"line\",\n        \"lineInterpolation\": \"stepBefore\",\n        \"barAlignment\": 0,\n        \"barWidthFactor\": 0.6,\n        \"lineWidth\": 1,\n        \"fillOpacity\": 25,\n        \"gradientMode\": \"none\",\n        \"spanNulls\": false,\n        \"insertNulls\": false,\n        \"showPoints\": \"never\",\n        \"pointSize\": 5,\n        \"stacking\": {\n          \"mode\": \"normal\",\n          \"group\": \"A\"\n        },\n        \"axisPlacement\": \"auto\",\n        \"axisLabel\": \"\",\n        \"axisColorMode\": \"text\",\n        \"axisBorderShow\": false,\n        \"scaleDistribution\": {\n          \"type\": \"linear\"\n        },\n        \"axisCenteredZero\": false,\n        \"hideFrom\": {\n          \"tooltip\": false,\n          \"viz\": false,\n          \"legend\": false\n        },\n        \"thresholdsStyle\": {\n          \"mode\": \"off\"\n        }\n      },\n      \"color\": {\n        \"mode\": \"palette-classic\"\n      },\n      \"mappings\": [],\n      \"thresholds\": {\n        \"mode\": \"absolute\",\n        \"steps\": [\n          {\n            \"color\": \"green\",\n            \"value\": null\n          },\n          {\n            \"color\": \"red\",\n            \"value\": 80\n          }\n        ]\n      },\n      \"fieldMinMax\": false\n    },\n    \"overrides\": []\n  },\n  \"pluginVersion\": \"11.3.0\",\n  \"targets\": [\n    {\n      \"datasource\": {\n        \"type\": \"loki\",\n        \"uid\": \"ce0gjtocsolq8f\"\n      },\n      \"editorMode\": \"code\",\n      \"expr\": \"count by(level) (count_over_time({exporter=\\\"OTLP\\\"} [1h]))\",\n      \"legendFormat\": \"{{.level}}\",\n      \"queryType\": \"range\",\n      \"refId\": \"A\",\n      \"step\": \"\"\n    }\n  ],\n  \"datasource\": {\n    \"default\": false,\n    \"type\": \"loki\",\n    \"uid\": \"ce0gjtocsolq8f\"\n  },\n  \"options\": {\n    \"tooltip\": {\n      \"mode\": \"multi\",\n      \"sort\": \"desc\"\n    },\n    \"legend\": {\n      \"showLegend\": true,\n      \"displayMode\": \"list\",\n      \"placement\": \"bottom\",\n      \"calcs\": []\n    }\n  }\n}\n```\n\n\u003c/details\u003e\n\n![](docs/logs-per-hour-by-level.png)\n\n### view logs by VM\n\nQuery: `count by(job) (count_over_time({exporter=\"OTLP\"} [1h]))`\n\n\u003cdetails\u003e\n\u003csummary\u003eClick for complete panel JSON\u003c/summary\u003e\n\n```json\n{\n  \"id\": 9,\n  \"type\": \"timeseries\",\n  \"title\": \"Logs per hour by job\",\n  \"gridPos\": {\n    \"x\": 12,\n    \"y\": 32,\n    \"h\": 8,\n    \"w\": 12\n  },\n  \"fieldConfig\": {\n    \"defaults\": {\n      \"custom\": {\n        \"drawStyle\": \"line\",\n        \"lineInterpolation\": \"linear\",\n        \"barAlignment\": -1,\n        \"barWidthFactor\": 0.6,\n        \"lineWidth\": 1,\n        \"fillOpacity\": 25,\n        \"gradientMode\": \"none\",\n        \"spanNulls\": false,\n        \"insertNulls\": false,\n        \"showPoints\": \"auto\",\n        \"pointSize\": 5,\n        \"stacking\": {\n          \"mode\": \"normal\",\n          \"group\": \"A\"\n        },\n        \"axisPlacement\": \"auto\",\n        \"axisLabel\": \"\",\n        \"axisColorMode\": \"text\",\n        \"axisBorderShow\": false,\n        \"scaleDistribution\": {\n          \"type\": \"linear\"\n        },\n        \"axisCenteredZero\": false,\n        \"hideFrom\": {\n          \"tooltip\": false,\n          \"viz\": false,\n          \"legend\": false\n        },\n        \"thresholdsStyle\": {\n          \"mode\": \"off\"\n        },\n        \"lineStyle\": {\n          \"fill\": \"solid\"\n        }\n      },\n      \"color\": {\n        \"mode\": \"palette-classic\"\n      },\n      \"mappings\": [],\n      \"thresholds\": {\n        \"mode\": \"absolute\",\n        \"steps\": [\n          {\n            \"color\": \"green\",\n            \"value\": null\n          },\n          {\n            \"color\": \"red\",\n            \"value\": 80\n          }\n        ]\n      }\n    },\n    \"overrides\": []\n  },\n  \"pluginVersion\": \"11.3.0\",\n  \"targets\": [\n    {\n      \"datasource\": {\n        \"type\": \"loki\",\n        \"uid\": \"ce0gjtocsolq8f\"\n      },\n      \"editorMode\": \"builder\",\n      \"expr\": \"count by(job) (count_over_time({exporter=\\\"OTLP\\\"} [1h]))\",\n      \"legendFormat\": \"{{.job}}\",\n      \"queryType\": \"range\",\n      \"refId\": \"A\"\n    }\n  ],\n  \"datasource\": {\n    \"default\": false,\n    \"type\": \"loki\",\n    \"uid\": \"ce0gjtocsolq8f\"\n  },\n  \"options\": {\n    \"tooltip\": {\n      \"mode\": \"multi\",\n      \"sort\": \"desc\"\n    },\n    \"legend\": {\n      \"showLegend\": true,\n      \"displayMode\": \"list\",\n      \"placement\": \"bottom\",\n      \"calcs\": []\n    }\n  }\n}\n```\n\n\u003c/details\u003e\n\n![](docs/logs-per-hour-by-job.png)\n\n### show the last logs\n\nQuery:\n\n```loki\n{exporter=\"OTLP\"} | json | __error__=`` | line_format `{{.service_name}} {{.severity}}: {{.body_MESSAGE}}`\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eClick for complete panel JSON\u003c/summary\u003e\n\n```json\n{\n  \"id\": 11,\n  \"type\": \"logs\",\n  \"title\": \"Last logs\",\n  \"gridPos\": {\n    \"x\": 0,\n    \"y\": 40,\n    \"h\": 8,\n    \"w\": 12\n  },\n  \"fieldConfig\": {\n    \"defaults\": {},\n    \"overrides\": []\n  },\n  \"pluginVersion\": \"11.3.0\",\n  \"targets\": [\n    {\n      \"datasource\": {\n        \"type\": \"loki\",\n        \"uid\": \"ce0gjtocsolq8f\"\n      },\n      \"editorMode\": \"builder\",\n      \"expr\": \"{exporter=\\\"OTLP\\\"} | json | __error__=`` | line_format `{{.service_name}} {{.severity}}: {{.body_MESSAGE}}`\",\n      \"maxLines\": 100,\n      \"queryType\": \"range\",\n      \"refId\": \"A\"\n    }\n  ],\n  \"datasource\": {\n    \"default\": false,\n    \"type\": \"loki\",\n    \"uid\": \"ce0gjtocsolq8f\"\n  },\n  \"options\": {\n    \"showTime\": true,\n    \"showLabels\": false,\n    \"showCommonLabels\": false,\n    \"wrapLogMessage\": false,\n    \"prettifyLogMessage\": false,\n    \"enableLogDetails\": true,\n    \"dedupStrategy\": \"none\",\n    \"sortOrder\": \"Descending\"\n  }\n}\n```\n\n\u003c/details\u003e\n\n![](docs/last-logs.png)\n\n### drill-down the content of a single log entry\n\n![](docs/drill-down-log-entry.png)\n\n## Copyright and license\n\n2024 Davide Alberani \u003cda@mimante.net\u003e\n\nReleased under the Apache 2 license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falberanid%2Fpve2otelcol","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falberanid%2Fpve2otelcol","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falberanid%2Fpve2otelcol/lists"}