{"id":19945953,"url":"https://github.com/cicerops/monitoring-check-grafana","last_synced_at":"2026-05-08T15:47:08.036Z","repository":{"id":118414917,"uuid":"127580208","full_name":"cicerops/monitoring-check-grafana","owner":"cicerops","description":"Monitor a Grafana datasource against data becoming stale to detect data loss or other dropout conditions.","archived":false,"fork":false,"pushed_at":"2022-01-09T01:07:40.000Z","size":37,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-12T05:26:04.954Z","etag":null,"topics":["data","database","freshness","grafana","grafana-datasource","icinga2","icinga2-plugin","influxdb","monitoring","stale"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cicerops.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-31T23:43:38.000Z","updated_at":"2023-05-08T15:12:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"70491770-beaa-4324-be25-dd9b5c423dd6","html_url":"https://github.com/cicerops/monitoring-check-grafana","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cicerops%2Fmonitoring-check-grafana","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cicerops%2Fmonitoring-check-grafana/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cicerops%2Fmonitoring-check-grafana/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cicerops%2Fmonitoring-check-grafana/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cicerops","download_url":"https://codeload.github.com/cicerops/monitoring-check-grafana/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241374538,"owners_count":19952545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","database","freshness","grafana","grafana-datasource","icinga2","icinga2-plugin","influxdb","monitoring","stale"],"created_at":"2024-11-13T00:27:34.420Z","updated_at":"2026-05-08T15:47:07.982Z","avatar_url":"https://github.com/cicerops.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":".. image:: https://img.shields.io/github/tag/daq-tools/monitoring-check-grafana.svg\n    :target: https://github.com/daq-tools/monitoring-check-grafana\n.. image:: https://img.shields.io/badge/monitoring%20platform-Icinga2-green.svg\n    :target: https://www.icinga.com/\n.. image:: https://img.shields.io/badge/os%20platform-Linux%20%7C%20OS%20X-blue.svg\n    :target: #\n\n|\n\n########################\nmonitoring-check-grafana\n########################\n\n\n*****\nAbout\n*****\nA monitoring sensor for checking a Grafana datasource\nagainst data becoming stale. This will let you detect\ndata loss or other dropout conditions of feeds into\nyour datasources.\n\n\n*****\nGoals\n*****\nTired of operations or engineering messing with the datasources\nor the sensors under the hood you are just watching in Grafana?\n\nThis plugin is an attempt to have a basic end-to-end monitoring\nprobe covering the whole track of data flowing from arbitrary\nsensors into a (timeseries) database and then being displayed\nin Grafana. So, this probe basically checks for success in:\n\n- Acquisition: Data is received by the DAQ system.\n- Storage: Measurements are stored into the database.\n- Retrieval: Measurements are retrieved from the database.\n- Display: Data is displayed in Grafana (almost).\n\nThis is nearly to-the-glass monitoring as it probes the very\nsame Grafana API endpoints as the frontend uses for fetching\nmetric data from, just before rendering it to the display.\n\n\n**********\nReferences\n**********\nWe are currently using this plugin for monitoring freshness of\ndata flows from different sources into InfluxDB:\n\n- Particulate matter measurements from the grassroots project `luftdaten.info`_\n  are imported into the data acquisition system running at `Lufdaten in Grafana`_.\n  You might also enjoy reading `about Lufdaten in Grafana`_.\n\n- Weather data measurements from the `DWD CDC FTP server`_ are imported into the\n  data acquisition system running at `weather.hiveeyes.org`_, see also\n  `Datenerfassungssystem für Wetterdaten`_.\n\n- Measurements from the `Open Hive Teststand`_ are received from the\n  data acquisition system running at `swarm.hiveeyes.org`_, see also\n  the `Hiveeyes project`_  and the `Hiveeyes data collection platform`_.\n\nKudos to all the people working behind the scenes for\nproviding these great open data resources to the community!\n\n.. _luftdaten.info: https://luftdaten.info/\n.. _Lufdaten in Grafana: https://luftdaten.getkotori.org/\n.. _about Lufdaten in Grafana: https://getkotori.org/docs/applications/luftdaten.info/\n\n.. _DWD CDC FTP server: ftp://ftp-cdc.dwd.de/\n.. _weather.hiveeyes.org: https://weather.hiveeyes.org/\n.. _Datenerfassungssystem für Wetterdaten: https://community.hiveeyes.org/t/open-weather-data/113/22\n\n.. _Open Hive Teststand: https://community.hiveeyes.org/t/temperaturkompensation-fur-waage-notig-datensammlung/245/2\n.. _swarm.hiveeyes.org: https://swarm.hiveeyes.org/\n.. _Hiveeyes project: https://hiveeyes.org/\n.. _Hiveeyes data collection platform: https://getkotori.org/docs/applications/hiveeyes.html\n\n\n*****\nUsage\n*****\n::\n\n    $ ./check-grafana-datasource-stale.sh --help\n\n    Options:\n    -u, --uri           Grafana API datasource proxy URI\n    -d, --database      Database name\n\n    -t, --table         Table name\n    -w, --warning       Maximum age threshold of data to result in warning status\n    -c, --critical      Maximum age threshold of data to result in critical status\n\n    -h, --help          Print detailed help\n    -V, --version       Print version information\n    -v, --verbose       Turn on verbose output\n\n\n*******\nExample\n*******\nSensor invocation::\n\n    ./check-grafana-datasource-stale.sh \\\n        --uri https://datahub.example.org/grafana/api/datasources/proxy/42/query \\\n        --database testdrive \\\n        --table temperature \\\n        --warning 12h \\\n        --critical 3d \\\n        --verbose\n\nSensor output::\n\n    INFO:  Checking testdrive:temperature for data not older than 3d\n    INFO:  Checking testdrive:temperature for data not older than 12h\n    WARNING - Data in testdrive:temperature is stale for 12h or longer\n\n\n**********\nScreenshot\n**********\nData acquisition from luftdaten.info triggered a data loss warning\n\n.. image:: https://raw.githubusercontent.com/daq-tools/monitoring-check-grafana/main/screenshot-datasource-stale.jpg\n\nwhen the people operating the platform had to perform some maintenance work on the database\n\n.. raw:: html\n\n    \u003cblockquote class=\"twitter-tweet\" data-lang=\"en\"\u003e\n        \u003cp lang=\"en\" dir=\"ltr\"\u003e\n            If someone is wondering: The API is down for maintenance. Today we received\n            value no. ‘2^31+1’ . But the database was defined with a maximum of 2^31 values.\n            We are currently changing this to 2^63. But this may need some time.\n        \u003c/p\u003e\n        \u0026mdash; OK Lab Stuttgart (@codeforS)\n        \u003ca href=\"https://twitter.com/codeforS/status/980017103976763392\"\u003eMarch 31, 2018\u003c/a\u003e\n    \u003c/blockquote\u003e\n    \u003cscript async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"\u003e\u003c/script\u003e\n\n\n*********************\nInstall prerequisites\n*********************\nThis sensor uses the fine programs HTTPie_ and jq_,\nplease install them on your system.\n\n\nDebian\n======\n::\n\n    apt install httpie jq\n\n    # Optionally\n    pip install httpie\n\n\nmacOS\n=====\n::\n\n    brew install httpie jq\n\n\n.. _HTTPie: https://httpie.org/\n.. _jq: https://stedolan.github.io/jq/\n\n\n\n*******************\nSetup Icinga plugin\n*******************\n\nPlugin environment\n==================\n::\n\n    mkdir -p /usr/local/lib/icinga2/plugins\n\nEdit ``/etc/icinga2/constants.conf``::\n\n    const CustomPluginDir = \"/opt/monitoring/plugins\"\n\nInstallation\n============\n::\n\n    git clone https://github.com/daq-tools/monitoring-check-grafana /opt/monitoring-check-grafana\n    ln -s /opt/monitoring-check-grafana/check-grafana-datasource-stale.sh /opt/monitoring/plugins/check-grafana-datasource-stale\n    ln -s /opt/monitoring-check-grafana/icinga-command-check-grafana.conf /etc/icinga2/conf.d/command-check-grafana.conf\n\n\nConfiguration\n=============\nA blueprint for a usual configuration object::\n\n    object Service \"Grafana datasource freshness for testdrive:temperature\" {\n      import \"generic-service\"\n      check_command         = \"check-grafana-datasource-stale\"\n\n      host_name             = \"datahub.example.org\"\n      vars.sla              = \"24x7\"\n\n      vars.grafana_uri      = \"https://datahub.example.org/grafana/api/datasources/proxy/42/query\"\n      vars.grafana_database = \"testdrive\"\n      vars.grafana_table    = \"temperature\"\n      vars.grafana_warning  = \"1h\"\n      vars.grafana_critical = \"12h\"\n\n      # Optionally assign this service exclusively to these notification recipients only\n      #vars.notification.mail.users  = [ \"bruce-lee\", \"chuck-norris\" ]\n      #vars.notification.mail.groups = [ \"null\" ]\n    }\n\n\nSee also `icinga-service-check-grafana.example.conf`_.\n\n.. _icinga-service-check-grafana.example.conf: https://github.com/daq-tools/monitoring-check-grafana/blob/main/icinga-service-check-grafana.example.conf\n\n\n*******************\nProject information\n*******************\n\nAbout\n=====\nThe \"monitoring-check-grafana\" sensor program is released under the GNU AGPL license.\nIts source code lives on `GitHub \u003chttps://github.com/daq-tools/monitoring-check-grafana\u003e`_.\n\nIf you'd like to contribute you're most welcome!\nSpend some time taking a look around, locate a bug, design issue or\nspelling mistake and then send us a pull request or create an issue.\n\nThanks in advance for your efforts, we really appreciate any help or feedback.\n\nLicense\n=======\nLicensed under the GNU AGPL license. See LICENSE_ file for details.\n\n.. _LICENSE: https://github.com/daq-tools/monitoring-check-grafana/blob/main/LICENSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcicerops%2Fmonitoring-check-grafana","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcicerops%2Fmonitoring-check-grafana","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcicerops%2Fmonitoring-check-grafana/lists"}