{"id":21351740,"url":"https://github.com/serverfarmer/heartbeat-linux","last_synced_at":"2025-09-07T00:41:31.833Z","repository":{"id":92623467,"uuid":"153635378","full_name":"serverfarmer/heartbeat-linux","owner":"serverfarmer","description":"Heartbeat is a Server Farmer subproject, aiming at extensible server monitoring, with or without Server Farmer installed.","archived":false,"fork":false,"pushed_at":"2025-03-01T15:55:34.000Z","size":70,"stargazers_count":6,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-07T00:13:57.717Z","etag":null,"topics":["alerting","heartbeat","icinga","linux","monitoring","nagios","pingdom","prtg","server-monitoring","smart","statuscake","uptime","uptimerobot"],"latest_commit_sha":null,"homepage":"http://serverfarmer.org/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serverfarmer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-18T14:14:57.000Z","updated_at":"2025-03-15T11:58:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"a3e4b7eb-9148-42fb-ba2e-ba2d9ea5a211","html_url":"https://github.com/serverfarmer/heartbeat-linux","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/serverfarmer/heartbeat-linux","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serverfarmer%2Fheartbeat-linux","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serverfarmer%2Fheartbeat-linux/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serverfarmer%2Fheartbeat-linux/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serverfarmer%2Fheartbeat-linux/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serverfarmer","download_url":"https://codeload.github.com/serverfarmer/heartbeat-linux/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serverfarmer%2Fheartbeat-linux/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273983073,"owners_count":25202092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alerting","heartbeat","icinga","linux","monitoring","nagios","pingdom","prtg","server-monitoring","smart","statuscake","uptime","uptimerobot"],"created_at":"2024-11-22T03:10:46.994Z","updated_at":"2025-09-07T00:41:31.807Z","avatar_url":"https://github.com/serverfarmer.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Overview\n\nHeartbeat is a Server Farmer subproject, that extends functionally your chosen monitoring/alerting solution by providing abilities to monitor:\n- services listening on known ports\n- running Docker containers\n- running libvirt-based virtual machines\n- SMART for local drives (also SAS and all drives connected through hardware RAID controllers)\n- free space under critical directories (eg. `/var/lib/mysql` - directories are detected automatically, see below)\n- mounted LUKS encrypted drives (or just any device mapper based)\n- MySQL database replication status (master-slave only)\n- custom conditions defined per monitored host\n\nHeartbeat can work with any monitoring/alerting system, that supports http(s) keyword monitoring, including:\n- public: StatusCake, Uptimerobot, Pingdom etc.\n- local: Nagios, Icinga, Zabbix, PRTG etc.\n\nThis versions is compatible with:\n- Linux (all distributions, possibly except some minimal ones)\n- FreeBSD 9.x or later\n- OpenBSD 5.x or later\n- NetBSD 6.x or later\n\n\n## Installation\n\nHeartbeat can be installed in 2 modes: with or without Server Farmer.\n\n1. With Server Farmer it:\n- is installed automatically on all Linux/FreeBSD-based hosts\n- installs `/etc/heartbeat/hooks/smart.sh` hook for Cacti and NewRelic (see below)\n- uses Heartbeat server address from `/opt/farm/config/get-url-heartbeat.sh` script (see below)\n\n2. Manual installation without Server Farmer:\n\n```\ngit clone https://github.com/serverfarmer/heartbeat-linux /opt/heartbeat\n/opt/heartbeat/setup.sh\n```\n\nNext, put your Heartbeat instance url into `/etc/heartbeat/server.url` file (unless you want to use the public instance, eg. for testing).\n\n### OS specific notes\n\n#### FreeBSD, OpenBSD\n\nMake sure that `bash`, `curl`, `flock` and `smartmontools` packages are installed. Update CA root certificates if needed.\n\n#### NetBSD, OpenBSD, Slackware\n\nExecute `crontab -e` as root and add this line to crontab:\n\n`*/2 * * * * /opt/heartbeat/scripts/cron/update.sh`\n\n#### NetBSD earlier than 7\n\nMake sure that `bash`, `netcat`, `mozilla-rootcerts` and `smartmontools` packages are installed. Installing `smartmontools` will require changing `PKG_PATH` variable first, to point to release 7 repository, eg.\n\n`PKG_PATH=ftp://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/7.0/All`\n\nAfter installing (and each upgrade of) `mozilla-rootcerts` package, execute `mozilla-rootcerts install` as root to refresh CA root certificates.\n\n\n## How it works\n\n#### Local part:\n\n1. Cron job `/opt/heartbeat/scripts/cron/update.sh` is run every 2 minutes.\n2. It runs `/opt/heartbeat/scripts/checks/all.sh` script to collect detected items to report. This script can be also run manually for debugging purposes - it just prints all found items on console.\n3. Cron job sends the collected list to Heartbeat server.\n\n##### Heartbeat server address detection:\n\n1. Cron job looks for `/etc/heartbeat/server.url` file - if it exists, it should contain the full path to Heartbeat server.\n2. If Server Farmer is also installed, cron job uses server address returned by `/opt/farm/config/get-url-heartbeat.sh` script. So if you cloned Server Farmer main repository and changed mentioned script to point to your private instance, it will be automatically detected here.\n3. Otherwise it uses hardcoded `https://serverfarmer.home.pl/heartbeat/` (public Heartbeat instance).\n\n#### Remote part:\n\n1. Your chosen monitoring/alerting platform is querying [Heartbeat server](https://github.com/serverfarmer/heartbeat-server) for particular item on particular monitored host.\n2. Heartbeat server responds with either `ALIVE` or `DEAD` keyword, where `ALIVE` means that this item was last reported no longer than 270 seconds ago.\n\nItems are reported every 120 seconds, so 270 seconds means tolerance for 1 failed request + up to 30 seconds overall network lag. And this limit can be easily adjusted in repository with server part.\n\n## Query URL format\n\nAssuming that:\n- your Heartbeat server has address `http://heartbeat.yourdomain.com/heartbeat/`\n- your example monitored host has hostname `yourserver.yourdomain.com`\n\nthis is the complete URL that checks for `ssh` service running on this host:\n\n`http://heartbeat.yourdomain.com/heartbeat/query.php?id=ssh_yourserver_yourdomain_com`\n\nRules:\n- everything is converted to lowercase\n- underlines, colons and slashes are replaced with dashes\n- dots in hostnames are replaced with underlines\n- network service names are listed in `/opt/heartbeat/scripts/checks/services.sh` script\n\n\n## Performance\n\nSingle AWS `t2.micro` instance, storing temporary files on `tmpfs` filesystem, can handle over 3000 individual checks without any performance issues, assuming that queries from monitoring system are done via http (no encryption), every 1 minute.\n\nNote that you can use different addresses for reporting data from monitored hosts, and for querying (in particular, you can use https for reporting and http for querying over internal network).\n\n\n## SMART monitoring details\n\nHeartbeat automatically detects all local drives, even ones not supported by udev:\n- SATA drives connected straight, or via USB or eSATA (including with port multiplier), or even as passthrough from hypervisor to virtual machine\n- NVMe drives connected straight, or via USB (note: not all USB bridge chipsets are supported)\n- SATA/SAS drives connected to MegaRAID controller\n- SATA/SAS drives connected to any custom hardware controller, assuming that such drives are exposed via `/dev/sg*` interfaces\n\n##### Server Farmer hook for Cacti and NewRelic\n\nFor each detected and not excluded drive (not necessarily meeting conditions described below), Heartbeat executes `/etc/heartbeat/hooks/smart.sh` script, with the full name of SMART dump file as the only argument. When Heartbeat is installed by Server Farmer, this file is installed automatically, you can however replace it with your own one.\n\nVersion provided by Server Farmer:\n\n- parses the SMART dump again and pushes the drive metrics to NewRelic (assuming that `sf-monitoring-newrelic` extension is installed and NewRelic license key is properly configured)\n- copies this dump using scp to Cacti server (assuming that `sf-monitoring-cacti` extension is installed)\n\n##### Handling known drive defects\n\nIn highly professional use, drives mostly work in stable physical conditions for all their lifetime. This is often not the case for smaller companies or private use, where physical conditions (eg. temperature, cables etc.) can change from time to time.\n\nBecause of that, some particular SMART errors can happen and shouldn't be considered a problem. For example, non-zero `UDMA_CRC_Error_Count` is often a result of bad eSATA cable/connector, and it stays non-zero even after the cable is replaced. And there are numerous similar exceptions, where certain defects doesn't yet mean that drive should be replaced.\n\nIn file `/etc/heartbeat/known-smart-defects.conf` you can store such exceptions, eg.:\n\n`WDC_WD121KRYZ-01W0RB0_XXXXXXXX:Temperature_Celsius:50`\n\nmeans that this particular drive is allowed to run with allowed temperature increased by 2 degrees from standard (which is not recommended anyway, but it's a better solution than simply dropping such drive).\n\n##### Excluding problematic drives\n\nThere are certain cases, where you want to exclude particular drives from being detected and checked every 2 minutes, for example:\n\n- unstable RAID controller, causing random system crashes during SMART read attempts\n- USB drives in external enclosures, meant to be either disconnected or put in `standby` condition for most of the time, that might overheat otherwise\n\nYou can add such drives to these files to exclude them from being detected:\n- `/etc/heartbeat/skip-smart.nvme` (drives recognized and handled by udev)\n- `/etc/heartbeat/skip-smart.sata` (drives recognized and handled by udev)\n- `/etc/heartbeat/skip-smart.raid` (drives connected to hardware RAID controllers)\n\n##### Required SMART conditions for SATA drives\n\n- `Temperature_Celsius` - max 48 degrees for magnetic drives, or 55 degrees for SSD\n- `Reallocated_Sector_Ct` - 0\n- `End-to-End_Error` - 0\n- `UDMA_CRC_Error_Count` - 0\n- `Spin_Retry_Count` - 0\n- `Runtime_Bad_Block` - max 10\n- `Current_Pending_Sector` - max 2\n- `Reported_Uncorrect` - 0\n- `Offline_Uncorrectable` - 0\n- `Calibration_Retry_Count` - 0\n- `Power_On_Hours` - max 70000 (which is around 8 years)\n\n##### Required SMART conditions for NVMe drives\n\n- `temperature` - max 65 degrees\n- `critical_warning` - 0\n- `media_errors` - max 10\n- `power_on_hours` - max 25000 (which is around 2.5 years)\n\n##### Required SMART conditions for SAS drives\n\nDrive temperature is not monitored, since SAS drives have Drive Trip Temperature mechanism.\n\n- `Elements in grown defect list` (similar to `Reallocated_Sector_Ct`) - max 4\n- `Non-medium error count` (similar to `UDMA_CRC_Error_Count`) - max 10\n- ECC-corrected reads - max 6\n- ECC-corrected writes - max 2\n- ECC-corrected verifications - max 2\n- `number of hours powered up` - max 70000 (only for Seagate and Hitachi drives)\n\n\n## Free space monitoring details\n\n`/opt/heartbeat/config/common-data-directories.list` file contains the list of directories commonly used to storage bigger amounts of data, eg. by databases, queues, (para)virtualization etc. This file is processed during Heartbeat setup and any directories from this list that actually exist on current host, are added to `/etc/heartbeat/detected-data-directories.conf` file. Next, they are checked every 2 minutes, if they have at least 12 GB of free disk space.\n\nAdditionally, root filesystem is required to have 512 MB of free space, and `/boot` directory - 80 MB.\n\nSuch limits are designed to give system administrators just enough time to safely deal with the problem - not to assure that system will be able to run for next weeks or months. You can however implement your own limits, just by adding the following line to custom check script:\n\n```\n/opt/heartbeat/scripts/checks/space-check.sh /var/lib 491520000\n```\n\nThis example check will require `/var/lib` directory to have at least 480 GB of free space, or otherwise it will fail.\n\n\n## Implementing custom checks\n\nYou can implement custom checks just by adding them to `/etc/heartbeat/hooks/custom.sh` script. It just needs to print the list of passed checks on console, one per line. For example, the above check for `/var/lib` directory free space should just print:\n\n`space-var-lib`\n\nIt is important that this script, or scripts/libraries/etc. that you invoke from it, should not print anything else on console - otherwise it will be sent to Heartbeat server and might interfere with other checks.\n\nTo simplify your custom logic, you can also use `/opt/heartbeat/scripts/checks/custom/count-processes.sh` script, that counts the processes with given name pattern, eg.:\n\n`/opt/heartbeat/scripts/checks/custom/count-processes.sh app/console 34 my-symfony-app-console`\n\nSuch script will print `my-symfony-app-console` if there will be at least 34 running processes with `app/console` in their names. Note that you can use spaces in the first argument\n\n`/opt/heartbeat/scripts/checks/custom/count-processes.sh \"app/console rabbitmq:consumer\" 31 rabbit-consumer`\n\n\n## Debugging\n\nTo see, what is reported to Heartbeat server, just run:\n\n- `/opt/heartbeat/scripts/checks/all.sh` - to see the list of reported checks (running it with `--debug` argument will disable SMART hook script and removing temporary files with SMART dumps)\n- `/opt/heartbeat/scripts/facts/get-reported-hostname.sh` - to see the hostname used for reporting\n\nIf you added/removed drives or directories to monitor free space, run `/opt/heartbeat/setup.sh` to scan system for changes.\n\nAll Heartbeat settings are stored in `/etc/heartbeat` directory, and temporary files in `/var/cache/heartbeat` (which should be mounted as `tmpfs`).\n\n#### Common problems\n\n##### `/opt/heartbeat/scripts/checks/all.sh` script shows many errors, when run manually\n\nMake sure that you have installed all required system packages. See [notes](#os-specific-notes) above.\n\n##### `/opt/heartbeat/scripts/cron/update.sh` script is added to `/etc/crontab`, but doesn't run\n\nAdd it to root crontab manually using `crontab -e` command. See [notes](#os-specific-notes) above.\n\n##### `/opt/heartbeat/scripts/cron/update.sh` script runs, but can't contact Heartbeat server\n\nMake sure that your CA root certificates are up to date (this is the most common problem on old systems).\n\n\n## How to contribute\n\nWe are welcome to contributions of any kind: bug fixes, added code comments,\nsupport for new operating system versions or hardware etc.\n\nIf you want to contribute:\n- fork this repository and clone it to your machine\n- create a feature branch and do the change inside it\n- push your feature branch to github and create a pull request\n\n\n## License\n\n|                      |                                          |\n|:---------------------|:-----------------------------------------|\n| **Author:**          | Tomasz Klim (\u003copensource@tomaszklim.pl\u003e) |\n| **Copyright:**       | Copyright 2016-2024 Tomasz Klim          |\n| **License:**         | MIT                                      |\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserverfarmer%2Fheartbeat-linux","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserverfarmer%2Fheartbeat-linux","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserverfarmer%2Fheartbeat-linux/lists"}