https://github.com/iot-salzburg/dtz_watchdog
The watchdog monitors each service on the dtz cluster and notifies via Slack. Additonally, a second watchdog watches the main one.
https://github.com/iot-salzburg/dtz_watchdog
Last synced: about 1 month ago
JSON representation
The watchdog monitors each service on the dtz cluster and notifies via Slack. Additonally, a second watchdog watches the main one.
- Host: GitHub
- URL: https://github.com/iot-salzburg/dtz_watchdog
- Owner: iot-salzburg
- License: apache-2.0
- Created: 2018-09-07T17:57:42.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2023-05-01T20:56:49.000Z (about 2 years ago)
- Last Synced: 2023-05-01T22:01:23.036Z (about 2 years ago)
- Language: Python
- Size: 55.7 KB
- Stars: 0
- Watchers: 6
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Watchdog
The watchdog monitors each service on the dtz cluster and notifies
via Slack. Additionally, a second watchdog watches the main one.
## Contents
1. [Requirements](#requirements)
2. [Usage](#usage)
## Requirements
Clone the repository and set the environment:
```bash
git clone https://github.com/iot-salzburg/dtz_watchdog.git
cd dtz_watchdog
virtualenv -p $(which python3) setup/venv
source setup/venv/bin/activate
python -m pip install -r requirements.txt
```
## Usage
### Cluster Watchdog
The Watchdog uses a slack webhook to notify about cluster issues. Therefore open in Slack a new channel, then `add app`,
look for `Incoming WebHooks`, `Add Configuration` and select the desired Slack Channel. Then a new configuration will
show the WebHook-Url in the form: `https://hooks.slack.com/services/id1/id2/id3`. This URL should be set as
environment variable on the host. Note that the **url is in quotes**, so that
it can be accessed better within python. Additionally set the ip-address and hostname of the cluster watchdog, and also the ip-address of the meta watchdog.
```bash
echo "SLACK_URL=https://hooks.slack.com/services/id1/id2/id3" >> .env
echo "CLUSTER_WATCHDOG_HOSTNAME=il071" >> .env
echo "SWARM_MAN_IP=192.168.48.71" >> .env
echo "META_WATCHDOG_URL=192.168.48.50" >> .env
echo "PORT=8081" >> .env
cat .env
```
Now, the Watchdogs can be started.
```bash
user@il071:dtz_watchdog$ python3 src/cluster-watchdog.py
user@il050:dtz_watchdog$ python3 src/meta-watchdog.py
```
View if the cluster is healthy [http://il071:8081/](http://192.168.48.71:8081/).
### Deployment
Adjust the settings like user name and absolute path to the repository
and copy the systemd service into the proper directory:
```
sudo nano setup/watchdog.service
sudo cp setup/watchdog.service /etc/systemd/system/watchdog.service
```
An execute to enable:
```
sudo systemctl enable watchdog.service
sudo systemctl start watchdog
sudo systemctl status watchdog
```
### Meta Watchdog Deployment
As there would be no notifications if the host server itself crashes,
we deploy a meta watchdog, that watches only on the cluster watchdog.
Therefore, adjust the settings like user name and absolute path to the repository
and copy the systemd service into the proper directory:
```
sudo nano setup/meta-watchdog.service
sudo cp setup/meta-watchdog.service /etc/systemd/system/meta-watchdog.service
```
An execute to enable:
```
sudo systemctl enable meta-watchdog.service
sudo systemctl start meta-watchdog
sudo systemctl status meta-watchdog
```
Both watchdogs should now be up and running:
* [cluster-watchdog](http://192.168.48.71:8081/)
* [meta-watchdog](http://192.168.48.50:8081/)