{"id":19623495,"url":"https://github.com/dotheevo/veeam-prometheus-grafana","last_synced_at":"2025-04-28T04:31:54.228Z","repository":{"id":89882064,"uuid":"540912561","full_name":"DoTheEvo/veeam-prometheus-grafana","owner":"DoTheEvo","description":"Monitoring Veeam B\u0026R Community Edition with prometheus and grafana","archived":false,"fork":false,"pushed_at":"2024-05-05T08:18:13.000Z","size":188,"stargazers_count":15,"open_issues_count":1,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-05T12:31:56.328Z","etag":null,"topics":["docker","grafana","prometheus","pushgateway","veeam"],"latest_commit_sha":null,"homepage":"","language":"PowerShell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DoTheEvo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-24T17:33:13.000Z","updated_at":"2024-05-05T08:18:15.000Z","dependencies_parsed_at":"2024-05-04T12:30:13.212Z","dependency_job_id":null,"html_url":"https://github.com/DoTheEvo/veeam-prometheus-grafana","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DoTheEvo%2Fveeam-prometheus-grafana","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DoTheEvo%2Fveeam-prometheus-grafana/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DoTheEvo%2Fveeam-prometheus-grafana/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DoTheEvo%2Fveeam-prometheus-grafana/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DoTheEvo","download_url":"https://codeload.github.com/DoTheEvo/veeam-prometheus-grafana/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224096177,"owners_count":17255048,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","grafana","prometheus","pushgateway","veeam"],"created_at":"2024-11-11T11:34:10.740Z","updated_at":"2024-11-11T11:34:11.728Z","avatar_url":"https://github.com/DoTheEvo.png","language":"PowerShell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Veeam B\u0026R dashboard\r\n\r\n###### guide-by-example\r\n\r\n![logo](https://i.imgur.com/EEExOB0.png)\r\n\r\n-----------------\r\n\r\n# Purpose\r\n\r\nCentralized **monitoring dashboard** with **alerts** for Veeam B\u0026R.\u003cbr\u003e\r\nWorks with community edition.\r\nRelatively easily adjusted to any backup solution that can report basic info.\r\n\r\n* [Veeam Backup \u0026 Replication Community Edition](\r\nhttps://www.veeam.com/virtual-machine-backup-solution-free.html)\r\n* [Prometheus](https://prometheus.io/)\r\n* [Grafana](https://grafana.com/)\r\n\r\nA **powershell script** periodically runs on machines running VBR,\r\ngathering information about backup-jobs and repositories.\r\nThis info gets pushed to a **prometheus pushgateway**, where it gets scraped\r\nin to prometheus.\r\nGrafana **dashboard** then visualizes the gathered information.\u003cbr\u003e\r\n\r\n![dashboard_pic](https://i.imgur.com/00xPhwQ.png)\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e\u003ch1\u003eBasic info on Veeam Backup \u0026 Replication\u003c/h1\u003e\u003c/summary\u003e\r\n\r\n* VBR is installed on a windows machine. Can be physical or virtual.\r\n* It needs a repository where to store backups.\r\n  Can be local drives, network storage, cloud,..\r\n* Job logs are in `C:\\ProgramData\\Veeam\\Backup`\r\n* Various types of jobs are created that regularly run, creating backups.\r\n\r\n#### Virtual machines backup\r\n\r\n* [Official documentation](https://helpcenter.veeam.com/docs/backup/vsphere/backup.html)\r\n\r\nFor Hyper-V / VMware.\u003cbr\u003e\r\nVeeam has admin credentails for the hypervisor.\r\nIt initiates the backup process at schedule, creates a snapshot of a VM,\r\nprocess the VM's data, copies them in to a repository, deletes the snapshot.\u003cbr\u003e\r\nVM's data are stored in a single file, `vbk` for full backup,\r\n`vib` for incremental backup.\u003cbr\u003e\r\nVeeam by default creates weekly\r\n[synthetic full backup,](https://helpcenter.veeam.com/docs/backup/vsphere/synthetic_full_hiw.html)\r\nwhich combines previous backups in to a new standalone `vbk`.\r\n\r\n#### Fileshare backup\r\n\r\n* [Official documentation](https://helpcenter.veeam.com/docs/backup/vsphere/file_share_support.html)\r\n\r\nFor network shares, called also just `File Backup`.\u003cbr\u003e\r\nDiffers from VM backup in a way files are stored, no vbk and vib files,\r\nbut bunch of `vblob` files.\u003cbr\u003e\r\nAlso, long term retention requires an archive repository,\r\nnot available in community edition.\r\n\r\n#### Agent backup - Managed by server \r\n\r\n* [Official documentation](https://helpcenter.veeam.com/docs/backup/agents/agents_job.html)\r\n\r\nFor physical machines, intented for the ones that run 24/7\r\nand should be always accessible by Veeam.\u003cbr\u003e\r\nVery similar to VMs backup. The VBR server initiates the backup,\r\nthe agent that is installed on the machine creates VSS snapshot,\r\nand data end up in a repository, either in a `vbk` file or `vib` file.\r\n\r\n#### Agent backup - Managed by agent - Backup policy\r\n\r\n* [Official documentation](https://helpcenter.veeam.com/docs/backup/agents/agents_policy.html)\r\n\r\nIntended for use with workstations that dont have regular connectivity\r\nwith the VBR server. VBR installs an agent on the machine,\r\nhands it XML configuration, a **backup policy**, that tells it how and where\r\nto regularly backup and then its hands off, the agent is in charge.\u003cbr\u003e\r\nVeeam periodically tries to sync the current policy settings with the already\r\ndeployed agents during protection group rescans.\r\n\r\nThis one was bit tricky to monitor, as job's history contains not just backup\r\nsessions, but also the policy updates.\r\nSome extra steps are needed in the powershell script to get backup runs without\r\npolicy updates.\r\n\u003c/details\u003e\r\n\r\n---\r\n---\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e\u003ch1\u003ePrometheus and Grafana Setup in Docker\u003c/h1\u003e\u003c/summary\u003e\r\n\r\n[Here](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/prometheus_grafana_loki)\r\nis a guide-by-example for monitoring using Prometheus, Grafana, Loki.\r\nMight be useful as it goes in to more details.\r\n\r\n## Files and directory structure\r\n\r\n```\r\n/home/\r\n└── ~/\r\n    └── docker/\r\n        └── veeam_monitoring/\r\n            ├── 🗁 grafana_data/\r\n            ├── 🗁 prometheus_data/\r\n            ├── 🗋 .env\r\n            ├── 🗋 docker-compose.yml\r\n            └── 🗋 prometheus.yml\r\n```\r\n\r\n* `grafana_data/` - a directory where grafana stores its data\r\n* `prometheus_data/` - a directory where prometheus stores its database and data\r\n* `.env` - a file containing environment variables for docker compose\r\n* `docker-compose.yml` - a docker compose file, telling docker how to run the containers\r\n* `prometheus.yml` - a configuration file for prometheus\r\n\r\nThe 3 files must be provided.\u003c/br\u003e\r\nThe directories are created by docker compose on the first run.\r\n\r\n## docker-compose\r\n\r\nThree containers to spin up.\u003c/br\u003e\r\n\r\n* **Prometheus** - prometheus server, pulling, storing, evaluating metrics.\r\n* **Pushgateway** - web server ready to receive pushed information.\r\n* **Grafana** - web GUI visualization of the collected metrics in nice dashboards.\r\n\r\nOf note for prometheus container is **data retention** set to 45 days,\r\nand **admin api** being enabled.\u003cbr\u003e\r\nPushgateway has **admin api** enabled too, to be able to execute wipes.\r\n\r\n`docker-compose.yml`\r\n```yml\r\nservices:\r\n\r\n  prometheus:\r\n    image: prom/prometheus:v2.43.1\r\n    container_name: prometheus\r\n    hostname: prometheus\r\n    restart: unless-stopped\r\n    user: root\r\n    command:\r\n      - '--config.file=/etc/prometheus/prometheus.yml'\r\n      - '--storage.tsdb.path=/prometheus'\r\n      - '--web.console.libraries=/etc/prometheus/console_libraries'\r\n      - '--web.console.templates=/etc/prometheus/consoles'\r\n      - '--storage.tsdb.retention.time=45d'\r\n      - '--web.enable-lifecycle'\r\n      - '--web.enable-admin-api'\r\n    volumes:\r\n      - ./prometheus.yml:/etc/prometheus/prometheus.yml\r\n      - ./prometheus_data:/prometheus\r\n    ports:\r\n      - \"9090:9090\"\r\n\r\n  grafana:\r\n    image: grafana/grafana:9.5.2\r\n    container_name: grafana\r\n    hostname: grafana\r\n    restart: unless-stopped\r\n    env_file: .env\r\n    user: root\r\n    volumes:\r\n      - ./grafana_data:/var/lib/grafana\r\n    ports:\r\n      - \"3000:3000\"\r\n\r\n  pushgateway:\r\n    image: prom/pushgateway:v1.5.1\r\n    container_name: pushgateway\r\n    hostname: pushgateway\r\n    restart: unless-stopped\r\n    command:\r\n      - '--web.enable-admin-api'    \r\n    ports:\r\n      - \"9091:9091\"\r\n\r\nnetworks:\r\n  default:\r\n    name: $DOCKER_MY_NETWORK\r\n    external: true\r\n```\r\n\r\n`.env`\r\n\r\n```bash\r\n# GENERAL\r\nDOCKER_MY_NETWORK=caddy_net\r\nTZ=Europe/Bratislava\r\n\r\n# GRAFANA\r\nGF_SECURITY_ADMIN_USER=admin\r\nGF_SECURITY_ADMIN_PASSWORD=admin\r\nGF_USERS_ALLOW_SIGN_UP=false\r\nGF_SERVER_ROOT_URL=https://grafana.example.com\r\n# GRAFANA EMAIL SETTINGS\r\nGF_SMTP_ENABLED=true\r\nGF_SMTP_HOST=smtp-relay.sendinblue.com:587\r\nGF_SMTP_USER=example@gmail.com\r\nGF_SMTP_PASSWORD=xzu0dfFhn3eqa\r\nstartTLS_policy=NoStartTLS\r\n# GRAFANA CUSTOM SETTINGS\r\n# DATE FORMATS SWITCHED TO NAMES OF THE DAYS OF THE WEEK\r\n#GF_DATE_FORMATS_INTERVAL_HOUR = dddd\r\n#GF_DATE_FORMATS_INTERVAL_DAY = dddd\r\n```\r\n\r\nThe containers must be on a **custom named docker network**,\r\nalong with caddy reverse proxy. This allows **hostname resolution**.\u003c/br\u003e\r\nThe network name is set in the `.env` file, in `DOCKER_MY_NETWORK` variable.\u003c/br\u003e\r\nIf one does not exist yet: `docker network create caddy_net`\r\n\r\nIn the `.env` file, there are also two date settings for grafana commented out.\r\nUncomment to show full name of days in the week instead of exact date.\u003cbr\u003e\r\n\r\n## prometheus.yml\r\n\r\n[Official documentation.](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)\r\n\r\nA config file for prometheus, bind mounted in to the prometheus container.\u003cbr\u003e\r\nOf note is **honor_labels** set to true,\r\nwhich means that **conflicting labels**, like `job`, set during push\r\nare kept over labels set by `prometheus.yml` for that scrape job.\r\n[Docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).\r\n\r\n`prometheus.yml`\r\n```yml\r\nglobal:\r\n  scrape_interval:     15s\r\n  evaluation_interval: 15s\r\n\r\nscrape_configs:\r\n  - job_name: 'pushgateway-scrape'\r\n    scrape_interval: 60s\r\n    honor_labels: true\r\n    static_configs:\r\n      - targets: ['pushgateway:9091']\r\n```\r\n\r\n## Reverse proxy\r\n\r\nCaddy v2 is used, details\r\n[here](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/caddy_v2).\u003c/br\u003e\r\n\r\n`Caddyfile`\r\n```php\r\ngrafana.{$MY_DOMAIN} {\r\n    reverse_proxy grafana:3000\r\n}\r\n\r\npush.{$MY_DOMAIN} {\r\n    reverse_proxy pushgateway:9091\r\n}\r\n\r\n# prom.{$MY_DOMAIN} {\r\n#     reverse_proxy prometheus:9090\r\n# }\r\n```\r\n\r\n## Start the containers \r\n\r\n* `docker compose up -d`\r\n\r\n## Grafana configuration\r\n\r\n* First run login with admin/admin.\r\n* In Preferences \u003e Datasources set `http://prometheus:9090` for url.\u003cbr\u003e\r\n  Save and test should be green.\r\n* Once some metrics are pushed to prometheus,\r\n  they should be searchable in Explore section in Grafana.\r\n\r\n![prometheus_working_pic_confirmation](https://i.imgur.com/hO8eERV.png)\r\n\r\n\u003c/details\u003e\r\n\r\n---\r\n---\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e\u003ch1\u003eLearning in small steps\u003c/h1\u003e\u003c/summary\u003e\r\n\r\nA section written during first testing\r\n\r\nwhat should work at this moment\r\n\r\n* \\\u003cdocker-host-ip\u003e:3000 - grafana\r\n* \\\u003cdocker-host-ip\u003e:9090 - prometheus \r\n* \\\u003cdocker-host-ip\u003e:9091 - pushgateway \r\n\r\n### Learning and testing how to push data to pushgateway\r\n\r\n* metrics must be floats\r\n* naming [convention](https://prometheus.io/docs/practices/naming/)\r\n  is to end the metric names with units\r\n* labels in url are used to pass strings info and to mark the metrics\r\n* The idea what\r\n [job and instance](https://prometheus.io/docs/concepts/jobs_instances/) represent.\r\n  In pushgateway I guess the job is still just overal main idea\r\n  and instance is about final unique, err instance.\r\n\r\n\r\nPrometheus requires linux [line endings.](\r\nhttps://github.com/prometheus/pushgateway/issues/144)\u003cbr\u003e\r\nThe \"\\`n\" in the `$body` is to simulate it in windows powershell.\r\n\r\nAlso in powershell the grave(backtick) character - \\` \r\nis for [escaping stuff](https://ss64.com/ps/syntax-esc.html)\u003cbr\u003e\r\nHere it is also used to escape new line. This allows breaking a command\r\nin to multiple easier to read lines.\r\nThough it caused issues, introducing space where it should not be,\r\nthats why `-uri` is always full length in the final script.\r\nGod damn fragile powershell.\r\n\r\n`test.ps1`\r\n```ps1\r\n$body = \"storage_diskC_free_space_bytes 32`n\"\r\n\r\nInvoke-RestMethod `\r\n    -Method PUT `\r\n    -Uri \"http://10.0.19.4:9091/metrics/job/veeam_report/instance/PC1\" `\r\n    -Body $body\r\n```\r\n\r\n* in the $body we have name of the metrics - `storage_diskC_free_space_bytes`\u003cbr\u003e\r\n  and the value of that metrics - `32`\u003cbr\u003e\r\n* in the url, after `10.0.19.4:9091/metrics/`, we have two labels defined\u003cbr\u003e\r\n `job=veeam_report` and `instance=PC1`\u003cbr\u003e\r\n  note the pattern, name of a label and value of it, they always must be in pair.\r\n  They can be named whatever, but `job` and `instance` are customary\r\n\r\nHeres how the data look in prometheus when executing `storage_diskC_free_space_bytes` query\r\n\r\n![first_put](https://i.imgur.com/ZycWmHz.png)\r\n\r\nThe labels help us target the data in grafana.\r\n\r\n### first dashobard\r\n\r\n* create **new dashboard**, panel\r\n* switch type to **Status history**\r\n* select metric - `storage_diskC_free_space_bytes`\r\n* [query options](https://grafana.com/docs/grafana/next/panels-visualizations/query-transform-data/#query-options)\r\n  * min interval - 1h\r\n  * relative time - now-10h/h\r\n* to not deal with long ugly names add transformation - Rename by regex\u003cbr\u003e\r\n  Match - `.+instance=\"([^\"]*).*` - [explained](https://stackoverflow.com/questions/2013124/regex-matching-up-to-the-first-occurrence-of-a-character)\u003cbr\u003e\r\n  Replace - `$1`\r\n* can also play with transparency, legend, treshold for pretty colors\r\n\r\nshould look in the end somewhat like this\r\n\r\n![first_graph](https://i.imgur.com/KW3B9dd.png)\r\n\r\n*extra info*\u003cbr\u003e\r\n[Examples.](https://prometheus.io/docs/prometheus/latest/querying/examples/)\r\nthis command deletes all metrics on prometheus, assuming api is enabled\u003cbr\u003e\r\n`curl -X POST -g 'http://10.0.19.4:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~\".*\"}'`\r\n\r\nSo theres the proof of concept of being able to send data to pushgateway\r\nand visualize them in grafana\r\n\r\n### PromQL basics\r\n\r\n[Here's](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/prometheus_grafana_loki#promql-basics)\r\nmy basic understanding.\u003cbr\u003e\r\nHow prometheus stores data, how to query, difference between instant vector\r\nand range vector, some links.\r\n\r\n\u003c/details\u003e\r\n\r\n---\r\n---\r\n\r\n# The powershell script\r\n\r\n![script_pic](https://i.imgur.com/Vb2Olmx.png)\r\n\r\n**The Script: [veeam_prometheus_info_push.ps1](https://github.com/DoTheEvo/veeam-prometheus-grafana/blob/main/veeam_prometheus_info_push.ps1)**\r\n\r\nThe script itself should be pretty informative with the comments in it.\u003cbr\u003e\r\n\r\nTested with VBR **v12**\u003cbr\u003e\r\nMight work with v11, except for agent-based backups as there were bugs\r\nin new cmdlets in that version.\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eChangelog\u003c/summary\u003e\r\n\r\n* v0.4\r\n  * added $ErrorActionPreference = \"Stop\"\r\n    which will terminate script's execution on any error\r\n  * job run time window calculation changed from the endtime to startime\r\n  * detection of a job being a full backup is now separate part and\r\n    done after the backup ends\r\n* v0.3\r\n  * huge rewrite\r\n* v0.2\r\n  * added pushing of repository disk usage info\r\n  * changed metrics name to include units\r\n  * general cleanup\r\n* v0.1 - the initial script\r\n\r\n\u003c/details\u003e\r\n\r\n#### Get-VBRJob and Get-VBRComputerBackupJob\r\n\r\nVeeam is now warning with every use of `Get-VBRJob` cmdlet that \r\nfuture versions will not be returning agent-based backup jobs.\r\nSo to avoid tech debt, the script uses\r\n`Get-VBRComputerBackupJob` and `Get-VBRComputerBackupJobSession`\r\nand got bigger and messier because of it, but should be more ready for that future.\r\n\r\n#### Job result codes\r\n\r\n* 0 = success\r\n* 1 = warning\r\n* 2 = failed\r\n* -1 =  running\r\n* -11 =  running full backup or full synthetic backup\r\n* 99 = disabled or not scheduled\r\n\r\nThe double digit ones are addition by the script.\u003cbr\u003e \r\nAlso agent based backups needed a rewrite of their return values,\r\nas they used different ones.\r\n\r\n#### Job run visualization\r\n\r\nThis visualization of runs is not precise, can be shifted some on the time line,\r\nbut it should be enough for general overview.\r\n\r\nJob themselves report that they are running but this can miss short running jobs.\r\nSo in adition the script checks the last jobs start time,\r\nif it was within the last hour the result code is set to `-1`. So every job\r\nis shown at least 1 hour long.\u003cbr\u003e\r\nThis also means that if the script would be scheduled to run periodically\r\nat intervals longer than an hour, it might miss runs.\r\nThe default deployment is every 30 minutes.\r\n\r\nUntil a job is finished we dunno if a run was a full backup or a full syntenthic,\r\nso there is also check of the last end time. If it was within the last hour\r\nand it was full/syntenthic_full, it changes the result code to `-11`.\r\n\r\n#### Data size and Backup size\r\n\r\n* Data size - The size of the data being backedup.\u003cbr\u003e\r\n  There is an issue of being unable to get the correct size for agent based\r\n  backups that target specific folders. If the backup target would be \r\n  entire machine or a partition, the data would be correct.\u003cbr\u003e\r\n  To get at least some approximation, the size of the last vbk file is used,\r\n  multiplied by `1.3` to account for some compression.\r\n* Backup size - the combined size of all backups of the job.\r\n\r\n# DEPLOY.cmd file\r\n\r\nTo ease the deployment.\r\n\r\n* Download [this repo.](https://github.com/DoTheEvo/veeam-prometheus-grafana/archive/refs/heads/main.zip)\r\n* Extract.\r\n* Edit `veeam_prometheus_info_push.ps1`\u003cbr\u003e\r\n  set `$BASE_URL` and `$GROUP` name.\r\n* Run `DEPLOY.cmd` as an administrator.\r\n* Done.\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eWhat happens under the hood:\u003c/summary\u003e\r\n\r\n* DEPLOY.cmd - checks if it runs as an administrator, ends if not.\r\n* DEPLOY.cmd - creates directory `C:\\Scripts` if it does not exists.\r\n* DEPLOY.cmd - checks if the script already exists, if it does,\r\n               renames it by adding a random suffix.\r\n* DEPLOY.cmd - copies veeam_prometheus_info_push.ps1 in to `C:\\Scripts`.\r\n* DEPLOY.cmd - imports taskscheduler xml task named veeam_prometheus_info_push.\r\n* TASKSCHEDULER - the task executes every 30 minutes, at xx:15 and xx:45,\r\n                  with random delay of 30 seconds.\r\n* TASKSCHEDULER - the task runs with the highest privileges as user - SYSTEM (S-1-5-18).\r\n* DEPLOY.cmd - enables powershell scripts execution on that windows PC.\r\n* DEPLOY.cmd - `Unblock-File` to allow the script execution when not created localy.\r\n\r\n\u003c/details\u003e\r\n\r\n# Pushgateway\r\n\r\n![pic_pushgateway](https://i.imgur.com/64Fqzfd.png)\r\n\r\nPushed data can be checked On Pushgateway's url.\r\n\r\nTo delete all data from pushgateway\r\n\r\n* from web interface theres a button\r\n* `curl -X PUT 10.0.19.4:9091/api/v1/admin/wipe`\r\n* `curl -X PUT https://push.example.com/api/v1/admin/wipe`\r\n\r\n### Periodily wiping clean the pushgateway\r\n\r\nWithout any action the pushed metrics sit on the pushgateway **forever**.\r\n[This is intentional.](https://github.com/prometheus/pushgateway/issues/19)\u003cbr\u003e\r\nIt is **essential** to wipe pushgateway clean daily to better visualize lack of\r\nnew reports coming in.\r\n\r\nFor this the dockerhost can have a simple systemd service and a timer.\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eHow to setup systemd pushgateway_wipe.service\u003c/summary\u003e\r\n\r\nIn `/etc/systemd/system/`\r\n\r\n`pushgateway_wipe.service`\r\n```ini\r\n[Unit]\r\nDescription=wipe clean prometheus pushgateway\r\n\r\n[Service]\r\nType=simple\r\nExecStart=curl -X PUT https://push.example.com/api/v1/admin/wipe\r\n```\r\n\r\n`pushgateway_wipe.timer`\r\n```ini\r\n[Unit]\r\nDescription=wipe clean prometheus pushgateway\r\n \r\n[Timer]\r\nOnCalendar=00:19:00\r\n \r\n[Install]\r\nWantedBy=timers.target\r\n```\r\n\r\nenable the timer: `sudo systemctl enable pushgateway_wipe.timer`\r\n\r\n\u003c/details\u003e\r\n\r\n# Prometheus\r\n\r\n![pic_prometheus](https://i.imgur.com/7uFdC6J.png)\r\n\r\nIn the compose file the data retention is set to 45 days.\r\n\r\n* `--storage.tsdb.retention.time=45d`\r\n\r\nNot much really to do once it runs. Checking values can be done through grafana,\r\nand for deletion one needs to use api.\u003cbr\u003e\r\nBut still, one can access its web gui from LAN side with `\u003cdockerhost\u003e:9090`,\r\nor can setup web access to it from the outside like for grafana and pushgateway.\r\n\r\n[Official documentation on queries](https://prometheus.io/docs/prometheus/latest/querying/basics/)\r\n\r\nTo query something just write plain metrics name, like `veeam_job_result_info`.\r\nIn the table tab it shows result from a recent time window. Switching to graph\r\ntab allows larger time range.\r\n\r\nMore targeted query, with the use of regex, signified by `=~`\r\n\r\n  * `veeam_job_result_info{instance=~\"Backup Copy Job.*\"}`\r\n\r\nTo delete all metrics on prometheus\r\n\r\n  * `curl -X POST -g 'http://10.0.19.4:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~\".*\"}'`\r\n\r\nTo delete metrics of an instance or group\r\n\r\n* `curl -X POST -g 'https://prom.example.com/api/v1/admin/tsdb/delete_series?match[]={instance=~\"^Backup.Copy.Job.*\"}'`\r\n* `curl -X POST -g 'https://prom.example.com/api/v1/admin/tsdb/delete_series?match[]={group=~\"CocaCola\"}'`\r\n\r\nTheres no white space in the query, so dots are used.\r\n\r\n# Grafana dashboard\r\n\r\n![dashboard](https://i.imgur.com/FjpN76I.png)\r\n\r\nThe json file in this repo can be imported in to grafana.\r\n\r\n* [VBR_dashboard_v2.json](https://github.com/DoTheEvo/veeam-prometheus-grafana/blob/main/VBR_dashboard_v2.json)\r\n* Dashboards \u003e New \u003e Import \u003e paste json\r\n\r\nChangelog\r\n\r\n* v2 - changed the initial time ranges, fixed last run and last report times\r\n* v1 - the initial dashboard \r\n\r\nTo set the dashboard to be shown right away when visiting the domain\u003cbr\u003e\r\nUser (right top corner) \u003e Profile \u003e Home Dashboard \u003e Set \u003e Save\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e\u003ch1\u003eSteps to manually recreate dashboard\u003c/h1\u003e\u003c/summary\u003e\r\n\r\n![panel-status-history](https://i.imgur.com/nbj9kJb.png)\r\n\r\n### Veeam Status History\r\n\r\nThe first panel is for seeing last X days backup history, at quick glance\r\n\r\n* Visualization = Status history\r\n* Data source = Prometheus\r\n* Query, switch from builder to code\r\n  `veeam_job_result_info{job=\"veeam_job_report\"}`\r\n* Query options \u003e Min interval = 1h\u003cbr\u003e\r\n  This sets the \"resolution\" of status history panel,\u003cbr\u003e\r\n  but data are renewed by default only every 30min.\u003cbr\u003e\r\n  During the first setup something smaller like 10min looks good.\r\n* two ways to have nice labels\r\n  * Query \u003e Options \u003e Legend \u003e switch from `Auto` to `Custom`\u003cbr\u003e\r\n    Legend = `{{name}} | {{group}}`\r\n  * Transform \u003e Rename by regex\u003cbr\u003e\r\n    Match = `.+group=\"([^\"]*).+instance=\"([^\"]*).*`\u003cbr\u003e\r\n    Replace = `$2 | $1`\r\n* Panel \u003e title = `Veeam Status History`\r\n* Status history \u003e Show values = never\r\n* Legend \u003e Visibility = off\r\n* Value mapping\r\n  * 0 = Successful; Green\r\n  * 1 = Warning; Yellow\r\n  * 2 = Failed; Red\r\n  * -1 = Running; Blue\r\n  * -11 = Full Backup; Purple\r\n  * 99 = Disabled | Unscheduled; Grey\r\n\r\n---\r\n\r\n![disk-use](https://i.imgur.com/Ijw2WoM.png)\r\n\r\n### Repositories Disk Use\r\n\r\nThis panel shows how full repositories are.\r\n\r\nUnfortunately grafana is not as capable as I hoped. While their example\r\n[shows](https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/bar-gauge/)\r\nexactly what I wanted, they cheated by picking the same max value for all disks.\r\nSo no nice GB and TB info, just percent.\u003cbr\u003e\r\nTried to [float](https://github.com/grafana/grafana/discussions/66159)\r\nthe idea of maybe addressing this in their discussion on github.\r\n\r\n* Visualization = Bar gauge\r\n* Data source = Prometheus\r\n* Query, switch from builder to code\r\n  ```\r\n  (veeam_repo_total_size_bytes{job=\"veeam_repo_report\"}\r\n  - veeam_repo_free_space_bytes{job=\"veeam_repo_report\"})\r\n  / ((veeam_repo_total_size_bytes{job=\"veeam_repo_report\"}) /100)\r\n  ```\r\n* Query \u003e Options \u003e Legend \u003e switch from `Auto` to `Custom`\u003cbr\u003e\r\n  Legend = `{{name}} | {{server}} | {{group}}`\r\n* Panel \u003e title = `Repositories Disk Use`\r\n* Bar gauge \u003e Display mode \u003e Basic\r\n* Standard options \u003e Unit = Misc \u003e Percent (0-100)\r\n* Standard options \u003e Min = 0\r\n* Standard options \u003e Max = 100\r\n* Standard options \u003e Decimals = 0\r\n* Standard options \u003e Display Name = `${__field.displayName}`\u003cbr\u003e\r\n  Needed [if only one repository](https://github.com/grafana/grafana/issues/48983),\r\n  to show the name under the bar.\r\n* Thresholds\r\n  * 90 = red\r\n  * 75 = Yellow\r\n  * base = green\r\n\r\n---  \r\n\r\n![panel-table](https://i.imgur.com/OCbIiBF.png)\r\n\r\n### Job's Details\r\n\r\nThis panel is a table with more details about jobs.\r\n\r\n* Visualization = Table\r\n* Data source = Prometheus\r\n* Query, switch from builder to code\r\n  `veeam_job_result_info{job=\"veeam_job_report\"}`\r\n  * Query options \u003e Format = Table\u003cbr\u003e\r\n* This results in a table where each job's last result is shown,\r\n  plus labels and their values.\u003cbr\u003e\r\n  One could start cleaning it up with a Transform,\r\n  but there are other metrics missing and the time stuff is in absolute values\r\n  instead of x minutes/hours ago.\u003cbr\u003e\r\n  So before cleaning, more mess will be added.\r\n* [Rename](https://i.imgur.com/2CVyvWQ.gif) the original query\r\n  from `A` to `result`.\u003cbr\u003e\r\n  This renaming will be used in all following queries so that the fields\r\n  are distinguishable in transformation later.\r\n* Create following queries, the first line is the new name,\r\n  the second is the query code itself.\u003cbr\u003e\r\n  Every query has in Options \u003e Type set to **table**.\r\n  * `data_size`\u003cbr\u003e\r\n    `veeam_job_data_size_bytes{job=\"veeam_job_report\"}`\r\n  * `backup_size`\u003cbr\u003e\r\n    `veeam_job_backup_size_bytes{job=\"veeam_job_report\"}`\r\n  * `restore_points`\u003cbr\u003e\r\n    `veeam_job_restore_points_total{job=\"veeam_job_report\"}`\r\n  * `job_runtime`\u003cbr\u003e\r\n     ```\r\n     veeam_job_end_time_timestamp_seconds{job=\"veeam_job_report\"} \r\n     - veeam_job_start_time_timestamp_seconds{job=\"veeam_job_report\"}\r\n     ```\r\n  * `last_job_run`\u003cbr\u003e\r\n    `time()-last_over_time(veeam_job_end_time_timestamp_seconds{job=\"veeam_job_report\"}[30d])`\r\n  * `last_report`\u003cbr\u003e\r\n    `time()-last_over_time(push_time_seconds{job=\"veeam_job_report\"}[30d])`\r\n* Now the results are there in many tables, switchable from a drop down menu,\r\n  but they need to be combined in to one table.\r\n* Transform \u003e Join by field \u003e Mode = OUTER; Field = instance\r\n* Now theres one long table with lot of duplication as every query brought \r\n  labels again. Now to clean it up.\r\n* Transform \u003e Organize fields\r\n  * Hide unwanted fields\u003cbr\u003e \r\n    Hiding anything with number 2, 3, 4, 5, 6, 7 in name works to get bulk of it gone\r\n  * Rename headers for fields that are kept.\r\n  * Reorder with drag and drop.\r\n* Panel options \u003e Title = `Job's Details`\r\n* Thresholds \u003e delete whatever is there; set Base to be transparent\r\n* Now the table will be modified using overrides\u003cbr\u003e\r\n  So that columns can be targeted separatly.\r\n* **Overrides**\r\n* Fields with name matching regex = `/Last Run|Runtime|Last Report/`\u003cbr\u003e\r\n  Standard options \u003e Unit = `seconds (s)`\u003cbr\u003e\r\n  Standard options \u003e Decimals = `0`\r\n* Fields with name matching regex = `/Data Size|Backup Size/`\u003cbr\u003e\r\n  Standard options \u003e Unit = `bytes(SI)`\u003cbr\u003e\r\n* Fields with name = `Result` \u003e Value mappings\u003cbr\u003e\r\n  * Value Mapping:\r\n    * 0 = Successful; Green\r\n    * 1 = Warning; Yellow\r\n    * 2 = Failed; Red\r\n    * -1 = Running; Blue\r\n    * -11 = Full Backup; Purple\r\n    * 99 = Disabled | Unscheduled; Grey\r\n    * the colors should be muted by transparency ~0.4\r\n  * Cell options \u003e Cell type\r\n    * `Colored background`\r\n    * `Gradient`\r\n* Fields with name = `Group` \u003e Value mappings\u003cbr\u003e\r\n  * Value Mapping:\r\n    * 0 = water; Green\r\n    * 1 = CocaCola; Yellow\r\n    * 2 = beer; Red\r\n    * the colors should be muted by transparency ~0.3\r\n  * Cell options \u003e Cell type\r\n    * `Colored background`\r\n    * `Gradient`\r\n* Save and look.\r\n* Adjusting column width will be creating overrides for that column.\u003cbr\u003e\r\n  Just to be aware, as it might be weird seeing like 12 overrides afterwards.\r\n\r\n\u003c/details\u003e\r\n\r\n----\r\n----\r\n\r\n# Grafana alerts\r\n\r\n![email_alert](https://i.imgur.com/Y01YoBw.png)\r\n\r\nGrafana alerts help with the reliability and danger of a failure going unnoticed.\u003cbr\u003e\r\nEspecially considering the dynamic nature of this setup, meaning that if reporting\r\nstops for any reason, after some time there is no indication that a job\r\neven existed, let alone failed.\r\n\r\nBefore getting to alerts, first the delivery mechanism and policy.\r\n\r\n### Contact points\r\n\r\nGrafana \u003e Alerting \u003e Contact points\r\n\r\n##### email\r\n\r\nJust needs corectly set some smtp stuff in the `.env` file for grafana,\r\nas can be seen in the setup section.\u003cbr\u003e\r\nThe contact point already exists, named `grafana-default-email`.\u003cbr\u003e\r\nCan be tested if it actually works when editing the contact point.\r\n\r\n\r\n##### ntfy\r\n\r\nPush notifications for a phone or desktop using selfhosted [ntfy](https://ntfy.sh/).\u003cbr\u003e\r\nDetailed setup of running ntfy as a docker container\r\n[here.](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/gotify-ntfy-signal#grafana-to-ntfy)\u003cbr\u003e\r\n\r\n* New contact point\r\n* Name = `ntfy`\r\n* Integration = `Webhook`\r\n* URL = `https://ntfy.example.com/veeam`\u003cbr\u003e\r\n  or if grafana-to-ntfy is already setup on the same docker network,\r\n  then URL = `http://grafana-to-ntfy:8080`\r\n* plain ntfy does not need credentials,\u003cbr\u003e\r\n  grafana-to-ntfy needs the ones from its `.env` file set.\r\n* Disable resolved message = check\r\n* Test\r\n* Save\r\n\r\nIssue I noticed now in testing with ntfy, is that if you get multiple failures\r\nit wont deliver. Could be solved by not letting it send the complex grafana\r\njson full of dynamic values, but just some generic static text about a failure.\u003cbr\u003e\r\nWill eventually look in to it, or report it to the dev.\r\n\r\n### Notification policies \r\n\r\nEditing the `Default policy`, making sure the contact point is the correct one\r\nis enough if just one contact point is planned to be used. Like just email.\r\n\r\nOf note are `Timing options` inside policy, that sets how often a firing alarm\r\nwill resend notification. Default is 4h, +5m for group interval.\r\n\r\nTo fire notification on multiple contact points,\r\nfor alerts in `veeam_alerts` folder:\r\n\r\n* Within the `Default policy` adding `+ New nested policy`.\r\n* Matching labels: `grafana_folder` `=` `veeam_alerts`\u003cbr\u003e\r\n  Select `Contact point` - `grafana-default-email`\u003cbr\u003e\r\n  Enable - `Continue matching subsequent sibling nodes`\u003cbr\u003e\r\n  Which means that after matching, it will continue to look for \r\n  other policies that would also match\r\n* Do the same again for a new nested policy, but use contact point to `ntfi`.\r\n\r\nThe `Default policy` is applied only if no other policy fits.\r\n\r\n## Alerts\r\n\r\nCurrently these alerts are not long term tested.\u003cbr\u003e\r\nThey should work, but should be considered in development.\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e\u003ch3\u003eAlert rule - Backup Failed or Warning\u003c/h3\u003e\u003c/summary\u003e\r\n\r\n- **1 Set an alert rule name**\r\n  - Rule name = `veaam_backup_failed_or_warning`\r\n- **2 Set a query and alert condition**\r\n  - **A** - Prometheus; set Last 2d\r\n    - Options \u003e Min step = 15m\r\n    - switch from builder to code\r\n    - `veeam_job_result_info{job=\"veeam_job_report\"}`\r\n  - **B** - Reduce\r\n    - Function = Last\r\n    - Input = A\r\n    - Mode = Strict\r\n  - **C** - Treshold\r\n    - Input = B\r\n    - is within range 0 to 3 (it's [not inclusive](https://github.com/grafana/grafana/issues/19193))\r\n    - Make this the alert condition\r\n- **3 Alert evaluation behavior**\r\n  - Folder = \"veeam_alerts\"\r\n  - Evaluation group (interval) = \"one_hour\"\u003cbr\u003e\r\n  - Evaluation interval = 1h\r\n  - For = 0s\r\n  - Configure no data and error handling\r\n    - Alert state if no data or all values are null = OK\r\n- **4 Add details for your alert rule**\r\n  - Metrics labels can be used here\r\n- **5 Notifications**\r\n  - nothing\r\n- Save and exit\r\n\r\n\u003c/details\u003e\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e\u003ch3\u003eAlert rule - Repo is 85% full\u003c/h3\u003e\u003c/summary\u003e\r\n\r\n- **1 Set an alert rule name**\r\n  - Rule name = `veaam_repo_full`\r\n- **2 Set a query and alert condition**\r\n  - **A** - Prometheus; set Last 2d\r\n    - Options \u003e Min step = 15m\r\n    - switch from builder to code\r\n      ```\r\n      (veeam_repo_total_size_bytes{job=\"veeam_repo_report\"}\r\n      - veeam_repo_free_space_bytes{job=\"veeam_repo_report\"})\r\n      / ((veeam_repo_total_size_bytes{job=\"veeam_repo_report\"}) /100)\r\n      ```\r\n  - **B** - Reduce\r\n    - Function = Last\r\n    - Input = A\r\n    - Mode = Strict\r\n  - **C** - Treshold\r\n    - Input = B\r\n    - is above `84`\r\n    - Make this the alert condition\r\n- **3 Alert evaluation behavior**\r\n  - Folder = \"veeam_alerts\"\r\n  - Evaluation group (interval) = \"one_hour\"\u003cbr\u003e\r\n  - Evaluation interval = 1h\r\n  - For = 0s\r\n  - Configure no data and error handling\r\n    - Alert state if no data or all values are null = OK\r\n- **4 Add details for your alert rule**\r\n  - Metrics labels can be used here\r\n- **5 Notifications**\r\n  - nothing\r\n- Save and exit\r\n\r\n\u003c/details\u003e\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e\u003ch3\u003eAlert rule - No report for 5 days\u003c/h3\u003e\u003c/summary\u003e\r\n\r\n- **1 Set an alert rule name**\r\n  - Rule name = `veaam_noreport_five_days`\r\n- **2 Set a query and alert condition**\r\n  - **A** - Prometheus; set Last 30 days (now-30d to now)\r\n    - switch from builder to code\r\n      `time()-last_over_time(push_time_seconds{job=\"veeam_job_report\"}[30d])`\r\n  - **B** - Reduce\r\n    - Function = Last\r\n    - Input = A\r\n    - Mode = Strict\r\n  - **C** - Treshold\r\n    - Input = B\r\n    - is above `432000`\r\n    - Make this the alert condition\r\n- **3 Alert evaluation behavior**\r\n  - Folder = \"veeam_alerts\"\r\n  - Evaluation group (interval) = \"twelve_hours\"\u003cbr\u003e\r\n  - Evaluation interval = 12h\r\n  - For = 0s\r\n  - Configure no data and error handling\r\n    - Alert state if no data or all values are null = Error\r\n- **4 Add details for your alert rule**\r\n  - nothing\r\n- **5 Notifications**\r\n  - nothing\r\n- Save and exit\r\n\r\n\u003c/details\u003e\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e\u003ch3\u003eAlert rule - No backup done for 5 days\u003c/h3\u003e\u003c/summary\u003e\r\n\r\n- **1 Set an alert rule name**\r\n  - Rule name = `veaam_nobackup_five_days`\r\n- **2 Set a query and alert condition**\r\n  - **A** - Prometheus; set Last 30 days (now-30d to now)\r\n    - switch from builder to code\r\n      `time()-last_over_time(veeam_job_end_time_timestamp_seconds{job=\"veeam_job_report\"}[30d])`\r\n  - **B** - Reduce\r\n    - Function = Last\r\n    - Input = A\r\n    - Mode = Strict\r\n  - **C** - Treshold\r\n    - Input = B\r\n    - is above `432000`\r\n    - Make this the alert condition\r\n- **3 Alert evaluation behavior**\r\n  - Folder = \"veeam_alerts\"\r\n  - Evaluation group (interval) = \"twelve_hours\"\u003cbr\u003e\r\n  - Evaluation interval = 12h\r\n  - For = 0s\r\n  - Configure no data and error handling\r\n    - Alert state if no data or all values are null = Error\r\n- **4 Add details for your alert rule**\r\n  - Metrics labels can be used here\u003cbr\u003e\r\n    nothing\r\n- **5 Notifications**\r\n  - nothing\r\n- Save and exit\r\n\r\n\u003c/details\u003e\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdotheevo%2Fveeam-prometheus-grafana","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdotheevo%2Fveeam-prometheus-grafana","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdotheevo%2Fveeam-prometheus-grafana/lists"}