{"id":15509427,"url":"https://github.com/bastelfreak/prometheusdemo","last_synced_at":"2026-01-31T05:02:47.831Z","repository":{"id":146050466,"uuid":"216395959","full_name":"bastelfreak/prometheusdemo","owner":"bastelfreak","description":"My talk + demo about scaleable node_exporter rollout with Puppet","archived":false,"fork":false,"pushed_at":"2020-02-04T10:34:01.000Z","size":22319,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"production","last_synced_at":"2025-06-10T10:49:06.433Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://bastelfreak.de/prometheusdemo/#1","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bastelfreak.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"bastelfreak","custom":"https://www.paypal.me/bastelfreak"}},"created_at":"2019-10-20T16:57:29.000Z","updated_at":"2022-07-05T07:35:54.000Z","dependencies_parsed_at":"2023-07-31T00:15:30.868Z","dependency_job_id":null,"html_url":"https://github.com/bastelfreak/prometheusdemo","commit_stats":{"total_commits":129,"total_committers":1,"mean_commits":129.0,"dds":0.0,"last_synced_commit":"d43694406fae3d8c2def405f7ff096a5b502480f"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bastelfreak/prometheusdemo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bastelfreak%2Fprometheusdemo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bastelfreak%2Fprometheusdemo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bastelfreak%2Fprometheusdemo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bastelfreak%2Fprometheusdemo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bastelfreak","download_url":"https://codeload.github.com/bastelfreak/prometheusdemo/tar.gz/refs/heads/production","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bastelfreak%2Fprometheusdemo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28929864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T04:05:25.756Z","status":"ssl_error","status_checked_at":"2026-01-31T04:02:35.005Z","response_time":128,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T09:42:22.136Z","updated_at":"2026-01-31T05:02:47.817Z","avatar_url":"https://github.com/bastelfreak.png","language":"HTML","funding_links":["https://github.com/sponsors/bastelfreak","https://www.paypal.me/bastelfreak"],"categories":[],"sub_categories":[],"readme":"# Deploy Prometheus exporters with Puppet at scale\n\n## Table of contents\n\n* [What is this about](#what-is-this-about)\n* [Architecture](#architecture)\n* [Notes](#notes)\n* [How to get started](#how-to-get-started)\n* [Vagrant cheat sheet](#vagrant-cheat-sheet)\n* [docs](#docs)\n* [License](#license)\n\n## What is this about\n\nThis is a demo environment to show the automatic rollout of Prometheus\nexporters with Puppet. A Vagrantfile is available that allows you to test and\ndiscover all of this locally.\n\nWe will bootstrap a CentOS 7 server with:\n\n* Puppetserver\n* Consul\n* node\\_exporter\n* Nginx as reverse TLS proxy\n\nThe second machine is a CentOS 7 client with:\n\n* Consul\n* node\\_exporter\n\n## Architecture\n\n![image](prometheus03.svg)\n\nOn the server, we setup a Consul master for service registration and service\ndiscovery. This allows us to register each node\\_exporter as a service.\nPrometheus can connect to consul, get a list of services and scrape them.\n\nTo make the whole setup more secure, we wil do two things:\n\n### Authentication and Authorisation\n\nIt's important that only trusted sources are allowed to access the exporters.\nTo achive this, We bind them to localhost and install an nginx in front of\nthem. All the exporters don't support any type of authentication or\nauthorisation. But nginx is able to not only handle generic TLS traffic,\nit can also validate TLS client certificates. Since we already have Puppet\non each box, we can reuse the Puppet client certificates for this. Our\nPrometheus daemon will connect to each exporter with a client certificate.\n\n### Firewalling\n\nSince firewalling needs to be scaleable, and consul demands a meshed network\nbetween each agent. We will deploy custom iptables chains and ipsets.\n\n## Notes\n\nWe heavily utilize the Puppet TLS certificates. To do this, we require a\nPuppetserver. To make this work as easy as possible, we will use a CentOS 7\nbase image. CentOS 8 isn't well supported yet by most modules. To prove the\nawesomenes of this setup, we will use an Archlinux (and CentOS 7 as well)\nclient.\n\nOur default branch in this repository is `production`. We use r10k to deploy\npuppet environment. r10k matches git branches to those environments and the\ndefault environment name is production. Switching the default branch name from\n`master` to `production` is easer then updating the puppet configuration.\n\n### Known issues\n\n### VM provisioning order\n\nThe clients depend on the server. If you start this project with a\n`vagrant up centosclient`, it won't work properly. You need to start with the\nserver:\n\n```sh\nvagrant up server\n```\n\n### TLS errors\n\nOur virtual machines don't have a proper FQDN with dots in it, only a hostname.\nIf your local `/etc/resolv.conf` file has a `search` set, Virtualbox will\nappend it to the hostname during TLS certificate creation.\n\nThe error during provisioning might look liks this:\n\n```\n    server: Warning: Unable to fetch my node definition, but the agent run will continue:\n    server: Warning: Server hostname 'prometheus' did not match server certificate; expected one of prometheus.fritz.box, DNS:puppet, DNS:prometheus.fritz.box\n    server: Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': Server hostname 'prometheus' did not match server certificate; expected one of prometheus.fritz.box, DNS:puppet, DNS:prometheus.fritz.box\n```\n\nBecause of a `/etc/resolv.conf` on the host that looks like this:\n\n```sh\n# Generated by NetworkManager\nsearch fritz.box\nnameserver 192.168.178.1\nnameserver fd00::e228:6dff:fe76:231c\n```\n\nRemove the `search` option and provision the virtual machines again.\n\nLatest versions of this repository contain an sed statement during the\nprovisioning that will remove the mentioned `search` line.\n\n## How to get started\n\nSo, you're interested in the Vagrant environment and want to use it. First of\nyou need to install virtualbox and Vagrant. A simple `vagrant status` should\nlook like this:\n\n```\n~/prometheusdemo $ vagrant status\nCurrent machine states:\n\nserver                    not created (virtualbox)\ncentosclient              not created (virtualbox)\narchclient                not created (virtualbox)\nubuntuclient              not created (virtualbox)\n\nThis environment represents multiple VMs. The VMs are all listed\nabove with their current state. For more information about a specific\nVM, run `vagrant status NAME`.\n~/prometheusdemo $\n```\n\nThe server needs to be started first. It will provide us a Puppetserver +\nPrometheus:\n\n```sh\nvagrant up server\n```\n\nAfterwards we can provision any of the clients. You can choose between CentOS,\nArch Linux or Ubuntu. You cannot start multiple instance of one machine.\nDepending on the amount of memory on your box, you can two or all three\nclients:\n\n```sh\nvagrant up centosclient\nvagrant up archclient\nvagrant up ubuntuclient\n```\n\nVagrant wraps ssh for us. If you want to enter one machine, you can do it like\nthis:\n\n```sh\nvagrant ssh server\n```\n\nThe Server gets a dedicated role:\n[modules/roles/manifests/server.pp](https://github.com/bastelfreak/prometheusdemo/blob/production/modules/roles/manifests/server.pp)\n\nAll clients share the same role:\n[modules/roles/manifests/client.pp](https://github.com/bastelfreak/prometheusdemo/blob/production/modules/roles/manifests/client.pp)\n\nExercise: What is the roles and profiles pattern?\n\n## Vagrant cheat sheet\n\n### Status of a Vagrant environment\n\n```sh\nvagrant status\n```\n\n### Start one / all VMs\n\n```sh\n# start all VMs from a Vagrantfile\nvagrant up\n```\n\n```sh\n# start a specific one\nvagrant up server\n```\n\n### Sync /etc/hosts entries to all VMs\n\nSoooooooooo, technology. Why does software suck so hard? It's a common usecase\nfor Virtualbox users to spawn multiple virtual machines. In most of those\nsituations, you want to connect from one virtual machine to another. On\nWindows host systems, virtual box is smart enough and does local DNS\nresolution. However, it doesn't do it on linux....\n\nAs a workaround, you can write your IP-Addresses to every `/etc/hosts` file.\nThis sucks. This is manual work... Thankfully, somebody wrote a Vagrant plugin\nfor this job\n[Vagrant hostmanager plugin](https://github.com/devopsgroup-io/vagrant-hostmanager#vagrant-host-manager)\n\nYou need to install it:\n\n```sh\nvagrant plugin install vagrant-hostmanager\n```\n\nAfterwards you can add a few entries to your [Vagrantfile](./Vagrantfile):\n\n```\nconfig.hostmanager.enabled = true         # Update /etc/hosts with entries from other VMs\nconfig.hostmanager.manage_host = false    # Don't update /etc/hosts on the Hypervisor\nconfig.hostmanager.include_offline = true # Also document offline VMs\n```\n\nIf you still want to update the hosts file on all running machines, do:\n\n```sh\n$ vagrant hostmanager\n[vagrant-hostmanager:guest] Updating hosts file on the virtual machine server...\n[vagrant-hostmanager:guest] Updating hosts file on the virtual machine centosclient...\n[vagrant-hostmanager:guest] Updating hosts file on the virtual machine archclient...\n```\n\nThe `/etc/hosts` file will look like this:\n\n```\n$ cat /etc/hosts\n127.0.0.1 centosclient  centosclient\n127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4\n::1         localhost localhost.localdomain localhost6 localhost6.localdomain6\n\n## vagrant-hostmanager-start\n192.168.33.10 prometheus\n\n192.168.33.11 centosclient\n\n192.168.33.12 archclient\n\n## vagrant-hostmanager-end\n\n```\n\nThe Software is smart enough to detect if the IP-Addresses are already listed\nin the file. They won't be added multiple times (it's idempotent).\n\n### Vagrant fails to start a VM\n\nError:\n\n```\n$ vagrant up server\nBringing machine 'server' up with 'virtualbox' provider...\n==\u003e server: Importing base box 'centos/7'...\n==\u003e server: Matching MAC address for NAT networking...\n==\u003e server: Checking if box 'centos/7' version '1905.1' is up to date...\nA VirtualBox machine with the name 'server' already exists.\nPlease use another name or delete the machine with the existing\nname, and try again.\n```\n\nCheck if Vagrant knows anything about the VM:\n\n```\n$ vagrant global-status\nid       name   provider   state    directory\n------------------------------------------------------------------------\n02c90f7  server virtualbox poweroff /home/bastelfreak/prometheusdemo\n\nThe above shows information about all known Vagrant environments\non this machine. This data is cached and may not be completely\nup-to-date (use \"vagrant global-status --prune\" to prune invalid\nentries). To interact with any of the machines, you can go to that\ndirectory and run Vagrant, or you can use the ID directly with\nVagrant commands from any directory. For example:\n\"vagrant destroy 1a2b3c4d\"\n```\n\nIf so, delete the VM:\n\n```\n$ vagrant destroy 02c90f7\n    server: Are you sure you want to destroy the 'server' VM? [y/N] y\n==\u003e server: Destroying VM and associated drives...\n```\n\nIf that fails, virtualbox might still know it:\n\n```\n$ VBoxManage list vms\n\"\u003cinaccessible\u003e\" {57d4f57d-4dda-44fc-acf9-f01826f52e5e}\n\"server\" {0cec0d5b-6706-426f-8453-900a5ab7b91d}\n```\n\nif so, delete it within Virtualbox:\n\n```\n$ VBoxManage unregistervm 0cec0d5b-6706-426f-8453-900a5ab7b91d --delete\n0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%\n```\n\n## Docs\n\nThis repository contains the above explained vagrant environment and a\nslidedeck. I did this presentation at the OSMC 2019 and at the\nconfiguration management camp 2020 in Gent.\n\n### OSMC 2019\n\n* [online version](https://bastelfreak.de/osmc2019/#1)\n* [pdf](Rollout_all_your_Prometheus_exporters_with_puppet_OSMC2019.pdf)\n* [Video recording](https://www.youtube.com/watch?v=-ijO-g4_7rU)\n\n### configuration management camp 2019\n\n* [online version](https://bastelfreak.de/prometheusdemo/#1)\n* [pdf](Rollout_all_your_Prometheus_exporters_with_puppet_-_CFGMGMTCAMP2020.pdf)\n\n## License\n\nThis project contains two licenses. The code is licensed as\n[GNU Affero General Public License v3.0](LICENSE). The documentation/text is\nlicsensed as [CC BY-NC-SA 4.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbastelfreak%2Fprometheusdemo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbastelfreak%2Fprometheusdemo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbastelfreak%2Fprometheusdemo/lists"}