{"id":13562423,"url":"https://github.com/fhemberger/nomad-demo","last_synced_at":"2025-05-12T08:32:13.623Z","repository":{"id":40466133,"uuid":"254154745","full_name":"fhemberger/nomad-demo","owner":"fhemberger","description":"Vagrant based demo setup for running Hashicorp Nomad","archived":false,"fork":false,"pushed_at":"2023-07-01T23:40:50.000Z","size":962,"stargazers_count":130,"open_issues_count":1,"forks_count":20,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-01T04:26:53.219Z","etag":null,"topics":["consul","coredns","demo","nomad","prometheus","traefik","vagrant"],"latest_commit_sha":null,"homepage":"http://nomadproject.io","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fhemberger.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-04-08T17:23:19.000Z","updated_at":"2025-02-27T16:48:17.000Z","dependencies_parsed_at":"2024-01-08T01:44:56.257Z","dependency_job_id":"25bd6713-f03f-4ce3-980e-e75a1e86cb83","html_url":"https://github.com/fhemberger/nomad-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fhemberger%2Fnomad-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fhemberger%2Fnomad-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fhemberger%2Fnomad-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fhemberger%2Fnomad-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fhemberger","download_url":"https://codeload.github.com/fhemberger/nomad-demo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253702130,"owners_count":21949923,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul","coredns","demo","nomad","prometheus","traefik","vagrant"],"created_at":"2024-08-01T13:01:08.454Z","updated_at":"2025-05-12T08:32:13.178Z","avatar_url":"https://github.com/fhemberger.png","language":"HCL","funding_links":[],"categories":["HCL","Examples / Demos","Demo and Learning Resources"],"sub_categories":[],"readme":"# Nomad Demo\n\nKubernetes is ubiquitous and an excellent platform, especially if you use the integrated offers of the big cloud providers like Amazon Web Services, Google Cloud or Microsoft Azure. But installing it on-premise can be quite a challenge: Many new concepts and abstraction layers need to be understood and managed in production, including several options for software defined networking and storage adapters.\n\nSmaller administration teams in particular are often not immediately up to this challenge, and the volume of software and services deployed in the end does not always justify such complexity.\n\nIt may be worth considering alternatives, especially if not only containers are to be managed centrally, but also other traditional workloads such as Java applications or VMs.\n\nThis demo project uses Consul, Nomad and Vault from HashiCorp:\n\n\u003cimg src=\".images/consul-logo.svg\" align=\"left\" width=\"200\" alt=\"\"\u003e \nConsul is a service networking solution to connect services across any runtime platform. Service registry, integrated health checks, and DNS and HTTP interfaces enable any service to discover and be discovered by other services.\u003cbr clear=\"both\"\u003e\u003cbr\u003e\n\n\u003cimg src=\".images/nomad-logo.svg\" align=\"left\" width=\"200\" alt=\"\"\u003e Nomad is a workload orchestrator that deploys and manages containers and non-containerized applications at scale. It addresses the technical complexity of workload orchestration across the cloud, on-prem, and hybrid infrastructure.\n\n\u003cimg src=\".images/vault-logo.svg\" align=\"left\" width=\"200\" alt=\"\"\u003e Vault allows you to store, access, and deploy secrets across applications, systems, and infrastructure. Vault centrally manages and enforces access to secrets and systems based on trusted sources of application and user identity.\n\n\n## Overview\n\n\u003cimg src=\".images/vm-overview.drawio.svg\" align=\"left\" alt=\"Overview of the created VMs and the deployed components\"\u003e\n\n[Traefik](https://docs.traefik.io/): Load balancer using Consul Catalog service discovery. All services registered in Consul with the tag `http` will be exposed as `\u003cservice name\u003e.demo`. In context of this demo setup, you'll have to adjust your `/etc/hosts` file accordingly ([see below](#user-content-etc-hosts)).\n\n[CoreDNS](http://coredns.io/): Provides name resolution for the `consul.` root domain and forwards DNS queries to either the local `*.consul.` zone or Cloudflare 1.1.1.1 as upstream DNS.\n\n[Prometheus](https://prometheus.io/): Collects basic metrics form Traefik, Consul (via [consul_exporter](https://github.com/prometheus/consul_exporter/)), Nomad and applications tagged with `prometheus`.\n\nAll three nodes run [Consul](https://consul.io/), [Nomad](https://nomadproject.io/) and [Vault](https://www.vaultproject.io/), with Nomad configured both as server (= control plane) and client (= node running the actual workload). As both Docker and a Java Runtime Environment is installed, Nomad can run containerized workloads and jar files.\n\nAn example deployment configuration for each is included in this setup.\n\n\u003cbr clear=\"both\"\u003e\u003cbr\u003e\n\n```\n🚨 REMEMBER, THIS IS JUST A DEMO SET UP AND IN NO WAY INTENDED TO RUN IN PRODUCTION! 🚨\n```\n\n\n## Setup\n\n### Prerequisites\n\nYou need three components to get the setup running:\n\n1. [VirtualBox](https://www.virtualbox.org/wiki/Downloads)\n2. [Vagrant](https://www.vagrantup.com/downloads.html) \u003csup id=\"a1\"\u003e[1](#f1)\u003c/sup\u003e\n3. [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) \u003e= 2.10 \u003csup id=\"a2\"\u003e[2](#f2)\u003c/sup\u003e\n\nIf you're working on macOS and have [Homebrew](https://brew.sh/) installed, you can setup everything with two commands:\n\n```sh\nbrew cask install vagrant virtualbox\nbrew install ansible\n```\n\nAfterwards, install dependencies for running the Ansible playbook:\n\n```sh\nansible-galaxy collection install community.general\n```\n\nYour host system will need at least 6GB of RAM and about 20GB of free hard disk (sorry, Vagrant VM boxes 🤷‍♂️).\n\nAlthough the Vagrant boxes are running Ubuntu LTS, Debian should work just as fine (PRs gladly accepted for RedHat Linux). The setup supports both Intel (`x86_64`/`amd64`) and ARM (`arm64`) architectures.\n\n### Installation\n\n1. **Clone this Git repository:**\n\n   ```sh\n   git clone --depth=1 https://github.com/fhemberger/nomad-demo.git\n   cd nomad-demo\n   ```\n\n2. **Create and provision virtual machines with Vagrant:**  \n   Vagrant will create four virtual machines with IPs _192.168.56.20–23_. This is the default IP range in VirtualBox. If your local network already uses this address range, you can define an alternate range in the `Vagrantfile` before continuing with the installation.\n\n   As there are some logical dependencies in the setup, it is split up in multiple parts:\n\n     1. Create the four VMs in VirtualBox with Vagrant\n     2. Setup the Consul cluster and elect a leader\n     3. Setup Vault (which requires the working Consul cluster),  \n        Nomad (which in turn requires a token from Vault) and the load balancer VM.\n     4. Deploy the provided demo jobs on Nomad\n\n     \u003cbr\u003e\n\n   ```sh\n   make\n   make run-jobs\n\n   # You can run `make help` to get a list of all available tasks.\n   ```\n\n3. \u003cstrong id=\"etc-hosts\"\u003eConfigure host names for all services:\u003c/strong\u003e  \n   Add the following lines to `/etc/hosts` on your machine. If you changed the IP range before in your `Vagrantfile`, make sure to adjust it here as well:\n\n   ```\n   192.168.56.20 traefik.demo\n   192.168.56.20 consul.demo\n   192.168.56.20 nomad.demo\n   192.168.56.20 vault.demo\n   192.168.56.20 grafana.demo\n   192.168.56.20 prometheus.demo\n   192.168.56.20 alertmanager.demo\n   192.168.56.20 hello-docker.demo\n   192.168.56.20 hello-java.demo\n   192.168.56.20 hello-vault.demo\n   ```\n\nTo check if everything is working correctly, go to http://traefik.demo, you should see the UI of the load balancer with a list of registered services:\n\n\u003ca href=\".images/screenshot-traefik01.png\"\u003e\u003cimg src=\".images/screenshot-traefik01.png\" align=\"left\" width=\"47.5%\" alt=\"Screenshot of the Traefik UI\"\u003e\u003c/a\u003e\n\u003ca href=\".images/screenshot-traefik02.png\"\u003e\u003cimg src=\".images/screenshot-traefik02.png\" align=\"right\" width=\"47.5%\" alt=\"Screenshot of the Traefik UI: All HTTP routes\"\u003e\u003c/a\u003e\n\n\u003cbr clear=\"both\"\u003e\u003cbr\u003e\n\nSites available after installation:\n\n- http://traefik.demo - Load balancer UI, see all registered services\n- http://consul.demo - Consul UI\n- http://nomad.demo - Nomad UI\n- http://vault.demo - Vault UI\n- http://grafana.demo - Grafana Dashboards\n- http://prometheus.demo - Prometheus metrics UI\n- http://alertmanager.demo - Prometheus Alertmanager UI\n\n### Removing all demo resources\n\nRun `make clean` to shut down and delete all created VMs. Afterwards remove the demo host names from `/etc/hosts`.\n\n\n## Working with Nomad\n\n### Deploying jobs\n\nIf you followed step 2 above, Prometheus, Grafana and the demo jobs will already be running on Nomad.\n\nIf you want to edit those jobs or deploy one of your own, there are two ways to do so:\n\n#### Running jobs from the UI\n\nThree example applications are included with this demo: [`hello-world-docker.nomad`](nomad_jobs/hello-world-docker.nomad), [`hello-world-java.nomad`](nomad_jobs/hello-world-java.nomad) and [`hello-world-vault.nomad`](nomad_jobs/hello-world-vault.nomad). Go to http://nomad.demo/ui/jobs/run, copy and paste one of the jobs into the editor and click \"Plan\".\n\nNomad performs a syntax check by dry-running the job on the scheduler without applying the changes yet. If you change settings in your job file later on, this step will also show a diff of all the changes (e.g. number of instances):\n\n\u003cimg src=\".images/screenshot-nomad.png\" alt=\"Screenshot of Nomad: Planning jobs\"\u003e\n\nClick \"Run\" to deploy the job to the Nomad cluster.\n\n#### Running jobs using the CLI\n\nIf you prefer to run the demos from the command line, you can use `vagrant ssh` to login to one of the nodes and the `nomad` CLI command to deploy them directly from the VM. The three example jobs are copied to the vagrant user's home directory on all instances, the node number doesn't matter, e.g.:\n\n```sh\nvagrant ssh consul-nomad-node1\n\n# Inside the VM\nnomad job run ~/nomad_jobs/hello-world-docker.nomad\n```\n\n### Stopping jobs\n\nGo to the [Job overview page](http://nomad.demo/ui/jobs), select a job, click \"Stop\" and confirm. Stopped jobs don't disappear immediately but remain in the \"Dead\" state until the garbage collection removes them completely.\n\n### Removing dead/completed jobs\n\nDead/completed jobs are cleaned up in accordance to the garbage collection interval (default: `1h`). You can force garbage collection using the System API endpoint which will run the global garbage collector:\n\n```sh\nvagrant ssh consul-nomad-node1 -c 'curl -X PUT http://localhost:4646/v1/system/gc'\n```\n\nIf you wish to lower the GC interval permanently for jobs, you can use the [`job_gc_threshold`](https://www.nomadproject.io/docs/agent/configuration/server.html#job_gc_threshold) configuration parameter within Nomad's [server config stanza](https://github.com/fhemberger/nomad-demo/blob/master/roles/nomad/templates/nomad.hcl.j2#L27).\n\n\n## What is Vault?\n\nVault is the Swiss Army knife for managing secrets and access rights across your entire platform, e.g.:\n\n- Store arbitrary passwords, API keys, etc. for your applications in the [key/value store](https://www.vaultproject.io/docs/secrets/kv).\n- Create custom, short lived credentials for [databases](https://www.vaultproject.io/docs/secrets/databases) and rotate root passwords regularly.\n- Generate dynamic API tokens for [Consul](https://www.vaultproject.io/docs/secrets/consul) and [Nomad](https://www.vaultproject.io/docs/secrets/nomad).\n- Create [X.509 certificates](https://www.vaultproject.io/docs/secrets/pki) and manage the [SSH access](https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-certificates) to your machines.\n- Manage access to your cloud environment ([AWS](https://www.vaultproject.io/docs/secrets/aws), [Azure](https://www.vaultproject.io/docs/secrets/azure), [Google Cloud](https://www.vaultproject.io/docs/secrets/gcp), etc.) for users and applications.\n- Authenticate users through [LDAP](https://www.vaultproject.io/docs/auth/ldap), [username/password](https://www.vaultproject.io/docs/auth/userpass), [GitHub](https://www.vaultproject.io/docs/auth/github), [Okta](https://www.vaultproject.io/docs/auth/okta), etc.\n\nAnd all this with granular access control and full audit trail.\n\n\n### Working with Vault\n\n\u003e This Vault integration is based on a [blog post](https://medium.com/hashicorp-engineering/nomad-integration-with-vault-42b0e5feca78) by Patrick Gryzan from HashiCorp.\n\nThe process of [initializing](https://www.vaultproject.io/docs/commands/operator/init) and [unsealing](https://www.vaultproject.io/docs/commands/operator/unseal) vault are already automated in Ansible for this demo, however you should familiarize yourself with the concept:\n\n\u003e When a Vault server is started, it starts in a *sealed* state. In this state, Vault is configured to know where and how to access the physical storage, but doesn't know how to decrypt any of it.\n\u003e \n\u003e *Unsealing* is the process of constructing the master key necessary to read the decryption key to decrypt the data, allowing access to the Vault.\n\n– [»Concepts: Seal/Unseal«](https://www.vaultproject.io/docs/concepts/seal) (from the Vault documentation)\n\nWhenever you stop the Vault service on the server or shut down the VM, the node will be sealed again, making sure your encrypted data is always safe.\nThe Consul dashboard will show a failed service check for that particular Vault node.\n\nTo unseal Vault again, run the following Ansible playbook:\n\n```sh\nansible-playbook \\\n  -i .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory \\\n  unseal-vault.yml\n```\n\nThe keys for unsealing and the user tokens to access Vault are stored under `credentials`. \n\n#### Workflow for requesting secrets from Vault in your app\n\n```mermaid\n%%{init: { 'sequence': { 'actorFontFamily': 'inherit', 'actorFontWeight': 'bold' }}}%%\nsequenceDiagram\n    participant App\n    participant Nomad\n    participant Vault\n    App-\u003e\u003eNomad: Requires Vault access\u003cbr\u003e('hello-vault' policy in \u003cbr\u003ejob manifest)\n    Nomad-\u003e\u003eVault: Requests new access token\u003cbr\u003ewith 'hello-vault' policy\n    Vault-\u003e\u003eNomad: Sends new token\n    Nomad-\u003e\u003eVault: Requests secrets (specified in\u003cbr\u003e'template' of job manifest)\n    Vault-\u003e\u003eNomad: Sends secrets\n    Nomad-\u003e\u003eApp: Writes secrets to ENV vars\n```\n\n\n## Collecting application metrics with Prometheus\n\nThis setup also includes a [Prometheus](https://prometheus.io/) instance, which uses Consul for service discovery. If your application's Nomad job is tagged with `prometheus` and has a metrics endpoint under `/metrics`, it will be scraped automatically and appear in the Prometheus target overview under http://prometheus.demo.\n\n\n## Grafana Dashboards\n\nThere are some basic dashboards for Consul, Nomad and Traefik available under http://grafana.demo. Username/password is `admin`.\n\n\n## Taking it further\n\nDive deeper into the [Job specification](https://nomadproject.io/docs/job-specification/): learn about the [`artifact`](https://nomadproject.io/docs/job-specification/artifact/), [`template`](https://nomadproject.io/docs/job-specification/template/) and [`volume`](https://nomadproject.io/docs/job-specification/volume/) stanzas to add config files and storage to your jobs. Starting 0.11 beta, Nomad also supports [Container Storage Interface (CSI)](https://www.hashicorp.com/blog/hashicorp-nomad-container-storage-interface-csi-beta/).\n\nYou can launch jobs that claim storage volumes from AWS Elastic Block Storage (EBS) or Elastic File System (EFS) volumes, GCP persistent disks, Digital Ocean droplet storage volumes, Ceph, vSphere, or vendor-agnostic third-party providers like Portworx. This means that the same plugins written by storage providers to support Kubernetes also support Nomad out of the box.\n\n\n## On Security\n\nFor this demo I tried to keep the setup simple, but already a bit closer to a practical use than \"just start it in development mode on your laptop\". I think that it provides a good, stable basis which can be extended (especially through Ansible) if the following security considerations are taken into account:\n\n- Separate the control plane from the worker pool running the applications, so faulty or malicious workloads have less impact on the overall system integrity and stability. Three nodes running a Consul and Nomad server are required to run in high availability (HA) mode. This allows Consul to reach a quorum, even in one node should go down. Each worker node however should run a Consul and Nomad Client, with the Nomad client talking to the Consul client and the Consul client talking to the Consul server.\n\n- Access Control Lists (ACLs) are indispensable to secure UI, API, CLI, service and agent communications on [Consul](https://www.consul.io/docs/acl/index.html), [Nomad](https://learn.hashicorp.com/nomad/acls/fundamentals) and [Vault](https://www.vaultproject.io/docs/concepts/policies). \n\n- Apply thorough network security measures (firewall, isolation, etc.) both from the outside and in between the machines.\n\n- Whatever workloads you run, please familiarize yourself with the appropriate security measures and implement them on all machines (and no, just putting applications into containers does not automatically make them secure).\n\n- Make all web-services only available via HTTPS. Traefik offers great support for [automatic certificate management with Let's Encrypt](https://docs.traefik.io/https/acme/), for example.\n\n- Monitor [Vault's audit log](https://www.vaultproject.io/docs/audit). All Vault API requests (internal and external) are logged to detect if someone tries to gain access to your secrets or tamper with them. This demo uses the `syslog` audit backend to send those logs to the `loadbalancer`, in lack of a different VM – **In production of course, audit logs *never* belong on an outside facing machine!**\n  \n  They are stored in `/var/log/vault/audit.log` and the file is rotated daily. However, it is better to send them to [Elasticsearch](https://www.elastic.co/elasticsearch/) for example and have them analyzed with [Kibana](https://www.elastic.co/elasticsearch/).\n\n\n## Contributing\n\nContributions and bug fixes are always welcome!\n\n\n## License\n\n[MIT](LICENSE)\n\n---\n\n\u003csup id=\"f1\"\u003e1\u003c/sup\u003e Deployment is also possible _without_ Vagrant if the VMs are provided by other means. All you need is an Ansible inventory file in the following format:\n\n```\nloadbalancer           ansible_host=your.vm.ip.address\n\n[consul_nomad]\nconsul-nomad-node1     ansible_host=your.vm.ip.address\n...\nconsul-nomad-nodeN     ansible_host=your.vm.ip.address\n```\n\nAfterwards run the provisioning step with `ansible-playbook -i \u003cinventory file\u003e playbook.yml` [↩](#user-content-a1)\n\n---\n\n\u003csup id=\"f2\"\u003e2\u003c/sup\u003e There is a [bug in older versions](https://github.com/ansible/ansible/pull/71824) regarding the new FQDN notation for builtin tasks (`ansible.builtin.module` vs. the old `module` notation) that prevents the playbook from running. [↩](#user-content-a2)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffhemberger%2Fnomad-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffhemberger%2Fnomad-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffhemberger%2Fnomad-demo/lists"}