{"id":15176357,"url":"https://github.com/hashicorp/nomad-auto-join","last_synced_at":"2025-10-01T14:31:45.501Z","repository":{"id":48273055,"uuid":"95550916","full_name":"hashicorp/nomad-auto-join","owner":"hashicorp","description":"Terraform config to automatically bootstrap a Nomad cluster","archived":true,"fork":false,"pushed_at":"2022-11-10T23:25:21.000Z","size":27,"stargazers_count":70,"open_issues_count":4,"forks_count":18,"subscribers_count":29,"default_branch":"master","last_synced_at":"2024-09-28T13:02:55.897Z","etag":null,"topics":["nomad","terraform"],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hashicorp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-27T11:24:22.000Z","updated_at":"2023-06-21T14:23:13.000Z","dependencies_parsed_at":"2022-09-04T15:01:57.501Z","dependency_job_id":null,"html_url":"https://github.com/hashicorp/nomad-auto-join","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fnomad-auto-join","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fnomad-auto-join/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fnomad-auto-join/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashicorp%2Fnomad-auto-join/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hashicorp","download_url":"https://codeload.github.com/hashicorp/nomad-auto-join/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234878308,"owners_count":18900676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nomad","terraform"],"created_at":"2024-09-27T13:03:05.768Z","updated_at":"2025-10-01T14:31:40.144Z","avatar_url":"https://github.com/hashicorp.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Auto-bootstrapping a Nomad cluster\n\nIn a previous post, we explored how Consul discovers other agents [using cloud metadata to bootstrap a cluster](/blog/consul-auto-join-with-cloud-metadata/). This post looks at Nomad's auto-joining functionality and how we can use Terraform to create an autoscaled cluster.\n\nUnlike Consul, Nomad's auto bootstrapping functionality does not use cloud metadata because when Nomad pairs with Consul, we inherit the functionality.  Consul's service discovery and health checking is the perfect platform to use for bootstrapping Nomad.\n\nThe startup process for the Nomad server or agent is as follows:\n\n1. The instance bootstraps and installs Nomad and Consul Agent\n2. The init system starts Consul Agent\n2. Consul Agent discovers the Consul cluster using AWS Metadata\n3. The init system starts Nomad with the location of a locally running Consul agent\n4. On start, Nomad queries the service catalog in Consul to discover other instances\n5. Nomad joins the discovered instances\n\n![Nomad bootstrap process](https://hyzxph.media.zestyio.com/nomad_auto_join.png)\n\nFor servers, we still require the initial number of instances that is expected in the bootstrap process to ensure that we have a healthy cluster before we start to schedule work.  The requirement for the initial cluster size is to ensure that the cluster can elect a leader and establish a quorum.\n\n## Setting up a Nomad cluster\nThe repository at [https://github.com/hashicorp/nomad-auto-join](https://github.com/hashicorp/nomad-auto-join) includes an example Terraform configuration to demonstrate this functionality.  Clone this repository to your local filesystem:\n\n```bash\n$ git clone https://github.com/hashicorp/nomad-auto-join\n```\n\nTo initiate and bootstrap the cluster, first set the following environment variables with your AWS credentials.\n\n```bash\n$ export AWS_REGION = \"[AWS_REGION]\"\n$ export AWS_ACCESS_KEY = \"[AWS_ACCESS_KEY]\"\n$ export AWS_SECRET_ACCESS_KEY = \"[AWS_SECRET_ACCESS_KEY]\"\n```\n\nThere are many ways to authenticate to AWS with Terraform, for details of all the authentication options, please see the [Terraform AWS provider documentation](https://www.terraform.io/docs/providers/aws/) for more information.\n\nOnce we have the environment variables set we can then initialise the terraform modules using the `terraform get` command.  Using modules in our setup allows us to keep our code dry.  For more information on this features please see the Terraform documentation: [https://www.terraform.io/docs/modules/usage.html](https://www.terraform.io/docs/modules/usage.html)\n\n```bash\n$ terraform get\n```\n\nWe can then run `plan` to create check the configuration before running `apply` to create the resources\n\n```bash\n$ terraform plan\nRefreshing Terraform state in-memory prior to plan...\nThe refreshed state will be used to calculate this plan, but will not be\npersisted to local or remote state storage.\n\n#..\n.\nPlan: 30 to add, 0 to change, 0 to destroy.\n```\n\nRunning `terraform apply` will create 5 `t2.micro` instances, 3 servers to form a quorum and 2 agents which run the jobs.  These settings are configurable in the terraform.tfvars file if you would like to change them.\n\n```bash\n$ terraform apply\n```\n\nOnce this is all up and running, the Terraform output shows the details of the created servers and clients.\n\n```bash\nApply complete! Resources: 30 added, 0 changed, 0 destroyed.\n\nThe state of your infrastructure has been saved to the path\nbelow. This state is required to modify and destroy your\ninfrastructure, so keep it safe. To inspect the complete state\nuse the `terraform show` command.\n\nState path: \n\nOutputs:\n\nalb_arn = arn:aws:elasticloadbalancing:eu-west-1:773674589724:loadbalancer/app/nomad-consul-external/3cca7be97ab6d28d\nalb_dns = nomad-consul-external-723193890.eu-west-1.elb.amazonaws.com\nssh_host = 34.251.93.86\n```\n\nBecause the Nomad API is not exposed publicly, we need to log into the SSH host which is attached to the same VPC.\n\n```bash\nssh ubuntu@$(terraform output ssh_host)\n```\n\nWe can then run `nomad server-members` to see a list of the servers in the cluster.\n\n```bash\nubuntu@ip-10-1-1-216:~$ nomad server-members\nName                  Address     Port  Status  Leader  Protocol  Build  Datacenter  Region\nip-10-1-1-99.global   10.1.1.99   4648  alive   false   2         0.5.6  dc1         global\nip-10-1-2-107.global  10.1.2.107  4648  alive   false   2         0.5.6  dc1         global\nip-10-1-2-48.global   10.1.2.48   4648  alive   true    2         0.5.6  dc1         global\n```\n\nMoreover, we can run `nomad node-status` to see a list of the clients.\n\n```bash\nubuntu@ip-10-1-1-131:~$ nomad node-status\nID        DC   Name          Class   Drain  Status\nec268e26  dc1  ip-10-1-2-5   \u003cnone\u003e  false  ready\nec21954b  dc1  ip-10-1-1-44  \u003cnone\u003e  false  ready\n```\n\n## Running jobs on Nomad\nTo run a job on Nomad, we need to create a job specification which defines the schema.   The job file uses the HashiCorp Configuration Language (HCL) that aims to strike a balance between a human readable and editable, and machine-friendly format.\n\nThe overall hierarchy of our job file is as follows:\n* job\n    * group\n        * task\n        \nEach file only has a single job. However, a job can have multiple groups, and each group may have multiple tasks.  When a group contains multiple tasks, Nomad co-locates these on the same client.  This hierarchy allows you the flexibility of configuring a single service or groups of services which define an application in the one file.  \n\nLet's take a look at the file jobs/syslog.hcl  which defines a system for running syslog-ng.\n\n```hcl\njob \"system_utils\" {\n  datacenters = [\"dc1\"]\n  type        = \"system\"\n\n # ...\n\n  group \"system_utils\" {\n    constraint {\n      distinct_hosts = true\n    }\n\n    # ...\n\n    task \"syslog\" {\n      driver = \"docker\"\n\n      # ...\n      }\n    }\n  }\n}\n```\n\n###  Job Stanza\nThe job stanza is the top most configuration in the job specification, in our job we are assigning the name \"system_utils\" this name must be globally unique across the whole cluster.  \n\n```hcl\njob \"system_utils\" {\n  datacenters = [\"eu-west-1\", “eu-west-2”]\n  type        = \"system\"\n\n  update {\n    stagger      = \"10s\"\n    max_parallel = 1\n  }\n# ...\n```\n\nWe are then setting the datacenter, which is a required option and which has no default.  We could have also specified the region which is another attribute for job placement.  The datacenters and region attributes allow fine-grained control over where a job runs.  For example, you may have the following cluster deployed:\n* region: Europe-AWS\n        * datacenter: eu-west-1\n        * datacenter: eu-east-1\n\nThe type definition allows you to set the three different job types that are available to the scheduler:\n* service - long-lived services that should never go down\n* batch - short lived or periodic jobs\n* system - schedule job on all running on all clients which that meet the job’s constraint\n\nhttps://www.nomadproject.io/docs/runtime/schedulers.html\n\nThe update stanza configures how we would like the job to be updated for example when we want to deploy a new version.  In our case the settings give us the capability to do rolling deploys:\n* max_parallel configures the number of task groups updated at any one time\n* stagger specifies the delay between sets of updates\n\n[https://www.nomadproject.io/docs/job-specification/update.html](https://www.nomadproject.io/docs/job-specification/update.html)\n\n### Group Stanza\nThe next level is the group stanza, and the group stanza defines a series of tasks that should be co-located on the same Nomad client.  To determine on which client the tasks get placed, we can specify some constraints.  Constraints filter on attributes and metadata, \n\nIn our example we are only setting the constraint `distinct_host`, this means that we would never schedule any more than one group to the same host.  Constraints can also apply to the Task and Job level; this gives you complete flexibility over the placement, all constraints evaluate on a hierarchical level which starts at a Job level and filters down.\n\nFull details on this capability can is available in the Nomad documentation:\n\n[https://www.nomadproject.io/docs/job-specification/constraint.html](https://www.nomadproject.io/docs/job-specification/constraint.html)\n\n\n### Task Stanza\nThe task stanza is an individual unit of work, such as running a Docker container or application.  If we take a look at our example, we are going to run the docker image syslog-ng.\n\n```hcl\n    task \"syslog\" {\n      driver = \"docker\"\n\n      config {\n        image = \"balabit/syslog-ng:latest\"\n\n        port_map {\n          udp = 514\n          tcp = 601\n        }\n      }\n// ..\n}\n```\n\nThe first attribute in this stanza is the *driver*; we are using the value `docker` as we would like to run a Docker container however many different values are possible:\n* docker - run a docker container\n* exec - execute a particular command, isolating primitives of the operating system to limit the tasks access to resources\n* java - run a Java application packaged into a Jar file\n* lxc - run LXC application containers\n* qemu - execute any regular qemu image\n* raw_exec - execute a particular command with no isolation, runs the command as the same user as the Nomad process. Disabled by default\n* rkt - run CoreOS rkt application containers\n\nFinally, we are going to set the config for the task, which is unique to each driver.  For our task we are using the Docker driver, the minimum we can specify is the image attribute to select the docker image. However, the driver supports most of the parameters which could are passed directly to Docker with the `docker run` command.\n\nhttps://www.nomadproject.io/docs/drivers/docker.html\n\n### Resources\nTo ensure that Nomad correctly allocates the task to the client we need to specify the resources that it consumes.  The resources stanza allows us to set limits for the CPU and memory, and network that the task can consume.  By limiting the resources, we can bin pack more tasks onto a single instance. However, the tasks get throttled to these limits, if we set them too small, we could inadvertently reduce the performance of the running application.\n\n```hcl\n      resources {\n        cpu    = 500 # 500 MHz\n        memory = 256 # 256MB\n\n        network {\n          mbits = 10\n\n          port \"udp\" {\n            static = \"514\"\n          }\n\n          port \"tcp\" {\n            static = \"601\"\n          }\n        }\n```\n\nThe port section of the network stanza allows us to allocate ports either dynamically or statically; we reference the ports which were defined in the config stanza earlier.  In this example we are statically allocating the http and admin ports as we want to map these to the load balancer however if we only need ports available for internal use then we can omit the `static = “514”` from the configuration and Nomad with dynamically allocate the port number to the container.\n\n[https://www.nomadproject.io/docs/job-specification/resources.html](https://www.nomadproject.io/docs/job-specification/resources.html)\n\n### Running the jobs\nNow that we understand the job configuration let's now see how to execute a job on our cluster.  Log into the remote ssh server by running:\n\n```bash\n$ ssh ubuntu@$(terraform output ssh_host)\n```\n\nIn this example we are using the terraform output variable for the `ssh_host` direct in our command, there is no need to cut and paste the IP address.\n\nWhen the SSH host was provisioned it downloaded the two example job files from the GitHub repository, you should see two files, syslog.hcl and http_test.hcl in the home directory.\nLike Terraform Nomad has a plan command which checks the syntax of our config file and determines if it can allocate the resources on the server, let’s run this command:\n\n```bash\n$nomad plan syslog.hcl\n+ Job: \"system_utils\"\n+ Task Group: \"system_utils\" (2 create)\n  + Task: \"syslog\" (forces create)\n\nScheduler dry-run:\n- All tasks successfully allocated.\n\nJob Modify Index: 0\nTo submit the job with version verification run:\n\nnomad run -check-index 0 syslog.hcl\n\nWhen running the job with the check-index flag, the job will only be run if the\nserver side version matches the job modify index returned. If the index has\nchanged, another user has modified the job and the plan's results are\npotentially invalid.                                                                                 \n```\n\nWe can now use the `nomad run -check-index 0 syslog.hcl` command to execute the plan and provision the application.\n\n```bashl\n$nomad run syslog.hcl\n==\u003e Monitoring evaluation \"711e028a\"\n    Evaluation triggered by job \"system_utils\"\n    Allocation \"82d60ade\" created: node \"ec25ffd9\", group \"system_utils\"\n    Allocation \"a449140e\" created: node \"ec2d8914\", group \"system_utils\"\n    Evaluation status changed: \"pending\" -\u003e \"complete\"\n==\u003e Evaluation \"711e028a\" finished with status \"complete\"\n```\n\nWe can check the status of the job by running the `nomad status system_utils` command, you will see that there has been 2 allocations one for each Nomad client.\n\n```bash\n$nomad status system_utils\nID            = system_utils\nName          = system_utils\nType          = system\nPriority      = 50\nDatacenters   = dc1\nStatus        = running\nPeriodic      = false\nParameterized = false\n\nSummary\nTask Group    Queued  Starting  Running  Failed  Complete  Lost\nsystem_utils  0       0         2        0       0         0\n\nAllocations\nID        Eval ID   Node ID   Task Group    Desired  Status   Created At\n82d60ade  711e028a  ec25ffd9  system_utils  run      running  06/29/17 16:39:18 BST\na449140e  711e028a  ec2d8914  system_utils  run      running  06/29/17 16:39:18 BST\n```\n\nNow we have seen how we can create a system job let’s create a standard job for the http-echo server.\n\n```hcl\n$ nomad plan http_test.hcl \n# ...\n$ nomad run -check-index 0 http_test.hcl\n# ...\n```\n\n```bash\n$ nomad status\nID     Type     Priority  Status\nsystem_util   system   50        running\nhttp-test  service  50        running\n```\n\nIf you open another terminal window and curl the external alb address you should see the following response.\n\n```bash\n$curl $(terraform output external_alb_dns)\n'hello world'\n```\n\n## Destroying the cluster\nDon’t forget that running resources in AWS incur cost, once you have finished with your cluster you can destroying by running:\n\n```bash\n$ terraform destroy\n```\n\n## Summary\nThe auto-bootstrapping functionality built into Nomad and Consul makes it possible to manage and auto-scale our clusters easily and not restricted to AWS.  This capability is available for most major cloud providers.  We have barely scratched the surface on the capacity of Nomad, but I hope you can see how Nomads power and simplicity could be a real benefit to your infrastructure.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhashicorp%2Fnomad-auto-join","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhashicorp%2Fnomad-auto-join","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhashicorp%2Fnomad-auto-join/lists"}