{"id":20821935,"url":"https://github.com/internetarchive/hind","last_synced_at":"2025-10-29T18:17:44.576Z","repository":{"id":50592082,"uuid":"458423518","full_name":"internetarchive/hind","owner":"internetarchive","description":"Hashistack-IN-Docker (single container with nomad + consul + caddy)","archived":false,"fork":false,"pushed_at":"2025-03-08T02:40:29.000Z","size":2154,"stargazers_count":61,"open_issues_count":1,"forks_count":8,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-03-31T11:51:12.245Z","etag":null,"topics":["caddy","cicd","consul","consul-connect","docker","hashistack","nomad"],"latest_commit_sha":null,"homepage":"https://internetarchive.github.io/hind/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/internetarchive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-12T04:51:40.000Z","updated_at":"2025-03-22T22:25:27.000Z","dependencies_parsed_at":"2024-11-01T02:21:03.486Z","dependency_job_id":"9e53fae4-9158-45b5-99c2-6bb5c4a2c8b4","html_url":"https://github.com/internetarchive/hind","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Fhind","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Fhind/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Fhind/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/internetarchive%2Fhind/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/internetarchive","download_url":"https://codeload.github.com/internetarchive/hind/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252913967,"owners_count":21824287,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["caddy","cicd","consul","consul-connect","docker","hashistack","nomad"],"created_at":"2024-11-17T22:13:29.498Z","updated_at":"2025-10-29T18:17:44.569Z","avatar_url":"https://github.com/internetarchive.png","language":"Shell","funding_links":[],"categories":["Infrastructure setup"],"sub_categories":["Deployment and Cluster Setup"],"readme":"# HinD - Hashistack-in-Docker\n```diff\n+       ___                                              ·\n+      /\\  \\                      ___                    ·\n+      \\ \\--\\       ___          /\\  \\        __ __      ·\n+       \\ \\--\\     /\\__\\         \\ \\--\\     / __ \\__\\    ·\n+   ___ /  \\--\\   / /__/     _____\\ \\--\\   / /__\\ \\__\\   ·\n+  /\\_ / /\\ \\__\\ /  \\ _\\    / ______ \\__\\ / /__/ \\ |__|  ·\n+  \\ \\/ /_ \\/__/ \\/\\ \\ _\\__ \\ \\__\\  \\/__/ \\ \\__\\ / /__/  ·\n+   \\  /__/         \\ \\/\\__\\ \\ \\__\\        \\ \\__/ /__/   ·\n+    \\ \\ _\\          \\  /_ /  \\ \\__\\        \\ \\/ /__/    ·\n+     \\ \\__\\         / /_ /    \\/__/         \\  /__/     ·\n+      \\/__/         \\/__/                    \\/__/      ·\n+                                                        ·\n```\n\n![install](img/hind.gif)\n\n\n\nInstalls `nomad`, `consul`, and `caddyserver` (router) together as a mini cluster running inside a single `podman` container.\n\nNomad jobs will run as `podman` containers on the VM itself, orchestrated by `nomad`, leveraging `/run/podman/podman.sock`.\n\nThe _brilliant_ `consul-template` will be used as \"glue\" between `consul` and `caddyserver` -- turning `caddyserver` into an always up-to-date reverse proxy router from incoming requests' Server Name Indication (SNI) to running containers :)\n\n# Setup and run\nThis will \"bootstrap\" your cluster with a private, unique `NOMAD_TOKEN`,\nand `sudo podman run` a new container with the hind service into the background.\n[(source)](https://raw.githubusercontent.com/internetarchive/hind/main/install.sh)\n\n```bash\ncurl -sS https://internetarchive.github.io/hind/install.sh | sudo sh\n```\n\n## Minimal requirements:\n- VM you can `ssh` into\n- VM with [podman](https://podman.io/docs/installation) package\n- if using a firewall (like `ferm`, etc.) make sure the following ports are open from the VM to the world:\n  - 443  - https\n  - 80   - http  (load balancer will auto-upgrade/redir to https)\n@see [#VM-Administration](#VM-Administration) section for more info.\n\n## https\nThe ideal experience is that you point a dns wildcard at the IP address of the VM running your `hind` system.\n\nThis allows automatically-created hostnames from CI/CD pipelines [deploy] stage to use the [git group/organization + repository name + branch name] to create a nice semantic DNS hostname for your webapps to run as and load from - and everything will \"just work\".\n\nFor example, `*.example.com` DNS wildcard pointing to the VM where `hind` is running, will allow https://myteam-my-repo-name-my-branch.example.com to \"just work\".\n\nWe use [caddy](https://caddyserver.com) (which incorporates `zerossl` and Let's Encrypt) to on-demand create single host https certs as service discovery from `consul` announces new hostnames.\n\n\n\n### build locally - if desired (not required)\nThis is our [Dockerfile](Dockerfile)\n\n```bash\ngit clone https://github.com/internetarchive/hind.git\ncd hind\nsudo podman build --network=host -t ghcr.io/internetarchive/hind:main .\n```\n\n\n## Setting up jobs\nWe suggest you use the same approach mentioned in\n[nomad repo README.md](https://gitlab.com/internetarchive/nomad/-/blob/master/README.md)\nwhich will ultimately use a templated\n[project.nomad](https://gitlab.com/internetarchive/nomad/-/blob/master/project.nomad) file.\n\n\n## Nicely Working Features\nWe use this in multiple places for nomad clusters at archive.org.\nWe pair it with our fully templatized\n[project.nomad](https://gitlab.com/internetarchive/nomad/-/blob/master/project.nomad)\nWorking nicely:\n- secrets, tokens\n- persistent volumes\n- deploys with multiple public ports\n- and more -- everything [here](https://gitlab.com/internetarchive/nomad/-/blob/master/README.md#customizing)\n\n## Nomad credentials\nGet your nomad access credentials so you can run `nomad status` anywhere\nthat you have downloaded `nomad` binary (include home mac/laptop etc.)\n\nFrom a shell on your VM:\n```bash\nexport NOMAD_ADDR=https://$(hostname -f)\nexport NOMAD_TOKEN=$(sudo podman run --rm --secret NOMAD_TOKEN,type=env hind sh -c 'echo $NOMAD_TOKEN')\n```\nThen, `nomad status` should work.\n([Download `nomad` binary](https://www.nomadproject.io/downloads) to VM or home dir if/as needed).\n\nYou can also open the `NOMAD_ADDR` (above) in a browser and enter in your `NOMAD_TOKEN`\n\nYou can try a trivial website job spec from the cloned repo:\n```bash\n# you can manually set NOMAD_VAR_BASE_DOMAIN to your wildcard DNS domain name if different from\n# the domain of your NOMAD_ADDR\nexport NOMAD_VAR_BASE_DOMAIN=$(echo \"$NOMAD_ADDR\" |cut -f2- -d.)\nnomad run https://internetarchive.github.io/hind/etc/hello-world.hcl\n```\n\n## Optional ways to extend your setup\nHere are a few environment variables you can pass in to your intitial `install.sh` run above, eg:\n```sh\ncurl -sS https://internetarchive.github.io/hind/install.sh | sudo sh -s -- -e ON_DEMAND_TLS_ASK=...\n```\n\n- `-e TRUSTED_PROXIES=[CIDR IP RANGE]`\n  - optionally allow certain `X-Forwarded-*` headers, otherwise defaults to `private_ranges`\n    [more info](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#trusted_proxies)\n- `-e NOMAD_ADDR_EXTRA=[HOSTNAME]`\n  - For 1+ extra, nicer https:// hostname(s) you'd like to use to talk to nomad,\n    pass in hostname(s) in CSV format for us to setup.\n- `-e ON_DEMAND_TLS_ASK=[URL]`\n  - If you want to use caddy `on_demand_tls`, URL to use to respond with 200/400 status codes.\n  - @see https://caddy.community/t/11179\n- `-e CERTS_SELF_SIGNED=true`\n  - If you want to use caddy `tls internal`, this will make self-signed certs with caddy making\n    an internal Certificate Authority (CA).\n    @see [#self-signed-or-internal-ca](#self-signed-or-internal-ca) below\n- `-e CLIENT_ONLY_NODE=true`\n  - Set this if you want to setup a client only VM\n    (ie: can run jobs/containers, but doesn't participate in leader elections \u0026 consensus protocols)\n- `...`\n  - other command line arguments to pass on to the main container's `podman run` invocation.\n\n\n## GUI, Monitoring, Interacting\n- see [nomad repo README.md](https://gitlab.com/internetarchive/nomad/-/blob/master/README.md) for lots of ways to work with your deploys.  There you can find details on how to check a deploy's status and logs, `ssh` into it, customized deploys, and more.\n- You can setup an `ssh` tunnel thru your VM so that you can see `consul` in a browser, eg:\n\n```bash\nnom-tunnel () {\n  [ \"$NOMAD_ADDR\" = \"\" ] \u0026\u0026 echo \"Please set NOMAD_ADDR environment variable first\" \u0026\u0026 return\n  local HOST=$(echo \"$NOMAD_ADDR\" |sed 's/^https*:\\/\\///')\n  ssh -fNA -L 8500:localhost:8500 $HOST\n}\n```\n\n- Then run `nom-tunnel` and you can see with a browser:\n  - `consul` http://localhost:8500/\n\n\n## Add more Virtual Machines to make a HinD cluster\nThe process is very similar to when you setup your first VM.\nThis time, you pass in the first VM's hostname (already in cluster), copy 2 secrets,\nand run the installer.\nYou essentially run the shell commands below on your 2nd (or 3rd, etc.) VM.\n\n```sh\nFIRST=vm1.example.com\n# copy secrets from $FIRST to this VM\nssh $FIRST 'sudo podman run --rm --secret HIND_C,type=env hind sh -c \"echo -n \\$HIND_C\"' |sudo podman secret create HIND_C -\nssh $FIRST 'sudo podman run --rm --secret HIND_N,type=env hind sh -c \"echo -n \\$HIND_N\"' |sudo podman secret create HIND_N -\n\ncurl -sS https://internetarchive.github.io/hind/install.sh | sudo sh -s -- -e FIRST=$FIRST\n```\n\n\n## Inspiration\nDocker-in-Docker (dind) and `kind`:\n- https://kind.sigs.k8s.io/\n\nfor `caddyserver` + `consul-connect`:\n- https://blog.tjll.net/too-simple-to-fail-nomad-caddy-wireguard/\n\n\n## VM Administration\nHere are a few helpful admin scripts we use at archive.org\n-- some might be helpful for setting up your VM(s).\n\n- [bin/ports-unblock.sh](bin/ports-unblock.sh) firewalls - we use `ferm` and here you can see how we\n                                   open the minimum number of HTTP/TCP/UDP ports we need to run.\n- [bin/setup-pv-using-nfs.sh](bin/setup-pv-using-nfs.sh) we tend to use NFS to share a `/pv/` disk\n                                   across our nomad VMs (when cluster is 2+ VMs)\n- [bin/setup-consul-dns.sh](bin/setup-consul-dns.sh) - consul dns name resolving --\n                                   but we aren't using this yet\n\n\n## Problems?\n- Older OS (eg: `ubuntu` `focal`) may not enable `podman.socket`.\n  If bootstrapping fails, on linux, you can run:\n\n```sh\nsudo systemctl enable --now podman.socket\n```\n\n- If the main `podman run` is not completing, check your `podman` version to see how recent it is.  The `nomad` binary inside the setup container can segfault due to a perms change.  You can either _upgrade your podman version_ or try adding this `install.sh` CLI option:\n```sh\n--security-opt seccomp=unconfined\n```\n\n- `docker push` repeated fails and \"running out of memory\" deep errors?\n[Try](https://dzone.com/articles/tcp-out-of-memory-consider-tuning-tcp-mem\n):\n```sh\nsysctl net.core.netdev_max_backlog=30000\nsysctl net.core.rmem_max=134217728\nsysctl net.core.wmem_max=134217728\n\n# to persist across reboots:\necho '\nnet.core.netdev_max_backlog=30000\nnet.core.rmem_max=134217728\nnet.core.wmem_max=134217728' |sudo tee /etc/sysctl.d/90-tcp-memory.conf\n```\n\n\n# Miscellaneous\n- client IP addresses will be in request header 'X-Forwarded-For' (per `caddy`)\n- pop inside the HinD container:\n```\nsudo podman exec -it hind zsh\n```\n- get list of `consul` services:\n```\nwget -qO- 'localhost:8500/v1/catalog/services?tags=1' | jq .\n```\n- get `caddy` config:\n```\nwget -qO- localhost:2019/config/ | jq .\n```\n\n\n# Maintenance:\n- If your podman seems to be running out of locks:\nsee the `num_locks` part\nin [install.sh](install.sh) and consider increasing or opening a GitHub issue\n```sh\n# https://docs.podman.io/en/latest/markdown/podman-system-renumber.1.html\npodman -r system renumber\n```\n\n- If your HinD container seems to be unable to fork processes\nsee the `--pids-limit` CLI arg part\nin [install.sh](install.sh) and consider increasing or opening a GitHub issue\n```sh\n# check HinD container's current pids limit:\ncat /sys/fs/cgroup/$(podman inspect --format '{{.State.CgroupPath}}' hind)/pids.max\n```\n\n\n\n## Self-Signed or Internal CA\n- devs just need to trust Caddy's root CA cert once (Caddy can generate it for you)\n- this is easier for internal dev environments\n```ini\nhttps://*.example.com {\n    # use caddy's internal certificate authority -- no ACME challenges needed\n    tls internal\n    reverse_proxy ...\n}\n```\n\nWhen you use Caddy `tls internal`,\ncaddy automatically creates its own Certificate Authority (CA) with:\n- A root CA certificate\n- A private key for signing\n\nThis happens automatically on first run. The root CA cert is stored at:\n```sh\n/pv/CERTS/pki/authorities/local/root.crt\n```\n\n- Devs install/trust Caddy's root CA cert one time in their browser/OS.\n- Caddy's internal CA signs certificates for *.example.com, foo.example.com, bar-branch-123.example.com, etc.\n- Browser sees these certs are signed by the already-trusted Caddy CA.\n- Zero warnings, zero clicks, zero overrides for any hostname signed by that CA.\n\nThis is exactly how Let's Encrypt works - you trust their root CA once (built into browsers),\nand any cert they sign \"just works.\"\n\n### What Devs Need To Do (One Time Setup)\n1. Get the root cert from your Caddy server:\n```sh\n# On the Caddy VM\ncat /pv/CERTS/pki/authorities/local/root.crt\n```\n\n2. Devs install it in their OS/browser:\n- `macOS`:\n```sh\nsudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain root.crt\n```\n- `Windows`: Double-click `root.crt` → Install Certificate → Local Machine → Place in \"Trusted Root Certification Authorities\"\n- `Linux` (Chrome/Chromium):\n```sh\ncp root.crt /usr/local/share/ca-certificates/caddy-local.crt\nsudo update-ca-certificates\n```\n- `Firefox`: Preferences → Privacy \u0026 Security → Certificates → View Certificates → Authorities → Import\n\n3. Done forever\n\n- Every hostname shows a green padlock with zero warnings\n- Caddy signs certs on-demand for any matching hostname. devs never see warnings again.\n\nSuperior to clicking through certificate warnings, which:\n- trains devs to ignore security warnings (bad habit)\n- has to be done per **hostname**\n- doesn't actually work in some browsers anymore\n\nThe internal CA approach is the professional way to handle internal dev HTTPS.\nYou give devs a Slack message with instructions; your devs install one cert.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finternetarchive%2Fhind","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finternetarchive%2Fhind","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finternetarchive%2Fhind/lists"}