{"id":13469081,"url":"https://github.com/samrocketman/docker-compose-ha-consul-vault-ui","last_synced_at":"2025-04-09T09:09:38.198Z","repository":{"id":38360601,"uuid":"119766997","full_name":"samrocketman/docker-compose-ha-consul-vault-ui","owner":"samrocketman","description":"A docker-compose example of HA Consul + Vault + Vault UI","archived":false,"fork":false,"pushed_at":"2024-08-21T01:37:10.000Z","size":136,"stargazers_count":206,"open_issues_count":0,"forks_count":67,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-02T06:07:12.508Z","etag":null,"topics":["consul","docker","docker-compose","high-availability","vault"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/samrocketman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-01T01:41:06.000Z","updated_at":"2025-02-28T10:22:06.000Z","dependencies_parsed_at":"2024-08-21T02:44:35.099Z","dependency_job_id":null,"html_url":"https://github.com/samrocketman/docker-compose-ha-consul-vault-ui","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samrocketman%2Fdocker-compose-ha-consul-vault-ui","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samrocketman%2Fdocker-compose-ha-consul-vault-ui/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samrocketman%2Fdocker-compose-ha-consul-vault-ui/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samrocketman%2Fdocker-compose-ha-consul-vault-ui/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/samrocketman","download_url":"https://codeload.github.com/samrocketman/docker-compose-ha-consul-vault-ui/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248008630,"owners_count":21032556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul","docker","docker-compose","high-availability","vault"],"created_at":"2024-07-31T15:01:25.844Z","updated_at":"2025-04-09T09:09:38.181Z","avatar_url":"https://github.com/samrocketman.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"# HA Consul + Vault + Vault UI\n\n\u003cimg\nsrc=\"https://user-images.githubusercontent.com/875669/35621353-e78a6956-0638-11e8-8e07-3d96e9e91dd7.png\"\nheight=48 width=72 alt=\"Docker Logo\" /\u003e \u003cimg\nsrc=\"https://user-images.githubusercontent.com/875669/35658016-46572728-06b4-11e8-9e25-3629e8a9d64d.png\"\nheight=48 width=48 alt=\"Consul Logo\" /\u003e \u003cimg\nsrc=\"https://user-images.githubusercontent.com/875669/35658041-6c0105fc-06b4-11e8-9bdc-fc933303b5d2.png\"\nheight=48 width=48 alt=\"Vault Logo\" /\u003e \u003cimg\nsrc=\"https://user-images.githubusercontent.com/875669/35658057-84201b96-06b4-11e8-88a8-733b7a225144.png\"\nheight=48 width=48 alt=\"VaultBoy Logo\" /\u003e\n\n\nThis project is an example of using [Consul][c], [Vault][v], and [Vault UI][ui]\nin a high availability (HA) configuration.  Conveniently packaged as [Docker][d]\nservices for provisioning via [Docker Compose][dc].\n\nFeatures:\n\n- dnsmasq makes Consul DNS available to all containers.  A secondary dnsmasq\n  server is provided which grants HA to the DNS available to all containers.\n  This allows consul-template to update DNS with zero DNS downtime.\n  consul-template will create a lock to ensure it is not possible for both\n  primary and secondary DNS servers to be down during DNS configuration updates\n  as part of service discovery.\n- consul-template updates dnsmasq configuration and restarts dnsmasq when the\n  configuration has changed (e.g. consul cluster size is increased on the fly).\n  This makes consul DNS lookups HA.\n- Vault is registered via service discovery which is exposed via Consul DNS.\n- Persists data across restarts as long as the cluster is gracefully shut down.\n  See [`Starting and stopping` section][#starting-and-stopping].\n- Local docker infrastructure is able to anonymously authenticate with Vault via\n  approle method and its CIDR address.\n- Linux and Mac OS with docker supported.\n\n# Prerequisites\n\n* [Docker][d]\n* [Docker Compose][dc]\n\nSupplemental reading material:\n\n- [Hitchhiker's guide to administering Vault](docs/vault-for-humans.md)\n- [Vault Auth By CIDR](docs/vault-auth-by-cidr.md) enables anonymous login to\n  Vault from docker infrastructure.\n\n# Getting started\n\n### Start the cluster\n\n\u003e Remove `--scale vault=3` if you want to start one instance of Vault.\n\u003e `docker compose up -d` would bring only Consul up in HA configuration.\n\n    ./scripts/consul-agent.sh --bootstrap\n    docker compose up --scale vault=3 -d\n\n### Configure your web browser\n\nConfigure your browser to use the SOCKS5 proxy listening on `localhost:1080`.\nWith your browser configured to use the proxy visit\n`http://consul.service.consul:8500/` and wait for the cluster to be ready.\nAfter the vault service has all nodes available, it is time to initialize vault.\n\n### Initialize Vault\n\nIf you wish to secure `secret.txt` with GPG, then set the `recipient_list`\nenvironment variable.  For example, the following.\n\n    export recipient_list=\"\u003cgpg fingerprint to your secret gpg key\u003e\"\n\nIf you do not use GPG or do not want to, then skip setting `recipient_list`.\nInitialize vault witht he following command.\n\n    ./scripts/initialize-vault.sh\n\nThe credentials for vault are located in the file `secret.txt` which is created\nwhen Vault is initialized.  Alternately, `secret.txt.gpg` if using GPG\nencryption.\n\n# Visit the web UI\n\n### Configure your browser\n\nConfigure your web browser to use the SOCKS5 proxy listening on\n`localhost:1080`.\n\nIn Firefox, do the following:\n\n1. Edit [connections settings][firefox-socks]\n2. Set Manual proxy configuration\n3. Set SOCKS host to `localhost`, set Port to `1080`, and check `SOCKS v5`\n   boolean.\n\nAlternately install [FoxyProxy extension][foxyproxy] which is an extension for\nquickly switching proxies on or off.\n\nFor other browsers, web search how to configure proxy settings or see what\nextensions are available for managing proxy settings.\n\n### Visit services via Consul DNS\n\nVisit http://portal.service.consul/.  It provides links to other web UIs and if\nyou configure additional portal services, then they will also show up\nautomatically.\n\nAlternately, you can visit consul and vault directly at:\n\n* http://consul.service.consul:8500/\n* http://active.vault.service.consul:8200/\n\nTo log into Vault UI you must generate for yourself an admin token.\n\n    ./scripts/get-admin-token.sh\n\nThe root user token for Vault is stored in `secret.txt` at the root of this\nrepository after you initialize Vault.\n\n### Other portal services\n\nFor playing around with service discovery I have created other docker compose\nfiles which will automatically register with this consul cluster.  Here's a list\nof what I have created so far.\n\n- [consul-chronograf][consul-chronograf]\n- [consul-grafana][consul-grafana]\n- [consul-influxdb][consul-influxdb]\n- [consul-kapacitor][consul-kapacitor]\n- [consul-mysql][consul-mysql]\n- [consul-nexus3][consul-nexus3]\n\n# Experiment\n\nWith HA enabled, container instances of consul and vault can be terminated with\nminor disruptions.\n\nConsul can be scaled up on the fly.  `consul-template` will automatically update\ndnsmasq to include new services.  dnsmasq will experience zero downtime.\n\n    docker compose up --scale vault=3 --scale consul-worker=6 -d\n\nTo play with failover for killing consul instances, it is recommended to review\n[fault tolerance for consul HA deployments][ft].\n\n# Starting and stopping\n\nBecause high availability clusters have to gossip across nodes you can't execute\na simple `docker compose down` without corrupting the clusters.  Instead, you\nhave to gracefully shut down all clusters that depend on consul and then\ngracefully shutdown consul itself.  For this, I have provided a script.\n\nStop consul and vault cluster safely.\n\n    ./scripts/graceful-shutdown.sh\n\nStart the consul and vault clusters.\n\n    docker compose up -d\n\n# Troubleshooting\n\n### DNS\n\nCurrently, output from the `dnsmasq` and `dnsmasq-secondary` servers are\nminimal.  Verbosity of output can be increased for troubleshooting.  Edit\n`docker compose.yml` and add `--log-queries` to the dnsmasq command.\n\nDNS client troubleshooting using Docker.\n\n    docker compose run dns-troubleshoot\n\nUsing the `dig` command inside of the container.\n\n    # rely on the internal container DNS\n    dig consul.service.consul\n\n    # specify the dnsmasq hostname as the DNS server\n    dig @dnsmasq vault.service.consul\n\n    # reference vault DNS by tags\n    dig active.vault.service.consul\n    dig standby.vault.service.consul\n\n### Logs\n\nView vault logs.\n\n    docker compose logs vault\n\nUser `docker exec` to log into container names.  It allows you to poke around\nthe runtime of the container.\n\n### SOCKS5 proxy\n\nRun a [SOCKS5 proxy][socks] for use with your browser.\n\n    docker run --network docker-compose-ha-consul-vault-ui_internal --dns 172.16.238.2 --init -p 127.0.0.1:1080:1080 --rm serjs/go-socks5-proxy\n\nConfigure your browser to use SOCKS proxy at `127.0.0.1:1080`.\n\n### Recovering data\n\nIt's possible a cluster was shutdown uncleanly and put into an irrecoverable\nstate with no leader.  If you have ever cleanly shut down consul, then it's\npossible you have a backup in the `backups/` directory.\n\nIf you're in this leaderless state, then wipe out your old cluster data with the\nfollowing command (this will permanently delete all old data).\n\n    docker compose down -v\n\nStart a new cluster.\n\n    docker compose up -d\n\nThe latest backup can be restored via the following script.\n\n    ./scripts/restore-consul.sh\n\nIf you have a specific backup you wish to restore, then you can call it as an\nargument.\n\n    ./scripts/restore-consul.sh backups/backup.snap\n\n# Screenshots\n\n![show portal before services are available](https://user-images.githubusercontent.com/875669/69476734-cbeb8500-0dab-11ea-83a1-f46013438fc0.png)\n\n---\n\n![show portal after services are available](https://user-images.githubusercontent.com/875669/69476742-dad23780-0dab-11ea-9b01-ec01574facab.png)\n\n---\n\n![consul screenshot of all discovered services](https://user-images.githubusercontent.com/875669/69476746-e32a7280-0dab-11ea-99cb-d3a39426a299.png)\n\n---\n\n![consul screenshot of service metadata](https://user-images.githubusercontent.com/875669/69476747-e9b8ea00-0dab-11ea-9bda-1abf3303e1fd.png)\n\n---\n\n# License\n\n[MIT License](LICENSE)\n\n[c]: https://www.consul.io/\n[consul-chronograf]: https://github.com/samrocketman/consul-chronograf\n[consul-grafana]: https://github.com/samrocketman/consul-grafana\n[consul-influxdb]: https://github.com/samrocketman/consul-influxdb\n[consul-kapacitor]: https://github.com/samrocketman/consul-kapacitor\n[consul-mysql]: https://github.com/samrocketman/consul-mysql\n[consul-nexus3]: https://github.com/samrocketman/consul-nexus3\n[d]: https://www.docker.com/\n[dc]: https://docs.docker.com/compose/\n[firefox-socks]: https://support.mozilla.org/en-US/kb/connection-settings-firefox\n[foxyproxy]: https://addons.mozilla.org/en-US/firefox/addon/foxyproxy-standard/\n[ft]: https://www.consul.io/docs/internals/consensus.html#deployment-table\n[socks]: https://github.com/serjs/socks5-server\n[ui]: https://github.com/djenriquez/vault-ui\n[v]: https://www.vaultproject.io/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamrocketman%2Fdocker-compose-ha-consul-vault-ui","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamrocketman%2Fdocker-compose-ha-consul-vault-ui","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamrocketman%2Fdocker-compose-ha-consul-vault-ui/lists"}