{"id":21293045,"url":"https://github.com/criteo/consul-templaterb","last_synced_at":"2025-05-16T02:09:56.944Z","repository":{"id":37514102,"uuid":"124880134","full_name":"criteo/consul-templaterb","owner":"criteo","description":"consul-template-like with erb (ruby) template expressiveness","archived":false,"fork":false,"pushed_at":"2024-12-12T11:55:23.000Z","size":1153,"stargazers_count":77,"open_issues_count":6,"forks_count":30,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-05-11T09:46:25.106Z","etag":null,"topics":["consul","consul-template","consul-templaterb","consul-ui","criteo","erb","hashicorp","hashicorp-consul","ruby","rubygem"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/criteo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-12T11:40:17.000Z","updated_at":"2025-04-15T10:11:11.000Z","dependencies_parsed_at":"2025-01-05T06:02:08.084Z","dependency_job_id":null,"html_url":"https://github.com/criteo/consul-templaterb","commit_stats":{"total_commits":517,"total_committers":26,"mean_commits":"19.884615384615383","dds":0.2514506769825918,"last_synced_commit":"e3d287e0eaecb80ec7700cbcaad9f25132642f71"},"previous_names":[],"tags_count":163,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Fconsul-templaterb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Fconsul-templaterb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Fconsul-templaterb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Fconsul-templaterb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/criteo","download_url":"https://codeload.github.com/criteo/consul-templaterb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254453667,"owners_count":22073618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul","consul-template","consul-templaterb","consul-ui","criteo","erb","hashicorp","hashicorp-consul","ruby","rubygem"],"created_at":"2024-11-21T13:53:04.689Z","updated_at":"2025-05-16T02:09:56.881Z","avatar_url":"https://github.com/criteo.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# consul-templaterb\n\n[![Build Status](https://github.com/criteo/consul-templaterb/workflows/Ruby/badge.svg?branch=master)](https://github.com/criteo/consul-templaterb/actions?query=branch%3Amaster)\n[![Gem Version](https://badge.fury.io/rb/consul-templaterb.svg)](http://github.com/criteo/consul-templaterb/releases)\n[![GEM Downloads](https://img.shields.io/gem/dt/consul-templaterb.svg)](https://rubygems.org/gems/consul-templaterb)\n[![License](https://img.shields.io/badge/license-ApacheV2-yellowgreen.svg)](#license)\n\nThe ruby GEM [consul-templaterb](https://rubygems.org/gems/consul-templaterb)\nis both a library and an executable that allows generating files\nusing data from Consul (Discovery and Key/Value Store) easily using ruby's\n[ERB](https://en.wikipedia.org/wiki/ERuby) templates.\nIt also supports launching programs and babysitting processes\nwhen rendering the files, thus notifying programs when data do change.\n\nIt is intended for the users accustomed to expressiveness or Ruby templating,\nallowing for more flexibility and features than Go templates.\n\nIt also allows using all of the Ruby language, especially useful for generating\nfiles in several formats ([JSON](samples/consul_template.json.erb),\n[XML](samples/consul_template.xml.erb)) for which text substitutions are hard\nto get right (escaping, attributes encoding...).\n\nIt also focuses on providing good performance and lightweight usage of bandwidth,\nespecially for very large clusters and watching lots of services.\n\nFor the rendering of complex templates in large Consul Clusters, it usually\nrenders faster with a more predictable way the template than the original\nconsul-template.\n\nIt provides a very [simple API](TemplateAPI.md) to write your templates\nwith fully [working examples](samples/).\n\nIt also allows displaying a very nice and hi-performance HTML5 UI for Consul,\nsee [consul-ui](samples/consul-ui) for details.\n\nThere is an article [docs/article-06_Template-based_discovery_with_consul-templaterb.md](docs/article-06_Template-based_discovery_with_consul-templaterb.md) summarizing why we developped this tool (also available on [medium](https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457)).\n\n## Video introduction to consul-templaterb\n\n[![Introduction to Consul-templaterb](docs/images/consul-ui_001.png)](https://youtu.be/zLzrLGLLl4Q)\n\n## Differences with HashiCorp's consul-template\n\n[Hashicorp's Consul Template](https://github.com/hashicorp/consul-template)\ninspired strongly the creation of this GEM and this executable wants\nto achieve better results in some use cases, especially for very large\nConsul clusters with many nodes and servers.\n\nConsul Template uses Go templates which is very limited in its set of\nfeatures: it is complicated to sort, apply real transformations\nusing code and even interact with the OS (ex: get the current date, format\ntimestamps...).\n\nThe sort feature for instances allow you to create a predictable output (i.e: meaning\nthat the order of nodes is predictable), thus it might offer better performance\nsince the reload of processes if happening ONLY when the files are binary\ndifferent. Thus, if using consul-templaterb, you will reload less your HAProxy or\nload-balancer than you would do with consul-template.\n\nCompared to consul-template, consul-templaterb offers the following features:\n\n* Hot-Reload of template files\n* Bandwidth limitation per endpoint (will soon support dynamic bandwidth limiter)\n* Supports launch and supervision of multiple child processes\n* Supports launching commands when files do change on disk (reload commands...)\n* Supports all Ruby features (ex: base64, real JSON/XML generation...)\n* Information about bandwidth\n\nThe executable supports semantics and command-line flags and options similar to\nHashiCorp's Consul-template, so many flags you might use in consul-template will\nwork in a similar way. It also supports the same environment variable\n`CONSUL_HTTP_ADDR` to find the Consul Agent to query and `CONSUL_HTTP_TOKEN` to\nget the token.\n\n## Installation\n\nYou might either use the executable directly OR use this GEM as a library by\nadding this line to your application's `Gemfile`:\n\n```ruby\ngem 'consul-templaterb'\n```\n\nAnd then execute:\n\n```shell\n$ bundle\n[...]\n```\n\nOr install it yourself as:\n\n```shell\n$ gem install consul-templaterb\n[...]\n```\n\nIf you simply want to use the executable on your preferred Linux distribution, you\nhave to install first: `ruby` and `ruby-dev`.\n\n### Quick install on Ubuntu-Linux\n\n```shell\nsudo apt-get install ruby ruby-dev \u0026\u0026 sudo gem install consul-templaterb\n```\n\nYou can now use it directly using the binary `consul-templaterb` in your path.\n\n### Quick install on Windows\n\nOn Windows, a bug exists with Ruby greater than 2.4.\n\nTo make it work, use this command as explained in\nhttps://github.com/oneclick/rubyinstaller2/issues/96#issuecomment-434619796\n\n```\ngem install eventmachine consul-templaterb --platform ruby\n```\n\n### Run it with Docker\n\nA Docker image is also available https://hub.docker.com/r/discoverycriteo/consul-templaterb\nand allows to quickly have a working\n[Consul-UI](https://github.com/criteo/consul-templaterb/blob/master/samples/consul-ui/README.md)\nthat will serve the UI to explore your Consul Cluster.\n\n### Playing with the samples templates\n\nSamples are installed with the GEM, you can either\n[download](https://github.com/criteo/consul-templaterb/tree/master/samples) them or\nsimply use the ones installed with the gem. To figure out where the templates are\ninstalled:\n\n```shell\n$ gem contents consul-templaterb|grep samples\n[...]\n```\n\nWill output the path where the samples are being installed, you can copy the directory\nsomewhere and then issue the command:\n\n```shell\n$ consul-templaterb samples/*.html.erb\nUsing samples/checks.html output for samples/checks.html.erb\n[...]\n```\n\nIt will render a full web site you may browse to look in real-time the status of your\nConsul Cluster.\n\nYou can now have a look to the [API Documentation](TemplateAPI.md) to modify existing\ntemplates or write your owns, it is very easy!\n\n## Usage of consul-templaterb\n\n### Show help\n\n```shell\n$ consul-templaterb --help\nUSAGE: consul-templaterb [[options]]\n    -h, --help                       Show help\n    -v, --version                    Show Version\n        --retry, --consul-retry-attempts [RETRIES]\n                                     If consul fails after n retries, stop the program, default=10\n    -f, --[no-]fail-fast             If consul/vault endpoints fail at startup, fail immediately\n    -g, --no-gzip-compression        Disable GZIP compression in HTTP requests\n    -c, --consul-addr=\u003caddress\u003e      Address of Consul, eg: http://localhost:8500\n        --consul-cert-chain=\u003cpath/to/cert_chain\u003e\n                                     Path to Consul TLS client certificate chain to use\n        --consul-private-key=\u003cpath/to/private_key\u003e\n                                     Path to Consul TLS client private key to use\n        --skip-consul-verify-tls     Skip verifying Consul TLS via certificate authority (DANGEROUS)\n    -l, --log-level=\u003clog_level\u003e      Log level, default=info, any of none|error|info|debug\n        --consul-token=\u003ctoken\u003e       Use a token to connect to Consul\n    -V, --vault-addr=\u003caddress\u003e       Address of Vault, eg: http://localhost:8200\n        --vault-cert-chain=\u003cpath/to/cert_chain\u003e\n                                     Path to Vault TLS client certificate chain to use\n        --vault-private-key=\u003cpath/to/private_key\u003e\n                                     Path to Vault TLS client private key to use\n        --skip-vault-verify-tls      Skip verifying Vault TLS via certificate authority (DANGEROUS)\n        --vault-token=\u003ctoken\u003e        Token used to authenticate against vault.\n        --[no-]vault-renew           Control auto-renewal of the Vault token. Default: activated\n        --vault-retry, --vault-retry-attempts [RETRIES]\n                                     If vault fails after n retries, stop the program, default=10\n        --vault-lease-duration-factor=\u003cfactor\u003e\n                                     Wait at least \u003cfactor\u003e * lease time before updating a Vault secret. Default: 0.5\n    -w, --wait=\u003cmin_duration\u003e        Wait at least n seconds before each template generation\n    -r, --retry-delay=\u003cmin_duration\u003e Min Retry delay on Error/Missing Consul Index\n    -k, --hot-reload=\u003cbehavior\u003e      Control hot reload behaviour, one of :[die (kill daemon on hot reload failure), keep (on error, keep running), disable (hot reload disabled)]\n    -K, --sig-term=kill_signal       Signal to send to next --exec command on kill, default=TERM\n    -M, --debug-memory-usage         Display messages when RAM grows\n    -T, --trim-mode=trim_mode        ERB Trim mode to use (- by default)\n    -R, --sig-reload=reload_signal   Signal to send to next --exec command on reload (NONE supported), default=HUP\n    -W, --wait-signal=min_duration   Wait at least n seconds before each reload signal being sent to next --exec process\n    -e, --exec=\u003ccommand\u003e             Execute the following command in as a subprocess when all templates are ready\n    -d, --debug-network-usage        Debug the network usage\n    -t erb_file:[output]:[command]:[params_file],\n        --template                   Add a erb template, its output and optional reload command\n    -o, --once                       Do not run the process as a daemon\n```\n\nWhen launched with file arguments ending with .erb, the executable will assume\nthe file is a template and will render the corresponding file without the\n`.erb` extension.\n\nIt means that you can call consul-templaterb with `*.erb` arguments, the shell\nwill then substitute all files and render it by removing the `.erb` extension as\nif the `--template my_file.ext.erb:myfile.ext` was used.\n\nIf the program is run in an automatic context, it could be useful to stream\nlogs instead of the default interactive log version which erase last log line.\nTo configure this behavior, set the `STREAM_LOG` environment variable to any\nvalue.\n\n### Generate multiple templates\n\nIn the same way as consul-template, consul-templaterb supports multiple templates and executing\ncommands when the files do change. The parameter `--template \u003cERB\u003e:\u003cDEST\u003e:[reload_command]:params_file` works\nin the following way:\n\n* ERB: the ERB file to use as a template\n* DEST: the destination file\n* reload_command: optional shell command executed whenever the file has been modified\n* params_file: JSON or YAML file to load and to use as parameter for the template (see\n  [param() function](TemplateAPI.md#paramparameter_name-default_value-nil) to retrieve\n  the values)\n\nThe argument can be specified multiple times, ex:\n\nExample of usage:\n\n```shell\n$ consul-templaterb \\\\\n  --template \"samples/ha_proxy.cfg.erb:/opt/haproxy/etc/haproxy.cfg:sudo service haproxy reload\"\n  --template \"samples/consul_template.erb:consul-summary.txt\"\n```\n\n### Process management and signalization of configuration files\n\nWith the `--exec` argument (can be specified multiple times), consul-templaterb will launch\nthe process specified when all templates have been generated. When generated file(s) do change,\na POSIX signal (HUP by default) is sent to the spawned process. If you generate several files\nat the same time, the signal will be sent only once the rendering of all templates is completed,\nso if your process is using several configuration files, all files will be modified and\nconsistent before the signal is sent to process.\n\nIf any template does return nothing (aka use `return nil` in the code of a template),\nconsul-templaterb does consider that the template is not ready and will not launch the executable\nnor write the file.\n\nSignals can be customized per process. Two signals are supported with options `--sig-reload` and\n`--sig-term`. When the option is added, the next `--exec` options to start a process will use the\ngiven signal. By default, HUP will be sent to reload events (you can use NONE to avoid sending any\nreload signal), TERM will be used when leaving consul-templaterb. A minimum duration between reload\nsignals can be specified for each sub process by prepending `--wait-signal=min_duration` to `--exec`\ncommand.\nIn such case, the signal will be sent every `min_duration` as a maximum (very useful for templates\nchanging a lot, but you don't want to trigger too many reloads, for instance for a load-balancer).\n\n### Bandwidth limitation\n\nThis is actually the original reason for the creation of this GEM: on Criteo's large clusters,\nconsul-template generated several hundreds of Mb/s to the Consul-Agent which also\ngenerated several hundreds of Mb/s with the Consul servers.\n\nBy design, the GEM supports limiting the number of requests per endpoints (see code in\n`bin/consul-templaterb` file). It avoids using too much network to fetch data from Consul\nin large Consul Clusters (especially when watching lots of files).\n\nThe limitation is static for now, but fair dynamic bandwidth allocation will allow limiting\nthe bandwidth used to get information for all services by capping the global bandwidth used\nby consul-templaterb.\n\n### Samples\n\nHave a look into the [samples/](samples/) directory to browse example files which contains those\nexamples:\n\n1. [List all nodes on Cluster](samples/consul-ui/consul_nodes.json.erb)\n2. [Show all services in Cluster](samples/consul-ui/consul_services.json.erb)\n3. [Show all Key/Values nicely](samples/consul-ui/consul_keys.json.erb)\n4. [Services in XML](samples/consul_template.xml.erb)\n5. [Services in JSON or YAML](samples/consul_template.json.erb)\n6. [Generate HAProxy Configuration](samples/ha_proxy.cfg.erb)\n7. [Export Consul Statistics to Prometheus](samples/metrics.erb): count all services, their state,\n   datacenters and nodes and export it to Prometheus easily to trigger alerts.\n8. [List all services/Nodes with their statuses for all datacenters](samples/all_services.txt.erb)\n9. [Show all services/instances not passing on all DCs](samples/tools/find_all_failing_services.txt.erb)\n10. [List all RubyGems consul versions from remote server JSON](samples/list_ruby_versions_from_rubygems.txt.erb)\n\nIf you want to test it quickly, you might try with (assuming your consul agent is listening on\n`http://localhost:8500`):\n\n```shell\n$ bundle exec bin/consul-templaterb -c 'http://localhost:8500' samples/*.html.erb\n[...]\n```\n\nIt will generate a full website in samples/ directory with lots of Consul information ready to\nuse (website updated automagically when values to change).\n\nAll templates are validated using the Travis CI, so all should be working for your Consul\nConfiguration.\n\n### Structured text generation (YAML, JSON, XML...)\n\nSince ERB supports real language features, we recommend you to use Hashes or Arrays in Ruby and\nat the end to generate the output. It allows changing very quickly from JSON to YAML or XML and\navoiding all the pitfalls of structured language serialization (such as escaping attributes or\nindentation).\n\nSee [Services in JSON or YAML](samples/consul_template.json.erb) to look at a working example.\n\n## Template development\n\nPlease look at [the template API](TemplateAPI.md) to have a list of all functions you might use for your\ntemplates. Don't forget to have a look at the [samples/](samples/) directory to have full working examples.\n\n## Development\n\n### Quick start\n\nWe recommend using bundle using `bundle install`, you can now run `bundle exec bin/consul-templaterb`.\nHelp is available running `bundle exec bin/consul-templaterb --help`\n\nThe following example will generate static HTML pages and JSON data for `consul-ui`:\n```\nbundle exec bin/consul-templaterb -c your.consul.agent:8500 samples/consul-ui/*.erb\n```\n\nIf you need remote calls, you need an HTTP server. A simple way to have one is using Python's simple HTTP\nserver. Example for `consul-ui`:\n```\ncd samples/consul-ui\npython -m SimpleHTTPServer\n```\n\n### Installation\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the\nversion number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version,\npush git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org/gems/consul-templaterb).\n\n## Known bugs\n\nHere are the known bugs of the application:\n\n* [ ] `render_file` might create an infinite recursion if a template includes itself indirectly.\n* [ ] On Windows, the consul-templaterb cannot work with more than 2048 endpoints watched (see https://github.com/oneclick/rubyinstaller/issues/104#issuecomment-7681074)\n\nPlease consult [CHANGELOG.md](CHANGELOG.md) for fixed bugs.\n\n## TODO\n\n* [x] Hashi's Vault support\n* [ ] Implement automatic dynamic rate limit\n* [x] More samples: apache, nginx, a full website displaying consul information...\n* [x] Optimize rendering speed at start-up: an iteration is done every second by default, but it would be possible to speed\n      up rendering by iterating with higher frequency until the first write of result has been performed.\n* [ ] Allow tuning bandwidth using a simple configuration file (while it should not be necessary for 90% of use-cases)\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for guidance: Bug reports and pull requests are welcome on GitHub at https://github.com/criteo/consul-templaterb.\n\nThis project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the\n[Contributor Covenant](http://contributor-covenant.org) code of conduct. See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)\n\n## License\n\nThe gem is available as Open-Source under the terms of the Apache v2 license. See [LICENSE.txt](LICENSE.txt) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcriteo%2Fconsul-templaterb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcriteo%2Fconsul-templaterb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcriteo%2Fconsul-templaterb/lists"}