{"id":16288076,"url":"https://github.com/jdelic/consul-smartstack","last_synced_at":"2025-03-20T03:30:34.618Z","repository":{"id":151274393,"uuid":"65942012","full_name":"jdelic/consul-smartstack","owner":"jdelic","description":"A Smartstack architecture, as defined by AirBnb, built on Hashicorp Consul","archived":false,"fork":false,"pushed_at":"2024-04-17T10:14:29.000Z","size":79,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-28T22:48:37.704Z","etag":null,"topics":["haproxy-templates","hashicorp-consul","service-discovery","smartstack"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jdelic.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-08-17T21:06:02.000Z","updated_at":"2024-07-28T07:54:20.000Z","dependencies_parsed_at":"2024-10-27T21:43:53.879Z","dependency_job_id":"ab1660be-98f3-4f6b-9cbf-b27257f14384","html_url":"https://github.com/jdelic/consul-smartstack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdelic%2Fconsul-smartstack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdelic%2Fconsul-smartstack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdelic%2Fconsul-smartstack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdelic%2Fconsul-smartstack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jdelic","download_url":"https://codeload.github.com/jdelic/consul-smartstack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244047609,"owners_count":20389205,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["haproxy-templates","hashicorp-consul","service-discovery","smartstack"],"created_at":"2024-10-10T19:47:12.225Z","updated_at":"2025-03-20T03:30:34.371Z","avatar_url":"https://github.com/jdelic.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Consul Smartstack\n=================\n\n**Deprecation Notice**\nHashicorp released \n`Consul 1.2 \"Service Mesh\" \u003chttps://www.hashicorp.com/blog/consul-1-2-service-mesh\u003e`_, \nwhich supersedes a lot of the functionality offered in this repository. Using\nConsul Connect will also offer you automatic PKI management, traffic encryption\nand endpoint validation and a degree of service authentication. The external\nload balancer code is still useful, but with the Consul integration in HAProxy\n1.8, you can probably skip a lot of the complexity. YMMV. IANAL. etc.\n\nThis repository contains a recipe for running a AirBnB\n`Smartstack \u003chttp://nerds.airbnb.com/smartstack-service-discovery-cloud/\u003e`_\\ -like\ninfrastructure, but not using `Nerve \u003chttps://github.com/airbnb/nerve\u003e`_ and\n`Synapse \u003chttps://github.com/airbnb/synapse\u003e`_, but instead using Hashicorp's\n`Consul \u003chttps://consul.io/\u003e`_ and\n`Consul-template \u003chttps://github.com/hashicorp/consul-template\u003e`_.\n\nIt provides a very expressive integration of ``consul-template`` with Jinja2\ntemplates that I find much better than the built-in golang templates.\n\nSummarized, Smartstack routes requests to services used by applications in a\nruntime environment by running local `haproxy \u003chttp://www.haproxy.org/\u003e`_\ninstances on each host that route tcp/udp traffic to these services.\nApplications don't distinguish between local and remote services, instead\neverything looks like a local service and routing traffic to an available\ninstance of said service is managed by a separate system. In AirBnB's case that\nseparate system is Zookeeper, Nerve and Synapse. In this implementation it's\nConsul and Consul-template.\n\nI have extracted this from\n`my Saltstack configuration \u003chttps://github.com/jdelic/saltshaker\u003e`_ where I\nuse it to push services out to nodes.\n\nIt also contains a template for a HAProxy load balancer that offers services\nto the internet (``haproxy-external.conf``).\n\nRun Smartstack with consul and consul-template\n----------------------------------------------\nBasic steps:\n\n* run consul with ``-config-dir=/etc/consul/services.d``, see\n  ``consul-template.service`` for an example systemd config file that I use\n* register services in consul from nodes by putting service definitions in\n  ``/etc/consul/services.d`` or have `Nomad \u003chttps://nomadproject.io/\u003e`_ /\n  `docker registrator \u003chttps://github.com/gliderlabs/registrator\u003e`_ do it for\n  you\n* have consul-template listen to the service catalog by having a Python\n  script pose as a ``consul-template`` template using the\n  ``{{services}}`` catalog, therefor getting rerendered every time a service\n  gets added or removed\n* Said Python script is rendered and then immediately executed by\n  consul-template taking parameters and a Jinja template rerendering haproxy\n  configurations from said Jinja template and then the Python script reloads\n  or restarts haproxy\n* The included haproxy config templates define a local proxy\n  (Smartstack-like) for proxying internal services to your applications on\n  ``127.0.0.1`` and\n* an external haproxy that using the concept of consul service tags,\n  interpreted by Python, to create a loadbalancer haproxy that terminates SSL\n  and forwards traffic to apps registering themselves as consul services on\n  nodes\n\nI run 3 instances of this setup in parallel on my servers, as you can see in my\n`Salt Smartstack setup \u003chttps://github.com/jdelic/saltshaker/tree/master/srv/salt/haproxy\u003e`_:\n\n1. One that routes (micro)services and essential services on localhost\n2. One that routes the same (micro)services and essential services on the\n   local Docker bridge interface. This allows services managed by Nomad or,\n   for example, Docker Swarm to also reach services living outside the cluster\n   edges\n3. One that routes internet-facing services to internal endpoints (a\n   SNI-aware, optionally SSL-terminating loadbalancer)\n\n\nWhy not use consul-template directly for templating the haproxy configuration?\n------------------------------------------------------------------------------\nThere are limits to Golang's (and by extension consul-template's) templating\nlanguage that make this too hard. Much of the infrastructure in this\nrepository depends on using consul service tags to pass information from the\nservice definition in Consul to haproxy. An intermediate Python script is a\ngood solution to provide a more expressive template language.\n\nOriginally the consul-template language did not support setting variables\nor other constructs (and the `developers didn't want to change that\n\u003chttps://github.com/hashicorp/consul-template/issues/399\u003e`_\\ ).\n\nNow there is `Scratch storage\n\u003chttps://github.com/hashicorp/consul-template#scratch\u003e`_, which are\ntemplate-local variables. This allows you to do a lot of what this\nrepository provides. I still like the flexibility of filtering the list of\nprocessed services through command-line arguments and the expressiveness that\n``SmartStackServiceContainer`` affords in the Jinja template. If you want to\nuse consul-template's `Vault \u003chttps://vaultproject.io/\u003e`_ integration, then\nthere is a trade-off. You could render a consul-template template from a\nJinja2 template and then invoke consul-template from ``--command`` with\n``-once``.\n\n\nCommand-line arguments\n----------------------\n``servicerenderer.ctmpl.py`` as its filename suggests is a consul-template\ntemplate that renders into a Python script. The resulting Python script is\nmeant to be executed by the ``command`` directive of consul-template's template\nconfiguration. The script supports a number of command-line arguments modifying\nits behaviour. It's purpose is to select a set of services from the full\nservices catalog in Consul, passed into it through consul-template, pass that\nset into a Jinja2 template and execute a command.\n\nQuery Syntax\n++++++++++++\nQueries are expressed as a comma-separated list of criteria which consist of\nkeys (field names) from Consul's service catalog and values ((sub)strings or\nregular expressions) that must match the field's value. Regular expressions\nstart with ``regex=``. Using ``--include`` or ``--exclude`` multiple times\nallows to express boolean ``OR`` semantics.\n\nExamples:\n\n.. code-block:\n\n    --include 'tags=smartstack:internal,name=regex=^xyz$'\n    --exclude tags=udp\n    --exclude ip=192.168.56.\n    --include tags=mytag,tags=myothertag,port=2323\n\n\n====================== =======================================================\nParameter              Description\n====================== =======================================================\n--add-all              Add all services to the selected set. This allows you\n                       to subtract services from the full set.\n--include [query]      Add services matching the query to the filtered set of\n                       services that is passed into the template. (see Query\n                       Syntax)\n--exclude [query]      Remove services matching the query to t\n--smartstack-localip   Set the {{localip}} template variable. (Default:\n                       ``127.0.0.1``).\n--open-iptables TYPE   Execute ``iptables`` commands for all services selected\n                       by ``--has/--has-not/--match/--no-match`` and append\n                       INPUT and OUTPUT rules between the IP specified by\n                       ``--smartstack-localip`` and the network. TYPE can be\n                       either ``plain`` or ``conntrack``. ``plain`` results\n                       in *both* INPUT and OUTPUT rules from and to the local\n                       IP. ``contrack`` will limit these connections to those\n                       that have the ``NEW`` state, assuming that your firewall\n                       is already configured to route ``RELATED`` packets.\n--only-iptables        Do not render any template or execute any command, just\n                       set up iptables for matched services.\n-D, --define KEY=VAL   Set KEY to VAL in the rendered template.\n-o, --output           Name the output file for the rendered result. Default:\n                       ``stdout``.\n-c, --command          The command to invoke after rendering the template.\n                       Will be executed in a shell.\ntemplate               The only positional command-line argument: specifies\n                       the Jinja2 template to render.\n====================== =======================================================\n\n\nTemplate directives\n-------------------\nWhen consul-template executes the Python script it renders a Jinja template. It\nwraps the consul service catalog in ``SmartstackService`` objects which are\nmanaged by ``SmartstackServiceContainer`` instances. Using these allows easily\nchainable selectors for services and their entries in Consul.\n\nTemplate context\n++++++++++++++++\nThe default template context contains all key/value pairs defined on the\ncommand-line via ``-D``. It also always contains the following variables:\n\n================= ============================================================\nTemplate variable Description\n================= ============================================================\n``localip``       The value passed to ``--smartstack-localip`` and the IP used\n                  for all iptables operations.\n``services``      The pre-filtered list of services (``--has``, ``--has-not``,\n                  ``--match`` and ``--no-match`` already applied) list of\n                  services from Consul's service catalog wrapped in a\n                  ``SmartstackServiceContainer`` (see below).\n================= ============================================================\n\n\nSmartStackServiceContainer\n++++++++++++++++++++++++++\nWhenever a group of services is returned, they are wrapped in an instance of\n``SmartstackServiceContainer``. This versatile class behaves like a ``dict``\nwhen it represents a number of services grouped by a common property or it\nbehaves like a ``list`` when it represents an unfiltered number of services.\nEach service is itself represented by an instance of ``SmartstackService``.\n\n================== ===========================================================\nAttribute          Description\n================== ===========================================================\n``.services``      Either a ``dict`` representing the groups of services split\n                   into groups by ``.group_by()`` or ``.group_by_tagvalue()``\n                   or a ``list`` of services.\n``.all_services``  Always a list of all services this instance of\n                   ``SmartstackServiceContainer`` started out with. You will\n                   rarely access this directly, use ``.ungroup()`` instead.\n``.grouped_by``    A list of values the services contained in this\n                   ``SmartstackServiceContainer``instance have been sorted by,\n                   one after the other.\n``.group_by_type`` A list of the types of groupings used, one after the other.\n                   Each grouping can be of type ``field`` or ``tag``.\n``.filtered_to``   A list of the criteria leading to this group. In nested\n                   ``SmartstackServiceContainer`` instances, the\n                   ``.filtered_to`` attribute of a child container is\n                   equivalent to the ``.grouped_by`` property of the container\n                   it was created from.\n================== ===========================================================\n\n============================ =================================================\nMethod                       Description\n============================ =================================================\n``.ungroup()``               Returns an unfiltered/ungrouped top-level\n                             ``SmartstackServiceContainer`` representing all\n                             services. This allows you to undo all previous\n                             calls to ``.group_by()`` and\n                             ``.group_by_tagvalue()``.\n``.value_set(f)``            Return a ``Set[str]`` of all values of *f* in the\n                             Consul services contained in the current\n                             container. Valid values of *f* are all fields\n                             returned in the Consul service catalog.\n``.tagvalue_set(f)``         Return a ``Set[str]`` of all tags in the list of\n                             tags on a Consul service defnition for which\n                             ``tagvalue.startswith(f) is True``.\n``.group_by(f)``             Return a ``SmartstackServiceContainer`` instance\n                             which represents a\n                             ``Dict[str, SmartstackServiceContainer]`` where\n                             each existing value of field *f* in the Consul\n                             service catalog is a key resolving to a list-like\n                             container of all services where ``f == key``.\n``.group_by_tagvalue(part)`` Return a ``SmartstackServiceContainer`` instance\n                             which represents a\n                             ``Dict[str, SmartstackServiceContainer]`` where\n                             the keys are all tag values that started with\n                             *part* (with *part* cut off) and the value is a\n                             list-like container containing all\n                             ``SmartstackService`` instances having a tag\n                             ``part+key``.\n============================ =================================================\n\nYou will probably never have to use these methods, but I'll document them\nanyway:\n\n==================== =========================================================\nMethod               Description\n==================== =========================================================\n``.add(...)``        Add a service to a ``list``-like container (raises\n                     ``ValueError`` on a ``dict``-like container.\n``.iter_services()`` Return a generator to iterate over all\n                     ``SmartstackService`` instances contained. ``__iter__()``\n                     is also defined, so you'll need this rarely.\n``.keys()``          Returns the keys of a ``dict``-like container.\n``.items()``         Returns the items of a ``dict``-like container.\n``.count()``         Returns the numer of SmartstackService instances in a\n                     ``list``-like container and the number of keys in a\n                     ``dict``-like container.\n==================== =========================================================\n\n\nSmartStackService\n+++++++++++++++++\nEach individual Consul service is wrapped in a ``SmartstackService`` instance.\n\n================== ===========================================================\nAttribute          Description\n================== ===========================================================\n``.svc``           The \"service dictionary\". This is the deserialized JSON\n                   structure returned by Consul for each service from the\n                   Consul service catalog. This gives you direct access to all\n                   data from Consul.\n``.ip``            The service's IP address as defined in the Consul service\n                   definition.\n``.port``          The service's IP port as defined via the\n                   ``smartstack:port:*`` tag *or* if that is not defined, the\n                   service's IP port from its Consul service definition.\n``.name``          The name of the service as defined in the Consul service\n                   definition.\n``.tags``          Returns a ``List[str]`` of all tags defined for this\n                   service in the Consul service definition.\n================== ===========================================================\n\n==================== =========================================================\nMethod               Description\n==================== =========================================================\n``.tagvalue(part)``  If the service has a tag starting with *part*, returns\n                     the tag with *part* cut off.\n==================== =========================================================\n\n\nExamples\n--------\nLook at the included haproxy configuration templates for example code.\n\n* ``haproxy-external.jinja.cfg`` is a configuration template for a HTTP(S)\n  loadbalancer supporting tag-based configuration for SNI and HTTP\n  hostname-based backend routing.\n\n* ``haproxy-internal.jinja.cfg`` is a configuration template for running a\n  Smartstack infrastructure on every node in a cluster routing internal\n  services from ``localhost`` on predefined ports, thereby allowing\n  applications to be ignorant of where the services they are using are\n  running.\n\n* ``servicerenderer-internal.conf`` a consul-template configuration example.\n\n\nPredefined Consul service tags\n++++++++++++++++++++++++++++++\nThe example templates use a number of tags to configure basic attributes of\nSmartstack and the external loadbalancer role.\n\n=========================== ==================================================\nTag                         Description\n=========================== ==================================================\nsmartstack:mode:TYPE        The haproxy mode to use for this service. Can be\n                            any haproxy supported mode. Default: ``tcp``.\n                            This is only used in the internal smartstack\n                            templates.\nsmartstack:port:PORT        An optional override for the service's IP port.\nsmartstack:protocol:PROT    Used to configure the external load balancer role.\n                            Can be ``http`` or ``https`` or ``sni`` depending\n                            on the internet-facing service. ``https`` will\n                            terminate SSL on the loadbalancer, whereas ``sni``\n                            can be used to send SSL traffic directly to the\n                            backend and terminate it there. (loadbalancer only)\nsmartstack:https-redirect   A tag that creates a haproxy rule to redirect\n                            a request over HTTP to HTTPS (loadbalancer only)\nsmartstack:hostname:HOST    Attaches an internet-facing service to the\n                            hostname HOST via the HTTP Host header or SNI.\nsmartstack:internal         Marks services used for Smartstack configuration\n                            via ``haproxy-internal.jinja.cfg``.\nsmartstack:external         Marks services that are hooked to to the external\n                            load balancer via ``haproxy-external.jinja.cfg``.\nhaproxy:frontend:option:OPT Allows passing *OPT* to haproxy's *option* config.\nhaproxy:frontend:port:PORT  Forces haproxy to listen on PORT while sending\n                            traffic to the service's port from Consul. This\n                            allows you to fix frontend ports for dynamically\n                            assigned backend ports (like Nomad and other\n                            cluster schedulers use).\ncrt:CERT                    Adds *CERT* as a SSL certificate to the\n                            loadbalancer haproxy in\n                            ``haproxy-external.jinja.cfg`` so it can do SNI\n                            and SSL termination.\n=========================== ==================================================\n\n\nLicense\n=======\n\nCopyright (c) 2017, Jonas Maurus\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this\n   list of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice,\n   this list of conditions and the following disclaimer in the documentation\n   and/or other materials provided with the distribution.\n\n3. Neither the name of the copyright holder nor the names of its contributors\n   may be used to endorse or promote products derived from this software\n   without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdelic%2Fconsul-smartstack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjdelic%2Fconsul-smartstack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdelic%2Fconsul-smartstack/lists"}