{"id":14965337,"url":"https://github.com/jdelic/saltshaker","last_synced_at":"2025-05-07T16:06:27.276Z","repository":{"id":61371708,"uuid":"67513485","full_name":"jdelic/saltshaker","owner":"jdelic","description":"How I use Salt","archived":false,"fork":false,"pushed_at":"2024-10-11T11:46:01.000Z","size":3109,"stargazers_count":39,"open_issues_count":0,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-10-30T01:02:59.663Z","etag":null,"topics":["consul-cluster","salt-configuration","salt-states","saltstack","smartstack"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jdelic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-06T14:03:33.000Z","updated_at":"2024-09-08T20:28:41.000Z","dependencies_parsed_at":"2023-01-29T11:47:19.707Z","dependency_job_id":"b50a95ed-27c9-4789-904f-831f03375005","html_url":"https://github.com/jdelic/saltshaker","commit_stats":{"total_commits":1803,"total_committers":4,"mean_commits":450.75,"dds":0.2617859123682751,"last_synced_commit":"4027ab213d324cd2dd75b23c5fd019a9272814bc"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdelic%2Fsaltshaker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdelic%2Fsaltshaker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdelic%2Fsaltshaker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdelic%2Fsaltshaker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jdelic","download_url":"https://codeload.github.com/jdelic/saltshaker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233352224,"owners_count":18663257,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul-cluster","salt-configuration","salt-states","saltstack","smartstack"],"created_at":"2024-09-24T13:34:36.492Z","updated_at":"2025-01-10T13:34:01.033Z","avatar_url":"https://github.com/jdelic.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# jdelic's Saltshaker\n\nThis is a collection of saltstack formulae designed to bring up an small\nhosting environment for multiple applications and services. The hosting\nenvironment is reusable, the services are primarily there to fulfill my needs\nand allow experimentation with different DevOps setups.\n\nCloning this repository is a good basis for your own Salt setup as it\nimplements a number of best practices I discovered and includes a fully\nfledged [SmartStack](http://nerds.airbnb.com/smartstack-service-discovery-cloud/)\nimplementation for internal, external and cross-datacenter services.\n\nIt also builds on the principles I have documented in my\n[GoPythonGo](https://github.com/gopythongo/gopythongo) build and deployment\nprocess.\n\nIt has full support for [Vagrant](http://vagrantup.com/), allowing easy\ntesting of new functionality and different setups on your local machine before\ndeploying them. Personally, I'm deploying this configuration on my laptop\nusing Vagrant, on Digital Ocean and my own server on Hetzner which I configure\nwith a XEN Hypervisor running VMs for all my development needs.\n\nEverything in here is based around **Debian 9.0 Stretch** (i.e. requires\nsystemd and uses Debian package naming).\n\nUsing these salt formulae you can bring up:\n\n  * a primarily Python/Django based application environment\n\n  * a [Hashicorp Nomad](https://nomadproject.io/) based Docker/application\n    cluster that integrates with the Consul-Smartstack implementation and\n    provides easy application deployment.\n\n  * including a consul/consul-template/haproxy based\n    [smartstack](http://nerds.airbnb.com/smartstack-service-discovery-cloud/)\n    implementation for service discovery (you can find an extracted stand-\n    alone version of that in my\n    [consul-smartstack repository](https://github.com/jdelic/consul-smartstack).\n\n  * a PostgreSQL database configuration for a \"fast\" database and a separate\n    tablespace on an encrypted partition\n\n  * a [Concourse.CI](http://concourse.ci/) build server environment for\n    your projects\n\n  * an HAProxy based reverse proxying load balancer for applications based\n    around the same smartstack implementation\n\nIt also contains configuration for\n\n  * a fully fledged PIM+Mail server with encrypted storage (based on\n    [Radicale](http://radicale.org/), [Dovecot](http://dovecot.org) and\n    [OpenSMTPD](https://www.opensmtpd.org/))\n\n  * single-sign-on for Radicale, Dovecot and OpenSMTPD, other web applications\n    and even PAM using a single database for authentication and providing\n    OAuth2, CAS and SAML (through Dovecot). This is provided through\n    authserver, a separate Django application written by me.\n\nThe salt configuration is pretty modular, so you can easily just use this\nrepository to bring up a GoPythonGo build and deployment environment without\nany of the other stuff.\n\n\n## Configuration deployment\n\nDeploying this salt configuration requires you to:\n\n  1. create a bootstrap server (for example a Amazon EC2 instance, a\n     Dom0 VM on your own Xen server or a Digital Ooean droplet)\n\n  2. Assign that server the `saltmaster` and `consulserver` roles\n     ```\n     mkdir -p /etc/roles.d\n     touch /etc/roles.d/saltmaster\n     touch /etc/roles.d/consulserver\n     ```\n\n  3. check out the saltshaker repository\n     ```\n     cd /opt\n     git clone https://github.com/jdelic/saltshaker\n     ln -sv /opt/saltshaker/srv/salt /srv/salt\n     ln -sv /opt/saltshaker/srv/pillar /srv/pillar\n     ln -sv /opt/saltshaker/srv/reactor /srv/reactor\n     ln -sv /opt/saltshaker/srv/salt-modules /srv/salt-modules\n     mkdir -p /etc/salt/master.d /etc/salt/minion.d\n     ln -sv /opt/saltshaker/etc/salt-master/master.d/saltshaker.conf /etc/salt/master.d/saltshaker.conf\n     ln -sv /opt/saltshaker/etc/salt-minion/minion.d/saltshaker.conf /etc/salt/minion.d/saltshaker.conf\n     ```\n\n  4. Install Salt\n     ```\n     wget -O /tmp/install_salt.sh https://bootstrap.saltstack.com\n     chmod 700 /tmp/install_salt.sh\n     /tmp/install_salt.sh -M -P\n     ```\n\n  5. Edit the Pillar data in `/srv/pillar`. You **must** create a network\n     configuration for your environment (see *Networking* below) and assign\n     it to your systems in `top.sls`. It's especially important to select\n     a count of consul server instances (3 are recommended for a production\n     environment). You also **must** provide a `secrets` pillar that contains\n     SSL certificates and such things.\n\n  6. Run `salt-call state.highstate -l debug` on your master to bring it up.\n\n  7. Bring up additional nodes (at least the count of consul server instances)\n\n  8. Assign them roles, install the salt minion using `install_salt.sh -P` and\n     call state.highstate. It's obviously much better to *automate* this step.\n     I did so for the XEN Hypervisor for example using the scripts in `role.d`\n     together with `xen-create-image`.\n\n\n# The secrets pillar\n\nYou should clone the saltshaker repository and then as a first step, replace\nthe git submodule in `srv/pillar/shared/secrets` with your own **private Git\nrepository**.\n\nFor my salt states to work, you **must** provide your own`shared.secrets`\npillar in `srv/pillar/shared/secrets` that **must** contain the following\npillars, unless you rework the salt states to use different ones. I use a\nwildcard certificate for my domains and the configuration pillars for\nindividual servers allow you to specify pillar references to change what\ncertificates are used, but in the default configuration most services refer\nto the ``maincert``, which is the wildcard certificate:\n\nIn `shared.secrets.ssl`:\n  * `ssl:maincert:cert` - The public X.509 SSL certificate for your domain.\n    You can replace these with cert pillars for your individual domain.\n  * `ssl:maincert:key:` - The private key for your SSL certificate without a\n    passphrase. You can replace this with key pillars for your individual\n    domain.\n  * `ssl:maincert:certchain` - The X.509 certificates tying your CA to a\n    browser-accredited CA, if necessary.\n  * `ssl:maincert:combined` - A concatenation of `:cert` and `:certchain`.\n  * `ssl:maincert:combined-key` - A concatenation of `:cert`, `:certchain` and\n    `:key`.\n\nIn `shared.secrets.vault`:\n  * `ssl:vault:cert` - the public X.509 SSL certificate used by the `vault`\n    role/server. Should contain SANs for `vault.local` resolving to `127.0.0.1`\n    (see notes on the `.local` and `.internal` domains under \"Networking\"\n    below).\n  * `ssl:vault:key` - its private key\n  * `ssl:vault:certchain` - its CA chain\n  * `ssl:vault:combined` - A concatenation of `:cert` and `:certchain`\n  * `ssl:vault:combined-key` - A concatenation of `:cert` and `:certchain` and\n    `:key`.\n  * `vault:s3:aws-accesskey` - The access key for an IAM role that can be used\n    for the Vault S3 backend (if you want to use that). Must have read/write\n    access to a S3 bucket.\n  * `vault:s3:aws-secretkey` - the secret key corresponding to the access key\n    above.\n\nIn `shared.secrets.concourse`:\n  * `ssh:concourse:public` - A SSH2 public RSA key for the concourse.ci TSA SSH\n    host.\n  * `ssh:concourse:key` - The private key for the public host key.\n\nIn `shared.secrets.postgresql`:\n  * `ssl:postgresql:cert,key,certchain,combined,combined-key` in the same\n    structure as the SSL certs mentioned above, containing a SSL cert used to\n    encrypt database traffic through `postgresql.local`.\n\nI manage these pillars in a private Git repository that I clone to\n`srv/pillar/shared/secrets` as a Git submodule. To keep the PEM encoded\ncertificates and keys in the pillar file, I use the following trick:\n\n```\n# pillar file\n{% set mycert = \"\n-----BEGIN CERTIFICATE-----\nMIIFBDCCAuwCAQEwDQYJKoZIhvcNAQELBQAwgdgxCzAJBgNVBAYTAkRFMQ8wDQYD\n...\n-----END CERTIFICATE-----\"|indent(12) %}\n\n# because we're using the indent(12) template filter we can then do:\nssl:\n    maincert:\n        cert: | {{mycert}}\n```\n\n## shared.secrets: The managed GPG keyring\n\nThe salt config also contains states which manage a shared GPG keyring. All\nkeys added to the dict pillar `gpg:keys` are iterated by the `crypto.gpg`\nstate and put into a GPG keyring accessible only by `root` and the user\ngroup`gpg-access`. There is a parallel pillar called `gpg:fingerprints`\nthat is used to check whether a key has already been added. The file system\nlocation of the shared keyring is set by the `gpg:shared-keyring-location`\npillar, which by default is `/etc/gpg-managed-keyring`.\n\n\n# Server configuration\n\n### Pillar overrides\n\n## Disks\n\n### /secure\n\n\n## The roledir grain\n\n\n## Available roles\n\n\n# Salt modules\n\n\n## The dynamicsecrets Pillar\n\nThis is a Pillar module that can be configured on the master to provide\ndynamically generated passwords and RSA keys across an environment managed by\na Salt master. The secrets are stored in a single **unencrypted** sqlite \ndatabase file that can (and should) be easily backed up (default path: \n`/etc/salt/dynamicsecrets.sqlite`). This should be used for non-critical \nsecrets that must be shared between minions and/or where a more secure solution\nlike Hashicorp's Vault is not yet applicable.\n\n**To be clear:** The purpose of `dynamicsecrets` is to help bootstrap a cluster \nof servers where a number of initial secrets must be set and \"remembered\" \nacross multiple nodes that are configured with Salt. This solution is *better*\nthan putting hardcoded secrets in your configuration management, but you should\nreally only rely on it to bootstrap your way to an installation of Hashicorp\nVault or another solution that solves the secret introduction problem \ncomprehensively!\n\nSecrets from the pillar are assigned to the \"roles\" grain or minion ids and\nare therefor only rendered to assigned minions from the master.\n\n```yaml\n# Example configuration\n    # Extension modules\n    extension_modules: /srv/salt-modules\n\n    ext_pillar:\n    - dynamicsecrets:\n        config:\n            approle-auth-token:\n                type: uuid\n            concourse-encryption:\n                length: 32\n            concourse-hostkey:\n                length: 2048\n                type: rsa\n            consul-acl-token:\n                type: uuid\n                unique-per-host: True\n            consul-encryptionkey:\n                encode: base64\n                length: 16\n        grainmapping:\n            roles:\n                authserver:\n                    - approle-auth-token\n        hostmapping:\n            '*':\n                - consul-acl-token\n```\n\nFor `type: password` the Pillar will simply contain the random password string.\nFor `type: uuid` the Pilar will return a UUID4 built from a secure random \nsource (as long as the OS provides one).\nFor `type: rsa` the Pillar will return a `dict` that has the following\nproperties:\n * `public_pem` the public key in PEM encoding\n * `public` the public key in `ssh-rsa` format\n * `key` the private key in PEM encoding\n\nThe dynamicsecrets pillar has been extracted [into it's own project at \njdelic/dynamicsecrets/](https://github.com/jdelic/dynamicsecrets).\n\n\n# Deploying applications\n\n## Service deployment \"through\" salt and \"on\" servers configured by salt\n\nFirst off, don't get confused by the service configuration and discovery states\nthat seem to be \"interwoven\" in this repository. The whole setup is meant to\n\n  * allow applications to be deployed from .debs or Docker containers, being\n    discovered through consul and then automatically be registered with a\n    server that has the \"loadbalancer\" role\n\n  * **but also** allow salt to install and configure services (like opensmtpd,\n    dovecot, concourse.ci or a PHP application that can not be easily packaged\n    in a .deb) and register that with consul to then make it available through\n    a server that has the \"loadbalancer\" role\n\nI generally, if in any way possible, would always prefer deploying an\napplication **not through salt states**, but other means (for example:\ninstalling a .deb package on all servers that have the role \"apps\" through the\nsalt CLI client) or better yet, push Docker containers to Nomad / Kubernetes /\nDocker Swarm. But if you have to (for example when configuring a service, which\nis typically part of a Unix system like a mail server) you **totally can** use\nsalt states for that. This way you don't have to repackage services which are\nalready set up for your system. No need to repackage dovecot in a Docker\ncontainer, for example, if the Debian Maintainers do such an awesome job of\nalready providing ready-to-run packages anyway! (Also, repeat after me:\n\"Salt, Puppet, Chef and Ansible are not deployment tools!\")\n\nAs I see it: use the best tool for the job. There is no dogma requiring you to\nrun all services inside a container for example. And a container is not a VM,\nso services consisting of multiple daemons don't \"containerize\" easily anyway.\nAnd some services really expect to use all available resources on a server\n(databases, for example) and *shouldn't be containerized* for that reason. And so\non and so forth..... so use whatever feels natural. This salt setup is flexible\nenough to accommodate all of these options.\n\n## Deploying packaged services from .debs (GoPythonGo applications, for example)\n\n## Deploying containerized services\nUsing servers with the `nomadserver` and `apps` roles, you have two options:\n\n  1. Use Docker Swarm, which is built into Docker\n  2. Use Hashicorp Nomad\n\nThey have different pros and cons, but Nomad supports scheduling jobs what are\nnot containerized (as in plain binaries and Java applications). Nomad also\ndirectly integrates with Consul clusters *outside* of the Nomad cluster. So it\nmakes service discovery between applications living inside and outside of the\ncluster painless. Using the smartstack implementation built into this\nrepository, you can just add `smartstack:*` tags to the `service {}` stanza in\nyour [Nomad job configuration](https://www.nomadproject.io/docs/job-specification/service.html)\nand service discovery as well as linking services to the loadbalancer will be\ntaken care of.\n\nFor plain Docker or Docker Swarm, you will have to run a\n[docker registrator](https://github.com/gliderlabs/registrator) instance on\neach cluster node, which does the same job as including consul service\ndefinitions in Nomad jobs. It registers services run from a docker container\nwith consul, discovering metadata from environment variables in the container.\nDocker Swarm however, at least until Nomad 0.6 arrives, will have the benefit\nof supporting VXLAN Overlay networks that can easily be configured to use\nIPSEC encryption. This gives you a whole new primitive for separating logical\nnetworks between your conainerized applications and is something to think\nabout.\n\nRegardless of how the services get registered in Consul, it will in turn\npropagate the service information through `consul-template` to `haproxy` making\nthe services accessible through localhost/docker-bridge based smartstack\nrouters, or even setting up routing to them from nodes with the `loadbalancer`\nrole, giving you fully automated service deployment for internal and external\nservices.\n\n\n# SmartStack\n\nSee [consul-smartstack](https://github.com/jdelic/consul-smartstack).\n\n### Integrating external services (not implemented yet)\n**Question: But I run my service X on Heroku/Amazon Elastic Beanstalk with\nautoscaling/Amazon Container Service/Microsoft Azure/Google Compute Engine/\nwhatever... how do I plug this into this smartstack implementation?**\n\n**Answer:** You create a Salt state that registers these services as\n`smartstack:internal` services, assign them a port in your [port map](PORTS.md)\nand make sure haproxy instances can route to it.This will cause\n`consul-template` instances on your machines to pick them up and make them\navailable on `localhost:[port]`. The ideal machines to assign these states to\nin my opinion are all machines that have the `consulserver` role. Registering\nservices with consul that way can either be done by dropping service\ndefinitions into `/etc/consul/services.d` or perhaps use\n[use salt consul states](https://github.com/pravka/salt-consul).\n\nA better idea security-wise is to create an \"egress router\" role that runs a\nspecialized version of haproxy-external that sends traffic from its internal IP\nto your external services. Then announce the internal IP+Port as an internal\nsmartstack service. The external services can then still be registered with the\nlocal Consul cluster, but you also have a defined exit point for traffic to\nthe external network.\n\n\n# Vault\n\nThis salt configuration also runs an instance of\n[Hashicorp Vault](https://vaultproject.io/) for better management of secure\ncredentials. It's good practice to integrate your applications with that\ninfrastructure.\n\nVault will be made available on `127.0.0.1` as an internal smartstack service\nthrough haproxy via consul-template on port `8200` once it's been\ninitialized (depending on the backend) and\n[unsealed](https://vaultproject.io/docs/concepts/seal.html).\n\nServices, however, **must** access Vault through a local alias installed in\n`/etc/hosts/` configured in the `allenvs.wellknown:vault:smartstack-hostname`\npillar (default: vault.local), because Vault requires SSL and that in turn\nrequires a valid SAN, so you have to configure Vault with a SSL certificate for\na valid hostname. I use my own CA and give Vault a certificate for the SAN\n`vault.local` and then pin the CA certificate to my own CA's cert in the\n`allenvs.wellknown:vault:pinned-ca-cert` pillar for added security (no other CA\ncan issue such a certificate for any uncompromised host).\n\n## Backends\nYou can configure Vault through the `[hosting environment].vault` pillar to use\neither the *consul*, *mysql*, *S3* or *PostgreSQL* backends.\n\n### Vault database backend\nGenerally, if you run on multiple VMs sharing a physical server, choose the\n`postgresql` backend and choose backup intervals and Vault credential leases\nwith a possible outage in mind. Such a persistent backend will not be highly\navailable, but unless you distribute your VMs across multiple physical\nmachines, your setup will not be HA anyway. So it's better to fail in a way \nthat let's your restore service easily.\n\nRunning this setup from this Salt recipe requires at least one server in the\nlocal environment to have the `database` role as it will host the Vault\nPostgresSQL database. The Salt recipe will automatically set up a `vault`\ndatabase on the `database` role if the vault pillar has the backend\nset  `postgresql`, because the `top.sls` file shipped from this repo assigns\nthe `vault.database` state to the `database` role.\n\nTo sum up: To enable this backend, set the Pillar\n`[server environment].vault.backend` to `postgresql` and assign exactly one\nserver the `database` role (this salt configuration doesn't support database\nreplication) and at least one server the `vault` role.\n\n[More information at the Vault website.](https://vaultproject.io/docs/config/index.html)\n\n### Vault backend: consul\nIf you run your VMs in a Cloud or on multiple physical servers, running Vault\nwith the Consul cluster backend will offer high availability. In this case it\nalso makes sense to run at least two instances of Vault. Make sure to distribute\nthem across at least two servers though, otherwise a hardware failure might take\ndown the whole Consul cluster and thereby also erase all of the data.\n\n[More information at the Vault website.](https://vaultproject.io/docs/config/index.html)\n\n\n# Networking\n\n## sysctl config\nSysCTL is set up to accept\n\n * `net.ipv4.ip_forward` IPv4 forwarding so we can route packets to docker and\n   other isolated separate networks\n * `net.ipv4.ip_nonlocal_bind` to allow local services to bind to IPs and ports\n   even if those don't exist yet. This reduces usage of the `0.0.0.0` wildcard\n   allowing to write more secure configurations.\n * `net.ipv4.conf.all.route_localnet` allows packets to and from `localhost` to\n   pass through iptables, making it possible to route them. This is required so\n   we can make consul services available on `localhost` even though consul runs \n   on its own `consul0` dummy interface.\n\n## iptables states\niptables is configured by the `basics` and `iptables` states to use the\n`connstate`/`conntrack` module to allow incoming and outgoing packets in the\n`RELATED` state. So to enable new TCP services in the firewall on each\nindividual machine managed through this saltshaker, only the connection\ncreation needs to be managed in the machine's states.\n\nThe naming standard for states that enable ports that get contacted is:\n`(servicename)-tcp-in(port)-recv`. For example:\n\n```yaml\nopenssh-in22-recv:\n    iptables.append:\n        - table: filter\n        - chain: INPUT\n        - jump: ACCEPT\n        - source: '0/0'\n        - proto: tcp\n        - dport: 22\n        - match: state\n        - connstate: NEW\n        - save: True\n        - require:\n            - sls: iptables\n```\n\nThe naming standard for states that enable ports that initiate connections is:\n`(servicename)-tcp-out(port)-send`. For example:\n\n```yaml\ndns-tcp-out53-send:\n      iptables.append:\n          - table: filter\n          - chain: OUTPUT\n          - jump: ACCEPT\n          - destination: '0/0'\n          - dport: 53\n          - match: state\n          - connstate: NEW\n          - proto: tcp\n          - save: True\n```\n\nConnections *to* and *from* `localhost` are always allowed.\n\n## Address overrides and standard interfaces\nThis saltshaker expects specific interfaces to be used for either internal or\nexternal (internet-facing) networking. The interface names are assigned in the\n`local|hetzner.network` pillar states in the `ifassign` states. Commonly the\nnetwork assignments are this:\n\n  * `eth0` is either a local NAT interface (vagrant) or the lowest numbered\n    full network interface.\n  * The lowest numbered full network interface is commonly the \"internal\n    interface\". It's connected to a non-routed local network between the nodes.\n  * The next numbered full network interface is commonly the \"external\n    interface\". It's connected to the internet.\n  * The `consul0` interface is a dummy interface with a link-local IP of\n    `169.254.1.1`. It's used to provide consul HTTP API and DNS API access to\n    local services. A link-local address is used so it can be routed from\n    the docker bridge.\n  * `docker0` is the default bridge network set up by Docker that containers\n    usually attach to.\n\nSome configurations (like the mailserver states) can expect multiple external\nnetwork interfaces or at least multiple IP addresses to work correctly.\n\nTherefor many states check a configuration pillar for themselves to figure out\nwhether they should bind to a specific IP and if not, use the first IP assigned\nto the internal or external network interface. This is usually accomplished by\nthe following Jinja2 recipe:\n\n```jinja2\n{{\n# first, check whether we have a configuration pillar for the service,\n# otherwise return an empty dictionary\npillar.get('authserver', {}).get(\n    # check whether the configuration pillar has a configuration option\n    # 'bind-ip'. If not, return the first IP of the local internal interface\n    'bind-ip',\n    # query the machine's interfaces (ip_interfaces) and use the value returned\n    # by the interface assiged to be internal in the ifassign pillar\n    grains['ip_interfaces'][pillar['ifassign']['internal']][\n        # use the first IP or the IP designated in the network pillar\n        pillar['ifassign'].get('internal-ip-index', 0)|int()\n    ]\n)\n}}\n```\n\n**Please note that commonly external services meant to be reached by the\ninternet listen to internal interfaces on their application servers and are\nrouted to the internet through the smartstack routing built into this\nconfiguration.**.\n\n## Reserved top-level domains\n\nThis configuration relies on three internal reserved domain suffixes, which\n**must be replaced if they're ever brought up as a TLD on the global DNS**.\nThose are:\n  * `.local` which **must** resolve to an address in 127.0.0.1/24\n  * `.internal` which **must** only be used within the non-publically-routed\n    network (i.e. on an \"internal\" network interface)\n  * `.consul.service` which is the suffix used by Consul for DNS based\n    \"service discovery\" (repeat after me: *DNS is not a service discovery\n    protocol*! Use Smartstack instead.)\n\n\n# Configuration\n\n## PostgreSQL\n\n### Accumulators\n\n**PostgreSQL**\n\nThe PostgreSQL configuration uses two accumulators that record `database user`\npairs (separated by a single space) for the `pg_hba.conf` file. These\naccumulators are called:\n\n  * `postgresql-hba-md5users-accumulator` and\n  * `postgresql-hba-certusers-accumulator`\n\nEach line appended to them creates a line in `pg_hba.conf` that is hardcoded\nto start with `hostssl`, use the PostgreSQL server's internal network and\nhas the authentication method `md5` or `cert`. These accumulators are meant for\nservice configuration to automatically add login rights to the database after\ncreating database roles for the service.\n\nThe `filename` attribute for such `file.accumulated` states *must* be set to\n`{{pillar['postgresql']['hbafile']}}` which is the configuration pillar\nidentifying the `pg_hba.conf` file for the installed version of PostgreSQL in\nthe default cluster. The accumulator must also have a `require_in` directive\ntying it to the `postgresql-hba-config` state delivering the `pg_hba.conf` file\nto the node.\n\nExample:\n\n```yaml\nmydb-remote-user:\n    file.accumulated:\n        - name: postgresql-hba-md5users-accumulator\n        - filename: {{pillar['postgresql']['hbafile']}}\n        - text: {{pillar['vault']['postgres']['dbname']}} {{pillar['vault']['postgres']['dbuser']}}\n        - require_in:\n            - file: postgresql-hba-config\n```\n\n**Apache2**\n\nThe Apache2 configuration uses an acuumulator `apache2-listen-ports` to gather\nall listen directives for its `/etc/apache2/ports.conf` file. The filename\nattribute  for states setting up new `Listen` directives, should be set to \n`apache2-ports-config`. The values accumulated can be either just port numbers\nor `ip:port` pairs.\n\nExample:\n\n```yaml\napache2-webdav-port:\n    file.accumulated:\n        - name: apache2-listen-ports\n        - filename: /etc/apache2/ports.conf\n        - text: {{my_ip}}:{{my_port}}\n        - require_in:\n            - file: apache2-ports-config\n```\n\n## PowerDNS Recursor\n\nThis Salt config sets up a PowerDNS recursor on every node that serves as an\ninterface to the Consul DNS API. It's usually only available to local clients\non `127.0.0.1:53` and `169.254.1.1:53` from where it forwards to Consul on \n`169.254.1.1:8600`. However, sometimes it's useful to expose the DNS API to\nother services, for example on a Docker bridge or for other OCI containers.\n\n### Accumulators\n\nThe PowerDNS configuration uses two accumulators to allow the adding of IPs\nthat PowerDNS Recursor listens on and allow CIDR ranges that can query the \nrecursing DNS server on those IPs:\n\n  * `powerdns-recursor-additional-listen-ips` and\n  * `powerdns-recursor-additional-cidrs`\n  \nAs above, the `filename` attribute for each `file.accumulated` state that uses\none such accumulator *must* be set to `/etc/powerdns/recursor.conf` and it must\nhave a `require_in` directive tying it to the `pdns-recursor-config` state.\n\nExample:\n\n```yaml\nmynetwork-dns:\n    file.accumulated:\n          - name: powerdns-recursor-additional-listen-ips\n          - filename: /etc/powerdns/recursor.conf\n          - text: 10.0.254.1\n          - require_in:\n              - file: pdns-recursor-config\n```\n\n\n# Contributing\n\nThe following style is extraced from what has informally followed during\ndevelopment of this repository and is therefor needed to remain consistent.\n\n## General code style\n\n  * Indents are 4 spaces\n  * Top-level `yaml` elements have two newlines between them (just like Python\n    PEP8)\n  * Each file has a newline at the end of its last line\n  * If you find yourself repeating the same states over and over (like creating\n    an entry in `/etc/appconfig/` for each deployed application, write a custom\n    Salt state (see `srv/_states/` for examples)\n  * aside from MarkDown documentation, lines are up to 120 characters long\n    unless linebreaks are impossible for technical reasons (long URLs,\n    configuration files that don't support line breaks).\n  * When breaking lines, search for \"natural\" points for well-readable line\n    breaks. In natural language these usually are after punctuation. In code,\n    they are usually found after parentheses in function calls or other code\n    constructs, while indenting the next line, or at the end of statements.\n\n### yaml code style\n\n  * String values are in single quotes `'xyz'`\n  * `.sls` yaml files should end with a vim modeline `# vim: syntax=yaml`.\n\n### Documentation style\nUse MarkDown formatted files.\n\n  * formatted to 80 columns.\n  * List markers are indented 2 spaces leading to a text indent of 4 (bullets)\n    or 5 (numbers) on the first level.\n  * Text starts right below a headline unless it's a first level headline *or*\n    the paragraph starts off with a list.\n\n## State structure\n\n  * Jinja template files get the file extension `.jinja.[original extension]`\n  * Configuration files should be stored near the states configuring the\n    service\n  * Only use explicit state ordering when in line with [ORDER.md}(ORDER.md).\n  * Put states configuring services in the top-level `srv/salt/` folder then\n    create a single state namespace with configuration for specific\n    organizations (like the `mn` state).\n\n## Pillar structure\n\n  * Each state should get its configuration from a pillar.\n  * Reuse configuration values through Jinja's\n    `{% from 'file' import variable %}` since Salt does not support referencing\n    pillars from pillars yet.\n  * Use `{{salt['file.join'}(...)}}`, `{{salt['file.basename'}(...)}}`,\n    `{{salt['file.dirname'}(...)}}` to construct paths from imported variables.\n  * Pillars may import variables from things in `srv/pillars/shared/`,\n    `srv/pillar/allenvs/` or from their local environment. No cross-environment\n    imports.\n  * Each deployment environment (\"vagrant\", \"hetzner\", \"digialocean\", \"ec2\" are\n    all examples) get their own namespace in `srv/pillars/`.\n  * Each environment has a special state called `*/wellknown.sls` that is\n    assigned to *all* nodes in that environment for shared configuration values\n    that can reasonably be expected to stay the same across all nodes, are not\n    security critical and are needed or expected to be needed to run more than\n    one service or application.\n  * Pillars that are not security critical and are needed for multiple services\n    or applications and can reasonably be expected to stay the same across all\n    environments go into `allenvs.wellknown`.\n  * `srv/pillar/shared/` is the namespace for configuration that is shared\n    across environments (or can reasonably expected to be the same, so that it\n    makes sense to only override specific values in the pillars for a specific\n    environment), *but* is not assigned to all nodes because its contents may\n    be security critical or simply only needed on a single role.\n\n### SSL configuration in Pillars\nSSL configuration should use the well-known keys `sslcert`, `sslkey` and if\nsupported, clients should have access to a `pinned-ca-cert` pillar so the CA\ncan be verified. `sslcert` and `sslkey` should support the magic value\n`default` which should make the states render configuration files referring to\nthe pillars `ssl:default-cert(-combined)` and `ssl:default-cert-key`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdelic%2Fsaltshaker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjdelic%2Fsaltshaker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdelic%2Fsaltshaker/lists"}