Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dgengtek/salt-ext-pillar-vault-lookup
Saltstack pillar extension module for vault secrets
https://github.com/dgengtek/salt-ext-pillar-vault-lookup
python3 saltstack saltstack-extension secrets vault
Last synced: 18 days ago
JSON representation
Saltstack pillar extension module for vault secrets
- Host: GitHub
- URL: https://github.com/dgengtek/salt-ext-pillar-vault-lookup
- Owner: dgengtek
- License: mit
- Created: 2022-06-24T20:46:49.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2022-06-24T21:06:18.000Z (over 2 years ago)
- Last Synced: 2024-11-05T12:35:47.237Z (2 months ago)
- Topics: python3, saltstack, saltstack-extension, secrets, vault
- Language: Python
- Homepage:
- Size: 20.5 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Saltstack pillar extension module for vault
This repository contains an implementation and tests for mapping vault paths to
pillar paths dynamically during runtime.* filtering by keys of the vault data(select only keys you want in the pillar)
* filtering by matching keys and selected values
* supported lookup of grains and pillars
* in either vault path or pillar path
* for filling values of matchers
* templating of path via jinja and using lookup resultsThe module parses the configuration and collects all found lookups(grains G@, pillar I@) in a set. During its
first pass through it does a lookup for each unique lookup definition so that a
repeatedly defined lookup is not requested multiple times.The code could be potentially used for a vault renderer.
## Installation
Drop [this file](./vault_lookup.py) in the /pillar directory which is
[configured on the salt-master](https://docs.saltproject.io/en/latest/ref/configuration/master.html#extension-modules)## Run tests
$ pip3 install -r requirements.txt
$ pytest## Configuration
### In salt masterEither in salt-master config
ext_pillar:
- vault_lookup:
lookup_key: ext_pillar_vault_lookup
config:
"":
- path: secret/admin/id_admin
pillarpath: users/admin/key### In salt master and pillar
Or pull the configuration from pillar if defined in the default pillar key `ext_pillar_vault_lookup`.
ext_pillar:
- : {}
...
- vault_lookup: {}Somewhere set the config in pillar before vault_lookup module
ext_pillar_vault_lookup:
"":
- path: secret/admin/id_admin
pillarpath: users/admin/key## Examples
### Single lookupSingle lookup of mapping a vault path to a pillarpath
ext_pillar:
- vault_lookup:
config:
"simple lookup":
# single mapping
- path: secret/admin/id_admin
pillarpath: users/admin/keyvault lookup:
private:
public:
mapped to pillar:users:
admin:
key:
private:
public:### Wildcard
Example of using wildcards. There must be an equal amount of wildcards for both paths.
ext_pillar:
- vault_lookup:
config:
"wildcard 1.":
# map each entry from list lookup of ../workers/* to ../worker_keys/* in pillar
# eg list returns ../workers/{worker1, worker2} which will map to pillar ../worker_keys/{worker1, worker2}
- path: secret/workers/*
pillarpath: somepath/*
"wildcard 2.":
- path: secret/nested/*/secrets/*
pillarpath: mapped/*/secret/*/valueFor the item wildcard 1.:
path: secret/workers/*
pillarpath: somepath/*path wildcard of vault contains ["w1", "w2"]. For each item a lookup will be done and the result mapped to pillarpath
secret/workers/w1 -> somepath/w1
secret/workers/w2 -> somepath/w2For the item wildcard 2.:
The products of each result will be combined and mapped to the pillarpath
path: secret/*/items/*
pillarpath: mapped/*/secret/*/value* eg secret/* returns ["s1", "s2"]
* eg secret/s1/items/* returns ["i1", "i2"]
* eg secret/s2/items/* returns ["i3", "i4"]The resulting mapping from vault(path) -> to pillar(pillarpath)
secret/s1/items/i1 -> mapped/s1/secret/i2/value
secret/s1/items/i2 -> mapped/s1/secret/i2/value
secret/s2/items/i3 -> mapped/s2/secret/i3/value
secret/s2/items/i4 -> mapped/s2/secret/i4/value### Filtering with single lookup
Example of conditionally filtering with single lookup
ext_pillar:
- vault_lookup:
config:
"default":
- path: secret/path
pillarpath: path/value
# use all keys from the vault path lookup
keys: []
matches:
- keys:
- host
# if the value of the key 'host' matches any of the values defined here, keep them in the keyset from the vault lookup
values:
- host3
- host7
- host9
# otherwise remove the listed keys here from the keyset defined in the outer definition 'keys'
# the keys here are added with their values to the result only if 'host' matches any of the values
add_keys:
- privateeg the data the vault lookup of secret/path returns:
host: host3
private: pkey
anotherkey: anothervaluevalue of key host is in the values of the matchers so the key 'private' will not be removed:
path:
value:
host: host3
private: pkey
anotherkey: anothervalue
if the vault lookup was instead:
host: host1
private: pkey
anotherkey: anothervaluethen the mapped pillar would look:
path:
value:
host: host1
anotherkey: anothervalue### Using grains or pillars in values of matchers
ext_pillar:
- vault_lookup:
config:
"example 1":
- path: secret/path
pillarpath: path/value
# only select the keys named 'public' and 'private' from the vault lookup keyset
keys:
- public
- private
matches:
- keys:
- host
# lookup grain id to check against the value of the vault lookup
values:
- G@id
# add keys if any of the values matched the lookup of the path, otherwise remove listed keys from the list of keys to add
add_keys:
- private
- keys:
- location
# lookup pillar path 'locations:west:hosts' to check against the value of the vault lookup of the key 'location'
values:
- I@locations:west:hosts
- I@hosts
# add keys if any of the values matched the lookup of the path, otherwise remove listed keys from the list of keys to add
add_keys:
- public
"example 2":
- path: secret/app1
pillarpath: app1
keys:
- token
matches:
- keys:
- slave
values:
- I@appslaves
add_keys:
- tokenvault lookup of example 1. secret/path:
host: host3
public: pubkey
private: pkey
anotherkey: anothervalueif the value of the key host matches the result of G@id then the pillar would be:
path:
value:
public: pubkey
private: pkey
if the value of G@id was host1 instead then the mapped pillar would look:
path:
value:
public: pubkey### Adding all keys or none if the matcher is not true.
ext_pillar:
- vault_lookup:
config:
"default":
- path: "secret/path"
pillarpath: "path/values"
# use all keys from the vault path lookup
keys: []
matches:
# if keys empty and values are not empty then merge all keys
- keys: []
values:
# only add all keys if the pillar lookup path 'somevalue:defined' is not empty
# otherwise no keys will be added
- 'I@somevalue:defined'
add_keys: []### Matching against an exact value of a pillar lookup
ext_pillar:
- vault_lookup:
config:
"default":
- path: "secret/salt/concourse/master/session_signing_key"
pillarpath: "concourse/web/session_signing_key"
# use all keys from the vault path lookup
keys: []
matches:
# if keys empty and values are not empty then merge all keys, otherwise none are added
- keys: []
values:
# only do it if the pillar lookup of 'tags' contains 'concourse-master'
- 'I@tags=concourse-master'
add_keys: []### Using grains or pillars in the paths.
These will expand and create multiple pairs matching their result. A lookup
result which contains a list will iterate over it. For a dictionary only the
keynames will be replaced for it.If only either one of the paths contains a lookup definition its lookup value
must be a single value. If the lookup value contains a list or dict then both
paths need to have the same lookup definition otherwise it will be unable to
create pairs to map against.ext_pillar:
- vault_lookup:
config:
# single value lookup is fine
"example 1":
- path: "secret/salt/concourse/workers/G@id"
pillarpath: "concourse/worker/G@id/worker_key"
# single value lookup is fine even if the other path contains no lookups
"example 2":
- path: "secret/salt/concourse/workers/G@id"
pillarpath: "concourse/worker/secret"
# this lookup must be a single value
"example 3":
- path: "secret/personal/I@some:path"
pillarpath: "private/secret"
# if this lookup contains a list or dict then the other path must have the same lookup definition
"example 4":
- path: "secret/personal/I@persons"
pillarpath: "private/I@persons"
# if pillar clustermembers returns a list, run a vault lookup for each item
# or if it is a value only a single lookup
"example 5":
- path: hosts/I@clustermembers/secret
# if the lookup was a single value, map to pillar 'myapp'
# if it was a list then the lookup is required to be in pillarpath too
# pillarpath: myapp/I@clustermembers
# otherwise there is no way to know how to map
pillarpath: myappVault lookup for example 4:
I@persons returns a list of ["p1", "p2"] or if it returns a dict of {"p1": {...}, "p2": ...}secret/personal/p1 -> private/p1
secret/personal/p2 -> private/p2Mapped pillar would be merged to:
private:
p1:
# result from the vault lookup mapped
# key: value
...
p2:
...### Matching against a value of a lookup result in paths
Using multiple counts for lookups is fine since it will map to one path each.:
ext_pillar:
- vault_lookup:
config:
"single lookup result":
# do a lookup of the pillar I@cluster and continue if it resolves to master1
# do a lookup of the pillar I@role and continue only if it is 'backup'
- path: hosts/I@cluster=master1/webtoken
pillarpath: myapp/config/I@role=backup/token
# this will map to:
# path -> hosts/master1/webtoken
# pillarpath -> myapp/config/backup/token
"multiple single lookups":
- path: hosts/I@cluster=master1/webtoken
pillarpath: myapp/I@app=app1/I@role=backup/token
- path: hosts/I@backupserver=b1/secret
pillarpath: backup/hosts/G@id/I@location/tokenIf I@cluster returns ["master1", "master2"] and I@role returns ["server", "backup"] then it will be mapped hosts/master1/webtoken -> myapp/config/token
### Jinja template rendering
Templates are mainly used to do lookups of a result of a previous lookup(a template depends on a previous templates result).
The constraint for using the keys from a template definition in paths is that all keys are of the same root template definition and either
1. the used keys are from the same templates used
# k1 and k2 are from the same template
- path: secret/{{k1}}/{{k2}}
pillarpath: new/{{k2}}
template:
- keys:
- k1
- k2
lookup: "I@somelookup"
- keys:
- k3
lookup: "I@otherlookup"2. or all the templates of the keys used in pillarpath contain all the templates from path
ie is the set of keys from the templates of a path a subset of the keys from the templates of pillarpathPractically make sure that the templates can be resolved in order of the list they are defined in.
Example of using jinja2 templates in paths and template definitions which depend
on a previous template definition.ext_pillar:
- vault_lookup:
config:
"jinja template rendering":
# lookup keys for templates {{key}} defined in paths
- path: "secret/ssh/users/{{user}}/{{id}}"
pillarpath: "users/present/{{puser}}/ssh_keys/{{id}}"
keys:
- public
- private
matches:
- keys:
- host
values:
- G@id
add_keys:
- private
# keys from a lookup can be used for the following lookups in the list
template:
# keys gets the values from the lookup and is used for templating paths or templates which depend on this key
# if 'ext_pillar_vault_ssh_keys:present' is a dict, puser will be the keyname for each key of the dict lookup result
# if 'ext_pillar_vault_ssh_keys:present' is a list, puser will be the name for each item of the list
# invalid if there are multiple keys listed and the lookup only resolves to a single value, list of values and not a dict
# t1.
- keys:
- puser
lookup: "I@ext_pillar_vault_ssh_keys:present"
# multiple keys can be used to match the result
# if the lookup is a dict the listed keys will be used for the lookup of the dict
# if the lookup is a list of dict then the keys named here will be used from lookup for each dict in the list
# t2.
- keys:
- id
- user
lookup: "I@ext_pillar_vault_ssh_keys:present:{{puser}}"The pillar lookup of template t1 'ext_pillar_vault_ssh_keys:present' returns
admin:
- user: backup
id: id_backup
- user: root
id: intranet
root:
- user: root
id: id_rootThe values of the key 'puser' will be ["admin", "root"]
The pillar lookup of template t2 'ext_pillar_vault_ssh_keys:present:admin' returns
- user: backup
id: id_backup
- user: root
id: intranetKeys and value pairs will be [(id: id_backup, user: backup), (id: intranet, user: root)]
The pillar lookup of template t2 'ext_pillar_vault_ssh_keys:present:root' returns
- user: root
id: intranetKeys and value pairs will be [(id: id_root, user: root)]
mapping of values for each pair:
secret/ssh/users/backup/id_backup -> users/present/admin/ssh_keys/id_backup
secret/ssh/users/root/intranet -> users/present/admin/ssh_keys/intranetsecret/ssh/users/root/id_root -> users/present/root/ssh_keys/intranet
### Mixing of templates, lookups, wildcards.
Order of rendering to get the resolved paths for vault lookup and pillarpath:
1. templates in the order of definition will be resolved first
2. then the lookups will be rendered on the results
3. wildcards are iterated over lastThe config:
ext_pillar:
- vault_lookup:
config:
"mix":
- path: "secret/apps/*/{{worker}}/I@appslaves"
pillarpath: "app/*/users/{{keyid}}/ssh_keys/I@appslaves"
template:
# t1.
- keys:
- worker
lookup: "I@workers"
# t2.
- keys:
- keyid
lookup: "I@workers:{{worker}}:config:keyid"Order of processing:
1. Existing pillar :
workers:
w1:
config:
keyid: | ...
w2:
config:
keyid: | ...Result of template t1.
worker: ["w1", "w2"]
Result of template t2.
keyid: ["k1", "k2"]
2. Salt pillar lookup
I@appslaves = ["s1", "s2"]
3. wildcard vault listing
secret/apps/* -> ["app1", "app2"]
Result of rendering in order and their mapping
1. templates
path: "secret/apps/*/{{worker}}/I@appslaves"
pillarpath: "app/*/users/{{keyid}}/ssh_keys/I@appslaves"
secret/apps/*/w1/I@appslaves -> app/*/users/k1/ssh_keys/I@appslaves
secret/apps/*/w2/I@appslaves -> app/*/users/k2/ssh_keys/I@appslaves2. lookups
secret/apps/*/w1/s1 -> app/*/users/k1/ssh_keys/s1
secret/apps/*/w1/s2 -> app/*/users/k1/ssh_keys/s2
secret/apps/*/w2/s1 -> app/*/users/k2/ssh_keys/s1
secret/apps/*/w2/s2 -> app/*/users/k2/ssh_keys/s23. wildcard
secret/apps/app1/w1/s1 -> app/app1/users/k1/ssh_keys/s1
secret/apps/app2/w1/s1 -> app/app2/users/k1/ssh_keys/s1
secret/apps/app1/w1/s2 -> app/app1/users/k1/ssh_keys/s2
secret/apps/app2/w1/s2 -> app/app2/users/k1/ssh_keys/s2
secret/apps/app1/w2/s1 -> app/app1/users/k2/ssh_keys/s1
secret/apps/app2/w2/s1 -> app/app2/users/k2/ssh_keys/s1
secret/apps/app1/w2/s2 -> app/app1/users/k2/ssh_keys/s2
secret/apps/app2/w2/s2 -> app/app2/users/k2/ssh_keys/s2If anything fails the pair will not be considered for pillar merging.