https://github.com/jbox-web/stacker
Stacker - A lightweight file-based CMDB
https://github.com/jbox-web/stacker
cmdb crystal crystal-lang salt saltstack
Last synced: 3 months ago
JSON representation
Stacker - A lightweight file-based CMDB
- Host: GitHub
- URL: https://github.com/jbox-web/stacker
- Owner: jbox-web
- License: mit
- Created: 2020-05-05T11:41:17.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2023-12-10T00:06:49.000Z (over 1 year ago)
- Last Synced: 2024-03-26T11:24:06.979Z (over 1 year ago)
- Topics: cmdb, crystal, crystal-lang, salt, saltstack
- Language: Crystal
- Homepage:
- Size: 505 KB
- Stars: 10
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Stacker - A lightweight file-based CMDB
[](https://github.com/jbox-web/stacker/blob/master/LICENSE)
[](https://github.com/jbox-web/stacker/actions/workflows/ci.yml)
[](https://hub.docker.com/r/nicoladmin/stacker)Stacker is [Salt PillarStack](https://docs.saltstack.com/en/master/ref/pillar/all/salt.pillar.stack.html) in [Crystal](https://crystal-lang.org/).
It is implemented using [crinja](https://github.com/straight-shoota/crinja) which is Jinja2 in Crystal :)
[Documentation](https://jbox-web.github.io/stacker/index.html)
## Installation
### Manual installation
Grab the latest binary from the [releases](https://github.com/jbox-web/stacker/releases) page and run it with the [sample configuration project](https://github.com/jbox-web/stacker/tree/master/example).
If you use [asdf](https://github.com/asdf-vm/asdf) you can also install Stacker with [asdf-stacker](https://github.com/jbox-web/asdf-stacker).
### Docker
```sh
docker pull nicoladmin/stacker:latest
```**Note :** The Docker mode comes with a [wrapper script](https://github.com/jbox-web/stacker/blob/master/stacker.sh) to ease interactions with the container
```sh
curl -sSL -o stacker.sh https://raw.githubusercontent.com/jbox-web/stacker/master/stacker.sh
chmod +x stacker.sh
```Usage : `stacker.sh {start|stop|restart|status|kill|clean|fetch|info|logs}`
## The 10 seconds test in Web mode
First we need to start Stacker with sample data :
### Manual installation
```sh
stacker server --config example/stacker.yml
```### Docker
```sh
./stacker.sh start
```Then fetch pillars with Curl :
```sh
curl --no-progress-meter -X POST -H "Content-Type: application/json" -d @example/grains/server1.curl.json http://127.0.0.1:3000/server1.example.net | jq
```You can also navigate to http://127.0.0.1:3000/server1.example.net to see the generated pillars.
## The 10 seconds test in CLI mode
In this mode you don't need the webserver to be running :
### Manual installation
```sh
stacker fetch server1.example.net --config example/stacker.yml --grains example/grains/server1.json --pillar example/ext_pillar/server1.json | jq
```### Docker
```sh
./stacker.sh fetch server1.example.net --grains grains/server1.json --pillar ext_pillar/server1.json | jq
```## Usage
```sh
Usage:
stacker [flags...] [arg...]Stacker is Salt PillarStack in Crystal
Flags:
--help # Displays help for the current command.
--version # Displays the version of the current application.Subcommands:
fetch # Fetch host pillars
info # Show Stacker information
server # Run Stacker webserver
```## Configuration
By default Stacker looks for it's configuration file in the current directory (`stacker.yml`).
You can pass an alternative file by using `--config` flag.
The configuration file is a YAML file looking like this :
```yml
---
doc_root: example/doc_root
entrypoint: server-pillars
log_file: ./log/stacker.logstacks:
default:
- example/doc_root/server-pillars/stack1.cfg
dev:
- example/doc_root/server-pillars/stack1.cfg
- example/doc_root/pillars/stack1.cfg
prod:
- example/doc_root/server-pillars/stack1.cfg
- example/doc_root/pillars/stack1.cfg
- example/doc_root/server-pillars/stack2.cfgserver_host: 127.0.0.1
server_port: 3000
server_environment: development
```**Note :** You can use relative or absolute file path.
Config | Description
---------------------|------------
`doc_root` | the webserver document root (must be specified). Since pillar are also crinja templates, it means where are the template files?
`entrypoint` | the webserver entrypoint (must be specified). The directory in the `doc_root` where we shoud look for `.yml` file
`log_file` | the path to the log file
`stacks` | a hash of namespaced [stack configuration files](https://docs.saltstack.com/en/master/ref/pillar/all/salt.pillar.stack.html#list-of-config-files) (must be specified)
`server_host` | ip address to bind to (default `127.0.0.1`)
`server_port` | port to bind to (default `3000`)
`server_environment` | `development` or `production` (default `production`)## Salt integration
To integrate Stacker with Salt you first need to add the [stacker pillar module](/salt/stacker.py) in Salt :
1. Declare extension modules directory in Salt (`/etc/salt/master.conf` or `/etc/salt/master.d/f_defaults.conf`)
```yml
extension_modules: /data/salt/modules
```2. Create `/data/salt/modules/pillar` directory and puts [stacker module](https://github.com/jbox-web/stacker/blob/master/salt/stacker.py) in it
```sh
mkdir -p /data/salt/modules/pillar
wget -O /data/salt/modules/pillar/stacker.py https://raw.githubusercontent.com/jbox-web/stacker/master/salt/stacker.py
```3. Declare the new ext_pillar module in Salt
```yml
ext_pillar:
- stacker: http://127.0.0.1:3000
```You can also pass parameters to Stacker module :
```yml
ext_pillar:
- stacker:
host: http://127.0.0.1:3000
namespace: 'production'
log_level: 'debug'
```4. Restart Salt, you're done :)
## Namespaces
Use namespaces by using optional `n=` query parameter :
```sh
curl http://127.0.0.1:3000/server1.example.net?n=dev
curl http://127.0.0.1:3000/server1.example.net?n=prod
```Or with `--namespace` flag when using Stacker CLI.
The default namespace when query parameter or CLI flag is omited is `default`.
## Output format
Set output format by using optional `f=` query parameter :
```sh
curl http://127.0.0.1:3000/server1.example.net?f=json
curl http://127.0.0.1:3000/server1.example.net?f=yaml
```Or with `--output-format` flag when using Stacker CLI.
The default output format when query parameter or CLI flag is omited is `json`.
Only `json` and `yaml` are supported.
## Logs
* log level
The log level is dynamic. No need to restart the web server :)
Set log level by using optional `l=` query parameter :
```sh
curl http://127.0.0.1:3000/server1.example.net?l=debug
curl http://127.0.0.1:3000/server1.example.net?l=trace
```Or with `--log-level` flag when using Stacker CLI.
The default log level when query parameter or CLI flag is omited is `info`.
Log levels other than `debug` or `trace` are meaningless.
`trace` level is very verbose as it dumps data before and after merge operations. In this case you might need some filtering, see below...
debug level will render something like this :
```sh
2020-10-02T23:18:27.149678Z INFO - processor: Building stack for: server2.example.net (namespace: prod)
2020-10-02T23:18:27.149897Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/stack1.cfg
2020-10-02T23:18:27.149927Z DEBUG - processor: Loading: example/doc_root/server-pillars/01-base.yml
2020-10-02T23:18:27.149936Z DEBUG - processor: Compiling: example/doc_root/server-pillars/01-base.yml
2020-10-02T23:18:27.150030Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/01-base.yml
2020-10-02T23:18:27.150078Z DEBUG - processor: Merging: example/doc_root/server-pillars/01-base.yml
2020-10-02T23:18:27.150099Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net.yml
2020-10-02T23:18:27.150109Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net.yml
2020-10-02T23:18:27.150190Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net.yml
2020-10-02T23:18:27.150263Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net.yml
2020-10-02T23:18:27.150705Z DEBUG - renderer: Compiled: example/doc_root/pillars/stack1.cfg
2020-10-02T23:18:27.150775Z DEBUG - processor: Loading: example/doc_root/pillars/01-base/01-base.yml
2020-10-02T23:18:27.150784Z DEBUG - processor: Compiling: example/doc_root/pillars/01-base/01-base.yml
2020-10-02T23:18:27.150873Z DEBUG - renderer: Compiled: example/doc_root/pillars/01-base/01-base.yml
2020-10-02T23:18:27.150932Z DEBUG - processor: Merging: example/doc_root/pillars/01-base/01-base.yml
2020-10-02T23:18:27.150942Z DEBUG - processor: Loading: example/doc_root/pillars/01-base/app1.yml
2020-10-02T23:18:27.150949Z DEBUG - processor: Compiling: example/doc_root/pillars/01-base/app1.yml
2020-10-02T23:18:27.151042Z DEBUG - renderer: Compiled: example/doc_root/pillars/01-base/app1.yml
2020-10-02T23:18:27.151091Z DEBUG - processor: Merging: example/doc_root/pillars/01-base/app1.yml
2020-10-02T23:18:27.151108Z DEBUG - processor: Loading: example/doc_root/pillars/01-base/zdump.yml
2020-10-02T23:18:27.151115Z DEBUG - processor: Compiling: example/doc_root/pillars/01-base/zdump.yml
2020-10-02T23:18:27.151208Z DEBUG - renderer: Compiled: example/doc_root/pillars/01-base/zdump.yml
2020-10-02T23:18:27.151238Z DEBUG - processor: Merging: example/doc_root/pillars/01-base/zdump.yml
2020-10-02T23:18:27.151267Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/common/locale.yml
2020-10-02T23:18:27.151286Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/common/locale.yml
2020-10-02T23:18:27.151392Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/common/locale.yml
2020-10-02T23:18:27.151426Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/common/locale.yml
2020-10-02T23:18:27.151452Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/common/timezone.yml
2020-10-02T23:18:27.151462Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/common/timezone.yml
2020-10-02T23:18:27.151572Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/common/timezone.yml
2020-10-02T23:18:27.151597Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/common/timezone.yml
2020-10-02T23:18:27.151619Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/common/users.yml
2020-10-02T23:18:27.151627Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/common/users.yml
2020-10-02T23:18:27.151795Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/common/users.yml
2020-10-02T23:18:27.151858Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/common/users.yml
2020-10-02T23:18:27.151879Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/client/salt.yml
2020-10-02T23:18:27.151887Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/client/salt.yml
2020-10-02T23:18:27.152009Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/client/salt.yml
2020-10-02T23:18:27.152039Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/client/salt.yml
2020-10-02T23:18:27.152060Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/openssh.yml
2020-10-02T23:18:27.152069Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/openssh.yml
2020-10-02T23:18:27.152231Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/openssh.yml
2020-10-02T23:18:27.152331Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/openssh.yml
2020-10-02T23:18:27.152351Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/docker.yml
2020-10-02T23:18:27.152360Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/docker.yml
2020-10-02T23:18:27.152478Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/docker.yml
2020-10-02T23:18:27.152528Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/docker.yml
2020-10-02T23:18:27.152547Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/nodejs.yml
2020-10-02T23:18:27.152554Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/nodejs.yml
2020-10-02T23:18:27.152688Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/nodejs.yml
2020-10-02T23:18:27.152729Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/nodejs.yml
2020-10-02T23:18:27.152748Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/php.yml
2020-10-02T23:18:27.152756Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/php.yml
2020-10-02T23:18:27.152931Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/php.yml
2020-10-02T23:18:27.153030Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/php.yml
2020-10-02T23:18:27.153050Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/redis.yml
2020-10-02T23:18:27.153059Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/redis.yml
2020-10-02T23:18:27.153534Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/redis.yml
2020-10-02T23:18:27.153604Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/redis.yml
2020-10-02T23:18:27.153631Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/custom/php-app-server-common.yml
2020-10-02T23:18:27.153639Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/custom/php-app-server-common.yml
2020-10-02T23:18:27.153759Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/custom/php-app-server-common.yml
2020-10-02T23:18:27.153835Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/custom/php-app-server-common.yml
2020-10-02T23:18:27.153856Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/custom/php-app-server-development.yml
2020-10-02T23:18:27.153864Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/custom/php-app-server-development.yml
2020-10-02T23:18:27.154676Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/custom/php-app-server-development.yml
2020-10-02T23:18:27.154824Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/custom/php-app-server-development.yml
2020-10-02T23:18:27.154931Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/stack2.cfg
2020-10-02T23:18:27.154970Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net/clean.yml
2020-10-02T23:18:27.154978Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net/clean.yml
2020-10-02T23:18:27.155054Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net/clean.yml
2020-10-02T23:18:27.155076Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net/clean.yml
2020-10-02T23:18:27.155086Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net/stacker.yml
2020-10-02T23:18:27.155090Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net/stacker.yml
2020-10-02T23:18:27.155162Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net/stacker.yml
2020-10-02T23:18:27.155181Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net/stacker.yml
2020-10-02T23:18:27.155191Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net/users.yml
2020-10-02T23:18:27.155195Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net/users.yml
2020-10-02T23:18:27.155451Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net/users.yml
2020-10-02T23:18:27.155475Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net/users.yml
2020-10-02T23:18:27.155486Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net/zdump.yml
2020-10-02T23:18:27.155490Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net/zdump.yml
2020-10-02T23:18:27.155896Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net/zdump.yml
2020-10-02T23:18:27.156648Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net/zdump.yml
2020-10-02T23:18:27.156664Z INFO - processor: End of stack build for: server2.example.net (namespace: prod)
```* filter logs by template path
Set the file path to debug by using optional `p=` query parameter :
```sh
curl http://127.0.0.1:3000/server1.example.net?l=trace&p=doc_root/pillars/02-roles/server/openssh.yml
```Or with `--path` flag when using Stacker CLI.
* filter logs by steps
Set the step to debug by using optional `s=` query parameter :
```sh
curl http://127.0.0.1:3000/server1.example.net?l=trace&p=doc_root/pillars/02-roles/server/openssh.yml&s=compile,yaml-load
```Or with `--step` flag when using Stacker CLI.
Valid options for the `step` param are : `compile` | `yaml-load` | `before-merge` | `after-merge` | `final`
## Scaling
The Stacker's web design leads to great possibilities :
* You can move Stacker and the pillar rendering process out of Salt server :
```yml
ext_pillar:
- stacker: http://stacker.example.corp:3000
```* You can run multiple instances of Stacker and call them sequentially :
```yml
ext_pillar:
- stacker: http://127.0.0.1:3000?n=foo
- stacker: http://127.0.0.1:4000?n=bar
- stacker: http://127.0.0.1:5000?n=baz
```With each instance having it's own stack configuration :)
## Deployment
You can use the [provided systemd unit](https://github.com/jbox-web/stacker/blob/master/extra/stacker.service) to manage the Stacker daemon.
## Merging strategies
Stacker implements [merging strategies](https://docs.saltstack.com/en/master/ref/pillar/all/salt.pillar.stack.html#merging-strategies) like PillarStack so you can use them in Stacker too :)
It works the same way and it's [tested](/spec/stacker/value_spec.cr#L45).
## Template syntax
The [template syntax](https://github.com/straight-shoota/crinja/blob/master/TEMPLATE_SYNTAX.md) is almost the same than Jinja2.
[Like PillarStack](https://docs.saltstack.com/en/master/ref/pillar/all/salt.pillar.stack.html#overall-process) you have access to these variables :
* `stack`
* `pillar`
* `minion_id`
* `grains` (instead of `__grains__`)Stacker adds a bunch of filters and functions :
Filters :
* json filter (dump json without character escaping) (like the [Salt one](https://docs.saltstack.com/en/latest/topics/jinja/index.html#tojson))
* traverse filter (like the [Salt one](https://docs.saltstack.com/en/latest/topics/jinja/index.html#traverse))
* unique filter (like the [Salt one](https://docs.saltstack.com/en/latest/topics/jinja/index.html#unique))Functions :
* log function (like the [Salt one](https://docs.saltstack.com/en/latest/topics/jinja/index.html#logs))
* dump function (it dumps objects to YAML in log file)
* array_push function
* merge_dict functionYou can see examples of code in the [documentation](https://jbox-web.github.io/stacker/Stacker/Runtime/Filter/Json.html).
The following filters/tests/functions/tags/operators are supported :
```
filters:
abs()
append(string=)
attr(name=)
batch(linecount=2, fill_with=none)
capitalize()
center(width=80)
date(format=)
default(default_value='', boolean=false)
dictsort(case_sensitive=false, by='key')
escape()
filesizeformat(binary=false)
first()
float(default=0.0)
forceescape()
format()
groupby(attribute=)
indent(width=4, indentfirst=false)
int(default=0, base=10)
join(separator='', attribute=none)
json(indent=none)
last()
length()
list()
lower()
map()
pprint(verbose=false)
prepend(string=)
random()
reject()
rejectattr()
replace(old=, new=, count=none)
reverse()
round(precision=0, method='common', base=10)
safe()
select()
selectattr()
slice(slices=2, fill_with=none)
sort(reverse=false, case_sensitive=false, attribute=none)
string()
striptags()
sum(attribute=none, start=0)
title()
tojson(indent=none)
traverse(attribute=none, default=none)
trim()
truncate(length=255, killwords=false, end='...', leeway=none)
unique()
upper()
urlencode()
urlize(trim_url_limit=none, nofollow=false, target=none, rel=none)
wordcount()
wordwrap(width=79, break_long_words=true, wrapstring=none)
xmlattr(autoescape=true)tests:
callable()
defined()
divisibleby(num=)
equalto(other=)
escaped()
even()
greaterthan(other=0)
in(seq=[])
iterable()
lessthan(other=0)
lower()
mapping()
nil()
none()
number()
odd()
sameas(other=)
sequence()
string()
undefined()
upper()functions:
array_push(array=[], item=none)
cycler()
debug()
dict()
dump(object=none)
joiner(sep=', ')
log(object=none)
merge_dict(hash=none, other=none)
range(start=0, stop=0, step=1)
super()tags:
autoescape~endautoescape
block~endblock
call~endcall
do
elif
else
endautoescape
endblock
endcall
endfilter
endfor
endif
endmacro
endraw
endset
endwith
extends
filter~endfilter
for~endfor
from
if~endif
import
include
macro~endmacro
raw~endraw
set~endset
with~endwithoperators:
operator[!=]
operator[%]
operator[*]
operator[**]
operator[+]
operator[-]
operator[/]
operator[//]
operator[<]
operator[<=]
operator[==]
operator[>]
operator[>=]
operator[and]
operator[not]
operator[or]
operator[~]
```## Development
To compile Stacker you will need [Crystal](https://crystal-lang.org) compiler.
You can easily setup your development environment with [asdf](https://github.com/asdf-vm/asdf) :
```sh
git clone https://github.com/jbox-web/stacker
make setup # Install asdf Crystal plugin and install Crystal compiler
make deps # Install Stacker dependencies
make build # Build Stacker in development mode
make relase # Build Stacker in production mode
```## Extend Stacker
If you need to add filters (or functions) just drop a new class with a few lines of Crystal code in [/src/runtime](https://github.com/jbox-web/stacker/tree/master/src/runtime) and recompile Stacker with `make build` (dev mode) or `make release` (release mode).
Your custom filters (or functions) should be available in Jinja templates. To be sure run `stacker info` and check the Crinja environment info.
Then feel free to submit a PR if you think it will be useful for people.
## Roadmap
* implement [Select Stacker config through grains|pillar|opts matching](https://github.com/saltstack/salt/blob/a670b4ae72ec11f5485c216c54059e14223019b8/salt/pillar/stack.py#L77)
* add documentation about "Stacker - Sample Project"