Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/novotnyr/aedile

Consul configuration K/V importer and exporter in the spirit of `git2consul`
https://github.com/novotnyr/aedile

consul

Last synced: 2 months ago
JSON representation

Consul configuration K/V importer and exporter in the spirit of `git2consul`

Awesome Lists containing this project

README 

        
About

=====

Aedile as a Consul configuration K/V importer and exporter in the spirit of `git2consul`



Running the Aedile

==================



Download the `aedile.jar`.



It is a standalone JAR, executable via `java -jar aedile.jar`



There are following functionalities:



*   import to Consul K/V from a specific directory with a set of `.properties` files

*   import to Consul K/V from Git

*   import to Consul K/V from a set of datacenter-specific directories

*   export from Consul K/V to the directory tree



Importing from the filesystem

=============================



To import the files from the filesystem directory, run Aedile:



    java -jar aedile.jar import /etc/consul/config/ft-prod



The Consul configuration for the import is guided by Environment Variables.



By default, all subdirectories are imported as well, recursively.



For example, `/etc/consul/config/ft-prod/mt` and `/etc/consul/config/ft-prod/meta`

subdirectories will be imported as `mt` and `meta` subdirectories respectively,

under `ft-prod` directory in Consul K/V.



Renaming property files (`--remap-config-name`)

-----------------------------------------------



Property filenames can be remapped:



    --remap-config-name application-local=application,local



This will remap `application-local.properties` to `application,local` directory in Consul K/V.



Disabling recursion

-------------------

To disable recursive import, use the `--no-recurse` argument, such as:



    java -jar aedile.jar import --no-recurse /etc/consul/config/ft-prod



Exporting from the Consul K/V to the filesystem

===============================================



To export the whole Consul K/V configuration into the filesystem directory

hierarchy, run Aedile exporter:



    java -jar aedile.jar export /tmp



This will import the root folder and all recursive subfolders to the `/tmp`

folder in the filesystem.



All Consul K/V folders will be mapped to the filesystem directories.

The bottommost folders will be mapped to the `.properties` files, containing

keys and values separated by `=`.



Importing large datacenter structure

====================================

Imports multiple folders that corresponds to datacenter configurations.



To import, run Aedile:



    java -jar aedile.jar import-dc /etc/consul/config/datacenters



Imagine the following folder hierarchy:



    ./datacenters

        |- dc1

        |  |- config

        |     |- unittest

        |     |  |- application

        |     |     |- code.properties

        |     |     |- demo.properties

        |     |- unittest2

        |        |- application

        |           |- demo2.properties

        |- dc2

           |- config

              |- prod

                  |- application.properties

                  |- impl.properties



Importing the `datacenters` directory creates the following structure in Consul

K/V. The `dc1` and `dc2` directories will map to the *dc*

parameter in Consul K/V REST API requests. The first directory `dc1` will map to

the `dc=dc1` query parameter, and the second directory `dc2` will map to the

`dc=dc2` query param.



The datacenter `dc1` will contain the structure that corresponds to the filesystem

structure.



    |- config

    |   |- unittest

    |   |  |- application

    |   |     |- code

    |   |     |   |- key=value from code.properties

    |   |     |   |- another key=value from code.properties

    |   |     |- demo

    |   |         |- key=value from demo.properties

    |   |         |- another key=value demo code.properties

    |   |- unittest2

           |- application

              |- demo2

                  |- key=value from demo2.properties

                  |- another key=value from demo2.properties



### Ignoring Folders

The `CONSUL_IMPORT_EXCLUDE` variable contains a datacenter/folder specification

that shall be ignored upon import.



An example:



    dc1#config/unittest/application:dc1#config/unittest/application2



*   The datacenter `dc1` shall ignore the `config/unittest/application` folder

*   The datacenter `dc1` shall ignore the `config/unittest2/application` folder



#### Wildcard supports



Specification may contain Ant-based wildcards:



    dc1#config/**/*



This ignores any subfolder (recursively) in the `config` folder for datacenter `dc1`.



Git import

==========



To import the files from the Git directory, run Aedile:



    java -jar aedile.jar git config.json



Git config

----------



Git config is configured via `git2consul`-like configuration file.



    {

      "version": "1.0",

      "local_store" : "./target/gitrepo",

      "repos" : [{

        "name" : "aedile",

        "url" : "https://github.com/novotnyr/aedile.git",

        "source_root": "src/test/resources/datacenters/dc1/config/unittest/application",

        "mountpoint": "config"

      }]

    }



*   `version`: always 1.0

*   `local_store`: the local temporary file that will hold the cloned Git repo

*   `name`: name of the Git repo. Will be appended to the Consul K/V prefix.  

    When building the key name, git2consul will concatenate mountpoint, repo

    name, and the path of the file in your git repo.

*   `url`: URL of the Git repo

*   `source_root`: A `source_root` is a repo-level option instructing Aedile to

    navigate to a subdirectory in the git repo before mapping files to K/Vs. By

    default, Aedile mirrors the entire repo into Consul KVs.  

    If you have a repo configured with the `source_root`

    `src/test/resources/datacenters/dc1/`, the file

    `src/test/resources/datacenters/dc1/config/unittest/application/config.properties` would be

    mapped to the KV as `config/unittest/application/config.properties`.



Environment Variables

=====================



*   `CONSUL_ENDPOINT` represents a Consul agent host. It is `localhost` by default.

*   `CONSUL_PORT` represents a Consul agent port. It is 8500 by default.

*   `TOKEN` corresponds to the ACL token as specified by Consul ACL protection. It is empty by default.

*   `CONSUL_HTTP_USER` represents a user that is sent in the HTTP Basic Authentication with HTTP requests to Consul API.

*   `CONSUL_HTTP_PASSWORD` represents a password that is sent in the HTTP Basic authentication with HTTP requests.

*   `CONSUL_KEY_PREFIX` represents a default prefix for the configuration values.

*   `CONSUL_DATACENTER` represents a datacenter that is targeted with requests.

*   `CONSUL_IMPORT_EXCLUDE` contains a specification of ignored datacenters and subfolders