Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/redhat-cop/agnosticv

Organize and merge YAML files to manage a Catalog
https://github.com/redhat-cop/agnosticv

babylon catalog gpte yaml yaml-configuration yaml-files

Last synced: 6 months ago
JSON representation

Organize and merge YAML files to manage a Catalog

Awesome Lists containing this project

README 

          
:toc2:



= Agnostic Vars



== Description



Agnosticv is a generic way of organizing and merging YAML files to manage a Catalog.



Currently the project is composed of:



. link:cli[CLI] `agnosticv`, a generic implementation.

. link:https://github.com/redhat-gpte-devopsautomation/agnosticv-operator[agnosticv-operator] an OpenShift operator that uses the CLI and generates OpenShift Custom Resources for link:https://github.com/redhat-cop/babylon[Babylon] based on the content of the agnosticv repositories. The operator is more opinionated than the CLI as it expects a specific file structure in the agnosticV repository. We use it to manage the catalog of our demo portal RHPDS.



=== Features ===



- Merge YAML files automatically following a particular convention, see below.

- Support for includes.

- Inject information about last change (commit, author, date) when merging the vars of a catalog item

- When listing catalog items, filter catalog items using JMESPath expressions (`--has` flag)

- Configure custom policies or behavior, per repository:

** Configure merge strategies

** Validate variables using  link:https://www.openapis.org/[OpenAPI v3] schemas. Just place schemas in a `.schemas` directory at the top-level of the agnosticv repository.

** Configuration file `/.agnosticv.yaml`



=== Example of agnosticv repositories



. public: see in link:cli/fixtures[cli/fixtures] for an example of structure of a catalog

. private link:https://github.com/rhpds/agnosticv[GPTE agnosticv repository]

. private link:https://github.com/redhat-gpte/gpte_summit_2021[Red Hat Summit 2021 repository]

. private link:https://github.com/redhat-gpe/2020Summit-AgnosticV[2020Summit-AgnosticV repository]

. private link:https://github.com/redhat-gpe/RHTR_2020_agnosticV/[RHTR_2020_agnosticV repository]



== `agnosticv` CLI



=== Install



Download the binary for your architecture (linux, mac, windows, ..) from link:https://github.com/redhat-cop/agnosticv/releases/[release page].



=== Usage



The CLI `agnosticv` has the following capabilities:



- list all the catalog items present in a directory

- merge and print the vars of an item of the catalog



.Usage

----

Usage of agnosticv:

  -debug

    	Debug mode

  -git

    	Perform git operations to gather and inject information into the merged vars like 'last_update'. Git operations are slow so this option is automatically disabled for listing. (default true)

  -has value

    	Use with --list only. Filter catalog items using a JMESPath expression.

    	Can be used several times (act like AND).



    	Examples:

    	--has __meta__.catalog

    	--has "env_type == 'ocp-clientvm'"

    	--has "to_string(worker_instance_count) == '2'"



  -list

    	List all the catalog items present in current directory.

  -merge string

    	Merge and print variables of a catalog item.

  -or-related value

    	Use with --list only. Same as --related except it appends the related files to the list instead of reducing it.



    	Example:

    	--list --related dir/common.yaml --or-related includes/foo.yaml

    	   List all catalog items under dir/ and also all catalog items that include includes/foo.yaml



    	Can be used several times (act like OR).

  -related value

    	Use with --list only. Filter output and display only related catalog items.

    	A catalog item is related to FILE if:

    	- it includes FILE as a common file

    	- it includes FILE via #include

    	- FILE is description.adoc or description.html



    	Example:

    	--list --related dir/common.yaml --related includes/foo.yaml

    	   List all catalog items under dir/ that also include includes/foo.yaml



    	Can be used several times (act like AND).

  -root string

    	The top directory of the agnosticv files. Files outside of this directory will not be merged.

    	By default, it's empty, and the scope of the git repository is used, so you should not

    	need this parameter unless your files are not in a git repository, or if you want to use a subdir. Use -root flag with -merge.

  -validate

    	Validate variables against schemas present in .schemas directory. (default true)

  -version

    	Print build version.

----



.List catalog items in local directory

--------------

cli $ ./agnosticv --list

fixtures/gpte/OCP_CLIENTVM/dev.yaml

fixtures/gpte/OCP_CLIENTVM/prod.yaml

fixtures/test/BABYLON_EMPTY_CONFIG/dev.yaml

fixtures/test/BABYLON_EMPTY_CONFIG/prod.yaml

fixtures/test/BABYLON_EMPTY_CONFIG/test.yaml

fixtures/test/BABYLON_EMPTY_CONFIG_AWS/dev.yaml

fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml

fixtures/test/BABYLON_EMPTY_CONFIG_AWS/test.yaml

fixtures/test/BABYLON_EMPTY_CONFIG_OSP/dev.yaml

fixtures/test/BABYLON_EMPTY_CONFIG_OSP/prod.yaml

fixtures/test/BABYLON_EMPTY_CONFIG_OSP/test.yaml

--------------



.Merge and print the vars of a catalog item

--------------

cli $ ./agnosticv --merge fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml

---

# MERGED: <1>

# fixtures/common.yaml

# fixtures/test/account.yaml

# fixtures/test/account.meta.yaml

# fixtures/test/BABYLON_EMPTY_CONFIG_AWS/common.yaml

# fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml

__meta__:

  deployer:

    scm_ref: test-empty-config-test-0.5

    scm_type: git

    scm_url: https://github.com/redhat-cop/agnosticd.git

    type: agnosticd

  secrets:

  - from-top-common.yml

  - name: gpte

  [...] output omitted

--------------

<1> Merge list: gives information about how files were merged to produce the final set of variables



NOTE: `common.yaml` files are always included when merging. `agnosticv` searches for those files as long as it is in the same git repository. If the files are not versioned with git, it is possible to "chroot" the search using the `--root` parameter.



== Build



----

cd cli

go get

go build -o agnosticv

----



== File naming convention



=== Nomenclature ===



* _Catalog item_, or _Leaf_: a file considered a catalog item. It appears when listing an agnosticv repository with `agnosticv --list`. To print the content of a catalog item, you run `agnosticv --merge PATH`. That merges common files, meta files, included files, and finally, the leaf to produce the catalog item.

* _Common file_:  a file that is automatically included in the merge list. Ex: `common.yaml`

* _Related file_: a file that is related to a catalog item. Ex: `description.adoc`

* _Included file_: any file that is included in the merge list using the `#include PATH` feature

* _Meta file_: Any file ending with `.meta.yml` or `.meta.yaml`. It contains the value of the `__meta__` dictionary.



=== Common files ===



Some files are automatically included in the merge list to produce the final catalog item. The following names are valid common YAML files:



- `common.yaml`

- `common.yml`

- `account.yaml`

- `account.yml`



They can be placed at any level in the agnosticv repository.



=== Includes ===



Files included in the merge list using the `#include PATH` feature. See the dedicated section below.



Usually, we place them in an `includes/` directory at the top of the agnosticv repository.



==== `#include` merge feature ====



* syntax:  `#include FILENAME`

* where: In any file

* identation is ignored

+

[source,yaml]

----

#include /file.yaml

----

+

And:

+

[source,yaml]

----

    #include /file.yaml

----

+

are the same.

* `FILENAME` is added to the merge list right **before** current file regardless of the position of `#include` in the file. In other words, current file vars take precedence over included files vars.

** That's also why you should put all your includes at the top of the file.

* if `FILENAME` starts with `/` then path is absolute to the AgnosticV repo.

** if not, the path is relative to the current file



===== Example =====



[source,yaml]

.`gpte/OCP4_WORKSHOP/prod.yaml`

----

#include /includes/file1.yaml

#include /includes/file2.yaml



cloud_provider: ec2

key_name: opentlc_admin_backdoor

repo_method: file



subdomain_base_suffix: .example.opentlc.com

HostedZoneId: Z3IHLWJZOU9SRT



agnosticv_meta:

  deploy_with: babylon

----



[source,yaml]

.`includes/file1.yaml` with vars at the "agnosticd" level

----

var1: value1

var2: value2

----



[source,yaml]

.`includes/file2.yaml` with meta vars

----

agnosticv_meta:

  secrets:

    - somesecret



__meta__:

  secrets:

    - name: somesecret

      namespace: gpte

----



The merge list will be:



. `/common.yaml`

. `/gpte/account.yaml`

. `/gpte/OCP4_WORKSHOP/common.yaml`

. `/includes/file1.yaml`

. `/includes/file2.yaml`

. `/gpte/OCP4_WORKSHOP/prod.yaml`



=== Meta files ===



For any common file, leaf file, or included file, you can create an associated meta file to be automatically included.

The meta file can contain the value of the  `\\__meta__` dictionary. It is convenient to separate that special dictionary from the other variables.



For example, the following files are valid meta files:



* `common.meta.yml` meta file for `common.yml`

* `account.meta.yml` meta file for `account.yml`

* `dev.meta.yml` meta file for `dev.yml`



WARNING: you can only put the content of the `\\__meta__` variable in a meta file.



.example1: content of meta file

[source,yaml]

----

__meta__:

  secrets:

    - name: mysecret

----



.example2: with the content of `\\__meta__` directly, same as example1

[source,yaml]

----

secrets:

  - name: mysecret

----



.example3: *wrong* meta file - This will fail

[source,yaml]

----

__meta__:

  secrets:

    - name: mysecret



another_var: value <1>

----

<1> other variables than `\\__meta__` are not allowed



=== Related Files ===



* Related files are helpful in estimating when a catalog item was last changed. All catalog items related to that file are considered touched if a related file is touched.

* To print all catalog items related to a file, run `agnosticv --list --related-to RELATED_FILE`

* All included files of a catalog item are automatically considered related files.

* All common files of a catalog item are automatically considered related files.

* It is possible to add custom related files using the config file; see section below.

* It is possible to load related files into the merged vars using the config file; see section below.



==== Configuration ====



You can add custom related files using the `related_files` configuration option in a `.agnosticv.yaml` configuration file. It contains a list of filenames that if present in the same directory of a catalog item, will be considered as related to that catalog item.



You can decide to load the content of the relative file into a path by specifying the path as a link:https://www.rfc-editor.org/rfc/rfc6901[JSON Pointer] format. The destination path will be a dictionary and the content will be loaded into the 'content_key'.



Here is an example of configuration:



.`/.agnosticv.yaml`

[source,yaml]

----

# For any catalog item, consider those files in same directory as related files:

related_files_v2:

  - file: service-ready-message-template.html.j2

    load_into: /__meta__/catalog/message_templates/service_ready

    content_key: template

    set:

      format: jinja2

      output_format: html

  - file: description.txt

----



When merging, that will produce:



[source,yaml]

----

__meta__:

  catalog:

    message_templates:

      service_ready:

        format: jinja2

        output_format: html

        template: |

          <>

----



=== Leaf files ===



The "leaf" files, or catalog items, are just the rest of the YAML files, having one of these extensions:



- yml

- yaml



You can list all catalog items in a directory by using `--list` parameter: `agnosticv --list`



==== Files ignored ====



* Any dotfile is ignored. Ex: `.git`

* Any directory named `includes` is reserved to includes. The files in those directories are never considered as catalog items.

* Any file containing:

+

----

#agnosticv catalog_item false

----

+

is ignored. It is not considered a catalog item.



== Merging strategies



=== Default ===



When it comes to merging variables, there are different possible strategies.



The default is the following:



|========================

| What | Dictionaries | Lists | Strings / Numbers



|`\\__meta__` and `agnosticv_meta` dictionaries

| **merge**

| **append**

| **replace**



| Rest of the vars



Same behavior as if you were using ansible{nbsp}extra{nbsp}vars



| **replace**

| **replace**

| **replace**

|========================



=== Custom merge strategies ===



It is possible to extend agnosticV and define the merge strategy to use on what variable or part of a dictionary variable.



To do that, you can define the custom merge strategies in any schema in the `.schemas` directory. Just use the **`x-merge`** keyword at the beginning of the schema. `x-merge` is a list of strategies. Each strategy defines a `path` and a `strategy` name to apply to that path. `path` is a link:https://www.rfc-editor.org/rfc/rfc6901[JSON Pointer]. For the list of strategies, see section below.



[source,yaml]

.`.schema/schema.yaml`  example of `x-merge` custom strategy

----

type: object

x-merge:

  - path: /__meta__/access_control # <1>

    strategy: overwrite # <2>

properties:

----

<1> The path of the variable or key of dictionnary, as a link:https://www.rfc-editor.org/rfc/rfc6901[JSON Pointer], to apply the custom strategy against.

<2> When merging, agnosticv will overwrite the content of `\\__meta__.access_control` instead of merging it.



For example, with the schema above and following merge list:



----

# MERGED:

# fixtures/common.yaml

# fixtures/test/account.yaml

# fixtures/test/BABYLON_EMPTY_CONFIG/common.yaml

# fixtures/test/BABYLON_EMPTY_CONFIG/prod.yaml

----



The value of `\\__meta__.access_control` from `prod.yaml` will take precedence and overwrite.



Here are the available custom merge strategies:



|========================

| Strategy | Can be applied to | Dictionaries | Lists | Strings / Numbers



| `overwrite`

| List or Dict

| **replace**

| **replace**

| **replace**



| `merge`

| List or Dict

| **Merge**

| **Append**

| **replace**



| `merge-no-append`

| Dict

| **Merge**

| **replace**

| **replace**



| `strategic-merge`

| List or Dict

| **Strategic Merge** footnote:strategic-merge[Merge similar to kubernetes link:https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#notes-on-the-strategic-merge-patch[stategic merge patch]. The patch merge-key for list is `name`.]

| **Strategic Merge** footnote:strategic-merge[]

| **replace**

|========================



== See also



- link:https://github.com/redhat-cop/agnosticd[AgnosticD] deployer

- link:https://github.com/redhat-cop/babylon[Babylon] project