{"id":20155266,"url":"https://github.com/redhat-cop/agnosticv","last_synced_at":"2025-04-09T22:02:23.198Z","repository":{"id":40385966,"uuid":"221363437","full_name":"redhat-cop/agnosticv","owner":"redhat-cop","description":"Organize and merge YAML files to manage a Catalog","archived":false,"fork":false,"pushed_at":"2024-06-28T09:02:45.000Z","size":238,"stargazers_count":6,"open_issues_count":3,"forks_count":5,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-09T22:02:04.861Z","etag":null,"topics":["babylon","catalog","gpte","yaml","yaml-configuration","yaml-files"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redhat-cop.png","metadata":{"files":{"readme":"readme.adoc","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-13T03:21:14.000Z","updated_at":"2024-06-28T09:02:49.000Z","dependencies_parsed_at":"2023-12-16T00:10:31.789Z","dependency_job_id":"38e77ab7-08bc-490c-b216-f7b5fe527b7c","html_url":"https://github.com/redhat-cop/agnosticv","commit_stats":{"total_commits":51,"total_committers":3,"mean_commits":17.0,"dds":"0.039215686274509776","last_synced_commit":"f752710cba3f773f95863865b8b94b7237650ace"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redhat-cop%2Fagnosticv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redhat-cop%2Fagnosticv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redhat-cop%2Fagnosticv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redhat-cop%2Fagnosticv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redhat-cop","download_url":"https://codeload.github.com/redhat-cop/agnosticv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248119296,"owners_count":21050755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["babylon","catalog","gpte","yaml","yaml-configuration","yaml-files"],"created_at":"2024-11-13T23:31:08.518Z","updated_at":"2025-04-09T22:02:23.159Z","avatar_url":"https://github.com/redhat-cop.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":":toc2:\n\n= Agnostic Vars\n\n== Description\n\nAgnosticv is a generic way of organizing and merging YAML files to manage a Catalog.\n\nCurrently the project is composed of:\n\n. link:cli[CLI] `agnosticv`, a generic implementation.\n. link:https://github.com/redhat-gpte-devopsautomation/agnosticv-operator[agnosticv-operator] an OpenShift operator that uses the CLI and generates OpenShift Custom Resources for link:https://github.com/redhat-cop/babylon[Babylon] based on the content of the agnosticv repositories. The operator is more opinionated than the CLI as it expects a specific file structure in the agnosticV repository. We use it to manage the catalog of our demo portal RHPDS.\n\n=== Features ===\n\n- Merge YAML files automatically following a particular convention, see below.\n- Support for includes.\n- Inject information about last change (commit, author, date) when merging the vars of a catalog item\n- When listing catalog items, filter catalog items using JMESPath expressions (`--has` flag)\n- Configure custom policies or behavior, per repository:\n** Configure merge strategies\n** Validate variables using  link:https://www.openapis.org/[OpenAPI v3] schemas. Just place schemas in a `.schemas` directory at the top-level of the agnosticv repository.\n** Configuration file `/.agnosticv.yaml`\n\n=== Example of agnosticv repositories\n\n. public: see in link:cli/fixtures[cli/fixtures] for an example of structure of a catalog\n. private link:https://github.com/rhpds/agnosticv[GPTE agnosticv repository]\n. private link:https://github.com/redhat-gpte/gpte_summit_2021[Red Hat Summit 2021 repository]\n. private link:https://github.com/redhat-gpe/2020Summit-AgnosticV[2020Summit-AgnosticV repository]\n. private link:https://github.com/redhat-gpe/RHTR_2020_agnosticV/[RHTR_2020_agnosticV repository]\n\n== `agnosticv` CLI\n\n=== Install\n\nDownload the binary for your architecture (linux, mac, windows, ..) from link:https://github.com/redhat-cop/agnosticv/releases/[release page].\n\n=== Usage\n\nThe CLI `agnosticv` has the following capabilities:\n\n- list all the catalog items present in a directory\n- merge and print the vars of an item of the catalog\n\n\n.Usage\n----\nUsage of agnosticv:\n  -debug\n    \tDebug mode\n  -git\n    \tPerform git operations to gather and inject information into the merged vars like 'last_update'. Git operations are slow so this option is automatically disabled for listing. (default true)\n  -has value\n    \tUse with --list only. Filter catalog items using a JMESPath expression.\n    \tCan be used several times (act like AND).\n\n    \tExamples:\n    \t--has __meta__.catalog\n    \t--has \"env_type == 'ocp-clientvm'\"\n    \t--has \"to_string(worker_instance_count) == '2'\"\n\n  -list\n    \tList all the catalog items present in current directory.\n  -merge string\n    \tMerge and print variables of a catalog item.\n  -or-related value\n    \tUse with --list only. Same as --related except it appends the related files to the list instead of reducing it.\n\n    \tExample:\n    \t--list --related dir/common.yaml --or-related includes/foo.yaml\n    \t   List all catalog items under dir/ and also all catalog items that include includes/foo.yaml\n\n    \tCan be used several times (act like OR).\n  -related value\n    \tUse with --list only. Filter output and display only related catalog items.\n    \tA catalog item is related to FILE if:\n    \t- it includes FILE as a common file\n    \t- it includes FILE via #include\n    \t- FILE is description.adoc or description.html\n\n    \tExample:\n    \t--list --related dir/common.yaml --related includes/foo.yaml\n    \t   List all catalog items under dir/ that also include includes/foo.yaml\n\n    \tCan be used several times (act like AND).\n  -root string\n    \tThe top directory of the agnosticv files. Files outside of this directory will not be merged.\n    \tBy default, it's empty, and the scope of the git repository is used, so you should not\n    \tneed this parameter unless your files are not in a git repository, or if you want to use a subdir. Use -root flag with -merge.\n  -validate\n    \tValidate variables against schemas present in .schemas directory. (default true)\n  -version\n    \tPrint build version.\n----\n\n.List catalog items in local directory\n--------------\ncli $ ./agnosticv --list\nfixtures/gpte/OCP_CLIENTVM/dev.yaml\nfixtures/gpte/OCP_CLIENTVM/prod.yaml\nfixtures/test/BABYLON_EMPTY_CONFIG/dev.yaml\nfixtures/test/BABYLON_EMPTY_CONFIG/prod.yaml\nfixtures/test/BABYLON_EMPTY_CONFIG/test.yaml\nfixtures/test/BABYLON_EMPTY_CONFIG_AWS/dev.yaml\nfixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml\nfixtures/test/BABYLON_EMPTY_CONFIG_AWS/test.yaml\nfixtures/test/BABYLON_EMPTY_CONFIG_OSP/dev.yaml\nfixtures/test/BABYLON_EMPTY_CONFIG_OSP/prod.yaml\nfixtures/test/BABYLON_EMPTY_CONFIG_OSP/test.yaml\n--------------\n\n.Merge and print the vars of a catalog item\n--------------\ncli $ ./agnosticv --merge fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml\n---\n# MERGED: \u003c1\u003e\n# fixtures/common.yaml\n# fixtures/test/account.yaml\n# fixtures/test/account.meta.yaml\n# fixtures/test/BABYLON_EMPTY_CONFIG_AWS/common.yaml\n# fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml\n__meta__:\n  deployer:\n    scm_ref: test-empty-config-test-0.5\n    scm_type: git\n    scm_url: https://github.com/redhat-cop/agnosticd.git\n    type: agnosticd\n  secrets:\n  - from-top-common.yml\n  - name: gpte\n  [...] output omitted\n--------------\n\u003c1\u003e Merge list: gives information about how files were merged to produce the final set of variables\n\nNOTE: `common.yaml` files are always included when merging. `agnosticv` searches for those files as long as it is in the same git repository. If the files are not versioned with git, it is possible to \"chroot\" the search using the `--root` parameter.\n\n== Build\n\n----\ncd cli\ngo get\ngo build -o agnosticv\n----\n\n== File naming convention\n\n=== Nomenclature ===\n\n* _Catalog item_, or _Leaf_: a file considered a catalog item. It appears when listing an agnosticv repository with `agnosticv --list`. To print the content of a catalog item, you run `agnosticv --merge PATH`. That merges common files, meta files, included files, and finally, the leaf to produce the catalog item.\n* _Common file_:  a file that is automatically included in the merge list. Ex: `common.yaml`\n* _Related file_: a file that is related to a catalog item. Ex: `description.adoc`\n* _Included file_: any file that is included in the merge list using the `#include PATH` feature\n* _Meta file_: Any file ending with `.meta.yml` or `.meta.yaml`. It contains the value of the `__meta__` dictionary.\n\n=== Common files ===\n\nSome files are automatically included in the merge list to produce the final catalog item. The following names are valid common YAML files:\n\n- `common.yaml`\n- `common.yml`\n- `account.yaml`\n- `account.yml`\n\nThey can be placed at any level in the agnosticv repository.\n\n=== Includes ===\n\nFiles included in the merge list using the `#include PATH` feature. See the dedicated section below.\n\nUsually, we place them in an `includes/` directory at the top of the agnosticv repository.\n\n==== `#include` merge feature ====\n\n* syntax:  `#include FILENAME`\n* where: In any file\n* identation is ignored\n+\n[source,yaml]\n----\n#include /file.yaml\n----\n+\nAnd:\n+\n[source,yaml]\n----\n    #include /file.yaml\n----\n+\nare the same.\n* `FILENAME` is added to the merge list right **before** current file regardless of the position of `#include` in the file. In other words, current file vars take precedence over included files vars.\n** That's also why you should put all your includes at the top of the file.\n* if `FILENAME` starts with `/` then path is absolute to the AgnosticV repo.\n** if not, the path is relative to the current file\n\n===== Example =====\n\n[source,yaml]\n.`gpte/OCP4_WORKSHOP/prod.yaml`\n----\n#include /includes/file1.yaml\n#include /includes/file2.yaml\n\ncloud_provider: ec2\nkey_name: opentlc_admin_backdoor\nrepo_method: file\n\nsubdomain_base_suffix: .example.opentlc.com\nHostedZoneId: Z3IHLWJZOU9SRT\n\nagnosticv_meta:\n  deploy_with: babylon\n----\n\n[source,yaml]\n.`includes/file1.yaml` with vars at the \"agnosticd\" level\n----\nvar1: value1\nvar2: value2\n----\n\n[source,yaml]\n.`includes/file2.yaml` with meta vars\n----\nagnosticv_meta:\n  secrets:\n    - somesecret\n\n__meta__:\n  secrets:\n    - name: somesecret\n      namespace: gpte\n----\n\nThe merge list will be:\n\n. `/common.yaml`\n. `/gpte/account.yaml`\n. `/gpte/OCP4_WORKSHOP/common.yaml`\n. `/includes/file1.yaml`\n. `/includes/file2.yaml`\n. `/gpte/OCP4_WORKSHOP/prod.yaml`\n\n=== Meta files ===\n\nFor any common file, leaf file, or included file, you can create an associated meta file to be automatically included.\nThe meta file can contain the value of the  `\\\\__meta__` dictionary. It is convenient to separate that special dictionary from the other variables.\n\nFor example, the following files are valid meta files:\n\n* `common.meta.yml` meta file for `common.yml`\n* `account.meta.yml` meta file for `account.yml`\n* `dev.meta.yml` meta file for `dev.yml`\n\nWARNING: you can only put the content of the `\\\\__meta__` variable in a meta file.\n\n.example1: content of meta file\n[source,yaml]\n----\n__meta__:\n  secrets:\n    - name: mysecret\n----\n\n.example2: with the content of `\\\\__meta__` directly, same as example1\n[source,yaml]\n----\nsecrets:\n  - name: mysecret\n----\n\n.example3: *wrong* meta file - This will fail\n[source,yaml]\n----\n__meta__:\n  secrets:\n    - name: mysecret\n\nanother_var: value \u003c1\u003e\n----\n\u003c1\u003e other variables than `\\\\__meta__` are not allowed\n\n=== Related Files ===\n\n* Related files are helpful in estimating when a catalog item was last changed. All catalog items related to that file are considered touched if a related file is touched.\n* To print all catalog items related to a file, run `agnosticv --list --related-to RELATED_FILE`\n* All included files of a catalog item are automatically considered related files.\n* All common files of a catalog item are automatically considered related files.\n* It is possible to add custom related files using the config file; see section below.\n* It is possible to load related files into the merged vars using the config file; see section below.\n\n==== Configuration ====\n\nYou can add custom related files using the `related_files` configuration option in a `.agnosticv.yaml` configuration file. It contains a list of filenames that if present in the same directory of a catalog item, will be considered as related to that catalog item.\n\nYou can decide to load the content of the relative file into a path by specifying the path as a link:https://www.rfc-editor.org/rfc/rfc6901[JSON Pointer] format. The destination path will be a dictionary and the content will be loaded into the 'content_key'.\n\n\nHere is an example of configuration:\n\n.`/.agnosticv.yaml`\n[source,yaml]\n----\n# For any catalog item, consider those files in same directory as related files:\nrelated_files_v2:\n  - file: service-ready-message-template.html.j2\n    load_into: /__meta__/catalog/message_templates/service_ready\n    content_key: template\n    set:\n      format: jinja2\n      output_format: html\n  - file: description.txt\n----\n\nWhen merging, that will produce:\n\n[source,yaml]\n----\n__meta__:\n  catalog:\n    message_templates:\n      service_ready:\n        format: jinja2\n        output_format: html\n        template: |\n          \u003c\u003ccontent of the file\u003e\u003e\n----\n\n=== Leaf files ===\n\nThe \"leaf\" files, or catalog items, are just the rest of the YAML files, having one of these extensions:\n\n- yml\n- yaml\n\nYou can list all catalog items in a directory by using `--list` parameter: `agnosticv --list`\n\n==== Files ignored ====\n\n* Any dotfile is ignored. Ex: `.git`\n* Any directory named `includes` is reserved to includes. The files in those directories are never considered as catalog items.\n* Any file containing:\n+\n----\n#agnosticv catalog_item false\n----\n+\nis ignored. It is not considered a catalog item.\n\n\n\n== Merging strategies\n\n=== Default ===\n\nWhen it comes to merging variables, there are different possible strategies.\n\nThe default is the following:\n\n|========================\n| What | Dictionaries | Lists | Strings / Numbers\n\n|`\\\\__meta__` and `agnosticv_meta` dictionaries\n| **merge**\n| **append**\n| **replace**\n\n| Rest of the vars\n\nSame behavior as if you were using ansible{nbsp}extra{nbsp}vars\n\n| **replace**\n| **replace**\n| **replace**\n|========================\n\n=== Custom merge strategies ===\n\nIt is possible to extend agnosticV and define the merge strategy to use on what variable or part of a dictionary variable.\n\nTo do that, you can define the custom merge strategies in any schema in the `.schemas` directory. Just use the **`x-merge`** keyword at the beginning of the schema. `x-merge` is a list of strategies. Each strategy defines a `path` and a `strategy` name to apply to that path. `path` is a link:https://www.rfc-editor.org/rfc/rfc6901[JSON Pointer]. For the list of strategies, see section below.\n\n[source,yaml]\n.`.schema/schema.yaml`  example of `x-merge` custom strategy\n----\ntype: object\nx-merge:\n  - path: /__meta__/access_control # \u003c1\u003e\n    strategy: overwrite # \u003c2\u003e\nproperties:\n----\n\u003c1\u003e The path of the variable or key of dictionnary, as a link:https://www.rfc-editor.org/rfc/rfc6901[JSON Pointer], to apply the custom strategy against.\n\u003c2\u003e When merging, agnosticv will overwrite the content of `\\\\__meta__.access_control` instead of merging it.\n\nFor example, with the schema above and following merge list:\n\n----\n# MERGED:\n# fixtures/common.yaml\n# fixtures/test/account.yaml\n# fixtures/test/BABYLON_EMPTY_CONFIG/common.yaml\n# fixtures/test/BABYLON_EMPTY_CONFIG/prod.yaml\n----\n\nThe value of `\\\\__meta__.access_control` from `prod.yaml` will take precedence and overwrite.\n\nHere are the available custom merge strategies:\n\n|========================\n| Strategy | Can be applied to | Dictionaries | Lists | Strings / Numbers\n\n| `overwrite`\n| List or Dict\n| **replace**\n| **replace**\n| **replace**\n\n| `merge`\n| List or Dict\n| **Merge**\n| **Append**\n| **replace**\n\n| `merge-no-append`\n| Dict\n| **Merge**\n| **replace**\n| **replace**\n\n| `strategic-merge`\n| List or Dict\n| **Strategic Merge** footnote:strategic-merge[Merge similar to kubernetes link:https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#notes-on-the-strategic-merge-patch[stategic merge patch]. The patch merge-key for list is `name`.]\n| **Strategic Merge** footnote:strategic-merge[]\n| **replace**\n|========================\n\n\n== See also\n\n- link:https://github.com/redhat-cop/agnosticd[AgnosticD] deployer\n- link:https://github.com/redhat-cop/babylon[Babylon] project\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredhat-cop%2Fagnosticv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredhat-cop%2Fagnosticv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredhat-cop%2Fagnosticv/lists"}