{"id":28924081,"url":"https://github.com/dsillman2000/yaml-reference","last_synced_at":"2026-03-02T01:01:32.311Z","repository":{"id":295440011,"uuid":"990116397","full_name":"dsillman2000/yaml-reference","owner":"dsillman2000","description":"YAML tagging system for reading and writing modular YAML files with Python","archived":false,"fork":false,"pushed_at":"2025-06-01T15:17:12.000Z","size":45,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-29T06:44:49.911Z","etag":null,"topics":["python","ruamel-yaml","yaml"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dsillman2000.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-25T14:37:49.000Z","updated_at":"2025-05-26T20:17:02.000Z","dependencies_parsed_at":"2025-06-01T16:24:18.631Z","dependency_job_id":null,"html_url":"https://github.com/dsillman2000/yaml-reference","commit_stats":null,"previous_names":["dsillman2000/yaml-reference"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/dsillman2000/yaml-reference","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsillman2000%2Fyaml-reference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsillman2000%2Fyaml-reference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsillman2000%2Fyaml-reference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsillman2000%2Fyaml-reference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dsillman2000","download_url":"https://codeload.github.com/dsillman2000/yaml-reference/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsillman2000%2Fyaml-reference/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28991633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T20:57:35.821Z","status":"ssl_error","status_checked_at":"2026-02-01T20:57:29.580Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","ruamel-yaml","yaml"],"created_at":"2025-06-22T10:03:51.322Z","updated_at":"2026-03-02T01:01:32.134Z","avatar_url":"https://github.com/dsillman2000.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# yaml-reference\n\nUsing `ruamel.yaml`, support cross-file references and YAML composition in YAML files using tags `!reference`, `!reference-all`, `!flatten`, and `!merge`.\n\nInstall the package from PyPI with:\n\n```bash\n# pip\npip install yaml-reference\n# poetry\npoetry add yaml-reference\n# uv\nuv add yaml-reference\n```\n\n## Spec\n![Spec Status](https://img.shields.io/badge/spec%20v0.2.6--3-passing-brightgreen?link=https%3A%2F%2Fgithub.com%2Fdsillman2000%2Fyaml-reference-specs%2Ftree%2Fv0.2.6-3)\n\nThis Python library implements the YAML specification for cross-file references and YAML composition in YAML files using tags `!reference`, `!reference-all`, `!flatten`, and `!merge` as defined in the [yaml-reference-specs project](https://github.com/dsillman2000/yaml-reference-specs).\n\n## Example\n\n```yaml\n# root.yaml\nversion: \"3.1\"\nservices:\n  - !reference\n    path: \"services/website.yaml\"\n\n  - !reference\n    path: \"services/database.yaml\"\n\nnetworkConfigs:\n  !reference-all\n  glob: \"networks/*.yaml\"\n\ntags: !flatten\n  - !reference { path: \"common/tags.yaml\" }\n  - \"web\"\n  - \"service\"\n\nconfig: !merge\n  - !reference { path: \"config/defaults.yaml\" }\n  - !reference { path: \"config/overrides.yaml\" }\n\n```\n\nSupposing there are `services/website.yaml` and `services/database.yaml` files in the same directory as `root.yaml`, and a `networks` directory with YAML files, the above will be expanded to account for the referenced files with the following Python code:\n\n```python\nfrom yaml_reference import load_yaml_with_references\n\ndata = load_yaml_with_references(\"root.yaml\")\nprint(data)\n# {\"networkConfigs\": [{\"network\": \"vpn\",\"version\": \"1.1\"},{\"network\": \"nfs\",\"version\": \"1.0\"}],\"services\": [\"website\",\"database\"],\"version\": \"3.1\"}\n\n# With path restrictions for security\ndata = load_yaml_with_references(\"root.yaml\", allow_paths=[\"/allowed/path\"])\n```\n\nNote that the `load_yaml_with_references` function instantiates a `ruamel.yaml.YAML` loader class (`typ='safe'`) to perform the deserialization of the YAML files, and returns a Python dictionary with the recursively-expanded YAML data.\n\nIf you wish to resolve one \"layer\" of references without recursively exhausting the entire reference graph, the `parse_yaml_with_references` function can be used to obtain the original YAML document's contents with `!reference`/`!reference-all` tags as dedicated objects called `Reference` and `ReferenceAll`.\n\n```python\nfrom yaml_reference import parse_yaml_with_references\n\ndata = parse_yaml_with_references(\"root.yaml\")\nprint(data[\"networkConfigs\"])\n# ReferenceAll(glob=\"networks/*.yaml\", location=\"/path/to/root.yaml\")\n\n# With path restrictions for security\ndata = parse_yaml_with_references(\"root.yaml\", allow_paths=[\"/allowed/path\"])\n```\n\n### The `!merge` Tag\n\nThe `!merge` tag combines multiple YAML mappings (dictionaries) into a single mapping. This is useful for composing configuration from multiple sources or applying overrides. When you use `!merge`, you provide a sequence of mappings that will be merged together, with later mappings overriding keys from earlier ones.\n\n```yaml\n# Example: Merge default and override configurations\nconfig: !merge\n  - {host: \"localhost\", port: 8080, debug: false}\n  - {port: 9000, debug: true}  # Overrides port and debug from the first mapping\n```\n\nWhen loaded with `load_yaml_with_references`, this becomes `{\"host\": \"localhost\", \"port\": 9000, \"debug\": true}`. The `!merge` tag can also be nested and combined with `!reference` and `!flatten` tags for complex YAML composition scenarios.\n\nNote that, if a nested sequence of mappings is provided to `!merge`, the sequence argument will be flattened first, and then the resulting mappings will be merged together. For example:\n\n```yaml\nconfig: !merge\n  - - a: 1\n    - b: 2\n  - c: 3\n  - - [{c: 5, a: 5}]\n```\n\nWill be processed into `{\"config\": {\"a\": 5, \"b\": 2, \"c\": 5}}` because the nested sequence of mappings will be flattened into a single sequence of mappings before merging.\n\n### Using Anchors with `!reference` and `!reference-all`\n\nBoth `!reference` and `!reference-all` tags support an optional `anchor` parameter that allows you to import only a specific anchored section from a file, rather than the entire file contents. This is useful when you want to extract a particular part of a larger YAML document.\n\n```yaml\n# main.yaml\ndatabase_config: !reference\n  path: \"config.yaml\"\n  anchor: db_settings\n\napi_keys: !reference-all\n  glob: \"secrets/*.yaml\"\n  anchor: api_key\n```\n\nIn this example, if `config.yaml` contains multiple anchored sections, only the one labeled with `\u0026db_settings` will be imported. Similarly, `!reference-all` will extract the `\u0026api_key` anchor from each file matching the glob pattern.\n\nHere's a practical example:\n\n```yaml\n# config.yaml\napp_name: MyApplication\ndb_settings: \u0026db_settings\n  host: localhost\n  port: 5432\n  database: myapp\ncache_settings: \u0026cache_settings\n  ttl: 3600\n```\n\n```yaml\n# main.yaml\nconfig: !reference\n  path: \"config.yaml\"\n  anchor: db_settings\n```\n\nWhen loaded with `load_yaml_with_references(\"main.yaml\")`, the result will be:\n\n```python\n{\n  \"config\": {\n    \"host\": \"localhost\",\n    \"port\": 5432,\n    \"database\": \"myapp\"\n  }\n}\n```\n\nNote that the `app_name` and `cache_settings` fields from `config.yaml` are not included in the result because only the anchored section was imported. If the specified anchor is not found in the referenced file, a `ValueError` will be raised.\n\n### VSCode squigglies\n\nTo get rid of red squigglies in VSCode when using the `!reference`, `!reference-all`, `!flatten`, and `!merge` tags, you can add the following to your `settings.json` file:\n\n```json\n    \"yaml.customTags\": [\n        \"!reference mapping\",\n        \"!reference-all mapping\",\n        \"!flatten sequence\",\n        \"!merge sequence\"\n    ]\n```\n\n## CLI interface\n\nThere is a CLI interface for this package which can be used to read a YAML file which contains `!reference` tags and dump its contents as pretty-printed JSON with references expanded. This is useful for generating a single file for deployment or other purposes. Note that the keys of mappings will be sorted alphabetically. This CLI interface is used to test the contract of this package against the `yaml-reference-specs` project.\n\n```bash\n$ yaml-reference-cli -h\n  usage: yaml-reference-cli [-h] [--allow ALLOW_PATHS] input_file\n\n  Compile a YAML file containing !reference tags into a new YAML file with resolved references. Expects a YAML file to be provided via the \"input_file\" argument.\n  Outputs JSON content to stdout.\n\n  positional arguments:\n    input_file           Path to the input YAML file with references to resolve and print as JSON.\n\n  options:\n     -h, --help           show this help message and exit\n     --allow ALLOW_PATHS  Path to allow references from.\n\n$ yaml-reference-cli root.yaml\n  {\n    \"networkConfigs\": [\n      {\n        \"network\": \"vpn\",\n        \"version\": \"1.1\"\n      },\n      {\n        \"network\": \"nfs\",\n        \"version\": \"1.0\"\n      }\n    ],\n    \"services\": [\n      \"website\",\n      \"database\"\n    ],\n    \"tags\": [\n      \"common:aws\",\n      \"common:http\",\n      \"common:security\",\n      \"common:waf\",\n      \"web\",\n      \"service\"\n    ],\n    \"version\": \"3.1\"\n  }\n```\n\nIt's still possible to yield the results as a YAML file using the `yq` CLI tool ([mikefarah/yq](https://github.com/mikefarah/yq)).\n\n```bash\n$ yaml-reference-cli root.yaml | yq -P\nnetworkConfigs:\n  - network: vpn\n    version: 1.1\n  - network: nfs\n    version: 1.0\nservices:\n  - website\n  - database\ntags:\n  - common:aws\n  - common:http\n  - common:security\n  - common:waf\n  - web\n  - service\nversion: 3.1\n# Pipe it to a result file\n$ yaml-reference-cli root.yaml | yq -P \u003e .compiled/root.yaml\n```\n\n## Circular reference protection\n\nAs required by the yaml-reference-specs specification, this package includes circular reference detection to prevent infinite recursion. If a circular reference is detected (e.g., A references B, B references C, C references A), a `ValueError` will be raised with a descriptive error message. This protects against self-references and circular chains in both `!reference` and `!reference-all` tags.\n\n## Security considerations\n\n### Path restriction and `allow_paths`\n\nBy default, `!reference` and `!reference-all` tags can only reference files within the same directory as the source YAML file (or child subdirectories). To allow references to files in other disparate directory trees, you must explicitly specify allowed paths using the `allow_paths` parameter:\n\n```python\nfrom yaml_reference import load_yaml_with_references\n\n# Allow references from specific directories only\ndata = load_yaml_with_references(\n    \"config.yml\",\n    allow_paths=[\"/allowed/path1\", \"/allowed/path2\"]\n)\n```\n\nIn the CLI, use the `--allow` flag:\n\n```bash\nyaml-reference compile input.yml --allow /allowed/path1 --allow /allowed/path2\n```\n\nWhether or not `allow_paths` is specified, the default behavior is to allow references to files in the same directory as the source YAML file (or subdirectories). \"Back-navigating\" out of a the root directory is not allowed (\"..\" local references in a root YAML file). This provides a secure baseline to prevent unsafe access which is not explicitly allowed.\n\n### Absolute path restrictions\n\nReferences using absolute paths (e.g., `/tmp/file.yml`) are explicitly rejected with a `ValueError`. All reference paths must be relative to the source file's directory. If you absolutely must reference an absolute path, relative paths to symlinks can be used. Note that their target directories must be explicitly allowed to avoid permission errors (see the above section about \"Path restriction and `allow_paths`\").\n\n## Acknowledgements\n\nContributor(s):\n\n- David Sillman \u003cdsillman2000@gmail.com\u003e\n  - Personal website: https://www.dsillman.com\n- Ryan Johnson\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdsillman2000%2Fyaml-reference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdsillman2000%2Fyaml-reference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdsillman2000%2Fyaml-reference/lists"}