{"id":15022144,"url":"https://github.com/tanbro/pyyaml-include","last_synced_at":"2025-10-20T04:12:21.376Z","repository":{"id":54277650,"uuid":"136558067","full_name":"tanbro/pyyaml-include","owner":"tanbro","description":"yaml include other yaml","archived":false,"fork":false,"pushed_at":"2025-10-01T06:54:50.000Z","size":357,"stargazers_count":87,"open_issues_count":1,"forks_count":23,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-10-01T08:36:56.129Z","etag":null,"topics":["pyyaml","yaml"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/pyyaml-include/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tanbro.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-06-08T02:53:15.000Z","updated_at":"2025-10-01T06:54:42.000Z","dependencies_parsed_at":"2023-12-25T09:44:41.870Z","dependency_job_id":"edd2f570-d827-4496-afa8-c8f7af8406e5","html_url":"https://github.com/tanbro/pyyaml-include","commit_stats":{"total_commits":336,"total_committers":11,"mean_commits":"30.545454545454547","dds":"0.46726190476190477","last_synced_commit":"1c03dd14955074d904c10370a34e26a488fbe8ba"},"previous_names":[],"tags_count":43,"template":false,"template_full_name":null,"purl":"pkg:github/tanbro/pyyaml-include","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tanbro%2Fpyyaml-include","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tanbro%2Fpyyaml-include/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tanbro%2Fpyyaml-include/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tanbro%2Fpyyaml-include/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tanbro","download_url":"https://codeload.github.com/tanbro/pyyaml-include/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tanbro%2Fpyyaml-include/sbom","scorecard":{"id":867482,"data":{"date":"2025-08-11","repo":{"name":"github.com/tanbro/pyyaml-include","commit":"13343daddb39deddd73f0af8bca487bccb4ea431"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.8,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 1 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 1/24 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/python-package.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-package.yml:43: update your workflow using https://app.stepsecurity.io/secureworkflow/tanbro/pyyaml-include/python-package.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-package.yml:45: update your workflow using https://app.stepsecurity.io/secureworkflow/tanbro/pyyaml-include/python-package.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/python-package.yml:61: update your workflow using https://app.stepsecurity.io/secureworkflow/tanbro/pyyaml-include/python-package.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-package.yml:72: update your workflow using https://app.stepsecurity.io/secureworkflow/tanbro/pyyaml-include/python-package.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-package.yml:74: update your workflow using https://app.stepsecurity.io/secureworkflow/tanbro/pyyaml-include/python-package.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-package.yml:82: update your workflow using https://app.stepsecurity.io/secureworkflow/tanbro/pyyaml-include/python-package.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-package.yml:95: update your workflow using https://app.stepsecurity.io/secureworkflow/tanbro/pyyaml-include/python-package.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/python-package.yml:100: update your workflow using https://app.stepsecurity.io/secureworkflow/tanbro/pyyaml-include/python-package.yml/main?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/python-package.yml:51","Warn: pipCommand not pinned by hash: .github/workflows/python-package.yml:79","Info:   0 out of   6 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 third-party GitHubAction dependencies pinned","Info:   0 out of   2 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: GNU General Public License v3.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/python-package.yml:89"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 8 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-24T03:04:50.853Z","repository_id":54277650,"created_at":"2025-08-24T03:04:50.853Z","updated_at":"2025-08-24T03:04:50.853Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278957787,"owners_count":26075528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pyyaml","yaml"],"created_at":"2024-09-24T19:57:31.247Z","updated_at":"2025-10-20T04:12:21.356Z","avatar_url":"https://github.com/tanbro.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pyyaml-include\n\n[![GitHub tag](https://img.shields.io/github/tag/tanbro/pyyaml-include.svg)](https://github.com/tanbro/pyyaml-include)\n[![Python Package](https://github.com/tanbro/pyyaml-include/workflows/Python%20package/badge.svg)](https://github.com/tanbro/pyyaml-include/actions?query=workflow%3A%22Python+package%22)\n[![Documentation Status](https://readthedocs.org/projects/pyyaml-include/badge/?version=latest)](https://pyyaml-include.readthedocs.io/en/latest/)\n[![PyPI](https://img.shields.io/pypi/v/pyyaml-include.svg)](https://pypi.org/project/pyyaml-include/)\n[![codecov](https://codecov.io/gh/tanbro/pyyaml-include/graph/badge.svg?token=N4QBGv08EE)](https://codecov.io/gh/tanbro/pyyaml-include)\n\nAn extending constructor of [PyYAML][]: include other [YAML][] files into current [YAML][] document.\n\nIn version `2.0`, [fsspec][] was introduced. With it, we can even include files by HTTP, SFTP, S3 ...\n\n\u003e ⚠️ **Warning** \\\n\u003e “pyyaml-include” `2.0` is **NOT compatible** with `1.0`\n\n## Install\n\n```bash\npip install \"pyyaml-include\"\n```\n\nBecause [fsspec][] was introduced to open the including files since v2.0, an installation can be performed like below, if want to open remote files:\n\n- for files on website:\n\n  ```bash\n  pip install \"pyyaml-include\" fsspec[http]\n  ```\n\n- for files on S3:\n\n  ```bash\n  pip install \"pyyaml-include\" fsspec[s3]\n  ```\n\n- see [fsspec][]'s documentation for more\n\n\u003e 🔖 **Tip** \\\n\u003e “pyyaml-include” depends on [fsspec][], it will be installed no matter including local or remote files.\n\n## Basic usages\n\nConsider we have such [YAML][] files:\n\n```\n├── 0.yml\n└── include.d\n    ├── 1.yml\n    └── 2.yml\n```\n\n- `1.yml` 's content:\n\n  ```yaml\n  name: \"1\"\n  ```\n\n- `2.yml` 's content:\n\n  ```yaml\n  name: \"2\"\n  ```\n\nTo include `1.yml`, `2.yml` in `0.yml`, we shall:\n\n1. Register a `yaml_include.Constructor` to [PyYAML][]'s loader class, with `!inc`(or any other tags start with `!` character) as it's tag:\n\n   ```python\n   import yaml\n   import yaml_include\n\n   # add the tag\n   yaml.add_constructor(\"!inc\", yaml_include.Constructor(base_dir='/your/conf/dir'))\n   ```\n\n1. Use `!inc` tag(s) in `0.yaml`:\n\n   ```yaml\n   file1: !inc include.d/1.yml\n   file2: !inc include.d/2.yml\n   ```\n\n1. Load `0.yaml` in your Python program\n\n   ```python\n   with open('0.yml') as f:\n      data = yaml.full_load(f)\n   print(data)\n   ```\n\n   we'll get:\n\n   ```python\n   {'file1': {'name': '1'}, 'file2': {'name': '2'}}\n   ```\n\n1. (optional) the constructor can be unregistered:\n\n   ```python\n   del yaml.Loader.yaml_constructors[\"!inc\"]\n   del yaml.UnSafeLoader.yaml_constructors[\"!inc\"]\n   del yaml.FullLoader.yaml_constructors[\"!inc\"]\n   ```\n\n### Include in Mapping\n\nIf `0.yml` was:\n\n```yaml\nfile1: !inc include.d/1.yml\nfile2: !inc include.d/2.yml\n```\n\nWe'll get:\n\n```yaml\nfile1:\n  name: \"1\"\nfile2:\n  name: \"2\"\n```\n\n### Include in Sequence\n\nIf `0.yml` was:\n\n```yaml\nfiles:\n  - !inc include.d/1.yml\n  - !inc include.d/2.yml\n```\n\nWe'll get:\n\n```yaml\nfiles:\n  - name: \"1\"\n  - name: \"2\"\n```\n\n## Advanced usages\n\n### Wildcards\n\nFile name can contain shell-style wildcards. Data loaded from the file(s) found by wildcards will be set in a sequence.\n\nThat is, a list will be returned when including file name contains wildcards.\nLength of the returned list equals number of matched files:\n\nIf `0.yml` was:\n\n```yaml\nfiles: !inc include.d/*.yml\n```\n\nWe'll get:\n\n```yaml\nfiles:\n  - name: \"1\"\n  - name: \"2\"\n```\n\n- when only 1 file matched, length of list will be 1\n- when there are no files matched, an empty list will be returned\n\nWe support `**`, `?` and `[..]`. We do not support `^` for pattern negation.\nThe `maxdepth` option is applied on the first `**` found in the path.\n\n\u003e ❗ **Important**\n\u003e\n\u003e - Using the `**` pattern in large directory trees or remote file system (S3, HTTP ...) may consume an inordinate amount of time.\n\u003e - There is no method like lazy-load or iteration, all data of found files returned to the YAML doc-tree are fully loaded in memory, large amount of memory may be needed if there were many or big files.\n\n### Work with fsspec\n\nIn `v2.0`, we use [fsspec][] to open including files, thus we can include files from many different sources, such as local file system, S3, HTTP, SFTP ...\n\nFor example, we can include a file from website in YAML:\n\n```yaml\nconf:\n  logging: !inc http://domain/etc/app/conf.d/logging.yml\n```\n\nIn such situations, when creating a `Constructor` constructor, a [fsspec][] filesystem object shall be set to `fs` argument.\n\nFor example, if want to include files from website, we shall:\n\n1. create a `Constructor` with a [fsspec][] HTTP filesystem object as it's `fs`:\n\n   ```python\n   import yaml\n   import fsspec\n   import yaml_include\n\n   http_fs = fsspec.filesystem(\"http\", client_kwargs={\"base_url\": f\"http://{HOST}:{PORT}\"})\n\n   ctor = yaml_include.Constructor(fs=http_fs, base_dir=\"/foo/baz\")\n   yaml.add_constructor(\"!inc\", ctor, yaml.Loader)\n   ```\n\n1. then, write a [YAML][] document to include files from `http://${HOST}:${PORT}`:\n\n   ```yaml\n   key1: !inc doc1.yml    # relative path to \"base_dir\"\n   key2: !inc ./doc2.yml  # relative path to \"base_dir\" also\n   key3: !inc /doc3.yml   # absolute path, \"base_dir\" does not affect\n   key3: !inc ../doc4.yml # relative path one level upper to \"base_dir\"\n   ```\n\n1. load it with [PyYAML][]:\n\n   ```python\n   yaml.load(yaml_string, yaml.Loader)\n   ```\n\nAbove [YAML][] snippet will be loaded like:\n\n- `key1`: pared YAML of `http://${HOST}:${PORT}/foo/baz/doc1.yml`\n- `key2`: pared YAML of `http://${HOST}:${PORT}/foo/baz/doc2.yml`\n- `key3`: pared YAML of `http://${HOST}:${PORT}/doc3.yml`\n- `key4`: pared YAML of `http://${HOST}:${PORT}/foo/doc4.yml`\n\n\u003e 🔖 **Tip** \\\n\u003e Check [fsspec][]'s documentation for more\n\n---\n\n\u003e ℹ️ **Note** \\\n\u003e If `fs` argument is omitted, a `\"file\"`/`\"local\"` [fsspec][] filesystem object will be used automatically. That is to say:\n\u003e\n\u003e ```yaml\n\u003e data: !inc: foo/baz.yaml\n\u003e ```\n\u003e\n\u003e is equivalent to (if no `base_dir` was set in `Constructor()`):\n\u003e\n\u003e ```yaml\n\u003e data: !inc: file://foo/baz.yaml\n\u003e ```\n\u003e\n\u003e and\n\u003e\n\u003e ```python\n\u003e yaml.add_constructor(\"!inc\", Constructor())\n\u003e ```\n\u003e\n\u003e is equivalent to:\n\u003e\n\u003e ```python\n\u003e yaml.add_constructor(\"!inc\", Constructor(fs=fsspec.filesystem(\"file\")))\n\u003e ```\n\n### Parameters in YAML\n\nAs a callable object, `Constructor` passes YAML tag parameters to [fsspec][] for more detailed operations.\n\nThe first argument is `urlpath`, it's fixed and must-required, either positional or named.\nNormally, we put it as a string after the tag(eg: `!inc`), just like examples above.\n\nHowever, there are more parameters.\n\n- in a sequence way, parameters will be passed to python as positional arguments, like `*args` in python function. eg:\n\n  ```yaml\n  files: !inc [include.d/**/*.yaml, {maxdepth: 1}, {encoding: utf16}]\n  ```\n\n- in a mapping way, parameters will be passed to python as named arguments, like `**kwargs` in python function. eg:\n\n  ```yaml\n  files: !inc {urlpath: /foo/baz.yaml, encoding: utf16}\n  ```\n\nBut the format of parameters has multiple cases, and differs variably in different [fsspec][] implementation backends.\n\n- If a scheme/protocol(“`http://`”, “`sftp://`”, “`file://`”, etc.) is defined, and there is no wildcard in `urlpath`, `Constructor` will invoke [`fsspec.open`](https://filesystem-spec.readthedocs.io/en/stable/api.html#fsspec.open) directly to open it. Which means `Constructor`'s `fs` will be ignored, and a new standalone `fs` will be created implicitly.\n\n  In this situation, `urlpath` will be passed to `fsspec.open`'s first argument, and all other parameters will also be passed to the function.\n\n  For example,\n\n  - the [YAML][] snippet\n\n    ```yaml\n    files: !inc [file:///foo/baz.yaml, r]\n    ```\n\n    will cause python code like\n\n    ```python\n    with fsspec.open(\"file:///foo/baz.yaml\", \"r\") as f:\n        yaml.load(f, Loader)\n    ```\n\n  - and the [YAML][] snippet\n\n    ```yaml\n    files: !inc {urlpath: file:///foo/baz.yaml, encoding: utf16}\n    ```\n\n    will cause python code like\n\n    ```python\n    with fsspec.open(\"file:///foo/baz.yaml\", encoding=\"utf16\") as f:\n        yaml.load(f, Loader)\n    ```\n\n- If `urlpath` has wildcard, and also scheme in it, `Constructor` will:\n\n  Invoke [fsspec][]'s [`open_files`](https://filesystem-spec.readthedocs.io/en/stable/api.html#fsspec.open_files) function to search, open and load files, and return the results in a list. [YAML][] include statement's parameters are passed to `open_files` function.\n\n- If `urlpath` has wildcard, and no scheme in it, `Constructor` will:\n\n  1. invoke corresponding [fsspec][] implementation backend's [`glob`](https://filesystem-spec.readthedocs.io/en/stable/api.html#fsspec.spec.AbstractFileSystem.glob) method to search files,\n  1. then call [`open`](https://filesystem-spec.readthedocs.io/en/stable/api.html#fsspec.spec.AbstractFileSystem.open) method to open each found file(s).\n\n  `urlpath` will be passed as the first argument to both `glob` and `open` method of the corresponding [fsspec][] implementation backend, and other parameters will also be passed to `glob` and `open` method as their following arguments.\n\n  In the case of wildcards, what need to pay special attention to is that there are **two separated parameters** after `urlpath`, the first is for `glob` method, and the second is for `open` method. Each of them could be either sequence, mapping or scalar, corresponds single, positional and named argument(s) in python. For example:\n\n  - If we want to include every `.yml` file in directory `etc/app` recursively with max depth at 2, and open them in utf-16 codec, we shall write the [YAML][] as below:\n\n    ```yaml\n    files: !inc [\"etc/app/**/*.yml\", {maxdepth: !!int \"2\"}, {encoding: utf16}]\n    ```\n\n    it will cause python code like:\n\n    ```python\n    for file in local_fs.glob(\"etc/app/**/*.yml\", maxdepth=2):\n        with local_fs.open(file, encoding=\"utf16\") as f:\n            yaml.load(f, Loader)\n    ```\n\n  - Since `maxdepth` is the seconde argument after `path` in `glob` method, we can also write the [YAML][] like this:\n\n    ```yaml\n    files: !inc [\"etc/app/**/*.yml\", [!!int \"2\"]]\n    ```\n\n    The parameters for `open` is omitted, means no more arguments except `urlpath` is passed.\n\n    it will cause python code like:\n\n    ```python\n    for file in local_fs.glob(\"etc/app/**/*.yml\", 2):\n        with local_fs.open(file) as f:\n            yaml.load(f, Loader)\n    ```\n\n  - The two parameters can be in a mapping form, and name of the keys are `\"glob\"` and `\"open\"`. for example:\n\n    ```yaml\n    files: !inc {urlpath: \"etc/app/**/*.yml\", glob: [!!int \"2\"], open: {encoding: utf16}}\n    ```\n\n  \u003e ❗ **Important** \\\n  \u003e [PyYAML][] sometimes takes scalar parameter of custom constructor as string, we can use a ‘Standard YAML tag’ to ensure non-string data type in the situation.\n  \u003e\n  \u003e For example, following [YAML][] snippet may cause an error:\n  \u003e\n  \u003e ```yaml\n  \u003e files: !inc [\"etc/app/**/*.yml\", open: {intParam: 1}]\n  \u003e ```\n  \u003e\n  \u003e Because [PyYAML][] treats `{\"intParam\": 1}` as `{\"intParam\": \"1\"}`, which makes python code like `fs.open(path, intParam=\"1\")`. To prevent this, we shall write the [YAML][] like:\n  \u003e\n  \u003e ```yaml\n  \u003e files: !inc [\"etc/app/**/*.yml\", open: {intParam: !!int 1}]\n  \u003e ```\n  \u003e\n  \u003e where `!!int` is a ‘Standard YAML tag’ to force integer type of `maxdepth` argument.\n  \u003e\n  \u003e \u003e ℹ️ **Note** \\\n  \u003e \u003e `BaseLoader`, `SafeLoader`, `CBaseLoader`, `CSafeLoader` do **NOT** support ‘Standard YAML tag’.\n  \u003e ---\n  \u003e \u003e 🔖 **Tip** \\\n  \u003e \u003e `maxdepth` argument of [fsspec][] `glob` method is already force converted by `Constructor`, no need to write a `!!int` tag on it.\n\n- Else, `Constructor` will invoke corresponding [fsspec][] implementation backend's [`open`](https://filesystem-spec.readthedocs.io/en/stable/api.html#fsspec.spec.AbstractFileSystem.open) method to open the file, parameters beside `urlpath` will be passed to the method.\n\n### Absolute and Relative URL/Path\n\nWhen the path after include tag (eg: `!inc`) is not a full protocol/scheme URL and not starts with `\"/\"`, `Constructor` tries to join the path with `base_dir`, which is a argument of `Constructor.__init__()`.\nIf `base_dir` is omitted or `None`, the actually including file path is the path in defined in [YAML][] without a change, and different [fsspec][] filesystem will treat them differently. In local filesystem, it will be `cwd`.\n\nFor remote filesystem, `HTTP` for example, the `base_dir` can not be `None` and usually be set to `\"/\"`.\n\nRelative path does not support full protocol/scheme URL format, `base_dir` does not effect for that.\n\nFor example, if we register such a `Constructor` to [PyYAML][]:\n\n```python\nimport yaml\nimport fsspec\nimport yaml_include\n\nyaml.add_constructor(\n    \"!http-include\",\n    yaml_include.Constructor(\n        fsspec.filesystem(\"http\", client_kwargs={\"base_url\": f\"http://{HOST}:{PORT}\"}),\n        base_dir=\"/sub_1/sub_1_1\"\n    )\n)\n```\n\nthen, load following [YAML][]:\n\n```yaml\nxyz: !http-include xyz.yml\n```\n\nthe actual URL to access is `http://$HOST:$PORT/sub_1/sub_1_1/xyz.yml`\n\n### Flatten sequence object in multiple matched files\n\nConsider we have such a YAML:\n\n```yaml\nitems: !include \"*.yaml\"\n```\n\nIf every file matches `*.yaml` contains a sequence object at the top level in it, what parsed and loaded will be:\n\n```yaml\nitems: [\n    [item 0 of 1st file, item 1 of 1st file, ... , item n of 1st file, ...],\n    [item 0 of 2nd file, item 1 of 2nd file, ... , item n of 2nd file, ...],\n    # ....\n    [item 0 of nth file, item 1 of nth file, ... , item n of nth file, ...],\n    # ...\n]\n```\n\nIt's a 2-dim array, because YAML content of each matched file is treated as a member of the list(sequence).\n\nBut if `flatten` parameter was set to `true`, like:\n\n```yaml\nitems: !include {urlpath: \"*.yaml\", flatten: true}\n```\n\nwe'll get:\n\n```yaml\nitems: [\n    item 0 of 1st file, item 1 of 1st file, ... , item n of 1st file,  # ...\n    item 0 of 2nd file, item 1 of 2nd file, ... , item n of 2nd file,  # ...\n    # ....\n    item 0 of n-th file, item 1 of n-th file, ... , item n of n-th file,  # ...\n    # ...\n]\n```\n\n\u003e ℹ️ **Note**\n\u003e\n\u003e - Only available when multiple files were matched.\n\u003e - **Every matched file should have a Sequence object in its top level**, or a `TypeError` exception may be thrown.\n\n### Serialization\n\nWhen load [YAML][] string with include statement, the including files are parsed into python objects by default. That is, if we call `yaml.dump()` on the object, what dumped is the parsed python object, and can not serialize the include statement itself.\n\nTo serialize the statement, we shall first create an `yaml_include.Constructor` object whose **`autoload` attribute is `False`**:\n\n```python\nimport yaml\nimport yaml_include\n\nctor = yaml_include.Constructor(autoload=False)\n```\n\nthen add both Constructor for Loader and Representer for Dumper:\n\n```python\nyaml.add_constructor(\"!inc\", ctor)\n\nrpr = yaml_include.Representer(\"inc\")\nyaml.add_representer(yaml_include.Data, rpr)\n```\n\nNow, the including files will not be loaded when call `yaml.load()`, and `yaml_include.Data` objects will be placed at the positions where include statements are.\n\ncontinue above code:\n\n```python\nyaml_str = \"\"\"\n- !inc include.d/1.yaml\n- !inc include.d/2.yaml\n\"\"\"\n\nd0 = yaml.load(yaml_str, yaml.Loader)\n# Here, \"include.d/1.yaml\" and \"include.d/2.yaml\" not be opened or loaded.\n# d0 is like:\n# [Data(urlpath=\"include.d/1.yaml\"), Data(urlpath=\"include.d/2.yaml\")]\n\n# serialize d0\ns = yaml.dump(d0)\nprint(s)\n# ‘s’ will be:\n# - !inc 'include.d/1.yaml'\n# - !inc 'include.d/2.yaml'\n\n# de-serialization\nctor.autoload = True # re-open auto load\n# then load, the file \"include.d/1.yaml\" and \"include.d/2.yaml\" will be opened and loaded.\nd1 = yaml.load(s, yaml.Loader)\n\n# Or perform a recursive opening / parsing on the object:\nd2 = yaml_include.load(d0) # d2 is equal to d1\n```\n\n`autoload` can be used in a `with` statement:\n\n```python\nctor = yaml_include.Constructor()\n# autoload is True here\n\nwith ctor.managed_autoload(False):\n    # temporary set autoload to False\n    yaml.full_load(YAML_TEXT)\n# autoload restore True automatic\n```\n\n### Include JSON or TOML\n\nWe can include files in different format other than [YAML][], like [JSON][] or [TOML][] -- ``custom_loader`` is for that.\n\n\u003e 📑 **Example** \\\n\u003e For example:\n\u003e\n\u003e ```python\n\u003e import json\n\u003e import tomllib as toml\n\u003e import yaml\n\u003e import yaml_include\n\u003e\n\u003e # Define loader function\n\u003e def my_loader(urlpath, file, Loader):\n\u003e     if urlpath.endswith(\".json\"):\n\u003e         return json.load(file)\n\u003e     if urlpath.endswith(\".toml\"):\n\u003e         return toml.load(file)\n\u003e     return yaml.load(file, Loader)\n\u003e\n\u003e # Create the include constructor, with the custom loader\n\u003e ctor = yaml_include.Constructor(custom_loader=my_loader)\n\u003e\n\u003e # Add the constructor to YAML Loader\n\u003e yaml.add_constructor(\"!inc\", ctor, yaml.Loader)\n\u003e\n\u003e # Then, json files will can be loaded by std-lib's json module, and the same to toml files.\n\u003e s = \"\"\"\n\u003e json: !inc \"*.json\"\n\u003e toml: !inc \"*.toml\"\n\u003e yaml: !inc \"*.yaml\"\n\u003e \"\"\"\n\u003e\n\u003e yaml.load(s, yaml.Loader)\n\u003e ```\n\n[YAML]: http://yaml.org/ \"YAML: YAML Ain't Markup Language™\"\n[PyYaml]: https://pypi.org/project/PyYAML/ \"PyYAML is a full-featured YAML framework for the Python programming language.\"\n[fsspec]: https://github.com/fsspec/filesystem_spec/ \"Filesystem Spec (fsspec) is a project to provide a unified pythonic interface to local, remote and embedded file systems and bytes storage.\"\n[JSON]: https://json.io/ \"JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write\"\n[TOML]: https://toml.io/ \"TOML aims to be a minimal configuration file format that's easy to read due to obvious semantics.\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftanbro%2Fpyyaml-include","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftanbro%2Fpyyaml-include","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftanbro%2Fpyyaml-include/lists"}