{"id":21686472,"url":"https://github.com/danlooo/metayaml","last_synced_at":"2026-05-17T12:09:02.914Z","repository":{"id":161001452,"uuid":"626465676","full_name":"danlooo/metayaml","owner":"danlooo","description":"Attributes of files and directories stored in YAML sidecar files","archived":false,"fork":false,"pushed_at":"2023-05-26T08:00:49.000Z","size":388,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-25T12:25:03.619Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danlooo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-11T14:20:04.000Z","updated_at":"2024-01-30T14:23:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"d1f59d49-b689-44b6-a933-30c7cf5d42b1","html_url":"https://github.com/danlooo/metayaml","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danlooo%2Fmetayaml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danlooo%2Fmetayaml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danlooo%2Fmetayaml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danlooo%2Fmetayaml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danlooo","download_url":"https://codeload.github.com/danlooo/metayaml/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244609458,"owners_count":20480782,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T16:29:56.741Z","updated_at":"2026-05-17T12:08:57.873Z","avatar_url":"https://github.com/danlooo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# metayaml\n\nAttributes of files and directories stored in YAML sidecar files\n\n\nThis is a proof of concept on how to easily create meta data / attributes for files and directories.\nThis works recursively, so attributes of a file are inherited from those from a parent directory if not specified directly.\n\nUse cases:\n\n- Assign users to directories / files so they can be notified when the directory is scheduled for archiving\n- Assign a due date so the data can be deleted or archived afterwards\n\nAdvantages:\n\n- Machine and human readable\n- Independent on file system and OS\n- Allows people to write the meta data but not the file itself (e.g. add him/herself as project members)\n- Works with Git\n- Very flexible: Can add attributes of any type including string, numbers and lists of them\n\n## Usage\n\nTo create attributes of a directory, just create a file `meta.yml` into that directory.\nCreate a file `/path/foo.txt.yml` to create attributes to just the file `/path/foo.txt`.\n\nFiles matching these recursively defined attributes can be then retrieved with the command `metayaml find`.\nOn the other hand, attributes of a given file or directory can be retrieved with the command `metayaml get`.\n\n## Installation\n\nFirst, ensure that [rclone](https://github.com/rclone/rclone) is available on the machine. Then, install metayaml using:\n\n```\npython -m pip install 'metayaml @ git+https://github.com/danlooo/metayaml'\n```\n\nGet help using `metayaml --help`\n\n```\nUsage: metayaml [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n  -v, --verbose\n  --help         Show this message and exit.\n\nCommands:\n  filter  Create rclone filter rules for files and directories matching a...\n  find    Find files matching a specific attribute stored in YAML meta...\n  get     Retrieves attributes of a directory or file based on YAML meta...\n```\n\nHelp page for a specific command: `metayaml find --help`\n\n```\nUsage: metayaml.py find [OPTIONS] QUERY\n\n  Find files matching a specific attribute stored in YAML meta data sidecar\n  files.\n\n  Examples:\n  metayaml find \"score \u003e 5\"\n  metayaml find \"djohn in users\"\n  metayaml find \"is_example = True\"\n\nOptions:\n  -d, --directory PATH  Search everything recursiveley inside this root\n                        directory  [default: .]\n  -a, --abs-path        Use absolute paths\n  --help                Show this message and exit.\n```\n\n## Forward search: Get attributes of a given directory or file\n\nGet attributes about a directory using `metayaml get example/Americas`\n\n```\ndescription: all data\nis_example: true\nscore: 5\n```\n\nor `metayaml get example/EU`\n\n```\ndescription: EU data\nis_example: true\nscore: 3.5\nusers:\n- dloos\n- fgans\n```\n\nVerbose mode to get warnings in case a value was overwritten by a child directory: `metayaml -v get example/EU`\n\n```\nINFO:root:overwrite description from 'all data' to 'EU data'\nINFO:root:overwrite score from '5' to '3.5'\ndescription: EU data\nis_example: true\nscore: 3.5\nusers:\n- dloos\n- fgans\n```\n\nGet attributes of a single file: `metayaml get example/EU/de.txt`\n\n```\ndescription: Germany data\nhas_sidecar_meta_file: true\nis_example: true\nscore: 3.5\nusers:\n- djohn\n```\n\n## Reverse search: Find files matching giving attributes\n\nFind all files for which a particular user is associated:\n`metayaml find \"dloos in users\"`\n\n```\nexample/EU/be.txt\nexample/EU/de.txt\nexample/EU/nl.txt\n```\n\nFiles in which the score is less than a given number: `metayaml find \"score \u003c 4\"`\n\n```\nexample/EU/be.txt\nexample/EU/de.txt\nexample/EU/de.txt.yml\nexample/EU/nl.txt\n```\n\nand also `metayaml find \"score \u003e= 4\"`\n\n```\nexample/Americas/South_America/ar.txt\nexample/Americas/South_America/br.txt\nexample/Americas/South_America/cl.txt\nexample/Americas/North_America/ca.txt\nexample/Americas/North_America/us.txt\n```\n\nFor boolean values, i.e., tags will show no file, because all files were examples: `metayaml find \"is_example = False\"`\n\nMetayaml uses [rclone](https://github.com/rclone/rclone) to find and filter files.\nIt creates [rclone filter rules](https://rclone.org/filtering/) that can be also exported: `metayaml filter \"score \u003c 4\"`\n\n```\n# rclone filter rules for searching 'score \u003c 4.0' inside '/home/dloos/lab/021-metayaml'\n- **/meta.yml\n+ example/EU/**\n- example/**\n- **\n```\n\n## Thoughts\n\n- Just use TOML?\n  - TOML uses a table by default. Yaml allows us to set primitives and it is also a superset of JSON\n- Can we just merge lists if there are specified to the file and a parent directory?\n  - It would be convenient to just put people A and B to the entire directory and person C to just a subdir. We then put just person C in the child meta file. Persons A and B will be then inferred from the parent meta file. However, doing so, there is no way to really overwrite this attribute. It would be also more confusing\n- Can we use [tagfs](https://github.com/loglob/tagfs)?\n  - Tagfs is a file system in user space (FUSE). We can add tags to a dir to let it appear in the FUSE mounted dir under the tag. However, this is binary only and the tags are stored at a very different location compared to the files\n- Can we use xattrs or Extended Attributes on Linux?\n  - This would be only accessible on SSH but not on Windows shared folders with SMB. This would also require write access to the file and is very hidden feature (Need a Terminal to access attributes)\n- We can also say that the parent overwrites the children. This is way less flexible but allows much faster reverse search to find files that match a certain attribute. Very bad. And finding all meta data fiels on /Net/Groups/BGI/work_1 takes just 6min\n\n## Contributing\n\nInterested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.\n\n## License\n\n`metayaml` was created by Daniel Loos. It is licensed under the terms of the MIT license.\n\n## Credits\n\n`metayaml` was created with [`cookiecutter`](https://cookiecutter.readthedocs.io/en/latest/) and the `py-pkgs-cookiecutter` [template](https://github.com/py-pkgs/py-pkgs-cookiecutter).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanlooo%2Fmetayaml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanlooo%2Fmetayaml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanlooo%2Fmetayaml/lists"}