{"id":13738487,"url":"https://github.com/vict0rsch/minydra","last_synced_at":"2025-04-22T08:30:45.532Z","repository":{"id":44886253,"uuid":"319061923","full_name":"vict0rsch/minydra","owner":"vict0rsch","description":"🦎 Minimal Python command-line parser inspired by Facebook's Hydra. Handles and parses arbitrary arguments into dot-accessible nested dictionaries. ","archived":false,"fork":false,"pushed_at":"2022-01-20T12:52:44.000Z","size":2356,"stargazers_count":21,"open_issues_count":2,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-01T21:11:23.006Z","etag":null,"topics":["command-line-tool","commandline","dictionaries","dot-access","minimal","no-dependencies","parser","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vict0rsch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-06T15:12:04.000Z","updated_at":"2024-12-04T21:46:08.000Z","dependencies_parsed_at":"2022-09-26T17:20:58.630Z","dependency_job_id":null,"html_url":"https://github.com/vict0rsch/minydra","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vict0rsch%2Fminydra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vict0rsch%2Fminydra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vict0rsch%2Fminydra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vict0rsch%2Fminydra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vict0rsch","download_url":"https://codeload.github.com/vict0rsch/minydra/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250206012,"owners_count":21392166,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line-tool","commandline","dictionaries","dot-access","minimal","no-dependencies","parser","python"],"created_at":"2024-08-03T03:02:23.838Z","updated_at":"2025-04-22T08:30:45.228Z","avatar_url":"https://github.com/vict0rsch.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# minydra 🦎\n\nMinimal Python command-line parser inspired by Facebook's Hydra + dot-accessible nested dictionaries.\n\nEasily parse arbitrary arguments from the command line without dependencies:\n\n![example code](assets/code.png)\n![example code](assets/run.png)\n\n![](https://img.shields.io/badge/coverage-99%25-success)\n![](https://img.shields.io/badge/version-0.1.5-informational)\n![](https://img.shields.io/badge/python-3.7%2B%20-orange)\n\n```bash\npip install minydra\n```\n\n`minydra` is tested on Python `3.7`, `3.8` and `3.9`.\n\n\u003cbr/\u003e\n\n\u003cp align=\"center\"\u003e\n \u003ca href=\"#getting-started\"\u003e\u003cstrong\u003eGetting Started\u003c/strong\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;•\u0026nbsp;\n \u003ca href=\"#forcing-types\"\u003e\u003cstrong\u003eForcing types\u003c/strong\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;•\u0026nbsp;\n \u003ca href=\"#minydict\"\u003e\u003cstrong\u003eMinyDict\u003c/strong\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;•\u0026nbsp;\n \u003ca href=\"#dumpingloading\"\u003e\u003cstrong\u003eSave config\u003c/strong\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;•\u0026nbsp;\n \u003ca href=\"#strict-mode\"\u003e\u003cstrong\u003ePrevent typos\u003c/strong\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;•\u0026nbsp;\n \u003ca href=\"#using-default-configurations\"\u003e\u003cstrong\u003eUse default configs\u003c/strong\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;•\u0026nbsp;\n \u003ca href=\"/examples\"\u003e\u003cstrong\u003eExamples\u003c/strong\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cbr/\u003e\n\n## Getting Started\n\n[`examples/parser.py`](examples/parser.py)\n\n```python\nfrom minydra.parser import Parser\n\nif __name__ == \"__main__\":\n    parser = Parser(\n        verbose=0, # print received args\n        allow_overwrites=False, # allow repeating args in the command-line\n        warn_overwrites=True, # warn repeating args if they are allowed\n        parse_env=True, # get environment variable\n        warn_env=True, # warn if an environment variable is specified but not found\n        defaults=None, # path to a MinyDict-loadable dictionary of default values for the args\n        strict=True, # if `defaults` is provided, whether to allow new keys in the command-line\n                     # or restrict to `defaults`' keys\n        keep_special_kwargs=True, # `defaults` and `strict` can be set from the command-line\n                                  # with `@defaults=` and `@strict=`. This argument decides if\n                                  # you want to keep those keys in the final arguments.\n    )\n    args = parser.args.pretty_print().resolve().pretty_print() # notice .resolve() transforms dotted.keys into nested dicts\n```\n\n[`examples/resolved_args.py`](examples/resolved_args.py)\n\n```python\nfrom minydra import resolved_args\n\nif __name__ == \"__main__\":\n    args = resolved_args()\n    args.pretty_print()\n```\n\n[`examples/demo.py`](examples/demo.py)\n[`examples/demo.json`](examples/demo.json)\n\n```python\nfrom minydra import MinyDict, resolved_args\nfrom pathlib import Path\n\nif __name__ == \"__main__\":\n    # parse arbitrary args in 1 line\n    args = resolved_args()\n\n    # override default conf\n    if args.default:\n        args = MinyDict.from_json(args.default).update(args)\n\n    # protect args in the rest of the code execution\n    args.freeze()\n\n    # print the args in a nice orderly fashion\n    args.pretty_print()\n\n    # access args with dot/attribute access\n    print(f'Using project \"{args.log.project}\" in {args.log.outdir}')\n\n    # save configuration\n    args.to_json(Path(args.log.outdir) / f\"{args.log.project}.json\")\n\n```\n\n\n\n[`examples/decorator.py`](examples/decorator.py)\n\n```python\nimport minydra\nfrom minydra.dict import MinyDict\n\n@minydra.parse_args(verbose=0, allow_overwrites=False) # Parser's init args work here\ndef main(args: MinyDict) -\u003e None:\n    args.resolve().pretty_print()\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Parsing\n\n* Simple strings are parsed to `float` and `int` automatically.\n* A single keyword will be interpreted as a positive flag.\n* A single keyword starting with `-` will be interpreted as a negative flag.\n* If `parse_env` is `True`, environment variables are evaluated.\n\n```text\n$ python examples/decorator.py outdir=$HOME/project save -log learning_rate=1e-4 batch_size=64\n╭───────────────────────────────────────────╮\n│ batch_size    : 64                        │\n│ learning_rate : 0.0001                    │\n│ log           : False                     │\n│ outdir        : /Users/victor/project     │\n│ save          : True                      │\n╰───────────────────────────────────────────╯\n```\n\n* dotted keys will be resolved to nested dictionary keys:\n\n```text\n$ python examples/decorator.py server.conf.port=8000\n╭────────────────────╮\n│ server             │\n│ │conf              │\n│ │ │port : 8000     │\n╰────────────────────╯\n```\n\n* Using `ast.literal_eval(value)`, `minydra` will try and parse more complex values for arguments as lists or dicts. Those should be specified as strings:\n\n```text\n$ python examples/decorator.py layers=\"[1, 2, 3]\" norms=\"{'conv': 'batch', 'epsilon': 1e-3}\"\n╭──────────────────────────────────────────────────╮\n│ layers : [1, 2, 3]                               │\n│ norms  : {'conv': 'batch', 'epsilon': 0.001}     │\n╰──────────────────────────────────────────────────╯\n```\n\n\u003cbr/\u003e\n\n### Forcing types\n\nAdding `___\u003ctype\u003e` to a key will force this type to the value. Notice how `01` is parsed to an integer `1` but `04` is parsed to a string (as specified) `\"04\"`, and `hello` is parsed to a `list`, not kept as a string\n\n```text\n$ python examples/decorator.py n_jobs___str=04 job=01 chips___list=hello\n╭────────────────────────────────────────╮\n│ chips  : ['h', 'e', 'l', 'l', 'o']     │\n│ job    : 1                             │\n│ n_jobs : 04                            │\n╰────────────────────────────────────────╯\n```\n\nKnown types are defined in `Parser.known_types` and the separator (`___`) in `Parser.type_separator`\n\n```python\nIn [1]: from minydra import Parser\n\nIn [2]: Parser.known_types\nOut[2]: {'bool', 'float', 'int', 'str'}\n\nIn [3]: Parser.type_separator\nOut[3]: '___'\n```\n\n\u003cbr/\u003e\n\n### Command-line configuration\n\nYou can configure the `Parser` from the command-line using special `@` arguments. In other words, all `__init__(self, ...)` arguments can be set from the command-line with `@argname=new_value`.\n\nIn particular if you run `python examples/decorator.py @defaults=./examples/demo.json` you will see:\n\n```\n╭──────────────────────────────────────╮\n│ @defaults : ./examples/demo.json     │\n│ log                                  │\n│ │logger                              │\n│ │ │log_level   : DEBUG               │\n│ │ │logger_name : minydra             │\n│ │outdir  : /some/path                │\n│ │project : demo                      │\n│ verbose   : False                    │\n╰──────────────────────────────────────╯\n```\n\n**But** if you add `@strict=false @keep_special_kwargs=false` you will now have:\n\n```\n$ python examples/decorator.py @defaults=./examples/demo.json @strict=false @keep_special_kwargs=false\n╭──────────────────────────────╮\n│ log                          │\n│ │logger                      │\n│ │ │log_level   : DEBUG       │\n│ │ │logger_name : minydra     │\n│ │outdir  : /some/path        │\n│ │project : demo              │\n│ verbose : False              │\n╰──────────────────────────────╯\n```\n\n(*you need to have `@strict=false` since `@keep_special_kwargs` is unknown in `demo.json`. It would not be the case if `strict=false` had been used in the script itself (but it can be overridden from the command-line!)*)\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## MinyDict\n\nMinydra's args are a custom lightweight wrapper around native `dict` which allows for dot access (`args.key`), resolving dotted keys into nested dicts and pretty printing sorted keys in a box with nested dicts indented. If a key does not exist, it will not fail, rather return None (as `dict.get(key, None)`).\n\na `MinyDict` inherits from `dict` so usual methods work `.keys()`, `.items()` etc.\n\n```python\n\nIn [1]: from minydra.dict import MinyDict\n\nIn [2]: args = MinyDict({\"foo\": \"bar\", \"yes.no.maybe\": \"idontknow\"}).pretty_print(); args\n╭──────────────────────────────╮\n│ foo          : bar           │\n│ yes.no.maybe : idontknow     │\n╰──────────────────────────────╯\nOut[2]: {'foo': 'bar', 'yes.no.maybe': 'idontknow'}\n\nIn [3]: args.resolve().pretty_print(); args\n╭──────────────────────────╮\n│ foo : bar                │\n│ yes                      │\n│ │no                      │\n│ │ │maybe : idontknow     │\n╰──────────────────────────╯\nOut[3]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}\n\nIn [4]: args.yes.no.maybe\nOut[4]: \"idontknow\"\n\nIn [5]: \"foo\" in args\nOut[5]: True\n\nIn [6]: \"rick\" in args\nOut[6]: False\n\nIn [7]: args.morty is None\nOut[7]: True\n\nIn [8]: args.items()\nOut[8]: dict_items([('foo', 'bar'), ('yes', {'no': {'maybe': 'idontknow'}})])\n```\n\n\u003cbr/\u003e\n\n### Dumping/Loading\n\nYou can save and read `MinyDict` to/from disk in 3 formats: `json` and `pickle` without dependencies, `yaml` with the `PyYAML` dependency (`pip install minydra[yaml]`).\n\nMethods `to_pickle`, `to_json` and `to_yaml` have 3 arguments:\n\n1. `file_path` as a `str` or `pathlib.Path` which is resolved:\n    1. expand env variable (`$MYDIR` for instance)\n    2. expand user (`~`)\n    3. make absolute\n2. `return_path` which defaults to `True`. If those methods return the path of the created file\n3. `allow_overwrites` which defaults to `True`. If `False` and `path` exists, a `FileExistsError` will be raised. Otherwise creates/overwrites the file at `file_path`\n4. `verbose` which defaults to `0`. If `\u003e0` prints the path of the created object\n\nNote:\n\n* `to/from_yaml` will fail with a `ModuleNotFoundError` if `PyYAML` is not installed.\n* the `json` standard does not accept ints as keys in dictionaries so `{3: 2}` would be dumped -- and therefore loaded -- as `{\"3\": 2}`.\n\n\n\n```python\nIn [1]: from minydra.dict import MinyDict\n\nIn [2]: args = MinyDict({\"foo\": \"bar\", \"yes.no.maybe\": \"idontknow\"}).resolve(); args\nOut[2]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}\n\nIn [3]: json_file_path = args.to_json(\"./args.json\")\n\nIn [4]: yaml_file_path = args.to_yaml(\"./args.yaml\")\n\nIn [5]: pkl_file_path = args.to_pickle(\"./args.pkl\")\n\nIn [6]: _ = args.to_json(\"./args.json\", verbose=1) # verbose argument prints the path\nJson dumped to: /Users/victor/Documents/Github/vict0rsch/minydra/args.json\n\nIn [7]: MinyDict.from_json(\"args.json\")\nOut[7]: {'foo': 'bar', 'yes': {'no': {'maybe': 'idontknow'}}}\n\nIn [8]: assert (\n    MinyDict.from_yaml(yaml_file_path)\n    == MinyDict.from_json(json_file_path)\n    == MinyDict.from_pickle(pkl_file_path)\n    == args\n)\n```\n\n[`examples/dumps.py`](examples/dumps.py)\n\n```text\npython examples/dumps.py path=\"./myargs.pkl\" format=pickle cleanup\n\n╭────────────────────────────╮\n│ cleanup : True             │\n│ format  : pickle           │\n│ path    : ./myargs.pkl     │\n╰────────────────────────────╯\nDumped args to /Users/victor/Documents/Github/vict0rsch/minydra/myargs.pkl\nCleaning up\n```\n\n\u003cbr/\u003e\n\n### Strict Mode\n\nTo prevent typos from the command-line, the `MinyDict.update` method has a strict mode: updating a `MinyDict` with another one using `strict=True` will raise a `KeyError` if the key does not already exist:\n\n```python\nfrom minydra import MinyDict, resolved_args\n\nif __name__ == \"__main__\":\n    # parse arbitrary args in 1 line\n    args = resolved_args()\n\n    # override default conf\n    if args.default:\n        path = args.default\n        # delete otherwise it will be used to update the conf which does not have\n        # \"default\" as a key, therefore raising a KeyError in strict mode\n        del args.default\n        args = MinyDict.from_json(path).update(args, strict=True)\n\n    args.pretty_print()\n```\n\nNo typo:\n\n```text\n$ python examples/strict.py default=./examples/demo.json log.logger.log_level=INFO\n╭──────────────────────────────╮\n│ log                          │\n│ │logger                      │\n│ │ │log_level   : INFO        │\n│ │ │logger_name : minydra     │\n│ │outdir  : /some/path        │\n│ │project : demo              │\n│ verbose : False              │\n╰──────────────────────────────╯\n```\n\nTypo:\n\n```\n$ python examples/strict.py default=./examples/demo.json log.logger.log_leveel=INFO\nTraceback (most recent call last):\n  File \"/Users/victor/Documents/Github/vict0rsch/minydra/examples/strict.py\", line 13, in \u003cmodule\u003e\n    args = MinyDict.from_json(path).update(args, strict=True)\n  File \"/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py\", line 111, in update\n    self[k].update(v, strict=strict)\n  File \"/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py\", line 111, in update\n    self[k].update(v, strict=strict)\n  File \"/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py\", line 100, in update\n    raise KeyError(\nKeyError: 'Cannot create a non-existing key in strict mode ({\"log_leveel\":INFO}).'\n```\n\n\u003cbr/\u003e\n\n### Using default configurations\n\nThe `minydra.Parser` class takes a `defaults=` keyword argument. This can be:\n\n* a `str` or a `pathlib.Path` to a `json` `yaml` or `pickle` file that `minydra.MinyDict` can load (`from_X`)\n* a `dict` or a `minydra.MinyDict`\n* a `list` of the above types, in which case the resulting defaults will be the result of sequential updates from those defaults, enabling hierarchical defaults (first defaults are the starting point, then each subsequent defaults updates it)\n\nWhen `defaults` is provided, the resulting `minydra.MinyDict` serves as a reference for the arguments parsed from the command-line:\n\n* **If** you setup the parser with `strict=True`, arguments from the command-line will still have a higher priority but they will **have to** be present in the `defaults` to prevent typos or unknown arguments (see [strict mode](#strict-mode))\n* arguments not present in the command-line with fallback to values in `defaults`\n\n`defaults` can actually be a `list` and the update order is the same as the list's. For instance:\n\n```python\nIn [1]: from minydra import Parser\n\nIn [2]: Parser(defaults=[\"./examples/demo.json\", \"./examples/demo2.json\"]).args.pretty_print();\n╭─────────────────────────────────╮\n│ log                             │\n│ │logger                         │\n│ │ │log_level   : INFO           │\n│ │ │logger_name : minydra        │\n│ │outdir  : /some/other/path     │\n│ │project : demo                 │\n│ new_key : 3                     │\n│ verbose : False                 │\n╰─────────────────────────────────╯\n```\n\nIf you need to set defaults from the command-line, there's a special `@defaults` keyword you can use:\n\n\n```text\n$ python examples/decorator.py @defaults=./examples/demo.json\n╭──────────────────────────────────────╮\n│ @defaults : ./examples/demo.json     │\n│ log                                  │\n│ │logger                              │\n│ │ │log_level   : DEBUG               │\n│ │ │logger_name : minydra             │\n│ │outdir  : /some/path                │\n│ │project : demo                      │\n│ verbose   : False                    │\n╰──────────────────────────────────────╯\n\n$ python examples/decorator.py @defaults=\"['./examples/demo.json', './examples/demo2.json']\"\n╭───────────────────────────────────────────────────────────────────╮\n│ @defaults : ['./examples/demo.json', './examples/demo2.json']     │\n│ log                                                               │\n│ │logger                                                           │\n│ │ │log_level   : INFO                                             │\n│ │ │logger_name : minydra                                          │\n│ │outdir  : /some/other/path                                       │\n│ │project : demo                                                   │\n│ new_key   : 3                                                     │\n│ verbose   : False                                                 │\n╰───────────────────────────────────────────────────────────────────╯\n```\n\n\u003cbr/\u003e\n\n### `pretty_print`\n\nPrints the `MinyDict` in a box, with dicts properly indented. A few arguments:\n\n1. `indents`, which defaults to `2`: the amount of indentation for nested dictionaries\n2. `sort_keys`, which defaults to `True`: whether or not to alphabetically sort the keys before printing\n\n\u003cbr/\u003e\n\n### `to_dict`\n\nTo produce a native Python `dict`, use `args.to_dict()`\n\n\u003cbr/\u003e\n\n### Protected attributes\n\n`MinyDict`'s methods (including the `dict` class's) are protected, they are read-only and you cannot therefore set _attributes_ with there names, like `args.get = 2`. If you do need to have a `get` argument, you can access it through _items_: `args[\"get\"] = 2`.\n\nTry with [`examples/protected.py`](examples/protected.py):\n\n```text\npython examples/protected.py server.conf.port=8000 get=3\n╭────────────────────╮\n│ get    : 3         │\n│ server             │\n│ │conf              │\n│ │ │port : 8000     │\n╰────────────────────╯\n\u003cbuilt-in method get of MinyDict object at 0x100ccd4a0\u003e\n3\ndict_items([('get', 3), ('server', {'conf': {'port': 8000}})])\n{'conf': {'port': 8000}}\n```\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Tests\n\nRun tests and pre-commit checks (`isort`, `black`, `flake8`) with\n\n```bash\n$ pip install -r requirements-test.txt\n$ pre-commit run --all-files\n$ pytest -vv --cov=minydra tests/\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvict0rsch%2Fminydra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvict0rsch%2Fminydra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvict0rsch%2Fminydra/lists"}