{"id":13738045,"url":"https://github.com/eladrich/pyrallis","last_synced_at":"2025-04-04T20:04:30.417Z","repository":{"id":38208406,"uuid":"435208875","full_name":"eladrich/pyrallis","owner":"eladrich","description":"Pyrallis is a framework for structured configuration parsing from both cmd and files. Simply define your desired configuration structure as a dataclass and let pyrallis do the rest!","archived":false,"fork":false,"pushed_at":"2023-12-14T14:06:46.000Z","size":4735,"stargazers_count":225,"open_issues_count":13,"forks_count":7,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-28T19:06:55.825Z","etag":null,"topics":["argparse","argparse-alternative","argument-parsing","configuration-management","dataclasses","deep-learning","hydra","machine-learning","python"],"latest_commit_sha":null,"homepage":"https://eladrich.github.io/pyrallis/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eladrich.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-05T15:44:45.000Z","updated_at":"2025-03-20T06:46:47.000Z","dependencies_parsed_at":"2024-06-18T21:26:48.423Z","dependency_job_id":"33d14180-233a-48a2-b158-8e3f5be5023b","html_url":"https://github.com/eladrich/pyrallis","commit_stats":{"total_commits":55,"total_committers":4,"mean_commits":13.75,"dds":0.1636363636363637,"last_synced_commit":"1e0586f9de9ed5d8d67d061dac1fb44c73f9d4a4"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eladrich%2Fpyrallis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eladrich%2Fpyrallis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eladrich%2Fpyrallis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eladrich%2Fpyrallis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eladrich","download_url":"https://codeload.github.com/eladrich/pyrallis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247242669,"owners_count":20907133,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["argparse","argparse-alternative","argument-parsing","configuration-management","dataclasses","deep-learning","hydra","machine-learning","python"],"created_at":"2024-08-03T03:02:09.716Z","updated_at":"2025-04-04T20:04:30.349Z","avatar_url":"https://github.com/eladrich.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/eladrich/pyrallis/master/docs/pyrallis_logo.png\" alt=\"logo\" width=\"70%\" /\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://badge.fury.io/py/pyrallis\"\u003e\u003cimg src=\"https://badge.fury.io/py/pyrallis.svg\" alt=\"PyPI version\" height=\"18\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/eladrich/pyrallis/actions/workflows/pytest.yml\"\u003e\u003cimg src=\"https://github.com/eladrich/pyrallis/actions/workflows/pytest.yml/badge.svg\" alt=\"PyTest\" height=\"18\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://pepy.tech/project/pyrallis\"\u003e\u003cimg src=\"https://pepy.tech/badge/pyrallis\" alt=\"Downloads\" height=\"18\"\u003e\u003c/a\u003e\n    \u003ca href=\"#contributors-\"\u003e\u003cimg src=\"https://img.shields.io/badge/all_contributors-2-orange.svg\" alt=\"All Contributors\" height=\"18\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" alt=\"License: MIT\" height=\"18\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n# Pyrallis - Simple Configuration with Dataclasses\n\n\u003e Pyrausta (also called pyrallis (πυραλλίς), pyragones) is a mythological insect-sized dragon from Cyprus.\n\n`Pyrallis` is a simple library, derived from `simple-parsing` and inspired by `Hydra`, for automagically creating project configuration from a dataclass.\n\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://github.com/eladrich/pyrallis/raw/master/docs/argparse2pyrallis.gif\" alt=\"GIF\" width=\"100%\" /\u003e\u003c/p\u003e\n\n## Why `pyrallis`?\n\nWith `pyrallis` your configuration is linked directly to your pre-defined `dataclass`, allowing you to easily create different configuration structures, including nested ones, using an object-oriented design. The parsed arguments are used to initialize your `dataclass`, giving you the typing hints and automatic code completion of a full `dataclass` object.\n\n\n## My First Pyrallis Example 👶\nThere are several key features to pyrallis but at its core pyrallis simply allows defining an argument parser using a dataclass.\n\n```python \nfrom dataclasses import dataclass\nimport pyrallis\n\n\n@dataclass\nclass TrainConfig:\n    \"\"\" Training config for Machine Learning \"\"\"\n    workers: int = 8 # The number of workers for training\n    exp_name: str = 'default_exp' # The experiment name\n\ndef main():\n    cfg = pyrallis.parse(config_class=TrainConfig)\n    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')\n\n```\n\nThe arguments can then be specified using command-line arguments, a `yaml` configuration file, or both.\n\n```console\n$ python train_model.py --config_path=some_config.yaml --exp_name=my_first_exp\nTraining my_first_exp with 42 workers...\n```\nAssuming the following configuration file\n```yaml\nexp_name: my_yaml_exp\nworkers: 42\n```\n\n### Key Features\nBuilding on that design `pyrallis` offers some really enjoyable features including \n\n* Builtin IDE support for autocompletion and linting thanks to the structured config. 🤓\n* Joint reading from command-line and a config file, with support for specifying a default config file. 😍\n* Support for builtin dataclass features, such as `__post_init__` and `@property` 😁\n* Support for nesting and inheritance of dataclasses, nested arguments are automatically created! 😲\n* A magical `@pyrallis.wrap()` decorator for wrapping your main class 🪄\n* Easy extension to new types using `pyrallis.encode.register` and `pyrallis.decode.register` 👽\n* Easy loading and saving of existing configurations using `pyrallis.dump` and `pyrallis.load` 💾\n* Magical `--help` creation from dataclasses, taking into account the comments as well! 😎\n* Support for multiple configuration formats (`yaml`, `json`,`toml`) using `pyrallis.set_config_type` ⚙️\n\n\n## Getting to Know The `pyrallis` API in 5 Simple Steps 🐲\n\nThe best way to understand the full `pyrallis` API is through examples, let's get started!\n\n###  🐲 1/5 `pyrallis.parse` for `dataclass` Parsing 🐲\n\nCreation of an argparse configuration is really simple, just use `pyrallis.parse` on your predefined dataclass.\n\n```python\nfrom dataclasses import dataclass, field\nimport pyrallis\n\n\n@dataclass\nclass TrainConfig:\n    \"\"\" Training config for Machine Learning \"\"\"\n    # The number of workers for training\n    workers: int = field(default=8)\n    # The experiment name\n    exp_name: str = field(default='default_exp')\n\n\ndef main():\n    cfg = pyrallis.parse(config_class=TrainConfig)\n    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')\n\n\nif __name__ == '__main__':\n    main()\n```\n\u003e Not familiar with `dataclasses`? you should probably check the [Python Tutorial](https://docs.python.org/3/library/dataclasses.html) and come back here.\n\nThe config can then be parsed directly from command-line\n```console\n$ python train_model.py --exp_name=my_first_model\nTraining my_first_model with 8 workers...\n```\nOh, and `pyrallis` also generates an `--help` string automatically using the comments in your dataclass 🪄\n\n```console\n$ python train_model.py --help\nusage: train_model.py [-h] [--config_path str] [--workers int] [--exp_name str]\n\noptional arguments:\n  -h, --help      show this help message and exit\n  --config_path str    Path for a config file to parse with pyrallis (default:\n                  None)\n\nTrainConfig:\n   Training config for Machine Learning\n\n  --workers int   The number of workers for training (default: 8)\n  --exp_name str  The experiment name (default: default_exp)\n```\n\n\n\n### 🐲 2/5 The `pyrallis.wrap` Decorator 🐲\nDon't like the `pyrallis.parse` syntax?\n```python\ndef main():\n    cfg = pyrallis.parse(config_class=TrainConfig)\n    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')\n```\nOne can equivalently use the `pyrallis.wrap` syntax 😎 \n```python\n@pyrallis.wrap()\ndef main(cfg: TrainConfig):\n    # The decorator automagically uses the type hint to parsers arguments into TrainConfig\n    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')\n```\nWe will use this syntax for the rest of our tutorial.\n\n\n### 🐲 3/5 Better Configs Using Inherent `dataclass` Features 🐲\nWhen using a dataclass we can add additional functionality using existing `dataclass` features, such as the `post_init` mechanism or `@properties` :grin:\n```python\nfrom dataclasses import dataclass, field\nfrom pathlib import Path\nfrom typing import Optional\nimport pyrallis\n\n\n@dataclass\nclass TrainConfig:\n    \"\"\" Training config for Machine Learning \"\"\"\n    # The number of workers for training\n    workers: int = field(default=8)\n    # The number of workers for evaluation\n    eval_workers: Optional[int] = field(default=None)\n    # The experiment name\n    exp_name: str = field(default='default_exp')\n    # The experiment root folder path\n    exp_root: Path = field(default=Path('/share/experiments'))\n\n    def __post_init__(self):\n        # A builtin method of dataclasses, used for post-processing our configuration.\n        self.eval_workers = self.eval_workers or self.workers\n\n    @property\n    def exp_dir(self) -\u003e Path:\n        # Properties are great for arguments that can be derived from existing ones\n        return self.exp_root / self.exp_name\n\n\n@pyrallis.wrap()\ndef main(cfg: TrainConfig):\n    print(f'Training {cfg.exp_name}...')\n    print(f'\\tUsing {cfg.workers} workers and {cfg.eval_workers} evaluation workers')\n    print(f'\\tSaving to {cfg.exp_dir}')\n```\n\n```console\n$ python -m train_model.py --exp_name=my_second_exp --workers=42\nTraining my_second_exp...\n    Using 42 workers and 42 evaluation workers\n    Saving to /share/experiments/my_second_exp\n```\n\u003e Notice that in all examples we use the explicit `dataclass.field` syntax. This isn't a requirement of `pyrallis` but rather a style choice. As some of your arguments will probably require `dataclass.field` (mutable types for example) we find it cleaner to always use the same notation.\n\n\n### 🐲 4/5 Building Hierarchical Configurations 🐲\nSometimes configs get too complex for a flat hierarchy 😕, luckily `pyrallis` supports nested dataclasses 💥\n\n```python\n\n@dataclass\nclass ComputeConfig:\n    \"\"\" Config for training resources \"\"\"\n    # The number of workers for training\n    workers: int = field(default=8)\n    # The number of workers for evaluation\n    eval_workers: Optional[int] = field(default=None)\n\n    def __post_init__(self):\n        # A builtin method of dataclasses, used for post-processing our configuration.\n        self.eval_workers = self.eval_workers or self.workers\n\n\n@dataclass\nclass LogConfig:\n    \"\"\" Config for logging arguments \"\"\"\n    # The experiment name\n    exp_name: str = field(default='default_exp')\n    # The experiment root folder path\n    exp_root: Path = field(default=Path('/share/experiments'))\n\n    @property\n    def exp_dir(self) -\u003e Path:\n        # Properties are great for arguments that can be derived from existing ones\n        return self.exp_root / self.exp_name\n\n# TrainConfig will be our main configuration class.\n# Notice that default_factory is the standard way to initialize a class argument in dataclasses\n\n@dataclass\nclass TrainConfig:\n    log: LogConfig = field(default_factory=LogConfig)\n    compute: ComputeConfig = field(default_factory=ComputeConfig)\n\n@pyrallis.wrap()\ndef main(cfg: TrainConfig):\n    print(f'Training {cfg.log.exp_name}...')\n    print(f'\\tUsing {cfg.compute.workers} workers and {cfg.compute.eval_workers} evaluation workers')\n    print(f'\\tSaving to {cfg.log.exp_dir}')\n```\nThe argument parse will be updated accordingly\n```console\n$ python train_model.py --log.exp_name=my_third_exp --compute.eval_workers=2\nTraining my_third_exp...\n    Using 8 workers and 2 evaluation workers\n    Saving to /share/experiments/my_third_exp\n```\n\n### 🐲 5/5 Easy Serialization with `pyrallis.dump` 🐲\nAs your config get longer you will probably want to start working with configuration files. Pyrallis supports encoding a dataclass configuration into a `yaml` file 💾\n\nThe command `pyrallis.dump(cfg, open('run_config.yaml','w'))` will result in the following `yaml` file\n```yaml\ncompute:\n  eval_workers: 2\n  workers: 8\nlog:\n  exp_name: my_third_exp\n  exp_root: /share/experiments\n```\n\u003e `pyrallis.dump` extends `yaml.dump` and uses the same syntax.\n\nConfiguration files can also be loaded back into a dataclass, and can even be used together with the command-line arguments.\n```python\ncfg = pyrallis.parse(config_class=TrainConfig,\n                              config_path='/share/configs/config.yaml')\n\n# or the decorator synrax\n@pyrallis.wrap(config_path='/share/configs/config.yaml')\n\n# or with the CONFIG argument\npython my_script.py --log.exp_name=readme_exp --config_path=/share/configs/config.yaml\n\n# Or if you just want to load from a .yaml without cmd parsing\ncfg = pyrallis.load(TrainConfig, '/share/configs/config.yaml')\n```\n\u003e Command-line arguments have a higher priority and will override the configuration file\n\n\nFinally, one can easily extend the serialization to support new types 🔥\n```python\n# For decoding from cmd/yaml\npyrallis.decode.register(np.ndarray,np.asarray)\n\n# For encoding to yaml \npyrallis.encode.register(np.ndarray, lambda x: x.tolist())\n\n# Or with the wrapper version instead \n@pyrallis.encode.register\ndef encode_array(arr : np.ndarray) -\u003e str:\n    return arr.tolist()\n```\n\n#### 🐲 That's it you are now a `pyrallis` expert! 🐲\n\n\n\n## Why Another Parsing Library?\n\u003cimg src=\"https://imgs.xkcd.com/comics/standards_2x.png\" alt=\"XKCD 927 - Standards\" width=\"70%\" /\u003e\n\n\u003e XKCD 927 - Standards \n\nThe builtin `argparse` has many great features but is somewhat outdated :older_man: with one its greatest weakness being the lack of typing. This has led to the development of many great libraries tackling different weaknesses of `argparse` (shout out for all the great projects out there! You rock! :metal:).  \n\nIn our case, we were looking for a library that would  support the vanilla `dataclass` without requiring dedicated classes, and would have a loading interface from both command-line and files. The closest candidates were `hydra` and `simple-parsing`, but they weren't exactly what we were looking for. Below are the pros and cons from our perspective:\n#### [Hydra](https://github.com/facebookresearch/hydra)\nA framework for elegantly configuring complex applications from Facebook Research.\n* Supports complex configuration from multiple files and allows for overriding them from command-line.\n* Does not support non-standard types, does not play nicely with `datclass.__post_init__`and requires a `ConfigStore` registration.\n#### [SimpleParsing](https://github.com/lebrice/SimpleParsing)\nA framework for simple, elegant and typed Argument Parsing by Fabrice Normandin\n* Strong integration with `argparse`, support for nested configurations together with standard arguments.\n* No support for joint loading from command-line and files, dataclasses are still wrapped by a Namespace, requires dedicated classes for serialization.\n\nWe decided to create a simple hybrid of the two approaches, building from `SimpleParsing` with some `hydra` features in mind. The result, `pyrallis`, is a simple library that that is relatively low on features, but hopefully excels at what it does.\n\nIf `pyrallis` isn't what you're looking for we strongly advise you to give `hydra` and `simpleParsing` a try (where other interesting option include `click`, `ext_argpase`, `jsonargparse`, `datargs` and `tap`). If you do :heart: `pyrallis` then welcome aboard! We're gonna have a great journey together! 🐲\n\n## Tips and Design Choices\n\n### Beware of Mutable Types (or use pyrallis.field)\nDataclasses are great (really!) but using mutable fields can sometimes be confusing. For example, say we try to code the following dataclass\n```python\n@dataclass\nclass OptimConfig:\n    worker_inds: List[int] = []\n    # Or the more explicit version\n    worker_inds: List[int] = field(default=[])\n```\nAs `[]` is mutable we would actually initialize every instance of this dataclass with the same list instance, and thus is not allowed. Instead `dataclasses` would direct you the default_factory function, which calls a factory function for generating the field in every new instance of your dataclass.\n\n```python\nworker_inds: List[int] = field(default_factory=list)\n```\n\nNow, this works great for empty collections, but what would be the alternative for\n```python\nworker_inds: List[int] = field(default=[1,2,3])\n```\nWell, you would have to create a dedicated factory function that regenerates the object, for example\n```python\nworker_inds: List[int] = field(default_factory=lambda : [1,2,3])\n```\nKind of annoying and could be confusing for a new guest reading your code :confused: Now, while this isn't really related to parsing/configuration we decided it could be nice to offer a sugar-syntax for such cases as part of `pyrallis`\n```python\nfrom pyrallis import field\nworker_inds: List[int] = field(default=[1,2,3], is_mutable=True)\n```\nThe `pyrallis.field` behaves like the regular `dataclasses.field` with an additional `is_mutable` flag. When toggled, the `default_factory` is created automatically, offering the same functionally with a more reader-friendly syntax.\n\n\n\n### Uniform Parsing Syntax\nFor parsing files we opted for `yaml` as our format of choice, following `hydra`, due to its concise format. \nNow, let us assume we have the following `.yaml` file which `yaml` successfully handles:\n```yaml\ncompute:\n  worker_inds: [0,2,3]\n```\nIntuitively we would also want users to be able to use the same syntax \n```cmd\npython my_app.py --compute.worker_inds=[0,2,3]\n```\n\nHowever, the more standard syntax for an argparse application would be \n```cmd\npython my_app.py --compute.worker_inds 0 2 3\n```\n\nWe decided to use the same syntax as in the `yaml` files to avoid confusion when loading from multiple sources.\n\nNot a `yaml` fun? `pyrallis` also supports `json` and `toml` formats using `pyrallis.set_config_type('json')` or `with pyrallis.config_type('json'):`\n\n\n\n# TODOs:\n- [x] Fix error with default Dict and List\n\u003e         Underlying error: No decoding function for type ~KT, consider using pyrallis.decode.register\n- [x] Refine the `--help` command\n\u003e For example the `options` argument is confusing there\n- [ ] Add a test to `omit_defaults`\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://about.me/ido.weiss\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/10072365?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eIdo Weiss\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#design-idow09\" title=\"Design\"\u003e🎨\u003c/a\u003e \u003ca href=\"#ideas-idow09\" title=\"Ideas, Planning, \u0026 Feedback\"\u003e🤔\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/yairf11\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/13931256?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eYair Feldman\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#design-yairf11\" title=\"Design\"\u003e🎨\u003c/a\u003e \u003ca href=\"#ideas-yairf11\" title=\"Ideas, Planning, \u0026 Feedback\"\u003e🤔\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feladrich%2Fpyrallis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feladrich%2Fpyrallis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feladrich%2Fpyrallis/lists"}