{"id":23068057,"url":"https://github.com/tjyuyao/cfgopt","last_synced_at":"2026-05-08T19:32:13.712Z","repository":{"id":37419447,"uuid":"505663871","full_name":"tjyuyao/cfgopt","owner":"tjyuyao","description":"You only configure your deep learning experiment once with cfgopt.","archived":false,"fork":false,"pushed_at":"2022-08-13T08:45:17.000Z","size":97,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-15T23:51:32.665Z","etag":null,"topics":["config","json","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tjyuyao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-21T02:31:12.000Z","updated_at":"2022-06-22T13:51:28.000Z","dependencies_parsed_at":"2022-07-12T13:00:57.292Z","dependency_job_id":null,"html_url":"https://github.com/tjyuyao/cfgopt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tjyuyao/cfgopt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjyuyao%2Fcfgopt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjyuyao%2Fcfgopt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjyuyao%2Fcfgopt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjyuyao%2Fcfgopt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tjyuyao","download_url":"https://codeload.github.com/tjyuyao/cfgopt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjyuyao%2Fcfgopt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32794612,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T08:22:46.396Z","status":"ssl_error","status_checked_at":"2026-05-08T08:22:45.650Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["config","json","pytorch"],"created_at":"2024-12-16T05:20:02.882Z","updated_at":"2026-05-08T19:32:13.693Z","avatar_url":"https://github.com/tjyuyao.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://github.com/tjyuyao/cfgopt/raw/main/cfgopt.png\" alt=\"Logo\"\u003e\u003c/p\u003e\n\n## Introduction\n\n`cfgopt` is a solution for elegance of configuring complicated deep learning projects. Here, you configure everything only once in json, without need to write configuration and commandline parsing boilerplate code in python anymore. The solution is extremly simple, elegant, and compact. Without any 3rd-party dependency, the core implementation of `cfgopt` is only around 200 LOC, and *will be* (todo) fully tested.\n\n## News\n\n- **2022/06/21** `cfgopt` code released.\n\n## Getting Started\n\n**Installation**\n\n```\npip install cfgopt\n```\n\n**Basic Usage**\n\nGenerally, you should have some json **c**on**f**i**g** files in a folder, and a python program that import `cfgopt` to parse them. The program can have command line **opt**ions that update specific field in the parsed results for this running. This is also why this library is called \"cfgopt\".\n\nTo be concrete, you have two choices using `cfgopt`:\n\n- Use the `cfgopt.parse_configs(cfg_root)` API, just pass in the directory path of all the configuration files, and explore with the results! Or you should read the examples described in the [Features](https://github.com/tjyuyao/cfgopt#features) section.\n- Use `cfgoptrun` command directly or wrap `cfgopt.main()` function, for details please refer to the follow-up [main function](https://github.com/tjyuyao/cfgopt#main-function) section.\n\n## Features\n\n\u003cdetails\u003e\u003csummary\u003e\u003ch3\u003ebasic json binding\u003c/h3\u003e\u003c/summary\u003e\u003cp\u003e\n\n`cfgopt` has a main api `parse_configs(cfg_root)` that accepts the path of *a folder of json files*. All files will be (recursively) load in python and added to a root `dict`, keeping the hierarchy and data types unchanged, with some exceptions described in follow-up sections.\n\n\u003e Feature first added in `v0.1`.\n\n\u003c/p\u003e\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003ch3\u003eblock-level reference\u003c/h3\u003e\u003c/summary\u003e\u003cp\u003e\n\n`cfgopt` release you from repeating yourself with a mountain pile of configuration files by borrowing the concept of `block-level reference and embedding` in many modern note-taking apps such as `Logseq`, or more well-known `hyperlinks` for webpages.\n\n**The `cfg://` format URI:**\n\n`cfgopt` follows and expands string values matching a special syntax `cfg://\u003cfile-path\u003e/\u003cintra-file-uri\u003e` in the configuration files during parsing. This is one of the most repealing feature for `cfgopt`. You can also specify *relative* uri which contains no substring \".json\" in it.\n\n**Example:**\n\n**file structure**\n\n```shell\n.\n└── test_blockref_in_list\n    ├── cfg\n    │   ├── data.json\n    │   └── recipes.json\n    └── test_blockref_in_list.py\n```\n\n**data.json**:\n\n```json\n{\n    \"data1\": {\n        \"meta\": {\n            \"location\": \"/data/1/loc\"\n        }\n    },\n    \"data2\": {\n        \"meta\": {\n            \"location\": \"/data/2/loc\"\n        }\n    }\n}\n```\n\n**recipes.json**:\n\n```json\n{\n    \"recipe1\": {\n        \"use_data\": [\n            \"cfg://data.json/data1\"\n        ]\n    },\n    \"recipe2\": {\n        \"use_data\": [\n            \"cfg://data.json/data1\",\n            \"cfg://data.json/data2\"\n        ]\n    }\n}\n```\n\n**test_blockref_in_list.py**\n```python\nimport cfgopt\n\ndef test_blockref_in_list():\n    cfg = cfgopt.parse_configs(cfg_root='test_blockref_in_list/cfg')\n\n    # following lines are equivalent\n    assert cfg[\"recipes.json\"][\"recipe2\"][\"use_data\"][1][\"meta\"][\"location\"] == \"/data/2/loc\"\n    assert cfg[\"recipes.json\"][\"recipe2/use_data/1/meta/location\"] == \"/data/2/loc\"\n    assert cfg[\"recipes.json/recipe2/use_data/1/meta/location\"] == \"/data/2/loc\"\n```\n\n\u003e `__hostname__` in URI will be translated into what you get by executing `hostname` in shell. Since `v0.5.5`.\n\u003e Relative URI support added in `v0.3.0`.\n\u003e Feature first added in `v0.1`.\n\n\u003c/p\u003e\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003ch3\u003ecommand-line update\u003c/h3\u003e\u003c/summary\u003e\u003cp\u003e\n\n`cfgopt` will automatically parse command line options matching the python regex format `^--(.*?)=(.*)`, and interpret it as an update of the parsed configuration folder. The right hand side of `=` should be valid json, and you might need to take care of shell escaping your special characters.\n\nFor example, you can write `--train.json/max_epochs=100` or `--train.json/max_epochs=\"100\"`, since your shell would escape double-quotes, you will get an integer `100` in both cases.\n\nBut if you write `--train.json/resume=\\\"/path/to/ckpt\\\"` or `--train.json/resume='\"/path/to/ckpt\"'`, you should probably get a string value, which depends on your shell implementation.\n\n\u003e Unmatched uri/keys are treated as errors since `v0.7.0`. Regex format updated as `^--(.*?)=(.*)` to match the first `=` symbol since `v0.7.2`.\n\n\u003e Behavior changed to update existing uri before expansion and then update thereafter, this makes both features, i.e., command-line specifying expansion and assigning values after expansion, work. Since `v0.5.0`.\n\n\u003e Updated regex format from `^--(.*)=(.*)` to `^--(.*\\.json.*)=(.*)` in `v0.2`. But then changed back since `v0.4.3`, be careful that unmatched uri/keys would just be ignored without warning.\n\n\u003e Feature first added in `v0.1`.\n\n\u003c/p\u003e\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003ch3\u003ejson objects inheritance\u003c/h3\u003e\u003c/summary\u003e\u003cp\u003e\n\nUsers can specify a json dict, that contains a `__base__` field, linking to another base json dict objects with the `cfg://` reference format (described in [block-level reference](https://github.com/tjyuyao/cfgopt#block-level-reference)). Then the current dict would inherit the base object, and also has its own values in normal fields. This feature is also critical to eliminate repeating, with which now you can develop multiple simillar configs from some prototypes.\n\n\u003e TODO: an example.\n\n\u003e Bugfix: changed from referencing to deepcopying the base dict in `v0.2`.\n\n\u003e Feature first added in `v0.1`.\n\n\u003c/p\u003e\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003ch3\u003eparse python objects\u003c/h3\u003e\u003c/summary\u003e\u003cp\u003e\n\n`cfgopt` has a extremly flexible feature, that parse an json dict to ALMOST any python objects defined in user's code or any code python can find in `PYTHON_PATH`.\n\nUsers can specify a json dict, that contains `__module__` and `__class__` field. The `__module__` field will be imported by `importlib.import_module()` during parsing, and `__class__` field naming any python `callable` in the imported module will be passed to `functools.partial()` along with other fields as keyword arguments. Finally, the mapped \"dict\" in python would be directly callable to instantiate corresponding class or get result of corresponding functions.\n\n**pseudo-code of parsing:**\n```python\nmodule = importlib.import_module(data[\"__module__\"])\nklass = getattr(module, data[\"__class__\"])\ndata[\"__class__\"] = partial(klass, **{k:v for k, v in data.items() if not k.startswith(\"__\")})\n```\n\n\u003e **NOTE**: The python object should not use VAR_POSITIONAL and POSITIONAL_ONLY arguments, and keywords arguments are always recommended than positional arguments.\n\nBy default, when arguments has nested object that is meant to be constructed by `cfgopt` and you call a higher level object, the nested ones in arguments will be **automatically and recursively instantiated** as long as **all its required arguments are defined**. If you want certain object not to be automatically created, add a `\"__as_type__\": true` field alongside the `__class__` field.\n\nLet us refer to these \"dict\"s with `__class__` and `__module__` fields as **python object builders**. You can call into these builders with extra arguments to instantiate them. You can also pass only partial arguments for multiple times, and it won't actually instantiate until all required arguments are ready, similar to above. This is very handy to pass around the builder everywhere deep into the code to collect scattered arguments. For example define a bachnorm builder, and pass it as argument to your networks, the latter can pass the channel argument when it is figured out from the preceding layer. The builder can be used multiple times.\n\nYou can also create a builder from python code, by using `cfgopt.PartialClass()` function, very similar to standard `functools.partial()`, but again it allows being called multiple times until all required args are given.\n\n\u003e TODO: an example.\n\n\u003e `__as_type__` keyword added in `v0.5.3`.\n\n\u003e Important Feature: `PartialClass` and lazily instantiation since `v0.5.0`.\n\n\u003e Supported auto instantiate of nested python objects since `v0.4.0`.\n\n\u003e API enhance: user now can directly call the mapped \"dict\" object instead of its `__class__` field in `v0.2`.\n\n\u003e Feature first added in `v0.1`.\n\n\u003c/p\u003e\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003ch3\u003emain function\u003c/h3\u003e\u003c/summary\u003e\u003cp\u003e\n\n```python\nimport argparse\nimport cfgopt\n\ndef main():\n    parser = argparse.ArgumentParser()\n    parser.add_argument(\n        \"recipe\",\n        help=\"a main function uri to be execute.\"\n    )\n    parser.add_argument(\n        \"-d\", \"--cfgdir\",\n        default=\"cfg\",\n        help=\"config directory that maps to cfg:// root. (default: `cfg`)\"\n    )\n    args, unknown_args = parser.parse_known_args()\n    cfgs = cfgopt.parse_configs(\n        cfg_root=args.cfgdir,\n        args=unknown_args,\n        args_root=args.recipe,\n    )\n    _main = cfgs[args.recipe]\n    return _main(recursive=False)\n\nif __name__ == \"__main__\":\n    main()\n```\n\nThis example `main()` function accepts the first argument as a previous described `cfg://`-format uri, that parses a python callable function, and call it as the main function. This is also implemented as `cfgopt.main()`, and user can use `cfgoptrun` command direct from shell, or just import this main function and call it in usercode with extra python arguments.\n\n\u003e TODO: an example.\n\n\u003e Behaviour changed: `cfgoptrun` interprets commandline arguments as relative path to `recipe` instead of absolute(full) uri since `v0.6.0`.\n\n\u003e Add `cfgoptrun` command (entrypoint) in `v0.2`.\n\n\u003e Feature first added in `v0.1`.\n\n\u003c/p\u003e\u003c/details\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftjyuyao%2Fcfgopt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftjyuyao%2Fcfgopt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftjyuyao%2Fcfgopt/lists"}