{"id":13738527,"url":"https://github.com/KevinMusgrave/easy-module-attribute-getter","last_synced_at":"2025-05-08T16:34:29.682Z","repository":{"id":57425176,"uuid":"218787854","full_name":"KevinMusgrave/easy-module-attribute-getter","owner":"KevinMusgrave","description":"Fetch and initialize objects in one line, without any if-statements or dictionaries. Merge and override complex config options at the command line.","archived":true,"fork":false,"pushed_at":"2021-01-21T03:32:23.000Z","size":138,"stargazers_count":19,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-04-28T08:25:01.125Z","etag":null,"topics":["attributes","clean-code","module","python","yaml"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KevinMusgrave.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-31T14:40:49.000Z","updated_at":"2023-10-20T09:50:06.000Z","dependencies_parsed_at":"2022-09-13T02:40:21.865Z","dependency_job_id":null,"html_url":"https://github.com/KevinMusgrave/easy-module-attribute-getter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinMusgrave%2Feasy-module-attribute-getter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinMusgrave%2Feasy-module-attribute-getter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinMusgrave%2Feasy-module-attribute-getter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinMusgrave%2Feasy-module-attribute-getter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KevinMusgrave","download_url":"https://codeload.github.com/KevinMusgrave/easy-module-attribute-getter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224746856,"owners_count":17363130,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attributes","clean-code","module","python","yaml"],"created_at":"2024-08-03T03:02:25.483Z","updated_at":"2024-11-15T07:31:19.133Z","avatar_url":"https://github.com/KevinMusgrave.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# easy-module-attribute-getter\n\n## Installation\n```\npip install easy-module-attribute-getter\n```\n\n## The Problem: unmaintainable if-statements and dictionaries\nIt's common to specify script parameters in yaml config files. For example:\n```yaml\nmodels:\n  modelA:\n    densenet121:\n      pretrained: True\n      memory_efficient: True\n  modelB:\n    resnext50_32x4d:\n      pretrained: True\n```\nUsually, the config file is loaded and then various if-statements or switches are used to instantiate objects etc. It might look something like this (depending on how the config file is organized):\n```python\nmodels = {}\nfor k in [\"modelA\", \"modelB\"]:\n\tmodel_name = list(args.models[k].keys())[0]\n\tif model_name == \"densenet121\":\n\t  models[k] = torchvision.models.densenet121(**args.models[k][model_name])\n\telif model_name == \"googlenet\":\n\t  models[k] = torchvision.models.googlenet(**args.models[k][model_name])\n\telif model_name == \"resnet50\":\n\t  models[k] = torchvision.models.resnet50(**args.models[k][model_name])\n\telif model_name == \"inception_v3\":\n\t  models[k] = torchvision.models.inception_v3(**args.models[k][model_name])\n\t...\n```\nThis is kind of annoying to do, and every time PyTorch adds new classes or functions that you want access to, you need to add new cases to your giant if-statement. An alternative is to make a dictionary:\n```\nmodel_dict = {\"densenet121\": torchvision.models.densenet121,\n                      \"googlenet\": torchvision.models.googlenet,\n                      \"resnet50\": torchvision.models.resnet50,\n                      \"inception_v3\": torchvision.models.inception_v3\n\t\t      ...}\nmodels = {}\nfor k in [\"modelA\", \"modelB\"]:\n\tmodel_name = list(args.models[k].keys())[0]\n\tmodels[k] = model_dict[model_name](**args.models[k][model_name])\n```\nThis is shorter than the if statement, but still requires you to manually spell out all the keys and classes. And you still have to update it yourself when the package updates.\n\n## The Solution\n### Fetch and initialize multiple models in one line\nWith this package, the above for-loop and if-statements get reduced to this:\n```python\nfrom easy_module_attribute_getter import PytorchGetter\npytorch_getter = PytorchGetter()\nmodels = pytorch_getter.get_multiple(\"model\", args.models)\n```\n\"models\" is a dictionary that maps from strings (\"modelA\" and \"modelB\") to the desired objects, which have already been initialized with the parameters specified in the config file.\n\n### Access multiple modules in one line\nSay you want access to the default package (torchvision.models), as well as the pretrainedmodels package, and two other custom model modules, X and Y. You can register these:\n```python\npytorch_getter.register('model', pretrainedmodels) \npytorch_getter.register('model', X)\npytorch_getter.register('model', Y)\n```\nNow you can still do the 1-liner:\n```python\nmodels = pytorch_getter.get_multiple(\"model\", args.models)\n```\nAnd pytorch_getter will try all 4 registered modules until it gets a match.\n\n### Automatically have yaml access to new classes\nIf you upgrade to a new version of PyTorch which has 20 new classes, you don't have to change anything. You automatically have access to all the new classes, and you can specify them in your yaml file.\n\n### Merge or override complex config options via the command line:\nThe example yaml file contains 'models' which maps to a nested dictionary containing modelA and modelB. It's easy to add another key to models at the command line, using the standard python notation for nested dictionaries.\n```\npython example.py --models {modelC: {googlenet: {pretrained: True}}}\n```\nThen in your script:\n```python\nimport argparse\nyaml_reader = YamlReader(argparse.ArgumentParser())\nargs, _, _ = yaml_reader.load_yamls({\"models\": ['models.yaml'], \"losses\": ['losses.yaml']}, max_merge_depth=float('inf'))\n```\nNow args.models contains 3 models.\n\nIf in general you'd like to merge config options, then in the load_yamls function, set the max_merge_depth argument to the number of sub-dictionaries you'd like the merge to apply to. \n\nWhat if you have max_merge_depth set to 1, but want to do a total override for a particular flag? In that case, just append \\~OVERRIDE\\~ to the flag:\n```\npython example.py --models~OVERRIDE~ {modelC: {googlenet: {pretrained: True}}}\n```\nNow args.models will contain just modelC, even though max_merge_depth is set to 1. \n\n### Load one or multiple yaml files into one args object\n```python\nfrom easy_module_attribute_getter import YamlReader\nyaml_reader = YamlReader()\nargs, _, _ = yaml_reader.load_yamls(['models.yaml'])\n```\nProvide a list of filepaths:\n```python\nargs, _, _ = yaml_reader.load_yamls(['models.yaml', 'optimizers.yaml', 'transforms.yaml'])\n```\nOr provide a root path and a dictionary mapping subfolder names to the bare filename\n```python\nroot_path = \"/where/your/yaml/subfolders/are/\"\nsubfolder_to_name_dict = {\"models\": \"default\", \"optimizers\": \"special_trial\", \"transforms\": \"blah\"}\nargs, _, _ = yaml_reader.load_yamls(root_path=root_path, subfolder_to_name_dict=subfolder_to_name_dict)\n```\n\n## Pytorch-specific features\n### Transforms\nSpecify transforms in your config file:\n```yaml\ntransforms:\n  train:\n    Resize:\n      size: 256\n    RandomResizedCrop:\n      scale: 0.16 1\n      ratio: 0.75 1.33\n      size: 227\n    RandomHorizontalFlip:\n      p: 0.5\n\n  eval:\n    Resize:\n      size: 256\n    CenterCrop:\n      size: 227\n```\nThen load composed transforms in your script:\n```python\ntransforms = {}\nfor k, v in args.transforms.items():\n    transforms[k] = pytorch_getter.get_composed_img_transform(v, mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225])\n```\nThe transforms dict now contains:\n```python\n{'train': Compose(\n    Resize(size=256, interpolation=PIL.Image.BILINEAR)\n    RandomResizedCrop(size=(227, 227), scale=(0.16, 1), ratio=(0.75, 1.33), interpolation=PIL.Image.BILINEAR)\n    RandomHorizontalFlip(p=0.5)\n    ToTensor()\n    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n), 'eval': Compose(\n    Resize(size=256, interpolation=PIL.Image.BILINEAR)\n    CenterCrop(size=(227, 227))\n    ToTensor()\n    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n)}\n```\n\n\n### Optimizers, schedulers, and gradient clippers\nOptionally specify the scheduler and gradient clipping norm, within the optimizer parameters.\n\nThe scheduler keys should be one of ```scheduler_by_epoch```, ```scheduler_by_iteration```, and ```scheduler_by_plateau```.\n\n```yaml\noptimizers:\n  modelA:\n    Adam:\n      lr: 0.00001\n      weight_decay: 0.00005\n    scheduler_by_epoch:\n      StepLR:\n        step_size: 2\n        gamma: 0.95\n    scheduler_by_iteration:\n      ExponentialLR:\n        gamma: 0.99\n    clip_grad_norm: 1\n  modelB:\n    Adam:\n      lr: 0.00001\n      weight_decay: 0.00005\n```\nCreate the optimizers:\n```python\noptimizers = {}\nschedulers = {}\ngrad_clippers = {}\nfor k, v in models.items():\n\toptimizers[k], schedulers[k], grad_clippers[k] = pytorch_getter.get_optimizer(v, yaml_dict=args.optimizers[k])\n```\n\n## Not just for PyTorch\nNote that the YamlReader and EasyModuleAttributeGetter classes are totally independent of PyTorch. I wrote the child class PyTorchGetter since that's what I'm using this package for, but the other two classes can be used in general cases and extended for your own purpose.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKevinMusgrave%2Feasy-module-attribute-getter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKevinMusgrave%2Feasy-module-attribute-getter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKevinMusgrave%2Feasy-module-attribute-getter/lists"}