{"id":13935587,"url":"https://github.com/negrinho/deep_architect","last_synced_at":"2025-08-03T01:31:53.302Z","repository":{"id":20435314,"uuid":"199943153","full_name":"negrinho/deep_architect","owner":"negrinho","description":"A general, modular, and programmable architecture search framework","archived":false,"fork":false,"pushed_at":"2023-03-24T23:41:07.000Z","size":1896,"stargazers_count":121,"open_issues_count":4,"forks_count":17,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-11-27T03:34:29.947Z","etag":null,"topics":["architecture-search","auto-ml","automatic-machine-learning","colab","colab-notebook","deep-learning","deep-neural-networks","hyperparameter-optimization","machine-learning","neural-architecture-search","neural-networks","pytorch","tensorflow"],"latest_commit_sha":null,"homepage":"https://deep-architect.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/negrinho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-31T23:36:31.000Z","updated_at":"2024-10-23T11:19:28.000Z","dependencies_parsed_at":"2024-11-27T03:41:01.733Z","dependency_job_id":null,"html_url":"https://github.com/negrinho/deep_architect","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/negrinho%2Fdeep_architect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/negrinho%2Fdeep_architect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/negrinho%2Fdeep_architect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/negrinho%2Fdeep_architect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/negrinho","download_url":"https://codeload.github.com/negrinho/deep_architect/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228511875,"owners_count":17931965,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["architecture-search","auto-ml","automatic-machine-learning","colab","colab-notebook","deep-learning","deep-neural-networks","hyperparameter-optimization","machine-learning","neural-architecture-search","neural-networks","pytorch","tensorflow"],"created_at":"2024-08-07T23:01:54.644Z","updated_at":"2024-12-06T18:34:19.205Z","avatar_url":"https://github.com/negrinho.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Overview\n\n**[[CODE](https://github.com/negrinho/deep_architect)]**\n**[[DOCUMENTATION](https://deep-architect.readthedocs.io/en/latest/)]**\n**[[PAPER](https://arxiv.org/abs/1909.13404)]**\n**[[BLOG POST](https://negrinho.github.io/2019/07/26/introducing-deep-architect.html)]**\n**[[GOOGLE GROUP](https://groups.google.com/forum/#!forum/deep_architect)]**\n**[[COLAB](https://colab.research.google.com/drive/14xZdGB-BbIivdgNje3Gu7ZmK0m4F02kE)]**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"400\" height=\"400\" src=\"docs/source/logo/logo.svg\"\u003e\n\u003c/p\u003e\n\n*DeepArchitect: Architecture search so easy you'll think it's magic!*\n\nCheck [colab](https://colab.research.google.com/drive/14xZdGB-BbIivdgNje3Gu7ZmK0m4F02kE) to play around with it and run examples.\n\n## Introduction\n\nDeepArchitect is a framework for automatically searching over computational graphs in arbitrary domains, designed with a focus on **modularity**, **ease of use**, **reusability**, and **extensibility**.\nDeepArchitect has the following **main components**:\n\n* a language for writing composable and expressive search spaces over computational\ngraphs in arbitrary domains (e.g., Tensorflow, Keras, Pytorch, and even non\ndeep learning frameworks such as scikit-learn and preprocessing pipelines);\n* search algorithms that can be used for arbitrary search spaces;\n* logging functionality to easily track search results;\n* visualization functionality to explore search results.\n\n\u003c!-- what is there in store for both researchers and practicioners --\u003e\nFor researchers, DeepArchitect aims to make architecture search research more reusable and reproducible by providing them with a modular framework that they can use to implement new search algorithms and new search spaces while reusing code.\nFor practitioners, DeepArchitect aims to augment their workflow by providing them with a tool to easily write search spaces encoding a large number of design choices and use search algorithms to automatically find good architectures.\n\n## Installation\n\nWe recommend playing with the code on [Colab](https://colab.research.google.com/drive/14xZdGB-BbIivdgNje3Gu7ZmK0m4F02kE) first.\n\nFor a local installation, run the following code snippet:\n\n```\ngit clone git@github.com:negrinho/deep_architect.git deep_architect\ncd deep_architect\nconda create --name deep_architect python=3.6\nconda activate deep_architect\npip install -e .\n```\n\nRun one of the examples to check for correctness, e.g., `python examples/framework_starters/main_keras.py` or `python examples/mnist_with_logging/main.py --config_filepath examples/mnist_with_logging/configs/debug.json`.\n\nWe have included [utils.sh](https://github.com/negrinho/deep_architect/blob/master/utils.sh) with useful development functionality, e.g., to build documentation, extract code snippets from documentation, and build Singularity containers.\n\n## A minimal DeepArchitect example with Keras\n\nWe adapt [this Keras example](https://github.com/keras-team/keras/blob/master/examples/mnist_cnn.py) by defining a search space of models and sampling a random model from it.\nThe original example has a single fixed three-layer neural network with ReLU activations in the hidden layers and dropout with rate equal to *0.2*.\nWe construct a search space by relaxing the number of layers that the network can have, choosing between sigmoid and ReLU activations, and the number of units for each dense layer.\nCheck this search space below:\n\n```python\nimport keras\nfrom keras.datasets import mnist\nfrom keras.models import Model\nfrom keras.layers import Dense, Dropout, Input\nfrom keras.optimizers import RMSprop\n\nimport deep_architect.helpers.keras_support as hke\nimport deep_architect.modules as mo\nimport deep_architect.hyperparameters as hp\nimport deep_architect.core as co\nimport deep_architect.visualization as vi\nfrom deep_architect.searchers.common import random_specify\n\nbatch_size = 128\nnum_classes = 10\nepochs = 20\n\n# the data, split between train and test sets\n(x_train, y_train), (x_test, y_test) = mnist.load_data()\n\nx_train = x_train.reshape(60000, 784)\nx_test = x_test.reshape(10000, 784)\nx_train = x_train.astype('float32')\nx_test = x_test.astype('float32')\nx_train /= 255\nx_test /= 255\nprint(x_train.shape[0], 'train samples')\nprint(x_test.shape[0], 'test samples')\n\n# convert class vectors to binary class matrices\ny_train = keras.utils.to_categorical(y_train, num_classes)\ny_test = keras.utils.to_categorical(y_test, num_classes)\n\n# model = Sequential()\n# model.add(Dense(512, activation='relu', input_shape=(784,)))\n# model.add(Dropout(0.2))\n# model.add(Dense(512, activation='relu'))\n# model.add(Dropout(0.2))\n# model.add(Dense(num_classes, activation='softmax'))\n\nD = hp.Discrete\n\n\ndef dense(h_units, h_activation):\n    return hke.siso_keras_module_from_keras_layer_fn(Dense, {\n        'units': h_units,\n        'activation': h_activation\n    })\n\n\ndef dropout(h_rate):\n    return hke.siso_keras_module_from_keras_layer_fn(Dropout, {'rate': h_rate})\n\n\ndef cell(h_units, h_activation, h_rate, h_opt_drop):\n    return mo.siso_sequential([\n        dense(h_units, h_activation),\n        mo.siso_optional(lambda: dropout(h_rate), h_opt_drop)\n    ])\n\n\ndef model_search_space():\n    h_activation = D(['relu', 'sigmoid'])\n    h_rate = D([0.0, 0.25, 0.5])\n    h_num_repeats = D([1, 2, 4])\n    return mo.siso_sequential([\n        mo.siso_repeat(\n            lambda: cell(\n                D([256, 512, 1024]), h_activation, D([0.2, 0.5, 0.7]), D([0, 1])\n            ), h_num_repeats),\n        dense(D([num_classes]), D(['softmax']))\n    ])\n\n\n(inputs, outputs) = mo.SearchSpaceFactory(model_search_space).get_search_space()\nrandom_specify(outputs)\ninputs_val = Input((784,))\nco.forward({inputs[\"in\"]: inputs_val})\noutputs_val = outputs[\"out\"].val\nvi.draw_graph(outputs, draw_module_hyperparameter_info=False)\nmodel = Model(inputs=inputs_val, outputs=outputs_val)\nmodel.summary()\n\nmodel.compile(\n    loss='categorical_crossentropy', optimizer=RMSprop(), metrics=['accuracy'])\n\nhistory = model.fit(\n    x_train,\n    y_train,\n    batch_size=batch_size,\n    epochs=epochs,\n    verbose=1,\n    validation_data=(x_test, y_test))\nscore = model.evaluate(x_test, y_test, verbose=0)\nprint('Test loss:', score[0])\n```\n\n\u003c!-- comments on the example. --\u003e\nThis example shows how to introduce minimal architecture search capabilities given an existing Keras example.\nOur search space encodes that our network will be composed of a sequence of *1*, *2*, or *4* cells, followed by a final dense module that outputs probabilities over classes.\nEach cell is a sub-search space (underlining the modularity and composability of DeepArchitect).\nThe choice of the type of activation for the dense layer in the cell search space is shared among all cell search spaces used.\nAll other hyperparameters of the cell search space are chosen independently for each occurrence of the cell search space in the sequence.\n\nThe original single Keras model is commented out in the code above to emphasize how little code is needed to support a nontrivial search space.\nWe encourage the reader to think about supporting the same search space using existing hyperparameter optimization tools or in an ad-hoc manner (e.g. how much code would be necessary to encode the search space and sample a random architecture from it).\n\n\u003c!-- suggestions on going forward. --\u003e\nThe [tutorials](https://deep-architect.readthedocs.io/en/latest/tutorials.html) and [examples](https://github.com/negrinho/deep_architect/tree/master/examples) cover additional aspects of DeepArchitect not shown in the code above.\n[This](https://github.com/negrinho/deep_architect/blob/master/examples/mnist_with_logging/main.py) is a slightly more complex example using searchers and logging.\n[These](https://github.com/negrinho/deep_architect/tree/master/examples/framework_starters) are minimal architecture search examples in DeepArchitect across deep learning frameworks.\nThey should be straightforward to adapt for your use cases.\n\n## Framework components\n\nThe main concepts in DeepArchitect are:\n\n* **Search spaces**: Search spaces are constructed by arranging modules (both basic and substitution) and hyperparameters (independent and dependent).\nModules are composed of inputs, outputs, and hyperparameters.\nA search space is passed around as a dictionary of inputs and a dictionary of outputs, allowing us to seamlessly deal with search spaces with multiple modules.\nSubstitution modules rely heavily on delayed evaluation.\nSearch space transitions result from value assignments to independent hyperparameters.\nRelevant code references to read for these ideas are [deep_architect/core.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/core.py) and [deep_architect/modules.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/modules.py).\n\n* **Searchers**:  Searchers interact with search spaces through a simple API.\nA searcher samples a model from the search space by assigning values to each independent hyperparameter, until there are no unassigned independent hyperparameters left.\nA searcher is instantiated with a search space.\nThe base API for the searcher has two methods `sample`, which samples an architecture from the search space, and `update`, which takes the results for a sampled architecture and updates the state of the searcher.\nExamples for the searcher API can be found at [deep_architect/searchers/common.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/searchers/common.py), [deep_architect/searchers/random.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/searchers/random.py), and [deep_architect/searchers/smbo.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/searchers/smbo.py).\nIt is also worth to look at [deep_architect/core.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/core.py) and for the traversal functionality to iterate over the independent hyperparameters in the search space.\n\n* **Evaluators**:  Evaluators take a sampled architecture from the search space and compute performance metrics for it. Evaluators often have a single method named `eval` that takes an architecture and returns a dictionary with evaluation results. In the simplest case, there is a single performance metric of interest (e.g., validation accuracy).\nSee [here](https://github.com/negrinho/deep_architect/blob/master/deep_architect/contrib/misc/evaluators/tensorflow/classification.py) for an example implementation of an evaluator.\n\n* **Logging**:  When we run an architecture search workload, we evaluate multiple architectures in the search space.\nWe maintain a folder per evaluation to keep track of the generated results (e.g., validation accuracy, number of parameters, example predictions, and model checkpoints).\nCode for logging can be found in [deep_architect/search_logging.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/search_logging.py).\nA simple example using logging is found [here](https://github.com/negrinho/deep_architect/blob/master/examples/mnist_with_logging/main.py).\n\n* **Visualization**:  Visualization allows us to inspect the structure of a search space and to visualize search space transitions.\nThese visualizations can be useful for debugging, e.g., checking if a search space was correctly encoded.\nThere are also visualizations to calibrate the necessary evaluation effort to recover the correct performance ordering for architectures in the search space, e.g., how many epochs do we need to invest to identify the best architecture (e.g., lies in the top 5).\nCode for visualization can be found in [deep_architect/visualization.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/visualization.py).\n\n## Main folder structure\n\nThe most important source files live in the [deep_architect folder](https://github.com/negrinho/deep_architect/tree/master/deep_architect).\nThe tutorials cover much of the information needed to extend the framework.\nSee below for a high-level tour of the repo.\n\n* [core.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/core.py): Most important classes to define search spaces.\n* [hyperparameters.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/hyperparameters.py): Basic hyperparameters and auxiliary hyperparameter sharer class.\n* [modules.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/modules.py): Definition of substitution modules along with auxiliary functionality to connect modules or construct larger search spaces from simpler search spaces.\n* [search_logging.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/search_logging.py): Functionality to keep track of the results of the architecture search workload, allowing us to maintain structured folders for each search experiment.\n* [utils.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/utils.py): Utility functions not directly related to architecture search, but useful in many related contexts such as logging and visualization.\n* [visualization.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/visualization.py): Simple visualizations to inspect search spaces as graphs or sequences of graphs.\n\nThere are also a few folders in the deep_architect folder.\n\n\u003c!-- * [communicators](https://github.com/negrinho/deep_architect/tree/master/deep_architect/communicators): Simple functionality to communicate between master and worker processes to relay the evaluation of an architecture and retrieve the results once finished. --\u003e\n* [contrib](https://github.com/negrinho/deep_architect/tree/master/deep_architect/contrib): Useful code that may or may not be maintained over time. Contributions by the community will live in this folder. See [here](https://github.com/negrinho/deep_architect/blob/master/CONTRIBUTING.md) for an in-depth explanation for the rationale behind the project organization and the contrib folder.\n* [helpers](https://github.com/negrinho/deep_architect/tree/master/deep_architect/helpers): Helpers for the current frameworks that we support. This allows us to take the base functionality defined in [core.py](https://github.com/negrinho/deep_architect/blob/master/deep_architect/core.py) and expand it to provide compilation functionality for computational graphs across frameworks. It should be instructive to compare support for different frameworks. One file per framework.\n* [searchers](https://github.com/negrinho/deep_architect/tree/master/deep_architect/searchers): Searchers that can be used for search spaces defined in DeepArchitect. One searcher per file.\n* [surrogates](https://github.com/negrinho/deep_architect/tree/master/deep_architect/surrogates): Surrogate functions over architectures in the search space. searchers based on sequential model based optimization are used frequently in DeepArchitect.\n\n## Roadmap for the future\n\nThe community will have a fundamental role in extending DeepAchitect.\nFor example, authors of existing architecture search algorithms can reimplement them in DeepArchitect, allowing the community to use them widely.\nThis sole fact will allow progress on architecture search to be measured more reliably.\nNew search spaces for new tasks can be implemented, allowing users to use them (either directly or in the construction of new search spaces) in their experiments.\nNew evaluators and visualizations can be implemented.\n\u003c!-- The ability to  may be added to easily construct ensembles from the many models that were explored as a result of the architecture search workload. --\u003e\nWilling contributors should reach out and check the [contributing guide](https://github.com/negrinho/deep_architect/blob/master/CONTRIBUTING.md).\nWe expect to continue extending and maintaining the DeepArchitect and use it for our research.\n\n## Reaching out\n\nYou can reach me at negrinho@cs.cmu.edu or at [\\@rmpnegrinho](https://twitter.com/rmpnegrinho).\nIf you tweet about DeepArchitect, please use the tag `#DeepArchitect` and/or mention me ([\\@rmpnegrinho](https://twitter.com/rmpnegrinho)) in the tweet.\nFor bug reports, questions, and suggestions, use [Github issues](https://github.com/negrinho/deep_architect/issues).\nUse the [Google group](https://groups.google.com/forum/#!forum/deep_architect) for more casual usage questions.\n\n## License\n\nDeepArchitect is licensed under the MIT license as found [here](https://github.com/negrinho/deep_architect/blob/master/LICENSE.md).\nContributors agree to license their contributions under the MIT license.\n\n## Contributors and acknowledgments\n\nThe lead researcher for DeepArchitect is [Renato Negrinho](https://www.cs.cmu.edu/~negrinho/).\n[Daniel Ferreira](https://github.com/dcferreira) played an important initial role in designing APIs through discussions and contributions.\nThis work benefited immensely from the involvement and contributions of talented CMU undergraduate students ([Darshan Patil](https://github.com/dapatil211), [Max Le](https://github.com/lethenghia18), [Kirielle Singajarah](https://github.com/ksingarajah), [Zejie Ai](https://github.com/aizjForever), [Yiming Zhao](https://github.com/startrails98), [Emilio Arroyo-Fang](https://github.com/fizzxed)).\nThis work benefited greatly from discussions with faculty (Geoff Gordon, Matt Gormley, Graham Neubig, Carolyn Rose, Ruslan Salakhutdinov, Eric Xing, and Xue Liu), and fellow PhD students (Zhiting Hu, Willie Neiswanger, Christoph Dann, and Matt Barnes).\nThis work was partially done while Renato Negrinho was a research scientist at [Petuum](https://petuum.com).\nThis work was partially supported by NSF grant IIS 1822831.\nWe thank a generous GCP grant for both CPU and TPU compute.\n\n## References\n\nIf you use this work, please cite:\n\n```\n@article{negrinho2017deeparchitect,\n  title={Deeparchitect: Automatically designing and training deep architectures},\n  author={Negrinho, Renato and Gordon, Geoff},\n  journal={arXiv preprint arXiv:1704.08792},\n  year={2017}\n}\n\n@article{negrinho2019towards,\n  title={Towards modular and programmable architecture search},\n  author={Negrinho, Renato and Patil, Darshan and Le, Nghia and Ferreira, Daniel and Gormley, Matthew and Gordon, Geoffrey},\n  journal={Neural Information Processing Systems},\n  year={2019}\n}\n```\n\nThe code for `negrinho2017deeparchitect` can be found [here](https://github.com/negrinho/deep_architect_legacy).\nThe ideas and implementation of `negrinho2017deeparchitect` evolved into the work of `negrinho2019towards`, found in this repo. See the [paper](https://arxiv.org/abs/1909.13404), [documentation](https://deep-architect.readthedocs.io/en/latest/), and [blog post](https://negrinho.github.io/2019/07/26/introducing-deep-architect.html). The code for the experiments reported in `negrinho2019towards` can be found [here](https://github.com/negrinho/negrinho2019towards), but it will not be actively maintained. For your work, please build on top of the `deep_architect` repo instead.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnegrinho%2Fdeep_architect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnegrinho%2Fdeep_architect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnegrinho%2Fdeep_architect/lists"}