{"id":13633375,"url":"https://github.com/joeddav/devol","last_synced_at":"2025-04-18T10:34:36.782Z","repository":{"id":74392434,"uuid":"81520737","full_name":"joeddav/devol","owner":"joeddav","description":"Genetic neural architecture search with Keras","archived":false,"fork":false,"pushed_at":"2023-05-25T14:45:47.000Z","size":85,"stargazers_count":950,"open_issues_count":7,"forks_count":112,"subscribers_count":43,"default_branch":"master","last_synced_at":"2025-04-18T03:27:19.313Z","etag":null,"topics":["automl","computer-vision","deep-learning","genetic-algorithm","keras","machine-learning","neural-architecture-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joeddav.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-02-10T03:07:54.000Z","updated_at":"2025-02-26T16:04:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"e369edf8-4fdb-46ec-9caf-45b6499582e3","html_url":"https://github.com/joeddav/devol","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeddav%2Fdevol","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeddav%2Fdevol/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeddav%2Fdevol/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joeddav%2Fdevol/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joeddav","download_url":"https://codeload.github.com/joeddav/devol/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249479056,"owners_count":21279187,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automl","computer-vision","deep-learning","genetic-algorithm","keras","machine-learning","neural-architecture-search"],"created_at":"2024-08-01T23:00:35.858Z","updated_at":"2025-04-18T10:34:36.483Z","avatar_url":"https://github.com/joeddav.png","language":"Python","readme":"# DEvol - Deep Neural Network Evolution\n\nDEvol (DeepEvolution) is a basic proof of concept for genetic architecture\nsearch in Keras. The current setup is designed for classification problems,\nthough this could be extended to include any other output type as well.\n\nSee `example/demo.ipynb` for a simple example.\n\n## Evolution\n\nEach model is represented as fixed-width genome encoding information about the\nnetwork's structure. In the current setup, a model contains a number of\nconvolutional layers, a number of dense layers, and an optimizer. The\nconvolutional layers can be evolved to include varying numbers of feature maps,\ndifferent activation functions, varying proportions of dropout, and whether to\nperform batch normalization and/or max pooling. The same options are available\nfor the dense layers with the exception of max pooling. The complexity of these\nmodels could easily be extended beyond these capabilities to include any\nparameters included in Keras, allowing the creation of more complex\narchitectures.\n\nBelow is a highly simplified visualization of how genetic crossover might take\nplace between two models.\n\n\u003cimg width=\"75%\" src=\"https://preview.ibb.co/gdMDak/crossover.png\"\u003e\n\u003ci\u003eGenetic crossover and mutation of neural networks\u003c/i\u003e\n\n## Results\n\nFor demonstration, we ran our program on the MNIST dataset (see `demo.ipynb` for\nan example setup) with 20 generations and a population size of 50. We allowed\nthe model up to 6 convolutional layers and 4 dense layers (including the softmax\nlayer). The best accuracy we attained with 10 epochs of training under these\nconstraints was 99.4%, which is higher than we were able to achieve when\nmanually constructing our own models under the same constraints. The graphic\nbelow displays the running maximum accuracy for all 1000 nets as they evolve\nover 20 generations.\n\nKeep in mind that these results are obtained with simple, relatively shallow\nneural networks with no data augmentation, transfer learning, ensembling,\nfine-tuning, or other optimization techniques. However, virtually any of these\nmethods could be incorporated into the genetic program. \n\n\u003cimg width=\"75%\" src=\"https://preview.ibb.co/i4BDak/running_max.png\"\u003e\n\u003ci\u003eRunning max of MNIST accuracies across 20 generations\u003c/i\u003e\n\n## Application\n\nThe most significant barrier in using DEvol on a real problem is the complexity\nof the algorithm. Because training neural networks is often such a\ncomputationally expensive process, training hundreds or thousands of different\nmodels to evaluate the fitness of each is not always feasible. Below are some\napproaches to combat this issue:\n\n- **Parallel Training** - The nature of evaluating the fitness of multiple\n  members of a population simultaneously is *embarassingly parallel*. A task\n  like this would be trivial to distribute among many GPUs and even machines.\n- **Early Stopping** - There's no need to train a model for 10 epochs if it\n  stops improving after 3; cut it off early.\n- **Train on Fewer Epochs** - Training in a genetic program serves one purpose:\n  to evaluate a model's fitness in relation to other models. It may not be\n  necessary to train to convergence to make this comparison; you may only need 2\n  or 3 epochs. However, it is important you exercise caution in decreasing\n  training time because doing so could create evolutionary pressure toward\n  simpler models that converge quickly. This creates a trade-off between\n  training time and accuracy which, depending on the application, may or may not\n  be desirable. \n- **Parameter Selection** - The more robust you allow your models to be, the\n  longer it will take to converge; i.e., don't allow horizontal flipping on a\n  character recognition problem even though the genetic program will eventually\n  learn not to include it. The less space the program has to explore, the faster\n  it will arrive at an optimal solution. \n\nFor some problems, it may be ideal to simply plug the data into DEvol and let\nthe program build a complete model for you, but for others, this hands-off\napproach may not be feasible. In either case, DEvol could give you insights into\noptimal model design that you may not have considered on your own. For the MNIST\ndigit classification problem, we found that ReLU does far better than a sigmoid\nfunction in convolutional layers, but they work about equally well in dense\nlayers. We also found that ADAGRAD was the highest-performing prebuilt optimizer\nand gained insight on the number of nodes to include in each dense layer. \n\nAt worst, DEvol could give you insight into improving your model architecture.\nAt best, it could give you a beautiful, finely-tuned model. \n\n## Wanna Try It?\n\n_Note: DEvol was created as an experiment and an interesting proof of concept.\nIt has been organized into a Python package for others to try, but is not\nactively developed, and should be used only for purposes of experimentation._\n\nTo setup, just clone the repo and run `pip install -e path/to/repo`. You should\nthen be able to access `devol` globally.\n\nDEvol is pretty straightforward to use for basic classification problems. See\n`example/demo.ipynb` for an example. There are three basic steps:\n\n1. **Prep your dataset.** DEvol expects a classification problem with labels\n   that are one-hot encoded as it uses `categorical_crossentropy` for its loss\n   function. Otherwise, you can prep your data however you'd like. Just pass\n   your input shape into `GenomeHandler`.\n2. **Create a `GenomeHandler`.** The `GenomeHandler` defines the constraints\n   that you apply to your models. Specify the maximum number of convolutional\n   and dense layers, the max dense nodes and feature maps, and the input shape.\n   You can also specify whether you'd like to allow batch_normalization,\n   dropout, and max_pooling, which are included by default. You can also pass in\n   a list of optimizers and activation functions you'd like to allow.\n3. **Create and run the DEvol.** Pass your `GenomeHandler` to the `DEvol`\n   constructor, and run it. Here you have a few more options such as the number\n   of generations, the population size, epochs used for fitness evaluation, the\n   evaluation metric to optimize (accuracy or loss) and an (optional) fitness\n   function which converts a model's accuracy or loss into a fitness score.\n\nSee `example/demo.ipynb` for a basic example.\n","funding_links":[],"categories":["AutoML","Profiling","Scheduling","超参数优化和AutoML","Python","神经网络结构搜索_Neural_Architecture_Search","Libraries","Table of Contents","Tools and projects"],"sub_categories":["Profiling","LLM"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoeddav%2Fdevol","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoeddav%2Fdevol","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoeddav%2Fdevol/lists"}