{"id":13738545,"url":"https://github.com/gmontamat/gentun","last_synced_at":"2025-04-15T14:12:11.439Z","repository":{"id":27110795,"uuid":"112526909","full_name":"gmontamat/gentun","owner":"gmontamat","description":"Hyperparameter tuning for machine learning models using a distributed genetic algorithm","archived":false,"fork":false,"pushed_at":"2024-09-25T22:56:46.000Z","size":637,"stargazers_count":88,"open_issues_count":7,"forks_count":19,"subscribers_count":4,"default_branch":"develop","last_synced_at":"2025-04-15T14:11:53.452Z","etag":null,"topics":["convolutional-neural-networks","cross-validation","distributed-algorithm","distributed-genetic-algorithm","gene-encoding","genetic-algorithm","genetic-algorithms","grid-search","hyperparameter-optimization","hyperparameter-tuning","keras","machine-learning","master-worker","scikit-learn","tensorflow","xgboost"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/gentun/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gmontamat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-29T20:56:01.000Z","updated_at":"2025-03-10T21:59:13.000Z","dependencies_parsed_at":"2024-04-16T22:50:50.341Z","dependency_job_id":"7a28e6d1-4b59-451d-951e-fd58323069a7","html_url":"https://github.com/gmontamat/gentun","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmontamat%2Fgentun","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmontamat%2Fgentun/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmontamat%2Fgentun/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gmontamat%2Fgentun/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gmontamat","download_url":"https://codeload.github.com/gmontamat/gentun/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249085427,"owners_count":21210267,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convolutional-neural-networks","cross-validation","distributed-algorithm","distributed-genetic-algorithm","gene-encoding","genetic-algorithm","genetic-algorithms","grid-search","hyperparameter-optimization","hyperparameter-tuning","keras","machine-learning","master-worker","scikit-learn","tensorflow","xgboost"],"created_at":"2024-08-03T03:02:26.235Z","updated_at":"2025-04-15T14:12:11.418Z","avatar_url":"https://github.com/gmontamat.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003ca name=\"readme-top\"\u003e\u003c/a\u003e\n\n\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/gmontamat/gentun\"\u003e\n    \u003cimg alt=\"plugin-icon\" src=\"https://github.com/gmontamat/gentun/blob/develop/assets/icon.png?raw=true\"\u003e\n  \u003c/a\u003e\n  \u003ch1 style=\"margin: 0;\" align=\"center\"\u003egentun\u003c/h1\u003e\n  \u003cp\u003e\n    Python package for distributed genetic algorithm-based hyperparameter tuning\n  \u003c/p\u003e\n\u003c/div\u003e\n\n[![PyPI](https://img.shields.io/pypi/v/gentun)](https://pypi.org/project/gentun/)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/gentun)](https://pypi.org/project/gentun/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/gentun)](https://pypi.org/project/gentun/)\n[![PyPI - License](https://img.shields.io/pypi/l/gentun)](https://pypi.org/project/gentun/)\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\n          \u003ca href=\"#single-node\"\u003eSingle Node\u003c/a\u003e\n          \u003cul\u003e\n            \u003cli\u003e\u003ca href=\"#adding-pre-defined-individuals\"\u003eAdding Pre-defined Individuals\u003c/a\u003e\u003c/li\u003e\n            \u003cli\u003e\u003ca href=\"#performing-a-grid-search\"\u003ePerforming a Grid Search\u003c/a\u003e\u003c/li\u003e\n          \u003c/ul\u003e\n        \u003c/li\u003e\n        \u003cli\u003e\n          \u003ca href=\"#multiple-nodes\"\u003eMultiple Nodes\u003c/a\u003e\n          \u003cul\u003e\n            \u003cli\u003e\u003ca href=\"#redis-setup\"\u003eRedis Setup\u003c/a\u003e\u003c/li\u003e\n            \u003cli\u003e\u003ca href=\"#controller-node\"\u003eController Node\u003c/a\u003e\u003c/li\u003e\n            \u003cli\u003e\u003ca href=\"#worker-nodes\"\u003eWorker Nodes\u003c/a\u003e\u003c/li\u003e\n          \u003c/ul\u003e\n        \u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#supported-models\"\u003eSupported Models\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#references\"\u003eReferences\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n## About The Project\n\nThe goal of this project is to create a simple framework\nfor [hyperparameter](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)) tuning of machine learning models,\nlike Neural Networks and Gradient Boosting Trees, using a genetic algorithm. Evaluating the fitness of an individual in\na population requires training a model with a specific set of hyperparameters, which is a time-consuming task. To\naddress this issue, we offer a controller-worker system: multiple workers can perform model training and\ncross-validation of individuals provided by a controller while this controller manages the generation of offspring\nthrough reproduction and mutation.\n\n*\"Parameter tuning is a dark art in machine learning, the optimal parameters of a model can depend on many scenarios.\"*\n~ [XGBoost tutorial](https://xgboost.readthedocs.io/en/stable/tutorials/param_tuning.html) on Parameter Tuning\n\n*\"The number of possible network structures increases exponentially with the number of layers in the network, which\ninspires us to adopt the genetic algorithm to efficiently traverse this large search space.\"* ~\n[Genetic CNN paper](https://arxiv.org/abs/1703.01513)\n\n## Installation\n\n```bash\npip install gentun\n```\n\nSome model handlers require additional libraries. You can also install their dependencies with:\n\n```bash\npip install \"gentun[xgboost]\"  # or \"gentun[tensorflow]\"\n```\n\nTo setup a development environment, run:\n\n```bash\npython -m pip install --upgrade pip\npip install 'flit\u003e=3.8.0'\nflit install --deps develop --extras tensorflow,xgboost\n```\n\n## Usage\n\n### Single Node\n\nThe most basic way to run the algorithm is using a single machine, as shown in the following example where we use it to\nfind the optimal hyperparameters of an [`xgboost`](https://xgboost.readthedocs.io/en/stable/) model. First, we download\na sample dataset:\n\n```python\nfrom sklearn.datasets import load_iris\n\ndata = load_iris()\nx_train = data.data\ny_train = data.target\n```\n\nNext, we need to define the hyperparameters we want to optimize:\n\n```python\nfrom gentun.genes import RandomChoice, RandomLogUniform\n\ngenes = [\n    RandomLogUniform(\"learning_rate\", minimum=0.001, maximum=0.1, base=10),\n    RandomChoice(\"max_depth\", [3, 4, 5, 6, 7, 8, 9, 10]),\n    RandomChoice(\"min_child_weight\", [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]),\n]\n```\n\nWe are using the `gentun.models.xgboost.XGBoost` handler, which performs k-fold cross validation with available train\ndata and returns an average metric over the folds. Thus, we need to define some static parameters which are shared\nacross the population over all generations:\n\n```python\nkwargs = {\n    \"booster\": \"gbtree\",\n    \"objective\": \"multi:softmax\",\n    \"metrics\": \"mlogloss\",  # The metric we want to minimize with the algorithm\n    \"num_class\": 3,\n    \"nfold\": 5,\n    \"num_boost_round\": 5000,\n    \"early_stopping_rounds\": 100,\n}\n```\n\nFinally, we are ready to run our genetic algorithm. `gentun` will check that all the model's required parameters are\npassed either through genes or keyword arguments.\n\n```python\nfrom gentun.algorithms import Tournament\nfrom gentun.models.xgboost import XGBoost\nfrom gentun.populations import Population\n\n# Run the genetic algorithm with a population of 50 for 100 generations\npopulation = Population(genes, XGBoost, 50, x_train, y_train, **kwargs)\nalgorithm = Tournament(population)\nalgorithm.run(100, maximize=False)\n```\n\nAs shown above, when the model and genes are implemented, experimenting with the genetic algorithm is simple. See for\nexample how easily can the Genetic CNN paper\nbe [defined on the MNIST handwritten digits set](examples/geneticcnn_mnist.py).\n\nNote that in genetic algorithms, the *fitness* of an individual is a number to be maximized. By default, this framework\nfollows this convention. Nonetheless, to make the framework more flexible, you can use the `maximize=False` parameter in\n`algorithm.run()` to override this behavior and minimize your fitness metric (e.g. when you want to minimize the loss,\nfor example *rmse* or *binary crossentropy*).\n\n#### Adding Pre-defined Individuals\n\nOftentimes, it's convenient to initialize the genetic algorithm with some known individuals instead of a random\npopulation. You can add custom individuals to the population before running the genetic algorithm if you already have\nan intuition of which hyperparameters work well with your model:\n\n```python\nfrom gentun.models.xgboost import XGBoost\nfrom gentun.populations import Population\n\n\n# Best known parameters\nhyperparams = {\n    \"learning_rate\": 0.1,\n    \"max_depth\": 9,\n    \"min_child_weight\": 1,\n}\n\n# Generate a random population and then add a custom individual\npopulation = Population(genes, XGBoost, 49, x_train, y_train, **kwargs)\npopulation.add_individual(hyperparams)\n```\n\n#### Performing a Grid Search\n\nGrid search is also widely used for hyperparameter optimization. This framework provides `gentun.populations.Grid`,\nwhich can be used to conduct a grid search over a single generation pass. You must use genes which define the `sample()`\nmethod, so that uniformly distributed hyperparameter values are obtained with it.\n\n```python\nfrom gentun.genes import RandomChoice, RandomLogUniform\nfrom gentun.models.xgboost import XGBoost\nfrom gentun.populations import Grid\n\n\ngenes = [\n    RandomLogUniform(\"learning_rate\", minimum=0.001, maximum=0.1, base=10),\n    RandomChoice(\"max_depth\", [3, 4, 5, 6, 7, 8, 9, 10]),\n    RandomChoice(\"min_child_weight\", [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]),\n]\n\ngene_samples = [10, 8, 11]  # How many samples we want to get from each gene\n\n# Generate a grid of individuals\npopulation = Grid(genes, XGBoost, gene_samples, x_train, y_train, **kwargs)\n```\n\nRunning the genetic algorithm on this population for just one generation is equivalent to doing a grid search over 10\n`learning_rate` values, all `max_depth` values between 3 and 10, and all `min_child_weight` values between 0 and 10.\n\n### Multiple Nodes\n\nYou can speed up the genetic algorithm by using several machines to evaluate individuals in parallel. One of node has to\nact as a *controller*, generating populations and running the genetic algorithm. Each time this *controller* node needs\nto evaluate an individual from a population, it will send a request to a job queue that is processed by *workers* which\nreceive the model's hyperparameters and perform model fitting through k-fold cross-validation. The more *workers* you\nrun, the faster the algorithm will evolve each generation.\n\n#### Redis Setup\n\nThe simplest way to start the Redis service that will host the communication queues is through `docker`:\n\n```shell\ndocker run -d --rm --name gentun-redis -p 6379:6379 redis\n```\n\n#### Controller Node\n\nTo run the distributed genetic algorithm, define a `gentun.services.RedisController` and pass it to the `Population`\ninstead of the `x_train` and `y_train` data. When the algorithm needs to evaluate the fittest individual, it will pass\nthe hyperparameters to a job queue in Redis and wait till all the individual's fitness are evaluated by worker\nprocesses. Once this is done, the mutation and reproduction steps are run by the controller and a new generation is\nproduced.\n\n```python\nfrom gentun.models.xgboost import XGBoost\nfrom gentun.services import RedisController\n\ncontroller = RedisController(\"experiment\", host=\"localhost\", port=6379)\n# ... define genes\npopulation = Population(genes, XGBoost, 100, controller=controller, **kwargs)\n# ... run algorithm\n```\n\n#### Worker Nodes\n\nThe worker nodes are defined using the `gentun.services.RedisWorker` class and passing the handler to it. Then, we use\nits `run()` method with train data to begin processing jobs from the queue. You can use as many nodes as desired as long\nas they have network access to the redis server.\n\n```python\nfrom gentun.models.xgboost import XGBoost\nfrom gentun.services import RedisWorker\n\nworker = RedisWorker(\"experiment\", XGBoost, host=\"localhost\", port=6379)\n\n# ... fetch x_train and y_train\nworker.run(x_train, y_train)\n```\n\n## Supported Models\n\nThis project supports hyperparameter tuning for the following models:\n\n- [x] XGBoost regressor and classifier\n- [x] Scikit-learn regressor and classifier\n- [x] [Genetic CNN](https://arxiv.org/pdf/1703.01513.pdf) with Tensorflow\n- [ ] [A Genetic Programming Approach to Designing Convolutional Neural Network Architectures](https://arxiv.org/pdf/1704.00764.pdf)\n\n## Contributing\n\nWe welcome contributions to enhance this library. You can submit your custom subclasses for:\n- [`gentun.models.Handler`](src/gentun/models/base.py#L11-L30)\n- [`gentun.genes.Gene`](src/gentun/genes.py#L11-L47)\n\nOur roadmap includes:\n- Training data sharing between the controller and worker nodes\n- Proof-of-work validation of what worker nodes submit\n\nYou can also help us speed up hyperparameter search by contributing your spare GPU time.\n\nFor more details on how to contribute, please check our [contribution guidelines](.github/CONTRIBUTING.md).\n\n## References\n\n### Genetic Algorithms\n\n* Artificial Intelligence: A Modern Approach. 3rd edition. Section 4.1.4\n* https://github.com/DEAP/deap\n* http://www.theprojectspot.com/tutorial-post/creating-a-genetic-algorithm-for-beginners/3\n\n### XGBoost Parameter Tuning\n\n* http://xgboost.readthedocs.io/en/latest/parameter.html\n* http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html\n* https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/\n\n### Papers\n\n* Lingxi Xie and Alan L. Yuille, [Genetic CNN](https://arxiv.org/abs/1703.01513)\n* Masanori Suganuma, Shinichi Shirakawa, and Tomoharu\n  Nagao, [A Genetic Programming Approach to Designing Convolutional Neural Network Architectures](https://arxiv.org/abs/1704.00764)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgmontamat%2Fgentun","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgmontamat%2Fgentun","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgmontamat%2Fgentun/lists"}