{"id":15065024,"url":"https://github.com/williamfalcon/test-tube","last_synced_at":"2026-02-23T23:18:44.669Z","repository":{"id":54985238,"uuid":"102551895","full_name":"williamFalcon/test-tube","owner":"williamFalcon","description":"Python library to easily log experiments and parallelize hyperparameter search for neural networks","archived":false,"fork":false,"pushed_at":"2022-07-22T06:10:37.000Z","size":1517,"stargazers_count":735,"open_issues_count":27,"forks_count":74,"subscribers_count":25,"default_branch":"master","last_synced_at":"2024-10-30T02:36:53.262Z","etag":null,"topics":["caffe","caffe2","chainer","data-science","deep-learning","grid-search","hyperparameter-optimization","keras","machine-learning","neural-networks","pytorch","random-search","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/williamFalcon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-06T02:14:57.000Z","updated_at":"2024-09-07T09:31:20.000Z","dependencies_parsed_at":"2022-08-14T08:10:43.123Z","dependency_job_id":null,"html_url":"https://github.com/williamFalcon/test-tube","commit_stats":null,"previous_names":["williamfalcon/test_tube"],"tags_count":127,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamFalcon%2Ftest-tube","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamFalcon%2Ftest-tube/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamFalcon%2Ftest-tube/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamFalcon%2Ftest-tube/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/williamFalcon","download_url":"https://codeload.github.com/williamFalcon/test-tube/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247135141,"owners_count":20889420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["caffe","caffe2","chainer","data-science","deep-learning","grid-search","hyperparameter-optimization","keras","machine-learning","neural-networks","pytorch","random-search","tensorflow"],"created_at":"2024-09-25T00:29:38.076Z","updated_at":"2026-02-23T23:18:44.642Z","avatar_url":"https://github.com/williamFalcon.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://williamfalcon.github.io/test-tube/\"\u003e\n    \u003cimg alt=\"react-router\" src=\"https://raw.githubusercontent.com/williamfalcon/test-tube/master/imgs/test_tube_logo.png\" width=\"50\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003ch3 align=\"center\"\u003e\n  Test Tube\n\u003c/h3\u003e\n\u003cp align=\"center\"\u003e\n  Log, organize and parallelize hyperparameter search for Deep Learning experiments\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://badge.fury.io/py/test-tube\"\u003e\u003cimg src=\"https://badge.fury.io/py/test-tube.svg\" alt=\"PyPI version\" height=\"18\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://travis-ci.org/williamFalcon/test-tube\"\u003e\u003cimg src=\"https://travis-ci.org/williamFalcon/test-tube.svg?branch=master\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://williamfalcon.github.io/test-tube/\"\u003e\u003cimg src=\"https://readthedocs.org/projects/test-tube/badge/?version=latest\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/williamFalcon/test-tube/blob/master/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\"\u003e\u003c/a\u003e\n\u003c/p\u003e   \n\n## Docs\n\n**[View the docs here](https://williamfalcon.github.io/test-tube/)**\n\n---   \n\nTest tube is a python library to track and parallelize hyperparameter\nsearch for Deep Learning and ML experiments. It's framework agnostic and\nbuilt on top of the python argparse API for ease of use.\n\n``` {.bash}\npip install test_tube\n```\n\n---   \n\n### Main test-tube uses\n\n-   [Parallelize hyperparameter\n    optimization](https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/)\n    (across multiple gpus or cpus).\n-   [Parallelize hyperparameter\n    optimization](https://williamfalcon.github.io/test-tube/hyperparameter_optimization/HyperOptArgumentParser/)\n    across HPC cluster using SLURM.   \n-   Log experiment hyperparameters and experiment data.   \n    [Experiments](https://williamfalcon.github.io/test-tube/experiment_tracking/experiment/)\n    across models.\n-   Visualize with [tensorboard](https://www.tensorflow.org/guide/summaries_and_tensorboard)\n\nCompatible with Python any Python ML library like Tensorflow, Keras, Pytorch, Caffe, Caffe2, Chainer, MXNet, Theano, Scikit-learn   \n\n---   \n### Examples   \nThe Experiment object is a subclass of Pytorch.SummaryWriter.  \n\n**Log and visualize with Tensorboard**     \n\n```{.python}\nfrom test-tube import Experiment\nimport torch\n\nexp = Experiment('/some/path')\nexp.tag({'learning_rate': 0.02, 'layers': 4})    \n\n# exp is superclass of SummaryWriter\nfeatures = torch.Tensor(100, 784)\nwriter.add_embedding(features, metadata=label, label_img=images.unsqueeze(1))\n\n# simulate training\nfor n_iter in range(2000):\n    e.log({'testtt': n_iter * np.sin(n_iter)})\n\n# save and close\nexp.save()\nexp.close()\n```\n\n```{.bash}\npip install tensorflow   \n\ntensorboard --logdir /some/path\n``` \n    \n**Run grid search on SLURM GPU cluster**    \n\n``` {.python}   \nfrom test_tube.hpc import SlurmCluster\n\n# hyperparameters is a test-tube hyper params object\nhyperparams = args.parse()\n\n# init cluster\ncluster = SlurmCluster(\n    hyperparam_optimizer=hyperparams,\n    log_path='/path/to/log/results/to',\n    python_cmd='python3'\n)\n\n# let the cluster know where to email for a change in job status (ie: complete, fail, etc...)\ncluster.notify_job_status(email='some@email.com', on_done=True, on_fail=True)\n\n# set the job options. In this instance, we'll run 20 different models\n# each with its own set of hyperparameters giving each one 1 GPU (ie: taking up 20 GPUs)\ncluster.per_experiment_nb_gpus = 1\ncluster.per_experiment_nb_nodes = 1\n\n# run the models on the cluster\ncluster.optimize_parallel_cluster_gpu(train, nb_trials=20, job_name='first_tt_batch', job_display_name='my_batch')   \n\n# we just ran 20 different hyperparameters on 20 GPUs in the HPC cluster!!    \n```    \n\n**Optimize hyperparameters across GPUs**\n\n``` {.python}\nfrom test_tube import HyperOptArgumentParser\n\n# subclass of argparse\nparser = HyperOptArgumentParser(strategy='random_search')\nparser.add_argument('--learning_rate', default=0.002, type=float, help='the learning rate')\n\n# let's enable optimizing over the number of layers in the network\nparser.opt_list('--nb_layers', default=2, type=int, tunable=True, options=[2, 4, 8])\n\n# and tune the number of units in each layer\nparser.opt_range('--neurons', default=50, type=int, tunable=True, low=100, high=800, nb_samples=10)\n\n# compile (because it's argparse underneath)\nhparams = parser.parse_args()\n\n# optimize across 4 gpus\n# use 2 gpus together and the other two separately\nhparams.optimize_parallel_gpu(MyModel.fit, gpu_ids=['1', '2,3', '0'], nb_trials=192, nb_workers=4)\n```\n\nOr... across CPUs\n\n``` {.python}\nhparams.optimize_parallel_cpu(MyModel.fit, nb_trials=192, nb_workers=12)\n```\n\nYou can also optimize on a *log* scale to allow better search over\nmagnitudes of hyperparameter values, with a chosen base (disabled by\ndefault). Keep in mind that the range you search over must be strictly\npositive.\n\n``` {.python}\nfrom test_tube import HyperOptArgumentParser\n\n# subclass of argparse\nparser = HyperOptArgumentParser(strategy='random_search')\n\n# Randomly searches over the (log-transformed) range [100,800).\n\nparser.opt_range('--neurons', default=50, type=int, tunable=True, low=100, high=800, nb_samples=10, log_base=10)\n\n\n# compile (because it's argparse underneath)\nhparams = parser.parse_args()\n\n# run 20 trials of random search over the hyperparams\nfor hparam_trial in hparams.trials(20):\n    train_network(hparam_trial)\n```\n\n### Convert your argparse params into searchable params by changing 1 line\n\n``` {.python}\nimport argparse\nfrom test_tube import HyperOptArgumentParser\n\n# these lines are equivalent\nparser = argparse.ArgumentParser(description='Process some integers.')\nparser = HyperOptArgumentParser(description='Process some integers.', strategy='grid_search')\n\n# do normal argparse stuff\n...\n```\n\n### Log images inline with metrics\n\n``` {.python}\n# name must have either jpg, png or jpeg in it\nimg = np.imread('a.jpg')\nexp.log('test_jpg': img, 'val_err': 0.2)\n\n# saves image to ../exp/version/media/test_0.jpg\n# csv has file path to that image in that cell\n```\n\n## Demos\n\n-   [Hyperparameter optimization for PyTorch across 20 cluster GPUs](https://github.com/williamFalcon/test-tube/blob/master/examples/pytorch_hpc_example.py)   \n-   [Hyperparameter optimization across 20 cluster CPUs](https://github.com/williamFalcon/test-tube/blob/master/examples/hpc_cpu_example.py)   \n-   [Experiments and hyperparameter optimization for tensorflow across 4 GPUs simultaneously](https://github.com/williamFalcon/test-tube/blob/master/examples/tensorflow_example.py)\n\n## How to contribute\n\nFeel free to fix bugs and make improvements! 1. Check out the [current\nbugs here](https://github.com/williamFalcon/test-tube/issues) or\n[feature\nrequests](https://github.com/williamFalcon/test-tube/projects/1). 2. To\nwork on a bug or feature, head over to our [project\npage](https://github.com/williamFalcon/test-tube/projects/1) and assign\nyourself the bug. 3. We'll add contributor names periodically as people\nimprove the library!\n\n## Bibtex\n\nTo cite the framework use:\n\n    @misc{Falcon2017,\n      author = {Falcon, W.A.},\n      title = {Test Tube},\n      year = {2017},\n      publisher = {GitHub},\n      journal = {GitHub repository},\n      howpublished = {\\url{https://github.com/williamfalcon/test-tube}}\n    }    \n    \n ## License    \n In addition to the terms outlined in the license, this software is U.S. Patent Pending.    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilliamfalcon%2Ftest-tube","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwilliamfalcon%2Ftest-tube","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilliamfalcon%2Ftest-tube/lists"}