https://github.com/adilzouitine/outfit
:dress: Tidy up your machine learning experiments
https://github.com/adilzouitine/outfit
experiments machine-learning
Last synced: about 1 month ago
JSON representation
:dress: Tidy up your machine learning experiments
- Host: GitHub
- URL: https://github.com/adilzouitine/outfit
- Owner: AdilZouitine
- Created: 2019-06-28T20:54:38.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2019-09-05T09:42:49.000Z (over 5 years ago)
- Last Synced: 2025-04-10T07:59:21.702Z (about 1 month ago)
- Topics: experiments, machine-learning
- Language: Jupyter Notebook
- Homepage:
- Size: 75.2 KB
- Stars: 17
- Watchers: 3
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# :dress: Outfit
Outfit is a lightweight library to **tidy up** your machine learning experiments in a simple way.
The idea of Outfit is to store in your `Wardrobe` your parameters, output file, scores and features in order to be able to make a request and find out which are your best experimentation according to a given criterion.
## How install outfit ?
**PyPI**:
```bash
pip install outfit
```
**Dev version**:
```bash
git clone https://github.com/AdilZouitine/outfit
cd outfit
pip install -r requirements.txt
pip install -e .
```
## How outfit works ?
- **Tutorial 1: Build a simple model management pipeline with outfit for a CNN with Pytorch on MNIST dataset** [](https://colab.research.google.com/drive/17vJSQ9WRv1OeagyayXBq_eHs9H_Qg5JP)
```python
import datetime
# Here import all the libraries you need for your experiment
from outfit import Wardrobe, getlog
wardrobe = Wardrobe(db_path='foo/bar/mnist.db')
exp = {
'experiment_name': 'ResNet18',
'comment': 'Use differential learning rate',
'date_experiment': datetime.datetime.now()
}
wardrobe.add_experiment(**exp)
param = {
'dropout': 0.20,
'kernel_size': '3x3',
'conv_block_1_lr': 0.01,
'conv_block_2_lr': 0.001
}
# Create the instance of your model here with your parameters
wardrobe.add_dict_parameter(param)
# Do your training phase here.
output = {'training log': '/result/training_log_resnet18.txt'}
@getlog(filepath=output['training log'])
def train_model(model, loaders, loss, lr_scheduler, n_epoch):
...
output.update({
'tensorboard': '/result/event.tb',
'model': 'diff_lr_resnet18.pth'
})
wardrobe.add_dict_output(output)
score = {
'train acc': 0.96,
'train loss': 0.430,
'val acc': 0.94,
'val loss': 0.460
}
wardrobe.add_dict_score(score)
wardrobe.tidy() # commit your experiment in database
```
```python
# If you want to get the best experiments
for exp in wardrobe.get_best_scores(mode='max',on_score='val acc'):
'''
Verbose is true by default and will print on the console
at each iteration the parameters, output file,
features and scores in a table format.
Also returns in dictionary the parameters, output file, features and scores.
'''
...
```
**Output**:
```
════════════════════
│ TOP 1 EXPERIMENT │
════════════════════
Table : Experiment
╒════╤═════════════════╤═══════════════════╤════════════════════════════════╤════════════════════╕
│ │ id_experiment │ experiment_name │ comment │ date_experiement │
╞════╪═════════════════╪═══════════════════╪════════════════════════════════╪════════════════════╡
│ 0 │ 1 │ ResNet18 │ Use differential learning rate │ │
╘════╧═════════════════╧═══════════════════╧════════════════════════════════╧════════════════════╛
Table : Parameter
╒════╤════════════════╤══════════════════╤═════════════╤══════════════╕
│ │ id_parameter │ parameter_name │ parameter │ experiment │
╞════╪════════════════╪══════════════════╪═════════════╪══════════════╡
│ 0 │ 1 │ dropout │ 0.2 │ 1 │
├────┼────────────────┼──────────────────┼─────────────┼──────────────┤
│ 1 │ 2 │ kernel_size │ 3x3 │ 1 │
├────┼────────────────┼──────────────────┼─────────────┼──────────────┤
│ 2 │ 3 │ conv_block_1_lr │ 0.01 │ 1 │
├────┼────────────────┼──────────────────┼─────────────┼──────────────┤
│ 3 │ 4 │ conv_block_2_lr │ 0.001 │ 1 │
╘════╧════════════════╧══════════════════╧═════════════╧══════════════╛
Table : Output
╒════╤═════════════╤═══════════════╤═══════════════════════════════════╤══════════════╕
│ │ id_output │ type_output │ path_output │ experiment │
╞════╪═════════════╪═══════════════╪═══════════════════════════════════╪══════════════╡
│ 0 │ 1 │ training log │ /result/training_log_resnet18.txt │ 1 │
├────┼─────────────┼───────────────┼───────────────────────────────────┼──────────────┤
│ 1 │ 2 │ tensorboard │ /result/event.tb │ 1 │
├────┼─────────────┼───────────────┼───────────────────────────────────┼──────────────┤
│ 2 │ 3 │ model │ diff_lr_resnet18.pth │ 1 │
╘════╧═════════════╧═══════════════╧═══════════════════════════════════╧══════════════╛
Table : Score
╒════╤════════════╤══════════════╤═════════╤══════════════╕
│ │ id_score │ type_score │ score │ experiment │
╞════╪════════════╪══════════════╪═════════╪══════════════╡
│ 0 │ 1 │ train acc │ 0.96 │ 1 │
├────┼────────────┼──────────────┼─────────┼──────────────┤
│ 1 │ 2 │ train loss │ 0.43 │ 1 │
├────┼────────────┼──────────────┼─────────┼──────────────┤
│ 2 │ 3 │ val acc │ 0.94 │ 1 │
├────┼────────────┼──────────────┼─────────┼──────────────┤
│ 3 │ 4 │ val loss │ 0.46 │ 1 │
╘════╧════════════╧══════════════╧═════════╧══════════════╛
Table : Feature
```
## Other solution:
[mlflow](https://github.com/mlflow/mlflow) & [dvc](https://github.com/iterative/dvc).
These solutions are great, they also offer a user interface and have many more options than my library however for a simple use where you only want to organize your experimentation and make a simple query.
Both solutions seem to be overkill.