https://github.com/aai-institute/classwise-calibration-experiments
Experiments for the paper "Class-wise and reduced calibration methods", ICMLA 2022
https://github.com/aai-institute/classwise-calibration-experiments
calibration classifier-calibration icmla icmla-2022 machine-learning transferlab
Last synced: 5 months ago
JSON representation
Experiments for the paper "Class-wise and reduced calibration methods", ICMLA 2022
- Host: GitHub
- URL: https://github.com/aai-institute/classwise-calibration-experiments
- Owner: aai-institute
- License: lgpl-3.0
- Created: 2022-09-14T10:11:54.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-10-13T18:03:04.000Z (over 3 years ago)
- Last Synced: 2024-01-27T20:07:10.493Z (over 2 years ago)
- Topics: calibration, classifier-calibration, icmla, icmla-2022, machine-learning, transferlab
- Language: Python
- Homepage:
- Size: 454 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: COPYING.LESSER
Awesome Lists containing this project
README
# Experiments for "Class-wise and Reduced Calibration Methods"
[](https://github.com/appliedAI-Initiative/Class-wise-and-reduced-calibration-methods/actions/workflows/code-quality.yaml)
This repository contains the experiments for the paper by Panchenko, Benmerzoug and de Benito Delgado, *Class-wise and reduced calibration methods*, submitted to the 21st IEEE International Conference on Machine Learning and Applications ([ICMLA 2022](https://www.icmla-conference.org/icmla22)).
# Pre-requisites
This project uses [Poetry](https://python-poetry.org/) for dependency management. More specifically version 1.2.0 of Poetry.
Start by installing it and then proceed to installing the requirements:
```shell
poetry install --no-root
```
And then activating the created virtual environment:
```shell
poetry shell
```
As an alternative you could build a docker image from the Dockerfile:
```shell
docker build . -t classwise-calibration:local
```
And then simply start a container:
```shell
docker container run -it --rm classwise-calibration:local
```
To start the notebooks from within the container use:
```shell
docker run -it --rm -p 8888:8888 classwise-calibration:local jupyter notebook --NotebookApp.default_url=/lab/ --ip=0.0.0.0 --port=8888
```
# Experiments
| Experiment | Binder |
|----------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Random Forest on Synthetic Data](notebooks/random_forest_synthetic.ipynb) | [](https://mybinder.org/v2/gh/appliedAI-Initiative/Class-wise-and-reduced-calibration-methods/main?filepath=notebooks%2Frandom_forest_synthetic.ipynb) |
| [LightGBM on Sorel20M](notebooks/lightgbm_sorel20m.ipynb) | [](https://mybinder.org/v2/gh/appliedAI-Initiative/Class-wise-and-reduced-calibration-methods/main?filepath=notebooks%2Flightgbm_sorel20m.ipynb) |
| [ResNet56 on CIFAR10](notebooks/resnet56_cifar10.ipynb) | [](https://mybinder.org/v2/gh/appliedAI-Initiative/Class-wise-and-reduced-calibration-methods/main?filepath=notebooks%2Fresnet56_cifar10.ipynb) |
| [DistilBERT on IMDB](notebooks/distilbert_imdb.ipynb) | [](https://mybinder.org/v2/gh/appliedAI-Initiative/Class-wise-and-reduced-calibration-methods/main?filepath=notebooks%2Fdistilbert_imdb.ipynb) |
| [DeiT on RVL-CDIP](notebooks/deit_rvl_cdip.ipynb) | [](https://mybinder.org/v2/gh/appliedAI-Initiative/Class-wise-and-reduced-calibration-methods/main?filepath=notebooks%2Fdeit_rvl_cdipb.ipynb) |
## Running the Experiments
To run the experiments use:
```shell
python -m src.experiments.
```
Where you would replace with the name of one of experiments' module.
For example, to run the Random Forest experiment with Synthetic Data use:
```shell
python -m src.experiments.random_forest_synthetic
```
## Notebooks
You could also use the notebooks if we to interactively run the experiments.
They are generated from the experiment scripts as follows:
```shell
bash scripts/generate_notebooks.sh
```
# License
This repository is distributed under [LGPL-3.0](https://www.gnu.org/licenses/lgpl-3.0.html).
A complete version can be found in two files: [here](LICENSE) and [here](COPYING.LESSER).
All contributions will be distributed under this license.