https://github.com/helmholtz-ai-energy/ab-training
https://github.com/helmholtz-ai-energy/ab-training
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/helmholtz-ai-energy/ab-training
- Owner: Helmholtz-AI-Energy
- License: bsd-3-clause
- Created: 2024-04-25T11:20:13.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-04-29T14:01:17.000Z (over 1 year ago)
- Last Synced: 2024-04-29T15:26:37.199Z (over 1 year ago)
- Language: Python
- Size: 557 KB
- Stars: 0
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# madonna
# Description
Orthogonal DPNN training methods
# Quickstart
## Create the pipeline environment and install the madonna package
Before using the template, one needs to install the project as a package.
* First, create a virtual environment.
> You can either do it with conda (preferred) or venv.
* Then, activate the environment
* Finally, install the project as a package. Run:
```
pip install -e .
```
## Run the MNIST example
This pipeline comes with a toy example (MNIST dataset with a simple feedforward neural network). To run the training (resp. testing) pipeline, simply run:
```
python scripts/train.py
# or python scripts/test.py
```
Or, if you want to submit the training job to a submit (resp. interactive) cluster node via slurm, run:
```
sbatch job_submission.sbatch
# or sbatch job_submission_interactive.sbatch
```
> * The experiments, evaluations, etc., are stored under the `logs` directory.
> * The default experiments tracking system is mlflow. The `mlruns` directory is contained in `logs`. To view a user friendly view of the experiments, run:
> ```
> # make sure you are inside logs (where mlruns is located)
> mlflow ui --host 0000
> ```
> * When evaluating (running `test.py`), make sure you give the correct checkpoint path in `configs/test.yaml`
# Project Organization
```
├── configs <- Hydra configuration files
│ ├── callbacks <- Callbacks configs
│ ├── data <- Datamodule configs
│ ├── debug <- Debugging configs
│ ├── experiment <- Experiment configs
│ ├── hparams_search <- Hyperparameter search configs
│ ├── local <- Local configs
│ ├── log_dir <- Logging directory configs
│ ├── logger <- Logger configs
│ ├── model <- Model configs
│ ├── trainer <- Trainer configs
│ │
│ ├── test.yaml <- Main config for testing
│ └── train.yaml <- Main config for training
│
├── data <- Project data
│ └── MNIST
│ ├── processed <- Processed data
│ └── raw <- Raw data
│
├── docs <- Directory for Sphinx documentation in rst or md.
├── models <- Trained and serialized models, model predictions
├── notebooks <- Jupyter notebooks.
├── reports <- Generated analysis as HTML, PDF, LaTeX, etc.
│ └── figures <- Generated plots and figures for reports.
├── scripts <- Scripts used in project
│ ├── job_submission.sbatch <- Submit training job to slurm
│ ├── job_submission_interactive.sbatch <- Submit training job to slurm (interactive node)
│ ├── test.py <- Run testing
│ └── train.py <- Run training
│
├── src/madonna <- Source code
│ ├── datamodules <- Lightning datamodules
│ ├── models <- Lightning models
│ ├── utils <- Utility scripts
│ │
│ ├── testing_pipeline.py <- Model evaluation workflow
│ └── training_pipeline.py <- Model training workflow
│
├── tests <- Tests of any kind
│ ├── helpers <- A couple of testing utilities
│ ├── shell <- Shell/command based tests
│ └── unit <- Unit tests
│
├── .gitignore <- List of files/folders ignored by git
├── .pre-commit-config.yaml <- Configuration of pre-commit hooks for code formatting
├── requirements.txt <- File for installing python dependencies
├── setup.cfg <- Configuration of linters and pytest
├── LICENSE.txt <- License as chosen on the command-line.
├── pyproject.toml <- Build configuration. Don't change! Use `pip install -e .`
│ to install for development or to build `tox -e build`.
├── setup.cfg <- Declarative configuration of your project.
├── setup.py <- [DEPRECATED] Use `python setup.py develop` to install for
│ development or `python setup.py bdist_wheel` to build.
└── README.md
```