{"id":19606521,"url":"https://github.com/ankitamungalpara/hyperparametertuning-experimenttracking","last_synced_at":"2025-10-07T01:53:53.410Z","repository":{"id":259253858,"uuid":"875457031","full_name":"AnkitaMungalpara/HyperParameterTuning-ExperimentTracking","owner":"AnkitaMungalpara","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-26T19:25:30.000Z","size":460,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-26T16:49:43.930Z","etag":null,"topics":["aim","continuous-machine-learning","dvc","experiment-tracking","hydra","hyperparameter-tuning","mlflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AnkitaMungalpara.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-20T03:06:14.000Z","updated_at":"2024-10-27T03:13:28.000Z","dependencies_parsed_at":"2024-10-23T22:44:46.837Z","dependency_job_id":"a4ebcbdd-2edf-49d3-b83b-72a2b43478a5","html_url":"https://github.com/AnkitaMungalpara/HyperParameterTuning-ExperimentTracking","commit_stats":null,"previous_names":["ankitamungalpara/hyperparametertuning-experimenttracking"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AnkitaMungalpara/HyperParameterTuning-ExperimentTracking","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnkitaMungalpara%2FHyperParameterTuning-ExperimentTracking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnkitaMungalpara%2FHyperParameterTuning-ExperimentTracking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnkitaMungalpara%2FHyperParameterTuning-ExperimentTracking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnkitaMungalpara%2FHyperParameterTuning-ExperimentTracking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AnkitaMungalpara","download_url":"https://codeload.github.com/AnkitaMungalpara/HyperParameterTuning-ExperimentTracking/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnkitaMungalpara%2FHyperParameterTuning-ExperimentTracking/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278708028,"owners_count":26031932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aim","continuous-machine-learning","dvc","experiment-tracking","hydra","hyperparameter-tuning","mlflow"],"created_at":"2024-11-11T10:05:51.404Z","updated_at":"2025-10-07T01:53:53.394Z","avatar_url":"https://github.com/AnkitaMungalpara.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Hyperparameter Optimization and Report](https://github.com/AnkitaMungalpara/HyperParameterTuning-ExperimentTracking/actions/workflows/hparam_optim.yaml/badge.svg)](https://github.com/AnkitaMungalpara/HyperParameterTuning-ExperimentTracking/actions/workflows/hparam_optim.yaml)\n\n# Hyperparameter Optimization and Report 🎯\n\nThis project focuses on **Hyperparameter Optimization** for a **CatDog Classification** model, leveraging powerful tools like **Aim**, **MLflow**, **DVC**, **CML**, **Hydra**, and **Optuna**. It demonstrates how to streamline experiment tracking, model tuning, and reporting with modern ML tools.\n\n\n## Table of Contents 📑\n\n- [Overview](#overview)\n- [Tools and Frameworks](#tools-and-frameworks)\n  - [Aim](#aim)\n  - [MLflow](#mlflow)\n  - [DVC](#dvc)\n  - [CML](#cml)\n  - [Hydra](#hydra)\n  - [Optuna](#optuna)\n- [Hyperparameter Optimization Results](#hyperparameter-optimization-results)\n   - [Combined Metrics Plot](#combined-metrics-plot)\n   - [Best Hyperparameters](#best-hyperparameters)\n   - [Hyperparameters for Each Experiment](#hyperparameters-for-each-experiment)\n   - [Test Accuracy Across Experiments (Epoch-wise)](#test-accuracy-across-experiments-epoch-wise)\n- [References](#references)\n\n\n\n## **Overview** 📘\n\nThis repository applies **hyperparameter optimization** techniques. It uses a sample dataset of 100 images per class from the [Chest X-Ray Pneumonia dataset](https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia), and integrates cutting-edge ML tools to automate experiment tracking, model training, and reporting.\n\n\n## Tools and Frameworks 🔧\n\n### Aim 📝\n\n[Aim](https://aimstack.io/) is an open-source tool designed for experiment tracking and visualization. It provides fast and efficient tracking for large-scale experiments, with an interactive web-based dashboard to compare runs.\n\n#### Key Features:\n- Lightweight and scalable experiment tracking.\n- Rich visualizations for metrics and hyperparameter tuning.\n- Easy to integrate with frameworks like **PyTorch** and **TensorFlow**.\n\n#### Commands:\n1. Start Aim dashboard:\n   ```bash\n   aim up\n   ```\n2. Run an experiment and track metrics:\n   ```bash\n   aim run python src/train.py experiment=\u003cexperiment_name\u003e\n   ```\n\n### MLflow 📈\n\n[MLflow](https://mlflow.org/) is an open-source platform for managing the end-to-end machine learning lifecycle, including experimentation, model packaging, and deployment.\n\n#### Key Features:\n- Logs parameters, metrics, and artifacts.\n- Provides a model registry to store and version models.\n- Powerful UI for visualizing and comparing multiple runs.\n\n#### Commands:\n1. Start the MLflow UI:\n   ```bash\n   mlflow ui\n   ```\n2. Run an experiment and log metrics:\n   ```bash\n   mlflow run . --experiment-name \"\u003cexperiment_name\u003e\"\n   ```\n\n### **DVC** 📂\n\n[DVC](https://dvc.org/) (Data Version Control) is a version control system tailored for machine learning projects, handling datasets and models.\n\n#### Key Features:\n- Manages datasets and models with Git-like version control.\n- Integrates with cloud storage like Google Drive, AWS S3, and Azure.\n- Ensures reproducibility across experiments by tracking datasets and models.\n\n#### Commands:\n1. Install dependencies and pull the dataset:\n   ```bash\n   UV_EXTRA_INDEX_URL=https://download.pytorch.org/whl/cpu uv sync\n   ```\n2. Pull the data from DVC:\n   ```bash\n   uv run dvc pull\n   ```\n\n### CML (Continuous Machine Learning) 🚀\n\n[CML](https://cml.dev/) (Continuous Machine Learning) integrates machine learning workflows into **GitHub Actions** or **GitLab CI**, automating the entire pipeline.\n\n#### Key Features:\n- Automates ML pipelines within CI/CD systems.\n- Generates reports for each experiment.\n- Supports model evaluation and metrics reporting directly in GitHub/GitLab.\n\n#### Commands:\n1. Create reports and post results automatically during CI runs:\n   ```bash\n   uv run python scripts/generate_results.py\n   ```\n\n### Hydra ⚙️\n\n[Hydra](https://hydra.cc/) is a framework for managing complex configurations in machine learning experiments. It simplifies running multiple configurations and allows you to easily override settings on the command line.\n\n#### Key Features:\n- Dynamic configuration management for complex projects.\n- Supports multirun for running multiple experiments simultaneously.\n- Easily override configuration values from the command line.\n\n#### Commands:\n1. Run a training script with configuration overrides:\n   ```bash\n   python src/train.py experiment=\u003cexperiment_name\u003e +trainer.fast_dev_run=True\n   ```\n2. Run multiple experiments with different configurations:\n   ```bash\n   python src/train.py --multirun experiment=\u003cexperiment_name\u003e model.embed_dim=16,32,64 +trainer.log_every_n_steps=5\n   ```\n\n### Optuna 🧪\n\n[Optuna](https://optuna.org/) is an open-source hyperparameter optimization framework designed to efficiently search hyperparameter spaces for the best configurations.\n\n#### Key Features:\n- Efficient hyperparameter search using techniques like Bayesian optimization.\n- Parallelization for faster search.\n- Visualizations to help analyze search results.\n\n#### Commands:\n1. Run a hyperparameter search with Optuna:\n   ```bash\n   python src/train.py --multirun hparam=catdog_vit_hparam +trainer.log_every_n_steps=5 hydra.sweeper.n_jobs=4\n   ```\n\n# Hyperparameter Optimization Results\n\n### Combined Metrics Plot\n\n![Combined Metrics Plot](utils/images/combined_metrics_plot_5_epoch_10_trials.png)\n\n### Best Hyperparameters\n\n|    | Parameter                        | Value                  |\n|---:|:---------------------------------|:-----------------------|\n|  0 | name                             | optuna                 |\n|  1 | model.drop_path_rate             | 0.2089407556793585     |\n|  2 | model.head_init_scale            | 0.9292090024255693     |\n|  3 | model.lr                         | 0.00023458293902856107 |\n|  4 | model.weight_decay               | 0.0005513192559352005  |\n|  5 | data.batch_size                  | 64                     |\n|  6 | augmentation.horizontal_flip     | True                   |\n|  7 | augmentation.random_crop         | True                   |\n|  8 | augmentation.random_rotation     | True                   |\n|  9 | regularization.gradient_clipping | 0.15371010694861154    |\n| 10 | best_value                       | 1.0                    |\n\n### Hyperparameters for Each Experiment\n\n\n|    | base_model    |   num_classes | pretrained   |     lr | optimizer   |   weight_decay |   scheduler_patience |   scheduler_factor |   drop_path_rate |   head_init_scale | horizontal_flip   | random_crop   | random_rotation   |   val_acc |   val_loss |\n|---:|:--------------|--------------:|:-------------|-------:|:------------|---------------:|---------------------:|-------------------:|-----------------:|------------------:|:------------------|:--------------|:------------------|----------:|-----------:|\n|  0 | convnext_nano |             2 | False        | 0.0003 | AdamW       |         0.0010 |                    3 |             0.1000 |           0.2257 |            0.6574 | True              | True          | True              |    0.8000 |     0.5406 |\n|  1 | convnext_nano |             2 | False        | 0.0002 | AdamW       |         0.0006 |                    3 |             0.1000 |           0.2089 |            0.9292 | True              | True          | True              |    0.6000 |     0.5929 |\n|  2 | convnext_nano |             2 | False        | 0.0005 | AdamW       |         0.0007 |                    3 |             0.1000 |           0.0723 |            1.0152 | True              | True          | True              |    0.7000 |     0.6401 |\n|  3 | convnext_nano |             2 | False        | 0.0004 | AdamW       |         0.0000 |                    3 |             0.1000 |           0.2775 |            1.7625 | True              | True          | True              |    0.8000 |     0.5590 |\n|  4 | convnext_nano |             2 | False        | 0.0002 | AdamW       |         0.0002 |                    3 |             0.1000 |           0.1194 |            1.6070 | True              | True          | True              |    0.8000 |     0.4880 |\n|  5 | convnext_nano |             2 | False        | 0.0003 | AdamW       |         0.0009 |                    3 |             0.1000 |           0.2148 |            0.5226 | True              | True          | True              |    0.8000 |     0.4855 |\n|  6 | convnext_nano |             2 | False        | 0.0009 | AdamW       |         0.0005 |                    3 |             0.1000 |           0.1251 |            1.5220 | True              | True          | True              |    0.7000 |     0.6624 |\n|  7 | convnext_nano |             2 | False        | 0.0009 | AdamW       |         0.0003 |                    3 |             0.1000 |           0.0952 |            1.1222 | True              | True          | True              |    0.6000 |     0.7140 |\n|  8 | convnext_nano |             2 | False        | 0.0001 | AdamW       |         0.0004 |                    3 |             0.1000 |           0.0881 |            1.4465 | True              | True          | True              |    0.7000 |     0.5756 |\n|  9 | convnext_nano |             2 | False        | 0.0006 | AdamW       |         0.0007 |                    3 |             0.1000 |           0.0287 |            1.8280 | True              | True          | True              |    0.7000 |     0.4451 |\n\n\n### Test Accuracy Across Experiments (Epoch-wise)\n\n![Test Accuracy Plot (Epoch-wise)](utils/images/test_accuracy_plot_epoch_5_epoch_10_trials.png)\n\n\n## **References** 📚\n\n- [Aim](https://aimstack.io/)\n- [MLflow](https://mlflow.org/)\n- [DVC](https://dvc.org/)\n- [CML](https://cml.dev/)\n- [Hydra](https://hydra.cc/)\n- [Optuna](https://optuna.org/)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankitamungalpara%2Fhyperparametertuning-experimenttracking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fankitamungalpara%2Fhyperparametertuning-experimenttracking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankitamungalpara%2Fhyperparametertuning-experimenttracking/lists"}