{"id":27791119,"url":"https://github.com/adityyaraj/smart-hyperparameter-tuning-with-optimization","last_synced_at":"2025-07-30T13:10:55.945Z","repository":{"id":290324181,"uuid":"974040486","full_name":"adityyaraj/Smart-Hyperparameter-Tuning-with-Optimization","owner":"adityyaraj","description":"Implementation of a hybrid hyperparameter optimization framework integrating evolutionary strategies, probabilistic modeling, and meta-learning for enhanced model training on tasks like MNIST classification.","archived":false,"fork":false,"pushed_at":"2025-04-28T07:29:38.000Z","size":22,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-14T12:41:27.872Z","etag":null,"topics":["ai","genetic-algorithm","hyperparameter-tuning","optmization"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adityyaraj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-28T07:02:59.000Z","updated_at":"2025-04-28T09:10:47.000Z","dependencies_parsed_at":"2025-04-28T08:38:36.695Z","dependency_job_id":null,"html_url":"https://github.com/adityyaraj/Smart-Hyperparameter-Tuning-with-Optimization","commit_stats":null,"previous_names":["adityyaraj/smart-hyperparameter-tuning-with-optimization"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/adityyaraj/Smart-Hyperparameter-Tuning-with-Optimization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adityyaraj%2FSmart-Hyperparameter-Tuning-with-Optimization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adityyaraj%2FSmart-Hyperparameter-Tuning-with-Optimization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adityyaraj%2FSmart-Hyperparameter-Tuning-with-Optimization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adityyaraj%2FSmart-Hyperparameter-Tuning-with-Optimization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adityyaraj","download_url":"https://codeload.github.com/adityyaraj/Smart-Hyperparameter-Tuning-with-Optimization/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adityyaraj%2FSmart-Hyperparameter-Tuning-with-Optimization/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267874024,"owners_count":24158757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","genetic-algorithm","hyperparameter-tuning","optmization"],"created_at":"2025-04-30T18:59:36.623Z","updated_at":"2025-07-30T13:10:55.903Z","avatar_url":"https://github.com/adityyaraj.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Smart Hyperparameter Tuning Using Optimization\n\n## Overview\nThis repository contains an advanced hyperparameter optimization framework that combines multiple sophisticated techniques: Genetic Algorithms, Bayesian Bandits, and Meta-Learning. The framework is demonstrated on a simplified AlexNet model for MNIST classification and achieves excellent accuracy results (98.86% on the final model).\n\n## Key Features\n\n### Hybrid Optimization Approach\nThe framework combines three powerful optimization techniques:\n1. **Genetic Algorithm** - Evolutionary approach for broad parameter space exploration\n2. **Bayesian Bandits** - Thompson sampling for efficient exploitation of promising regions\n3. **Meta-Learning** - Neural network that learns to predict hyperparameter performance\n\n### Results\nThe framework achieved excellent performance on MNIST classification:\n- Final model accuracy: **98.86%**\n- Best hyperparameters:\n  - Learning rate: 0.0029\n  - Batch size: 180\n  - Dropout rate: 0.116\n  - Weight decay: 0.0002\n\n### Neural Network Architecture\n- Simplified AlexNet adapted for MNIST dataset (28x28 grayscale images)\n- Configurable dropout rates for regularization\n- Early stopping based on gradient flatness\n\n### Smart Training Features\n- Gradient-based early stopping to prevent overfitting\n- Optimized batch processing for faster hyperparameter evaluation\n- Performance visualization for tracking optimization progress\n\n## Technical Components\n\n### `SimplifiedAlexNet` Class\nA CNN based on AlexNet architecture but adapted for MNIST images:\n- Convolutional layers with ReLU activation\n- MaxPooling for downsampling\n- Dropout for regularization\n- Linear layers for classification\n\n### `MetaLearner` Class\nA neural network that learns to predict performance based on hyperparameter configurations:\n- Takes hyperparameters as input\n- Outputs predicted performance score\n- Learns from actual training results\n\n### `BayesianBandit` Class\nImplementation of Thompson sampling for hyperparameter selection:\n- Maintains historical performance of parameter combinations\n- Samples from posterior distributions to select promising configurations\n- Balances exploration and exploitation\n\n### `GeneticOptimizer` Class\nEvolutionary algorithm for hyperparameter optimization:\n- Population-based search through parameter space\n- Tournament selection for parent choice\n- Single-point crossover for recombination\n- Adaptive mutation based on parameter types\n- Elitism to preserve best solutions\n\n### `GradientEarlyStopping` Class\nEarly stopping mechanism based on loss gradient:\n- Monitors loss changes over time\n- Stops training when improvements plateau\n- Prevents overfitting and reduces training time\n\n## Optimization Progress\n\n### Generation 2 Results\nThe genetic algorithm showed promising convergence in the second generation:\n- Best Individual: Learning rate=0.0029, Batch size=254, Dropout rate=0.104, Weight decay=0.0008\n- Test Accuracy: 98.58%\n\nExample individual training progress:\n```\nIndividual 4/10: (0.002403342844568322, 189, 0.11592158181031818, 0.000206849274179782)\nParams: (0.002403342844568322, 189, 0.11592158181031818, 0.000206849274179782), Epoch: 1, Train Acc: 83.08%, Test Acc: 96.16%\nParams: (0.002403342844568322, 189, 0.11592158181031818, 0.000206849274179782), Epoch: 2, Train Acc: 96.62%, Test Acc: 97.86%\nParams: (0.002403342844568322, 189, 0.11592158181031818, 0.000206849274179782), Epoch: 3, Train Acc: 97.68%, Test Acc: 98.42%\nParams: (0.002403342844568322, 189, 0.11592158181031818, 0.000206849274179782), Epoch: 4, Train Acc: 97.96%, Test Acc: 98.25%\nParams: (0.002403342844568322, 189, 0.11592158181031818, 0.000206849274179782), Epoch: 5, Train Acc: 98.30%, Test Acc: 98.71%\n```\n\n### Generation 3 Results\nThe genetic algorithm further refined the parameters:\n- Best Individual: Learning rate=0.0029, Batch size=180, Dropout rate=0.116, Weight decay=0.0002\n- Test Accuracy: 98.44%\n\n### Final Model Performance\nAfter training the model with the best found hyperparameters for 10 epochs:\n```\nParams: (0.002854088011483561, 180, 0.11592158181031818, 0.000206849274179782), Epoch: 10, Train Acc: 98.69%, Test Acc: 98.86%\nFinal Model Performance - Loss: 0.0560, Accuracy: 98.86%\n```\n\n## Hyperparameters Optimized\nThe framework optimizes the following hyperparameters:\n- Learning rate (range: 0.0001 to 0.01)\n- Batch size (range: 32 to 256)\n- Dropout rate (range: 0.1 to 0.7)\n- Weight decay (range: 0.00001 to 0.001)\n\n## Usage\n\n```python\nif __name__ == \"__main__\":\n    print(\"Starting Hybrid Hyperparameter Tuning for AlexNet on MNIST\")\n    best_params, best_score, best_accuracy = smart_hyperparameter_tuning(max_iters=25, genetic_gens=3)\n    \n    print(\"\\n=== Final Results ===\")\n    print(f\"Best Hyperparameters: {best_params}\")\n    print(f\"Best Score: {-best_score:.4f}\")\n    print(f\"Best Accuracy: {best_accuracy:.2f}%\")\n    \n    # Train a final model with the best parameters\n    print(\"\\nTraining final model with best parameters...\")\n    final_score, final_accuracy = train_model_with_params(best_params, max_epochs=10)\n```\n\n## Optimization Process\nThe framework follows a two-phase approach:\n\n1. **Phase 1: Genetic Algorithm Exploration**\n   - Creates an initial population of hyperparameter combinations\n   - Evaluates each combination by training a model\n   - Uses tournament selection, crossover, and mutation to evolve better hyperparameters\n   - Tracks the best individual across generations\n\n2. **Phase 2: Bayesian Bandit Refinement**\n   - Takes the promising hyperparameter space identified by the genetic algorithm\n   - Uses Thompson sampling to balance exploration and exploitation\n   - Leverages the meta-learner to predict the performance of untested configurations\n   - Efficiently searches the narrowed hyperparameter space\n\n## Output and Visualization\nThe framework generates:\n- Detailed logs of the optimization process\n- A comprehensive visualization (`hybrid_tuning_results.png`) showing:\n  - Overall accuracy and loss trends\n  - Performance distributions across genetic generations\n  - Bandit refinement progress\n\n## Dependencies\n- PyTorch\n- NumPy\n- SciPy\n- Matplotlib\n- torchvision\n\n## Patent Pending\nThis advanced hyperparameter optimization approach is patent pending. The code is provided for research and educational purposes.\n\n## License\n[Your chosen license]\n\n## Citation\nIf you use this code in your research, please cite:\n```\n[Your citation information]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadityyaraj%2Fsmart-hyperparameter-tuning-with-optimization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadityyaraj%2Fsmart-hyperparameter-tuning-with-optimization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadityyaraj%2Fsmart-hyperparameter-tuning-with-optimization/lists"}