{"id":30065071,"url":"https://github.com/adrianscott/adamwtest","last_synced_at":"2025-08-08T05:47:25.761Z","repository":{"id":307543946,"uuid":"1029885064","full_name":"AdrianScott/adamwtest","owner":"AdrianScott","description":"Testing AdamW vs experimental optimizer w/ extra adaptive smoothing","archived":false,"fork":false,"pushed_at":"2025-07-31T18:34:59.000Z","size":35,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-07-31T21:56:49.948Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AdrianScott.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-31T18:21:00.000Z","updated_at":"2025-07-31T18:35:03.000Z","dependencies_parsed_at":"2025-07-31T22:09:48.298Z","dependency_job_id":null,"html_url":"https://github.com/AdrianScott/adamwtest","commit_stats":null,"previous_names":["adrianscott/adamwtest"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/AdrianScott/adamwtest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdrianScott%2Fadamwtest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdrianScott%2Fadamwtest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdrianScott%2Fadamwtest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdrianScott%2Fadamwtest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AdrianScott","download_url":"https://codeload.github.com/AdrianScott/adamwtest/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdrianScott%2Fadamwtest/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269372822,"owners_count":24406288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-08T05:46:58.251Z","updated_at":"2025-08-08T05:47:25.726Z","avatar_url":"https://github.com/AdrianScott.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AdamW-McGinley Dynamic Optimizer\n\nAn implementation of an AdamW variant using McGinley Dynamic-inspired adaptive momentum smoothing. This optimizer dynamically adapts the beta smoothing coefficients based on gradient change rates, potentially offering better performance on non-stationary or noisy training regimes.\n\n## Concept\n\nThe McGinley Dynamic indicator from technical analysis proposes adaptive smoothing based on the rate of change of the signal. This project applies that concept to the AdamW optimizer's beta coefficients, dynamically adjusting them based on gradient changes:\n\n- Instead of fixed β₁ and β₂, we use dynamic β₁ₜ and β₂ₜ computed per step\n- β₁ₜ adapts based on gradient changes: β₁ₜ = β₁ / (1 + |grad_change|²)\n- β₂ₜ adapts based on squared gradient changes: β₂ₜ = β₂ / (1 + |grad_sq_change|²)\n- Minimum thresholds for beta values ensure stability\n\n## Features\n\n- `CustomAdamW` optimizer with dynamic smoothing capability\n- Optional global gradient norm scaling\n- Beta statistics logging integration with Weights \u0026 Biases\n- Comprehensive sweep infrastructure for comparative analysis\n\n## Installation\n\nTo install the required dependencies, run:\n\n```bash\npip install -r requirements.txt\n```\n\nOr set up a virtual environment:\n\n```bash\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\npip install -r requirements.txt\n```\n\n## Usage\n\n### Basic Usage\n\n```python\nfrom custom_optimizer import CustomAdamW\n\n# Create optimizer with dynamic smoothing\noptimizer = CustomAdamW(\n    model.parameters(),\n    lr=0.001,\n    weight_decay=0.01,\n    dynamic_smoothing=True,  # Enable dynamic betas\n    min_beta1=0.5,          # Minimum value for beta1\n    min_beta2=0.9,          # Minimum value for beta2\n    log_betas=True          # Track beta statistics\n)\n```\n\n### Running Experiments\n\nBefore running experiments, make sure to set up your Weights \u0026 Biases API key:\n\n```bash\nexport WANDB_API_KEY=your_api_key_here\n```\n\nTo run experiments comparing CustomAdamW to standard AdamW:\n\n```bash\npython sweep_train.py\n```\n\nThis launches a Weights \u0026 Biases sweep that compares both optimizers across different configurations.\n\n## Results\n\n### Sample Comparison Test\n\n![Sample Comparison](sample-comparison.png)\n\nFull experiment results for this sweep on Weights \u0026 Biases are available; contact me for access.\n\n## Project Structure\n\n- `custom_optimizer.py`: Implementation of CustomAdamW optimizer\n- `sweep_train.py`: Main training and sweep script for CIFAR-10 experiments\n- `adamw-mcginley-dynamic-optimizer.md`: Detailed explanation of the algorithm\n\n## License\n\nCopyright (c) 2025 Adrian Scott.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadrianscott%2Fadamwtest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadrianscott%2Fadamwtest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadrianscott%2Fadamwtest/lists"}