{"id":31216978,"url":"https://github.com/koratahiu/advanced_optimizers","last_synced_at":"2025-10-08T22:34:28.944Z","repository":{"id":314503015,"uuid":"1054267110","full_name":"Koratahiu/Advanced_Optimizers","owner":"Koratahiu","description":"A family of highly efficient, lightweight yet powerful optimizers.","archived":false,"fork":false,"pushed_at":"2025-09-30T03:24:05.000Z","size":177,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-30T05:34:51.101Z","etag":null,"topics":["adam-optimizer","compression","fine-tuning","llm","low-rank-approximation","memory-efficient","optimizer","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Koratahiu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-10T15:41:18.000Z","updated_at":"2025-09-30T03:24:08.000Z","dependencies_parsed_at":"2025-09-12T22:07:38.648Z","dependency_job_id":"fcf613fc-74b0-4b3d-acae-2d8223353844","html_url":"https://github.com/Koratahiu/Advanced_Optimizers","commit_stats":null,"previous_names":["koratahiu/advanced_optimizers"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Koratahiu/Advanced_Optimizers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koratahiu%2FAdvanced_Optimizers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koratahiu%2FAdvanced_Optimizers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koratahiu%2FAdvanced_Optimizers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koratahiu%2FAdvanced_Optimizers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Koratahiu","download_url":"https://codeload.github.com/Koratahiu/Advanced_Optimizers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koratahiu%2FAdvanced_Optimizers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000786,"owners_count":26082851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adam-optimizer","compression","fine-tuning","llm","low-rank-approximation","memory-efficient","optimizer","pytorch"],"created_at":"2025-09-21T13:02:46.766Z","updated_at":"2025-10-08T22:34:28.906Z","avatar_url":"https://github.com/Koratahiu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Advanced Optimizers (AIO)\n\nA comprehensive, all-in-one collection of optimization algorithms for deep learning, designed for maximum efficiency, minimal memory footprint, and superior performance across diverse model architectures and training scenarios.\n\n[![PyPI](https://img.shields.io/pypi/v/adv_optm)](https://pypi.org/project/adv_optm/)\n\n---\n\n## 📦 Installation\n\n```bash\npip install adv_optm\n```\n\n---\n\n## 🧠 Core Innovations\n\nThis library integrates multiple state-of-the-art optimization techniques validated through extensive research and practical training, with 1-bit compression for optimizer states:\n\n### **Memory-Efficient Optimization (SMMF-inspired)**\n- **Paper**: [SMMF: Square-Matricized Momentum Factorization](https://arxiv.org/abs/2412.08894)\n- **Approach**: Uses rank-1 non-negative matrix factorization with reconstruction cycle (factor → reconstruct → update → factor)\n- **Innovation**: \n  - First moment split into **1-bit sign + absolute value**\n  - Final storage: **four factored vectors + one 1-bit sign state**\n  - Preserves Adam-like update quality with drastically reduced memory\n\n---\n\n## ⚡ Performance Characteristics\n\n### Memory Efficiency (SDXL Model - 6.5GB)\n| Optimizer | Memory Usage | Description |\n|-----------|--------------|-------------|\n| `Adopt_Factored` | 328 MB | 4 small vectors + 1-bit state |\n| `Adopt_Factored + AdEMAMix` | 625 MB | 6 small vectors + two 1-bit states |\n| `Simplified_AdEMAMix` | 328 MB | Same as standard factored (no extra state) |\n\n### Speed Comparison (SDXL, Batch Size 4)\n| Optimizer | Speed | Notes |\n|-----------|-------|-------|\n| `Adafactor` | ~8.5s/it | Baseline |\n| `Adopt_Factored` | ~10s/it | +18% overhead from compression |\n| `Adopt_Factored + AdEMAMix` | ~12s/it | +41% overhead (3 factored states) |\n\n---\n\n## 🧪 Available Optimizers\n\n### Standard Optimizers (All support `factored=True/False`)\n| Optimizer | Description | Best For |\n|-----------|-------------|----------|\n| `Adam_Adv` | Advanced Adam implementation | General purpose |\n| `Adopt_Adv` | Adam-variant with independent beta2 | Stable training for small batch size regimes |\n| `Prodigy_Adv` | Prodigy with D-Adaptation | Adam with automatic LR tuning |\n| `Simplified_AdEMAMix` | Adam variant with accumulator momentum | Small/large batch training when tuned correctly |\n| `Lion_Adv` | Advanced Lion implementation | Memory-constrained environments |\n| `Prodigy_Lion_Adv` | Prodigy + Lion combination | Lion with automatic LR tuning |\n\n### Feature Matrix\n| Feature | Adam_Adv | Adopt_Adv | Prodigy_Adv | Simplified_AdEMAMix | Lion_Adv |\n|---------|----------|-----------|-------------|---------------------|----------|\n| Factored | ✓ | ✓ | ✓ | ✓ | ✓ |\n| AdEMAMix | ✓ | ✓ | ✓ | ✗ | ✗ |\n| Simplified_AdEMAMix | ✗ | ✗ | ✓ | ✓ | ✗ |\n| OrthoGrad | ✓ | ✓ | ✓ | ✓ | ✓ |\n| Grams | ✓ | ✓ | ✓ | ✗ | ✗ |\n| Cautious | ✓ | ✓ | ✓ | ✗ | ✓ |\n| atan2 | ✓ | ✓ | ✓ | ✗ | ✗ |\n| Stochastic Rounding | ✓ | ✓ | ✓ | ✓ | ✓ |\n| Fused Backward Pass | ✓ | ✓ | ✓ | ✓ | ✓ |\n\n---\n\n## ⚙️ Key Features \u0026 Parameters\n\n### Comprehensive Feature Guide\n\n| Feature | Description | Recommended Usage | Performance Impact | Theoretical Basis | Compatibility |\n|---------|-------------|-------------------|--------------------|-------------------|--------------|\n| **Factored** | Memory-efficient optimization using rank-1 factorization | Enable for large models (\u003e1B params) or limited VRAM | +12-41% time overhead, 1-bit memory usage | [SMMF](https://arxiv.org/abs/2412.08894) | All optimizers |\n| **AdEMAMix** | Dual EMA system for momentum | Use for long training runs (10k+ steps) | +1 state memory. | [AdEMAMix](https://arxiv.org/abs/2409.03137) | Adam/Adopt/Prodigy |\n| **Simplified_AdEMAMix** | Accumulator-based momentum | Small batch training (≤32) | Same memory as standard, no extra overhead | [Schedule-Free Connections](https://arxiv.org/abs/2502.02431) | Adam/Prodigy |\n| **OrthoGrad** | Removes gradient component parallel to weights | Full finetuning without weight decay | +33% time overhead, no memory impact | [Grokking at Edge](https://github.com/LucasPrietoAl/grokking-at-the-edge-of-numerical-stability) | All optimizers |\n| **Stochastic Rounding** | Improves precision for BF16 training | BF16 training | Minimal overhead (\u003c5%) | [Revisiting BFloat16 Training](https://arxiv.org/abs/2010.06192) | All optimizers |\n| **atan2** | Robust eps replacement + built-in clipping | Use with Adopt or unstable training | No overhead | [Adam-atan2](https://github.com/lucidrains/adam-atan2-pytorch) | Adam/Adopt/prodigy |\n| **Cautious** | Update only when the direction align with the gradients | should faster the convergence | No overhead | [C-Optim](https://github.com/kyleliang919/C-Optim) | Adam/Adopt/prodigy |\n| **Grams** | Update direction from the gradients | should have a stronger effect than cautious | No overhead | [Grams](https://github.com/Gunale0926/Grams) | Adam/Adopt/prodigy |\n\n---\n\n## Simplified_AdEMAMix Parameters\nSimplified_AdEMAMix replaces standard momentum with an accumulator for better small-large batch performance.\n\n| Parameter | Recommended Values | Description |\n|-----------|---------------------|-------------|\n| `beta1` | 0.9 (large BS), 0.99-0.9999 (small BS) | Determines memory length of accumulator |\n| `alpha` | 100-10 (small BS), 1-0 (large BS) | Gradient smoothing factor |\n\n**Alpha Tuning Guide**:\n| Batch Size | Recommended α | Rationale |\n|------------|---------------|-----------|\n| Small (≤32) | 100, 50, 20, 10 | Emphasizes recent gradients for quick adaptation |\n| Medium (32-512) | 10, 5, 2, 1 | Balanced approach |\n| Large (≥512) | 1, 0.5, 0 | Emphasizes historical gradients for stability |\n\n⚠️ **Important**: Use **~100x smaller learning rate** with Simplified_AdEMAMix compared to AdamW (e.g., 1e-6 instead of 1e-4)\n\n### 📊 Performance Validation\nSmall Batch Training (SDXL, BS=2, 1.8K steps)\n![Training Comparison](https://github.com/user-attachments/assets/7eff0671-cc59-47fc-8b63-d5205456d649)\n\n- **🟢 Prodigy_adv** (beta1=0.9, d0=1e-5): Final LR=2.9e-4\n- **🔵 Prodigy_adv + Simplified_AdEMAMix** (beta1=0.99, α=100, d0=1e-7): Final LR=5.8e-6\n\n**Results**:\n- Simplified_AdEMAMix shows faster convergence and better final performance\n- D-Adaptation automatically handles aggressive updates (50x smaller LR)\n- Generated samples show significantly better quality with Simplified_AdEMAMix\n\n---\n\n## ⚠️ Known Limitations\n\n### 1. Prodigy_Adv Sensitivity\n- Highly sensitive to gradient modifications (Adopt normalization, low-rank factorization)\n- May fail to increase learning rate in some LoRA scenarios\n- **Fix**: Disable factorization or set beta1=0\n\n### 2. Aggressive Learning Rates\n- Can destabilize factored first moment\n- **Recommendation**: Check Prodigy learning rate as reference for safe LR threshold\n\n---\n\n## 📚 References\n\n1. [SMMF: Square-Matricized Momentum Factorization](https://arxiv.org/abs/2412.08894)\n2. [The AdEMAMix Optimizer: Better, Faster, Older](https://arxiv.org/abs/2409.03137)\n3. [Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants](https://arxiv.org/abs/2502.02431)\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoratahiu%2Fadvanced_optimizers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoratahiu%2Fadvanced_optimizers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoratahiu%2Fadvanced_optimizers/lists"}