{"id":48519278,"url":"https://github.com/adeelahmad/mlx-guided-grpo","last_synced_at":"2026-04-07T20:30:46.040Z","repository":{"id":336647754,"uuid":"1150559854","full_name":"adeelahmad/mlx-guided-grpo","owner":"adeelahmad","description":"Train reasoning models on your Mac. GRPO training framework for Apple Silicon with curriculum learning.","archived":false,"fork":false,"pushed_at":"2026-02-05T12:47:58.000Z","size":227,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-05T23:57:48.695Z","etag":null,"topics":["apple-silicon","curriculum-learning","deepseek-r1","fine-tuning","grpo","llm","lora","m","m1","m2","m3","machine-learning","macos","mlx","mlx-lm","reasoning","rlhf","sllm"],"latest_commit_sha":null,"homepage":"https://github.com/adeelahmad/mlx-guided-grpo","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adeelahmad.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"adeelahmad"}},"created_at":"2026-02-05T12:27:29.000Z","updated_at":"2026-02-05T12:45:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/adeelahmad/mlx-guided-grpo","commit_stats":null,"previous_names":["adeelahmad/mlx-guided-grpo"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/adeelahmad/mlx-guided-grpo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adeelahmad%2Fmlx-guided-grpo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adeelahmad%2Fmlx-guided-grpo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adeelahmad%2Fmlx-guided-grpo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adeelahmad%2Fmlx-guided-grpo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adeelahmad","download_url":"https://codeload.github.com/adeelahmad/mlx-guided-grpo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adeelahmad%2Fmlx-guided-grpo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31528205,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T16:28:08.000Z","status":"ssl_error","status_checked_at":"2026-04-07T16:28:06.951Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","curriculum-learning","deepseek-r1","fine-tuning","grpo","llm","lora","m","m1","m2","m3","machine-learning","macos","mlx","mlx-lm","reasoning","rlhf","sllm"],"created_at":"2026-04-07T20:30:45.141Z","updated_at":"2026-04-07T20:30:46.033Z","avatar_url":"https://github.com/adeelahmad.png","language":"Python","funding_links":["https://github.com/sponsors/adeelahmad"],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/🍎_Apple_Silicon-Optimized-black?style=for-the-badge\" alt=\"Apple Silicon\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/MLX-Native-orange?style=for-the-badge\" alt=\"MLX Native\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/GRPO-Training-blue?style=for-the-badge\" alt=\"GRPO\"/\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003e🧠 MLX Guided GRPO\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eTrain reasoning models on your Mac. No cloud needed.\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  The first production-ready GRPO training framework for Apple Silicon.\u003cbr/\u003e\n  Fine-tune LLMs to \u003cem\u003ethink step-by-step\u003c/em\u003e using your M1/M2/M3/M4 Mac.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/adeelahmad/mlx-guided-grpo/stargazers\"\u003e\u003cimg src=\"https://img.shields.io/github/stars/adeelahmad/mlx-guided-grpo?style=social\" alt=\"Stars\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/adeelahmad/mlx-guided-grpo/network/members\"\u003e\u003cimg src=\"https://img.shields.io/github/forks/adeelahmad/mlx-guided-grpo?style=social\" alt=\"Forks\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/adeelahmad/mlx-guided-grpo/issues\"\u003e\u003cimg src=\"https://img.shields.io/github/issues/adeelahmad/mlx-guided-grpo\" alt=\"Issues\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/adeelahmad/mlx-guided-grpo/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/github/license/adeelahmad/mlx-guided-grpo\" alt=\"License\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#-quick-start\"\u003eQuick Start\u003c/a\u003e •\n  \u003ca href=\"#-features\"\u003eFeatures\u003c/a\u003e •\n  \u003ca href=\"#-why-guided-grpo\"\u003eWhy Guided GRPO\u003c/a\u003e •\n  \u003ca href=\"#-installation\"\u003eInstallation\u003c/a\u003e •\n  \u003ca href=\"#-examples\"\u003eExamples\u003c/a\u003e •\n  \u003ca href=\"#-documentation\"\u003eDocs\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## 🎯 Train Your Own Reasoning Model in 5 Minutes\n\n```bash\n# Install\npip install mlx-guided-grpo\n\n# Train (yes, it's this simple)\nmlx-grpo --model mlx-community/Qwen2.5-3B-Instruct-4bit \\\n         --data ./your_data.jsonl \\\n         --train --train-type lora \\\n         --curriculum-enabled\n```\n\n**That's it.** Your Mac is now training a reasoning model with curriculum learning.\n\n---\n\n## 🤔 Why Guided GRPO?\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\n### The Problem\n\nTraining reasoning models (like DeepSeek-R1, o1) requires:\n- ❌ Expensive cloud GPUs ($$$)\n- ❌ Complex distributed setups\n- ❌ NVIDIA-only frameworks\n- ❌ Weeks of engineering\n\n**Most developers can't train reasoning models.**\n\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\n### The Solution\n\nMLX Guided GRPO gives you:\n- ✅ **Train on your Mac** - M1/M2/M3/M4\n- ✅ **One command** - No config hell\n- ✅ **Curriculum learning** - Progressive difficulty\n- ✅ **Production ready** - Crash recovery, logging\n\n**Train reasoning models on consumer hardware.**\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n---\n\n## ✨ Features\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n### 🎓 Curriculum Learning\nGradually reduce scaffolding so models learn to think independently. Start with 100% guidance, end with 0%.\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n### 🔄 Two-Phase Generation\nAutomatic recovery for incomplete `\u003cthink\u003e` outputs. Never lose a training sample.\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n### 🎯 Smart Token Masking\nOnly train on tokens the model generated. Scaffolded tokens are properly masked from loss.\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n### ⚡ Apple Silicon Native\nBuilt on MLX for maximum Metal GPU utilization. 2-3x faster than PyTorch on Mac.\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n### 🧠 Conditional Gradient Scaling\nTrain different layers for thinking vs answering. Fine-grained control over what the model learns.\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n### 💾 Crash Recovery\nAutomatic checkpointing and resume. Metal GPU crashes? Training continues.\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### Full Feature List\n\n- **Training**: GRPO, DR-GRPO, BNPO loss variants\n- **Adapters**: LoRA, DoRA, Full fine-tuning\n- **Type System**: Extensible type-aware rewards for tool calling, MCQ, and general Q\u0026A ([docs](TYPE_SYSTEM.md))\n- **Memory**: Gradient checkpointing, cache management\n- **Rewards**: Type-dispatched rewards, custom reward functions\n- **Logging**: WandB integration, rollout logging\n- **Monitoring**: Threshold-based early stopping\n\n---\n\n## 📊 Benchmarks\n\n| Model | Hardware | Tokens/sec | Memory |\n|-------|----------|------------|--------|\n| Qwen2.5-3B-4bit | M3 Max 64GB | ~150 | 12GB |\n| Qwen2.5-7B-4bit | M3 Max 64GB | ~80 | 24GB |\n| Llama-3.2-3B-4bit | M2 Pro 32GB | ~120 | 10GB |\n\n*GRPO training with group_size=4, batch_size=2*\n\n---\n\n## 🚀 Installation\n\n### From PyPI (Recommended)\n\n```bash\npip install mlx-guided-grpo\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/adeelahmad/mlx-guided-grpo.git\ncd mlx-guided-grpo\npip install -e \".[all]\"\n```\n\n### Requirements\n\n- macOS 13.5+ with Apple Silicon (M1/M2/M3/M4)\n- Python 3.10+\n- 16GB+ RAM recommended\n\n---\n\n## 🏃 Quick Start\n\n### 1. Prepare Your Data\n\nCreate a JSONL file with prompts and reasoning traces:\n\n```json\n{\"prompt\": \"What is 15 * 7?\", \"answer\": \"\u003cthink\u003e\\nI need to multiply 15 by 7.\\n15 * 7 = 105\\n\u003c/think\u003e\\n\\n\\\\boxed{105}\"}\n{\"prompt\": \"Solve: 2x + 5 = 13\", \"answer\": \"\u003cthink\u003e\\nSubtract 5 from both sides:\\n2x = 8\\nDivide by 2:\\nx = 4\\n\u003c/think\u003e\\n\\n\\\\boxed{4}\"}\n```\n\n### 2. Train Your Model\n\n```bash\nmlx-grpo \\\n    --model mlx-community/Qwen2.5-3B-Instruct-4bit \\\n    --data ./math_data.jsonl \\\n    --train \\\n    --train-type lora \\\n    --iters 1000 \\\n    --batch-size 2 \\\n    --group-size 4 \\\n    --curriculum-enabled \\\n    --adapter-path ./my-reasoning-model\n```\n\n### 3. Use Your Model\n\n```python\nfrom mlx_lm import load, generate\n\nmodel, tokenizer = load(\"mlx-community/Qwen2.5-3B-Instruct-4bit\",\n                        adapter_path=\"./my-reasoning-model\")\n\nprompt = \"What is 23 * 17?\"\nresponse = generate(model, tokenizer, prompt=prompt, max_tokens=500)\nprint(response)\n# \u003cthink\u003e\n# I need to multiply 23 by 17...\n# \u003c/think\u003e\n# \\boxed{391}\n```\n\n---\n\n## 📖 Examples\n\n### Basic GRPO Training\n\n```bash\nmlx-grpo \\\n    --model mlx-community/Qwen2.5-0.5B-Instruct-4bit \\\n    --data ./data \\\n    --train --train-type lora \\\n    --group-size 4 \\\n    --learning-rate 1e-5\n```\n\n### Curriculum Learning (Recommended for Reasoning)\n\n```bash\nmlx-grpo \\\n    --model mlx-community/Qwen2.5-3B-Instruct-4bit \\\n    --data ./reasoning_data \\\n    --train --train-type lora \\\n    --curriculum-enabled \\\n    --curriculum-start-ratio 1.0 \\\n    --curriculum-end-ratio 0.0 \\\n    --curriculum-warmup-iters 100 \\\n    --curriculum-taper-iters 500 \\\n    --enforce-thinking\n```\n\n### With WandB Logging\n\n```bash\nmlx-grpo \\\n    --model mlx-community/Qwen2.5-3B-Instruct-4bit \\\n    --data ./data \\\n    --train --train-type lora \\\n    --wandb my-experiment \\\n    --log-rollouts \\\n    --log-rollouts-to-wandb\n```\n\n### Advanced: Dual-Gradient Mode (CGS)\n\n```bash\nmlx-grpo \\\n    --model mlx-community/Qwen2.5-7B-Instruct-4bit \\\n    --data ./data \\\n    --train --train-type lora \\\n    --thinking-layers \"0-15\" \\\n    --answer-layers \"16-31\" \\\n    --thinking-gradient-weight 0.5 \\\n    --answer-gradient-weight 1.0\n```\n\n---\n\n## 🔧 Key Concepts\n\n### Curriculum Learning\n\nProgressive scaffolding teaches models to reason independently:\n\n```\nIteration 0-100:   [████████████] 100% scaffolding (model learns format)\nIteration 100-400: [████████░░░░]  66% scaffolding (gradual reduction)\nIteration 400-700: [████░░░░░░░░]  33% scaffolding (increasing independence)\nIteration 700+:    [░░░░░░░░░░░░]   0% scaffolding (full independence)\n```\n\n### Smart Token Masking\n\nOnly train on what the model actually generated:\n\n```\n[PROMPT] [SCAFFOLD PREFIX] [MODEL GENERATION]\n   ↓           ↓                  ↓\n masked      masked         LOSS COMPUTED\n```\n\nThis prevents the model from getting \"free credit\" for scaffolded tokens.\n\n### Two-Phase Generation\n\nAutomatic recovery for incomplete structured outputs:\n\n```\nPhase 1: Model generates → \"\u003cthink\u003eLet me solve this... 2+2=\"\n         (Incomplete! Missing \u003c/think\u003e)\n\nPhase 2: Inject \"\u003c/think\u003e\\n\\boxed{\" → Continue generation → \"4}\"\n         (Complete! Injected tokens masked from loss)\n```\n\n---\n\n## 📚 Documentation\n\n| Topic | Link |\n|-------|------|\n| Full CLI Reference | [docs/cli.md](docs/cli.md) |\n| Training Arguments | [docs/arguments.md](docs/arguments.md) |\n| Custom Rewards | [docs/rewards.md](docs/rewards.md) |\n| Type System | [TYPE_SYSTEM.md](TYPE_SYSTEM.md) |\n| Architecture | [docs/architecture.md](docs/architecture.md) |\n| API Reference | [docs/api.md](docs/api.md) |\n\n---\n\n## 🆚 Comparison\n\n| Feature | MLX Guided GRPO | TRL (HuggingFace) | OpenRLHF |\n|---------|-----------------|-------------------|----------|\n| Apple Silicon Native | ✅ | ❌ | ❌ |\n| Curriculum Learning | ✅ | ❌ | ❌ |\n| Scaffold Token Masking | ✅ | ❌ | ❌ |\n| Two-Phase Generation | ✅ | ❌ | ❌ |\n| Single GPU Training | ✅ | ✅ | ⚠️ |\n| Consumer Hardware | ✅ | ⚠️ | ❌ |\n| One-Command Training | ✅ | ❌ | ❌ |\n\n---\n\n## 🛠️ Troubleshooting\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eOut of Memory?\u003c/strong\u003e\u003c/summary\u003e\n\n```bash\n# Reduce memory usage\nmlx-grpo ... \\\n    --grad-checkpoint \\\n    --batch-size 1 \\\n    --group-size 2 \\\n    --max-completion-length 256\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eMetal GPU Crash?\u003c/strong\u003e\u003c/summary\u003e\n\nTraining auto-saves checkpoints. Just resume:\n\n```bash\nmlx-grpo ... --resume\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eSlow Training?\u003c/strong\u003e\u003c/summary\u003e\n\n```bash\n# Use quantized model\n--model mlx-community/Qwen2.5-3B-Instruct-4bit\n\n# Reduce group size\n--group-size 2\n```\n\n\u003c/details\u003e\n\n---\n\n## 🤝 Contributing\n\nContributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n```bash\n# Setup development environment\ngit clone https://github.com/adeelahmad/mlx-guided-grpo.git\ncd mlx-guided-grpo\npip install -e \".[dev]\"\n\n# Run formatting\nblack mlx_grpo/\nisort mlx_grpo/\n```\n\n---\n\n## 📜 Citation\n\nIf you use MLX Guided GRPO in your research, please cite:\n\n```bibtex\n@software{mlx_guided_grpo,\n  author = {Ahmad, Adeel},\n  title = {MLX Guided GRPO: Reasoning Model Training for Apple Silicon},\n  year = {2024},\n  url = {https://github.com/adeelahmad/mlx-guided-grpo}\n}\n```\n\n---\n\n## 📄 License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n---\n\n## 🙏 Acknowledgments\n\n- [MLX](https://github.com/ml-explore/mlx) - Apple's ML framework\n- [mlx-lm](https://github.com/ml-explore/mlx-examples) - MLX language model utilities\n- [DeepSeek](https://github.com/deepseek-ai) - GRPO algorithm\n- [Qwen](https://github.com/QwenLM) - Excellent base models\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eBuilt with ❤️ for the Mac ML community\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://www.linkedin.com/in/adeelahmadch\"\u003eLinkedIn\u003c/a\u003e •\n  \u003ca href=\"https://github.com/adeelahmad\"\u003eGitHub\u003c/a\u003e •\n  \u003ca href=\"mailto:adeel@adeelahmad.net\"\u003eContact\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003csub\u003eIf this project helps you, please ⭐ star the repo!\u003c/sub\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadeelahmad%2Fmlx-guided-grpo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadeelahmad%2Fmlx-guided-grpo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadeelahmad%2Fmlx-guided-grpo/lists"}