{"id":19492658,"url":"https://github.com/agora-lab-ai/m1","last_synced_at":"2025-06-16T11:08:51.552Z","repository":{"id":260112501,"uuid":"880297874","full_name":"Agora-Lab-AI/m1","owner":"Agora-Lab-AI","description":"M1 is a research project exploring large-scale music generation using diffusion transformers. This repository contains the implementation of our proposed architecture combining recent advances in diffusion models, transformer architectures, and music processing.","archived":false,"fork":false,"pushed_at":"2025-04-21T16:30:37.000Z","size":2274,"stargazers_count":5,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-25T20:38:50.332Z","etag":null,"topics":["ai","anthropic","audio-diffusion","dit","gen","meta","ml","music","openai","stability"],"latest_commit_sha":null,"homepage":"https://discord.com/servers/agora-999382051935506503","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Agora-Lab-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["kyegomez"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2024-10-29T13:29:30.000Z","updated_at":"2025-04-08T13:37:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"c6400951-c557-4e7a-bdec-d81b5924cf9f","html_url":"https://github.com/Agora-Lab-AI/m1","commit_stats":null,"previous_names":["agora-lab-ai/m1"],"tags_count":0,"template":false,"template_full_name":"kyegomez/Python-Package-Template","purl":"pkg:github/Agora-Lab-AI/m1","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agora-Lab-AI%2Fm1","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agora-Lab-AI%2Fm1/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agora-Lab-AI%2Fm1/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agora-Lab-AI%2Fm1/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Agora-Lab-AI","download_url":"https://codeload.github.com/Agora-Lab-AI/m1/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Agora-Lab-AI%2Fm1/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260148393,"owners_count":22965913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","anthropic","audio-diffusion","dit","gen","meta","ml","music","openai","stability"],"created_at":"2024-11-10T21:22:28.980Z","updated_at":"2025-06-16T11:08:51.541Z","avatar_url":"https://github.com/Agora-Lab-AI.png","language":"Python","funding_links":["https://github.com/sponsors/kyegomez"],"categories":[],"sub_categories":[],"readme":"[![Multi-Modality](agorabanner.png)](https://discord.com/servers/agora-999382051935506503)\n\n# M1: Music Generation via Diffusion Transformers 🎵🔬\n\n[![Join our Discord](https://img.shields.io/badge/Discord-Join%20our%20server-5865F2?style=for-the-badge\u0026logo=discord\u0026logoColor=white)](https://discord.gg/agora-999382051935506503) [![Subscribe on YouTube](https://img.shields.io/badge/YouTube-Subscribe-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@kyegomez3242) [![Connect on LinkedIn](https://img.shields.io/badge/LinkedIn-Connect-blue?style=for-the-badge\u0026logo=linkedin\u0026logoColor=white)](https://www.linkedin.com/in/kye-g-38759a207/) [![Follow on X.com](https://img.shields.io/badge/X.com-Follow-1DA1F2?style=for-the-badge\u0026logo=x\u0026logoColor=white)](https://x.com/kyegomezb)\n\n\nM1 is a research project exploring large-scale music generation using diffusion transformers. This repository contains the implementation of our proposed architecture combining recent advances in diffusion models, transformer architectures, and music processing.\n\n## 🔬 Research Overview\n\nWe propose a novel approach to music generation that combines:\n- Diffusion-based generative modeling\n- Multi-query attention mechanisms\n- Hierarchical audio encoding\n- Text-conditional generation\n- Scalable training methodology\n\n### Key Hypotheses\n\n1. Diffusion transformers can capture long-range musical structure better than traditional autoregressive models\n2. Multi-query attention mechanisms can improve training efficiency without sacrificing quality\n3. Hierarchical audio encoding preserves both local and global musical features\n4. Text conditioning enables semantic control over generation\n\n## 🏗️ Architecture\n\n```\n                                              ┌─────────────────┐\n                                              │  Time Encoding  │\n                                              └────────┬────────┘\n                                                      │\n┌──────────────┐                              ┌──────▼─────────┐\n│ Audio Input  ├──► mel spectrogram ──────────►               │\n└──────────────┘                              │   Diffusion    │\n                                              │  Transformer   │ ──► Generated Audio\n┌──────────────┐    ┌─────────────┐          │     Block      │\n│ Text Input   ├──► │ T5 Encoder  ├──────────►               │\n└──────────────┘    └─────────────┘          └───────────────┘\n```\n\n### Implementation Details\n\n```python\n# Key architectural dimensions\nMODEL_CONFIG = {\n    'dim': 512,          # Base dimension\n    'depth': 12,         # Number of transformer layers\n    'heads': 8,          # Attention heads\n    'dim_head': 64,      # Dimension per head\n    'mlp_dim': 2048,     # FFN dimension\n    'dropout': 0.1       # Dropout rate\n}\n\n# Audio processing parameters\nAUDIO_CONFIG = {\n    'sample_rate': 16000,\n    'n_mels': 80,\n    'n_fft': 1024,\n    'hop_length': 256\n}\n```\n\n## 📊 Proposed Experiments\n\n### Phase 1: Architecture Validation\n- [ ] Baseline model training on synthetic data\n- [ ] Ablation studies on attention mechanisms\n- [ ] Time embedding comparison study\n- [ ] Audio encoding architecture experiments\n\n### Phase 2: Dataset Construction\nWe plan to build a research dataset from multiple sources:\n\n1. **Initial Development Dataset**\n   - 10k Creative Commons music samples\n   - Focused on single-instrument recordings\n   - Clear genre categorization\n\n2. **Scaled Dataset** (Future Work)\n   - Spotify API integration\n   - SoundCloud API integration\n   - Public domain music archives\n\n### Phase 3: Training \u0026 Evaluation\nPlanned training configurations:\n```yaml\ninitial_training:\n  batch_size: 32\n  gradient_accumulation: 4\n  learning_rate: 1e-4\n  warmup_steps: 1000\n  max_steps: 100000\n  \nevaluation_metrics:\n  - spectral_convergence\n  - magnitude_error\n  - musical_consistency\n  - genre_accuracy\n```\n\n## 🛠️ Development Setup\n\n```bash\n# Clone repository\ngit clone https://github.com/Agora-Lab-AI/m1.git\ncd m1-music\n\n# Create environment\nconda create -n m1 python=3.10\nconda activate m1\n\n# Install dependencies\npip install -r requirements.txt\n\n# Run tests\npytest tests/\n```\n\n\n## Example\n\n```python\nimport torch\nfrom m1.model import ModelConfig, AudioConfig, MusicDiffusionTransformer, DiffusionScheduler, train_step, generate_audio\nfrom loguru import logger\n\n# Example usage\ndef main():\n    logger.info(\"Setting up model configurations\")\n    \n    # Configure logging\n    logger.add(\"music_diffusion.log\", rotation=\"500 MB\")\n    \n    # Set device\n    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n    logger.info(f\"Using device: {device}\")\n    \n    # Initialize configurations\n    model_config = ModelConfig(\n        dim=512,\n        depth=12,\n        heads=8,\n        dim_head=64,\n        mlp_dim=2048,\n        dropout=0.1\n    )\n    \n    audio_config = AudioConfig(\n        sample_rate=16000,\n        n_mels=80,\n        audio_length=1024,\n        hop_length=256,\n        win_length=1024,\n        n_fft=1024\n    )\n    \n    # Initialize model and scheduler\n    model = MusicDiffusionTransformer(model_config, audio_config).to(device)\n    scheduler = DiffusionScheduler(num_inference_steps=1000)\n    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)\n    \n    # Example forward pass\n    logger.info(\"Preparing example forward pass\")\n    batch_size = 4\n    example_audio = torch.randn(batch_size, audio_config.audio_length).to(device)\n    example_text = {\n        'input_ids': torch.randint(0, 1000, (batch_size, 50)).to(device),\n        'attention_mask': torch.ones(batch_size, 50).bool().to(device)\n    }\n    \n    # Training step\n    logger.info(\"Executing training step\")\n    loss = train_step(\n        model,\n        scheduler,\n        optimizer,\n        example_audio,\n        example_text,\n        device\n    )\n    logger.info(f\"Training loss: {loss:.4f}\")\n    generation_text = {\n        'input_ids': torch.randint(0, 1000, (1, 50)).to(device),\n        'attention_mask': torch.ones(1, 50).bool().to(device)\n    }\n    \n    # Generation example\n    logger.info(\"Generating example audio\")\n    generated_audio = generate_audio(\n        model,\n        scheduler,\n        generation_text,\n        device,\n        audio_config.audio_length\n    )\n    logger.info(f\"Generated audio shape: {generated_audio.shape}\")\n\nif __name__ == \"__main__\":\n    main()\n\n```\n\n\n## 📝 Project Structure\n\n```\nm1/\n├── configs/               # Training configurations\n├── m1/\n│   ├── models/           # Model architectures\n│   ├── diffusion/        # Diffusion scheduling\n│   ├── data/             # Data loading/processing\n│   └── training/         # Training loops\n├── notebooks/            # Research notebooks\n├── scripts/              # Training scripts\n└── tests/                # Unit tests\n```\n\n## 🧪 Current Status\n\nThis is an active research project in early stages. Current focus:\n- [ ] Implementing and testing base architecture\n- [ ] Setting up data processing pipeline\n- [ ] Designing initial experiments\n- [ ] Building evaluation framework\n\n## 📚 References\n\nKey papers informing this work:\n- \"Diffusion Models Beat GANs on Image Synthesis\" (Dhariwal \u0026 Nichol, 2021)\n- \"Structured Denoising Diffusion Models\" (Sohl-Dickstein et al., 2015)\n- \"High-Resolution Image Synthesis with Latent Diffusion Models\" (Rombach et al., 2022)\n\n## 🤝 Contributing\n\nWe welcome research collaborations! Areas where we're looking for contributions:\n- Novel architectural improvements\n- Efficient training methodologies\n- Evaluation metrics\n- Dataset curation tools\n\n## 📬 Contact\n\nFor research collaboration inquiries:\n- Submit an issue\n- Start a discussion\n- Email: research@m1music.ai\n\n## ⚖️ License\n\nThis research code is released under the MIT License. \n\n## 🔍 Citation\n\nIf you use this code in your research, please cite:\n```bibtex\n@misc{m1music2024,\n  title={M1: Experimental Music Generation via Diffusion Transformers},\n  author={M1 Research Team},\n  year={2024},\n  publisher={GitHub},\n  journal={GitHub repository},\n  howpublished={\\url{https://github.com/Agora-Lab-AI/m1}}\n}\n```\n\n## 🚧 Disclaimer\n\nThis is experimental research code:\n- Architecture and training procedures may change significantly\n- Not yet optimized for production use\n- Results and capabilities are being actively researched\n- Breaking changes should be expected\n\nWe're sharing this code to foster collaboration and advance the field of AI music generation research.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagora-lab-ai%2Fm1","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagora-lab-ai%2Fm1","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagora-lab-ai%2Fm1/lists"}