{"id":47896036,"url":"https://github.com/declare-lab/jamify","last_synced_at":"2026-04-04T03:45:52.412Z","repository":{"id":306919313,"uuid":"1027656400","full_name":"declare-lab/jamify","owner":"declare-lab","description":"JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment","archived":false,"fork":false,"pushed_at":"2025-08-07T09:01:15.000Z","size":74226,"stargazers_count":156,"open_issues_count":12,"forks_count":20,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-04T03:45:38.904Z","etag":null,"topics":["ai-song","ai-song-generator","song-generation","tts"],"latest_commit_sha":null,"homepage":"https://declare-lab.github.io/jamify","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/declare-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-28T10:36:18.000Z","updated_at":"2026-03-24T11:26:41.000Z","dependencies_parsed_at":"2025-07-28T13:26:46.953Z","dependency_job_id":null,"html_url":"https://github.com/declare-lab/jamify","commit_stats":null,"previous_names":["declare-lab/jamify"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/declare-lab/jamify","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fjamify","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fjamify/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fjamify/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fjamify/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/declare-lab","download_url":"https://codeload.github.com/declare-lab/jamify/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fjamify/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31387023,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T01:22:39.193Z","status":"online","status_checked_at":"2026-04-04T02:00:07.569Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-song","ai-song-generator","song-generation","tts"],"created_at":"2026-04-04T03:45:48.045Z","updated_at":"2026-04-04T03:45:52.405Z","avatar_url":"https://github.com/declare-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://declare-lab.github.io/jamify-logo-new.png\" width=\"200\"/ \u003e\n \u003cbr/\u003e\n  \u003ch1\u003eJAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment\u003c/h1\u003e\n  \u003cbr/\u003e\n \n  [![arXiv](https://img.shields.io/badge/Read_the_paper-blue?style=flat\u0026logoColor=blue\u0026link=https%3A%2F%2Farxiv.org%2Fabs%2F)](https://arxiv.org/abs/2507.20880) [![Static Badge](https://img.shields.io/badge/JAM-Huggingface-violet?style=flat\u0026logo=huggingface\u0026logoColor=blue\u0026link=https%3A%2F%2Fhuggingface.co%2Fdeclare-lab%2FJAM-0.5)](https://huggingface.co/declare-lab/JAM-0.5) [![Static Badge](https://img.shields.io/badge/JAME-Dataset-orange?style=flat\u0026logo=huggingface\u0026logoColor=blue\u0026link=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Fdeclare-lab%2FJAME)](https://huggingface.co/datasets/declare-lab/JAME) [![Static Badge](https://img.shields.io/badge/HuggingFace-Space-brightgreen?style=flat\u0026logo=huggingface\u0026link=https%3A%2F%2Fhuggingface.co%2Fspaces%2Fdeclare-lab%2FJAM)](https://huggingface.co/spaces/declare-lab/JAM) [![Github](https://img.shields.io/badge/Github-Code-yellow?style=flat\u0026logo=github\u0026link=https%3A%2F%2Fgithub.com%2Fdeclare-lab%2Fjamify)](https://github.com/declare-lab/jamify) [![Static Badge](https://img.shields.io/badge/Project-Jamify-pink?style=flat\u0026logo=homepage\u0026link=https%3A%2F%2Fdeclare-lab.github.io%2Fjamify)](https://declare-lab.github.io/jamify) \n \n\u003c/div\u003e\n\n\nJAM is a rectified flow-based model for lyrics-to-song generation that addresses the lack of fine-grained word-level controllability in existing lyrics-to-song models. Built on a compact 530M-parameter architecture with 16 LLaMA-style Transformer layers as the Diffusion Transformer (DiT) backbone, JAM enables precise vocal control that musicians desire in their workflows. Unlike previous models, JAM provides word and phoneme-level timing control, allowing musicians to specify the exact placement of each vocal sound for improved rhythmic flexibility and expressive timing.\n\n## News\n\u003e 📣 05/08/25: Training code has been released! You can now train your own JAM models from scratch.\n\n\u003e 📣 29/07/25: We have released JAM-0.5, the first version of the AI song generator from Project Jamify!\n\n\n## Features\n\n![cover-photo](jam-teaser.png)\n\n- **Fine-grained Word and Phoneme-level Timing Control**: The first model to provide word-level timing and duration control in song generation, enabling precise prosody control for musicians\n- **Compact 530M Parameter Architecture**: Less than half the size of existing models, enabling faster inference with reduced resource requirements\n- **Enhanced Lyric Fidelity**: Achieves over 3× reduction in Word Error Rate (WER) and Phoneme Error Rate (PER) compared to prior work through precise phoneme boundary attention\n- **Global Duration Control**: Controllable duration up to 3 minutes and 50 seconds.\n- **Aesthetic Alignment through Direct Preference Optimization**: Iterative refinement using synthetic preference datasets to better align with human aesthetic preferences, eliminating manual annotation requirements\n\n## The Pipeline\n\n![cover-photo](jam.png)\n\n## Table of Contents\n\n- [JAM Samples](#jam-samples)\n- [Requirements](#requirements)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Inference](#inference)\n  - [Command Line Interface](#command-line-interface)\n  - [Configuration Options](#configuration-options)\n  - [Input File Formats](#input-file-formats)\n  - [Output Structure](#output-structure)\n- [Training](#training)\n  - [Data Formats](#data-formats)\n    - [WebDataset Format (Pretrain \u0026 SFT)](#webdataset-format-pretrain--sft)\n    - [DPO JSON Format](#dpo-json-format)\n  - [Training Commands](#training-commands)\n    - [Pretraining](#pretraining)\n    - [Supervised Fine-tuning (SFT)](#supervised-fine-tuning-sft)\n    - [Direct Preference Optimization (DPO)](#direct-preference-optimization-dpo)\n  - [Configuration Options](#configuration-options-1)\n  - [Monitoring](#monitoring)\n- [Troubleshooting](#troubleshooting)\n  - [Common Issues](#common-issues)\n- [Model Downloads](#model-downloads)\n- [Citation](#citation)\n- [License](#license)\n- [Support](#support)\n\n## JAM Samples\n\nCheck out the example generated music in the `generated_examples/` folder to hear what JAM can produce:\n\n- **`Hybrid Minds, Brodie - Heroin.mp3`** - Electronic music with synthesized beats and electronic elements\n- **`Jade Bird - Avalanche.mp3`** - Country music with acoustic guitar and folk influences  \n- **`Rizzle Kicks, Rachel Chinouriri - Follow Excitement!.mp3`** - Rap music with rhythmic beats and hip-hop style\n\nThese samples demonstrate JAM's ability to generate high-quality music across different genres while maintaining vocal intelligence, style consistency and musical coherence.\n\n## Requirements\n\n- Python 3.10 or higher\n- CUDA-compatible GPU with sufficient VRAM (8GB+ recommended)\n\n\n## Installation\n\n### 1. Clone the Repository\n\n```bash\ngit clone https://github.com/declare-lab/jamify\ncd jamify\n```\n\n### 2. Run Installation Script\n\nThe project includes an automated installation script, run it in your own virtual environment:\n\n```bash\nbash install.sh\n```\n\nThis script will:\n- Initialize and update git submodules (DeepPhonemizer)\n- Install Python dependencies from `requirements.txt`\n- Install the JAM package in editable mode\n- Install the DeepPhonemizer external dependency\n\n### 3. Manual Installation (Alternative)\n\nIf you prefer manual installation:\n\n```bash\n# Initialize submodules\ngit submodule update --init --recursive\n\n# Install dependencies\npip install -r requirements.txt\n\n# Install JAM package\npip install -e .\n\n# Install DeepPhonemizer\npip install -e externals/DeepPhonemizer\n```\n\n## Quick Start\n\n### Simple Inference\n\nThe easiest way to run inference is using the provided `inference.py` script:\n\n```python\npython inference.py\n```\n\nThis script will:\n1. Download the pre-trained JAM-0.5 model from Hugging Face\n2. Run inference with default settings\n3. Save generated audio to the `outputs` directory\n\n### Input Format\n\nCreate an input file at `inputs/input.json` with your songs:\n\n```json\n[\n  {\n    \"id\": \"my_song\",\n    \"audio_path\": \"inputs/reference_audio.mp3\",\n    \"lrc_path\": \"inputs/lyrics.json\", \n    \"duration\": 180.0,\n    \"prompt_path\": \"inputs/style_prompt.txt\"\n  }\n]\n```\n\nRequired files:\n- **Audio file**: Reference audio for style extraction\n- **Lyrics file**: JSON with timestamped lyrics\n- **Prompt file**: Text description of desired style/genre. Text prompt is not used in the default setting where the audio reference is utilized.\n\n## Inference\n\n### Using `python -m jam.infer`\n\nFor more control over the generation process:\n\n```bash\n# Basic usage with custom checkpoint\npython -m jam.infer evaluation.checkpoint_path=path/to/model.safetensors\n\n# With custom output directory\npython -m jam.infer evaluation.checkpoint_path=path/to/model.safetensors evaluation.output_dir=my_outputs\n\n# With custom configuration file\npython -m jam.infer config=configs/my_config.yaml evaluation.checkpoint_path=path/to/model.safetensors\n```\n\n### Multi-GPU Inference\n\nUse Accelerate for distributed inference:\n\n```bash\n# Basic usage with custom checkpoint\naccelerate launch -m jam.infer config=configs/jam_infer.yaml\n\n# With custom configuration file\naccelerate launch -m jam.infer config=configs/jam_infer.yaml\n```\n\n### Configuration Options\n\n#### Evaluation Settings\n- `evaluation.checkpoint_path`: Path to model checkpoint (required)\n- `evaluation.output_dir`: Output directory (default: \"outputs\")\n- `evaluation.test_set_path`: Input JSON file (default: \"inputs/input.json\")\n- `evaluation.batch_size`: Batch size for inference (default: 1)\n- `evaluation.num_samples`: Only generate first n samples in test_set_path (null = all)\n- `evaluation.vae_type`: VAE model type (\"diffrhythm\" or \"stable_audio\")\n\n#### Style Control\n- `evaluation.ignore_style`: Ignore style prompts (default: false)\n- `evaluation.use_prompt_style`: Use text prompts for style (default: false)\n- `evaluation.num_style_secs`: Style audio duration in seconds (default: 30)\n- `evaluation.random_crop_style`: Randomly crop style audio (default: false)\n\n### Input File Formats\n\n#### Lyrics File (`*.json`)\n```json\n[\n    {\"start\": 2.2, \"end\": 2.5, \"word\": \"First word of lyrics\"},\n    {\"start\": 2.5, \"end\": 3.7, \"word\": \"Second word of lyrics\"},\n    {\"more lines ....\"}\n]\n```\n\n#### Style Prompt File (`*.txt`)\n```\nElectronic dance music with heavy bass and synthesizers\n```\n\n#### Input Manifest (`input.json`)\n```json\n[\n  {\n    \"id\": \"unique_song_id\",\n    \"audio_path\": \"path/to/reference.mp3\",\n    \"lrc_path\": \"path/to/lyrics.json\",\n    \"duration\": 180.0,\n    \"prompt_path\": \"path/to/style.txt\"\n  }\n]\n```\n\n### Output Structure\n\nGenerated files are saved to the output directory:\n\n```\noutputs/\n├── generated/          # Final trimmed audio files\n├── generated_orig/     # Original generated audio\n├── cfm_latents/       # Intermediate latent representations\n├── local_files/       # Process-specific metadata\n└── generation_config.yaml  # Configuration used for generation\n```\n\n## Training\n\nJAM supports three training stages: pretraining, supervised fine-tuning (SFT), and direct preference optimization (DPO). Each stage requires specific data formats and training commands.\n\n### Data Formats\n\n#### WebDataset Format (Pretrain \u0026 SFT)\n\nFor pretraining and SFT, JAM expects data in WebDataset format - tar files containing:\n\n```\nyour_dataset-000000.tar\n├── song_id_1/\n│   ├── latent.pt      # Audio latent representation (torch tensor)\n│   ├── style.pt       # MuQ style embedding (torch tensor, 512-dim or n*512-dim)  \n│   └── json           # Metadata with phoneme information\n├── song_id_2/\n│   ├── latent.pt\n│   ├── style.pt\n│   └── json\n└── ...\n```\n\n**JSON Structure:**\n```json\n{\n  \"word\": [\n    {\"start\": 2.2, \"end\": 2.5, \"phoneme\": \"ˈfɜrst\"},\n    {\"start\": 2.5, \"end\": 3.7, \"phoneme\": \"wɜrd\"},\n    ...\n  ]\n}\n```\n\n**ID List JSONL (Optional):**\nWhen provided via `id_list_jsonl`, enables advanced filtering and sampling:\n```jsonl\n{\"id\": \"song_id_1\", \"duration\": 180.5}\n{\"id\": \"song_id_2\", \"duration\": 95.2}\n```\n\nFeatures when ID list is provided:\n- Filter training to specific song IDs\n- Duration-based sampling (`resample_by_duration_threshold`)\n- Duration filtering (`ignore_by_duration_threshold`)\n\n**Style Embeddings:**\nStyle embeddings are precomputed MuQ (Music Understanding Query) representations:\n- **Single Style**: 512-dimensional tensor when `multiple_styles: false`\n- **Multiple Styles**: n×512-dimensional tensor when `multiple_styles: true`\n  - During training, one random style is selected from the n available styles\n  - Enables style variation and data augmentation\n\n#### DPO JSON Format\n\nFor DPO training, provide a JSON file with preference pairs:\n\n```json\n[\n  {\n    \"win_latent\": \"path/to/preferred_latent.pt\",\n    \"loss_latent\": \"path/to/dispreferred_latent.pt\", \n    \"transcription\": \"path/to/lyrics.json\",\n    \"style\": \"path/to/style.pt\",\n    \"gt_latent\": \"path/to/ground_truth.pt\"\n  }\n]\n```\n\n**Required Fields:**\n- `win_latent`: Path to preferred latent representation\n- `loss_latent`: Path to dispreferred latent representation  \n- `transcription`: Path to JSON file with lyrics/phoneme data (same format as WebDataset JSON)\n- `style`: Path to MuQ style embedding (same format as WebDataset style.pt)\n\n**Conditional Fields:**\n- `gt_latent`: Required when DPO mode is set to \"gt\" in configuration\n\n### Training Commands\n\n#### Pretraining\n\n```bash\naccelerate launch -m jam.train config=configs/pretrain.yaml\n```\n\nKey configuration parameters in `configs/pretrain.yaml`:\n- `data.train_dataset.pattern`: Path pattern to WebDataset tar files, e.g. `your_jam_dataset-{000000..000485}.tar`\n- `data.train_dataset.id_list_jsonl`: Optional ID list for filtering\n- `training.max_steps`: Total training steps\n- `training.checkpoint_path`: Output directory for checkpoints\n\n#### Supervised Fine-tuning (SFT)\n\n```bash\naccelerate launch -m jam.train config=configs/sft.yaml\n```\n\nKey differences from pretraining:\n- `training.resume_from_safetensors`: Path to pretrained model\n- Higher `max_frames` for full-song training\n- `model.dit.grad_ckpt: true` for memory efficiency\n\n#### Direct Preference Optimization (DPO)\n\n```bash\naccelerate launch -m jam.dpo.train_dpo config=configs/dpo.yaml\n```\n\nKey DPO parameters:\n- `data.train_dataset.dpo_json_path`: Path to DPO preference data\n- `training.resume_from_safetensors`: Path to SFT model\n- `training.beta_dpo`: DPO regularization strength\n- `model.cfm.sft`: Set to \"none\" for classic DPO, \"gt\" for additional sft loss using ground truth latents, \"win\" if winning candidates are used as ground truth\n\n### Configuration Options\n\n#### Training Parameters\n- `max_steps`: Total training steps\n- `learning_rate`: Learning rate (typically 7.5e-5 for pretrain/SFT, 5.0e-7 for DPO)\n- `grad_accumulation_steps`: Gradient accumulation steps\n- `save_per_updates`: Checkpoint saving frequency\n\n#### Data Parameters  \n- `max_frames`: Maximum sequence length\n- `batch_size`: Training batch size\n- `shuffle`: Whether to shuffle training data\n- `multiple_styles`: Whether to use multiple style embeddings (n×512-dim) with random selection\n\n#### Model Parameters\n- `grad_ckpt`: Enable gradient checkpointing for memory efficiency\n- `dual_drop_prob`: Dual dropout probabilities for CFM training\n\n### Monitoring\n\nTraining progress is logged to Weights \u0026 Biases. Configure in your YAML:\n\n```yaml\nwandb:\n  project: \"JAM\"\n  name: \"my_training_run\" \n  mode: online  # or offline\n```\n\n## Troubleshooting\n\n### Common Issues\n\n#### \"Checkpoint path not found\"\n```bash\n# Make sure to specify the checkpoint path\npython -m jam.infer evaluation.checkpoint_path=path/to/your/model.safetensors\n```\n\n#### \"CUDA out of memory\"\n```bash\n# Reduce batch size or use mixed precision\naccelerate launch --mixed_precision=fp16 -m jam.infer evaluation.checkpoint_path=model.safetensors\n```\n\n#### \"Test set not found\"\n```bash\n# Create input.json file in inputs/ directory or specify custom path\npython -m jam.infer evaluation.test_set_path=path/to/your/input.json evaluation.checkpoint_path=model.safetensors\n```\n\n## Model Downloads\n\nThe `inference.py` script automatically downloads the JAM-0.5 model. For manual download:\n\n```python\nfrom huggingface_hub import snapshot_download\nmodel_path = snapshot_download(repo_id=\"declare-lab/jam-0.5\")\n```\n## Citation\n\nIf you use JAM in your research, please cite:\n\n```bibtex\n@misc{liu2025jamtinyflowbasedsong,\n      title={JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment}, \n      author={Renhang Liu and Chia-Yu Hung and Navonil Majumder and Taylor Gautreaux and Amir Ali Bagherzadeh and Chuan Li and Dorien Herremans and Soujanya Poria},\n      year={2025},\n      eprint={2507.20880},\n      archivePrefix={arXiv},\n      primaryClass={cs.SD},\n      url={https://arxiv.org/abs/2507.20880}, \n}\n```\n\n## License\n\n**JAM** is the first open-sourced model released under **Project Jamify**, developed for facilitating academic research and creative exploration in AI-generated songs from lyrics. The model is subject to:\n\n1. **Project Jamify License**: Intended **solely for non-commercial, academic, and entertainment purposes**\n2. **Stability AI Community License Agreement**: Required due to use of Stability AI model components\n\n### Key Restrictions:\n- **No copyrighted material** was used in a way that would intentionally infringe on intellectual property rights\n- **JAM is not designed** to reproduce or imitate any specific artist, label, or protected work\n- Outputs generated by JAM must **not be used to create or disseminate content that violates copyright laws**\n- **Commercial use of JAM or its outputs is strictly prohibited**\n- **Attribution Required**: Must retain \"This Stability AI Model is licensed under the Stability AI Community License, Copyright © Stability AI Ltd. All Rights Reserved.\"\n\n### Responsibility\nResponsibility for the use of the model and its outputs lies entirely with the end user, who must ensure all uses comply with applicable legal and ethical standards.\n\nFor complete license terms, see [LICENSE.md](LICENSE.md) and [STABILITY_AI_COMMUNITY_LICENSE.md](STABILITY_AI_COMMUNITY_LICENSE.md).\n\nFor questions, concerns, or collaboration inquiries, please contact the Project Jamify team via the official repository.\n\n## Support\n\nFor issues and questions:\n- Open an issue on GitHub\n- Check the troubleshooting section above\n- Review the configuration options for parameter tuning \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fjamify","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeclare-lab%2Fjamify","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fjamify/lists"}