https://github.com/chenyuntc/csm-training
https://github.com/chenyuntc/csm-training
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/chenyuntc/csm-training
- Owner: chenyuntc
- Created: 2025-03-24T03:21:30.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2025-04-01T05:01:58.000Z (10 months ago)
- Last Synced: 2025-07-25T19:29:01.346Z (6 months ago)
- Language: Python
- Size: 13.7 KB
- Stars: 5
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# CSM
---
This repo contains the code for fine-tuning and using the CSM model.
CSM (Conversational Speech Model) is a speech generation model from [Sesame](https://www.sesame.com) that generates RVQ audio codes from text and audio inputs. The model architecture employs a [Llama](https://www.llama.com/) backbone and a smaller audio decoder that produces [Mimi](https://huggingface.co/kyutai/mimi) audio codes.
## Requirements
* A CUDA-compatible GPU
* The code has been tested on CUDA 12.4 and 12.6, but it may also work on other versions
* Similarly, Python 3.10 is recommended, but newer versions may be fine
* For some audio operations, `ffmpeg` may be required
* Access to the following Hugging Face models:
* [Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B)
* [CSM-1B](https://huggingface.co/sesame/csm-1b) (for English generation)
* [yycc/csm-1b-chinese](https://huggingface.co/yycc/csm-1b-chinese) (used in the provided training script)
### Setup
```bash
git clone git@github.com:SesameAILabs/csm.git
cd csm
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
# You will need access to CSM-1B and Llama-3.2-1B
huggingface-cli login
```
### Windows Setup
The `triton` package cannot be installed in Windows. Instead use `pip install triton-windows`.
## Usage
Generate a sentence
```python
from generator import load_csm_1b
import torchaudio
import torch
if torch.backends.mps.is_available():
device = "mps"
elif torch.cuda.is_available():
device = "cuda"
else:
device = "cpu"
generator = load_csm_1b(device=device)
audio = generator.generate(
text="你好, 我是sesame,我会说字正腔圆的普通话",
speaker=0,
context=[],
max_audio_length_ms=10_000,
)
torchaudio.save("audio.wav", audio.unsqueeze(0).cpu(), generator.sample_rate)
```
## Training
The `main.py` script provides an example of how to fine-tune the CSM model. It is currently configured to train on a Chinese dataset (`EmiliaIterableDataset`) using the `yycc/csm-1b-chinese` checkpoint as a base.
```bash
# Adjust parameters like NUM_GRAD_ACCUM, LR, batch_size in main.py as needed
torchrun --nproc_per_node=4 main.py
```
## FAQ
**Does this model come with any voices?**
The model open-sourced here is a base generation model. It is capable of producing a variety of voices, but it has not been fine-tuned on any specific voice.
**Can I converse with the model?**
CSM is trained to be an audio generation model and not a general-purpose multimodal LLM. It cannot generate text. We suggest using a separate LLM for text generation.
**Does it support other languages?**
The base `sesame/csm-1b` model has some capacity for non-English languages due to data contamination in the training data, but performance may vary.
The provided `main.py` script demonstrates fine-tuning for Chinese using the `yycc/csm-1b-chinese` checkpoint and the `EmiliaIterableDataset`. You can adapt this script for other languages and datasets.
## Misuse and abuse ⚠️
This project provides a high-quality speech generation model for research and educational purposes. While we encourage responsible and ethical use, we **explicitly prohibit** the following:
- **Impersonation or Fraud**: Do not use this model to generate speech that mimics real individuals without their explicit consent.
- **Misinformation or Deception**: Do not use this model to create deceptive or misleading content, such as fake news or fraudulent calls.
- **Illegal or Harmful Activities**: Do not use this model for any illegal, harmful, or malicious purposes.
By using this model, you agree to comply with all applicable laws and ethical guidelines. We are **not responsible** for any misuse, and we strongly condemn unethical applications of this technology.
---
## Authors
Johan Schalkwyk, Ankit Kumar, Dan Lyth, Sefik Emre Eskimez, Zack Hodari, Cinjon Resnick, Ramon Sanabria, Raven Jiang, and the Sesame team.