https://github.com/facebookresearch/spiritlm

Inference code for the paper "Spirit-LM Interleaved Spoken and Written Language Model".
https://github.com/facebookresearch/spiritlm

Last synced: about 7 hours ago
JSON representation

Inference code for the paper "Spirit-LM Interleaved Spoken and Written Language Model".

• Host: GitHub
• URL: https://github.com/facebookresearch/spiritlm
• Owner: facebookresearch
• License: other
• Created: 2024-09-06T14:18:20.000Z (2 months ago)
• Default Branch: main
• Last Pushed: 2024-10-28T10:56:02.000Z (16 days ago)
• Last Synced: 2024-11-06T12:12:37.397Z (7 days ago)
• Language: Python
• Size: 3.56 MB
• Stars: 748
• Watchers: 16
• Forks: 47
• Open Issues: 7
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
# Meta Spirit LM: Interleaved Spoken and Written Language Model

This repository contains the model weights, inference code and evaluation scripts for the Spirit LM [paper](https://arxiv.org/pdf/2402.05755.pdf). You can find more generation samples on our [demo page](https://speechbot.github.io/spiritlm/).

## Spirit LM Model Overview



## Installation Setup

### Conda

```

conda env create -f env.yml

pip install -e '.[eval]'

```

### Pip

```

pip install -e '.[eval]'

```

### Dev

(Optionally, use only if you want to run the tests.)

```

pip install -e '.[dev]'

```

## Checkpoints Setup

See [checkpoints/README.md](checkpoints/README.md)

## Quick Start

### Speech Tokenization

See [spiritlm/speech_tokenizer/README.md](spiritlm/speech_tokenizer/README.md)

### Spirit LM Generation

See [spiritlm/model/README.md](spiritlm/model/README.md)

### Speech-Text Sentiment Preservation benchmark (STSP)

See [spiritlm/eval/README.md](spiritlm/eval/README.md)

## Model Card

More details of the model can be found in [MODEL_CARD.md](MODEL_CARD.md).

## License

The present code is provided under the **FAIR Noncommercial Research License** found in [LICENSE](LICENSE).

## Citation

```

@misc{nguyen2024spiritlminterleavedspokenwritten,

      title={SpiRit-LM: Interleaved Spoken and Written Language Model},

      author={Tu Anh Nguyen and Benjamin Muller and Bokai Yu and Marta R. Costa-jussa and Maha Elbayad and Sravya Popuri and Paul-Ambroise Duquenne and Robin Algayres and Ruslan Mavlyutov and Itai Gat and Gabriel Synnaeve and Juan Pino and Benoit Sagot and Emmanuel Dupoux},

      year={2024},

      eprint={2402.05755},

      archivePrefix={arXiv},

      primaryClass={cs.CL},

      url={https://arxiv.org/abs/2402.05755},

}

```