{"id":13838322,"url":"https://github.com/facebookresearch/encodec","last_synced_at":"2025-05-14T00:09:40.611Z","repository":{"id":62049996,"uuid":"554807775","full_name":"facebookresearch/encodec","owner":"facebookresearch","description":"State-of-the-art deep learning based audio codec supporting both mono 24 kHz audio and stereo 48 kHz audio.","archived":false,"fork":false,"pushed_at":"2024-01-04T01:58:44.000Z","size":4294,"stargazers_count":3650,"open_issues_count":56,"forks_count":324,"subscribers_count":58,"default_branch":"main","last_synced_at":"2025-04-10T02:18:09.409Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookresearch.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-20T12:35:53.000Z","updated_at":"2025-04-09T11:22:51.000Z","dependencies_parsed_at":"2024-06-18T12:36:22.077Z","dependency_job_id":"84215f11-e764-4e21-bb02-5746e73f60ac","html_url":"https://github.com/facebookresearch/encodec","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fencodec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fencodec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fencodec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fencodec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookresearch","download_url":"https://codeload.github.com/facebookresearch/encodec/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254044190,"owners_count":22005099,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T15:01:50.621Z","updated_at":"2025-05-14T00:09:35.604Z","avatar_url":"https://github.com/facebookresearch.png","language":"Python","funding_links":[],"categories":["Python","HarmonyOS","Resources","语音识别与合成_其他"],"sub_categories":["Windows Manager","Image-Guided","资源传输下载"],"readme":"# EnCodec: High Fidelity Neural Audio Compression\n![linter badge](https://github.com/facebookresearch/encodec/workflows/linter/badge.svg)\n![tests badge](https://github.com/facebookresearch/encodec/workflows/tests/badge.svg)\n\nThis is the code for the EnCodec neural codec presented in the [High Fidelity Neural Audio Compression](https://arxiv.org/pdf/2210.13438.pdf) [[abs]](https://arxiv.org/abs/2210.13438).\npaper. We provide our two multi-bandwidth models:\n* A causal model operating at 24 kHz on monophonic audio trained on a variety of audio data.\n* A non-causal model operating at 48 kHz on stereophonic audio trained on music-only data.\n\nThe 24 kHz model can compress to 1.5, 3, 6, 12 or 24 kbps, while the 48 kHz model\nsupport 3, 6, 12 and 24 kbps. We also provide a pre-trained language model for each\nof the models, that can further compress the representation by up to 40% without\nany further loss of quality.\n\nFor reference, we also provide the code for our novel [MS-STFT discriminator](encodec/msstftd.py) and the [balancer](encodec/balancer.py).\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./architecture.png\" alt=\"Schema representing the structure of Encodec,\n    with a convolutional+LSTM encoder, a Residual Vector Quantization in the middle,\n    followed by a convolutional+LSTM decoder. A multiscale complex spectrogram discriminator is applied to the output, along with objective reconstruction losses.\n    A small transformer model is trained to predict the RVQ output.\"\nwidth=\"800px\"\u003e\u003c/p\u003e\n\n\n## Samples\n\nSamples including baselines are provided on [our sample page](https://ai.honu.io/papers/encodec/samples.html).\nYou can also have a quick demo of what we achieve for 48 kHz music with EnCodec, along with\nentropy coding, by clicking the thumbnail (original tracks provided by [Lucille Crew](https://open.spotify.com/artist/5eLv7rNfrf3IjMnK311ByP?si=X_zD9ackRRGjFP5Y6Q7Zng) and [Voyageur I](https://open.spotify.com/artist/21HymveeIhDcM4KDKeNLz0?si=4zXF8VpeQpeKR9QUIuck9Q)).\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://ai.honu.io/papers/encodec/final.mp4\"\u003e\n\u003cimg src=\"./thumbnail.png\" alt=\"Thumbnail for the sample video.\n\tYou will first here the ground truth, then ~3kbps, then 12kbps, for two songs.\"\u003e\u003c/a\u003e\u003c/p\u003e\n\n## 🤗 Transformers\n\nEncodec has now been added to Transformers. For more information, please refer to [Transformers' Encodec docs](https://huggingface.co/docs/transformers/main/en/model_doc/encodec).\n\nYou can find both the [24KHz](https://huggingface.co/facebook/encodec_24khz) and [48KHz](https://huggingface.co/facebook/encodec_48khz) checkpoints on the 🤗 Hub.\n\nUsing 🤗 Transformers, you can leverage Encodec at scale along with all the other supported models and datasets. ⚡️\nAlternatively you can also directly use the encodec package, as detailed in the Usage section. \n\nTo use first you'd need to set up your development environment!\n```\npip install -U datasets \npip install git+https://github.com/huggingface/transformers.git@main\n```\n\nThen, start embedding your audio datasets at scale!\n```python\nfrom datasets import load_dataset, Audio\nfrom transformers import EncodecModel, AutoProcessor\n\n# dummy dataset, however you can swap this with an dataset on the 🤗 hub or bring your own\nlibrispeech_dummy = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n\n# load the model + processor (for pre-processing the audio)\nmodel = EncodecModel.from_pretrained(\"facebook/encodec_24khz\")\nprocessor = AutoProcessor.from_pretrained(\"facebook/encodec_24khz\")\n\n# cast the audio data to the correct sampling rate for the model\nlibrispeech_dummy = librispeech_dummy.cast_column(\"audio\", Audio(sampling_rate=processor.sampling_rate))\naudio_sample = librispeech_dummy[0][\"audio\"][\"array\"]\n\n# pre-process the inputs\ninputs = processor(raw_audio=audio_sample, sampling_rate=processor.sampling_rate, return_tensors=\"pt\")\n\n# explicitly encode then decode the audio inputs\nencoder_outputs = model.encode(inputs[\"input_values\"], inputs[\"padding_mask\"])\naudio_values = model.decode(encoder_outputs.audio_codes, encoder_outputs.audio_scales, inputs[\"padding_mask\"])[0]\n\n# or the equivalent with a forward pass\naudio_values = model(inputs[\"input_values\"], inputs[\"padding_mask\"]).audio_values\n\n# you can also extract the discrete codebook representation for LM tasks\n# output: concatenated tensor of all the representations\naudio_codes = model(inputs[\"input_values\"], inputs[\"padding_mask\"]).audio_codes\n\n```\n\n## What's up?\n\nSee [the changelog](CHANGELOG.md) for details on releases.\n\n## Installation\n\nEnCodec requires Python 3.8, and a reasonably recent version of PyTorch (1.11.0 ideally).\nTo install EnCodec, you can run from this repository:\n```bash\npip install -U encodec  # stable release\npip install -U git+https://git@github.com/facebookresearch/encodec#egg=encodec  # bleeding edge\n# of if you cloned the repo locally\npip install .\n```\n\n**Supported platforms:** we officially support only Mac OS X (you might need XCode installed if running on a non Intel Mac), and recent versions of mainstream Linux distributions. We will try to help out on Windows but cannot provide strong support. Any other platform (iOS / Android / onboard ARM) are not supported.\n\n## Usage\n\nYou can then use the EnCodec command, either as\n```bash\npython3 -m encodec [...]\n# or\nencodec [...]\n```\n\nIf you want to directly use the compression API, checkout `encodec.compress`\nand `encodec.model`. See hereafter for instructions on how to extract the discrete\nrepresentation.\n\n### Model storage\n\nThe models will be automatically downloaded on first use using Torch Hub.\nFor more information on where those models are stored, or how to customize\nthe storage location, [checkout their documentation.](https://pytorch.org/docs/stable/hub.html#where-are-my-downloaded-models-saved)\n\n### Compression\n\n```bash\nencodec [-b TARGET_BANDWIDTH] [-f] [--hq] [--lm] INPUT_FILE [OUTPUT_FILE]\n```\nGiven any audio file supported by torchaudio on your platform, compresses\nit with EnCodec to the target bandwidth (default is 6 kbps, can be either 1.5, 3, 6, 12 or 24).\nOUTPUT_FILE must end in `.ecdc`. If not provided it will be the same as `INPUT_FILE`,\nreplacing the extension with `.ecdc`.\nIn order to use the model operating at 48 kHz on stereophonic audio, use the `--hq` flag.\nThe `-f` flag is used to force overwrite an existing output file.\nUse the `--lm` flag to use the pretrained language model with entropy coding (expect it to\nbe much slower).\n\nIf the sample rate or number of channels of the input doesn't match that of the model,\nthe command will automatically resample / reduce channels as needed.\n\n### Decompression\n```bash\nencodec [-f] [-r] ENCODEC_FILE [OUTPUT_WAV_FILE]\n```\nGiven a `.ecdc` file previously generated, this will decode it to the given output wav file.\nIf not provided, the output will default to the input with the `.wav` extension.\nUse the `-f` file to force overwrite the output file (be carefull if compress then decompress,\nnot to overwrite your original file !). Use the `-r` flag if you experience clipping, this will\nrescale the output file to avoid it.\n\n### Compression + Decompression\n```bash\nencodec [-r] [-b TARGET_BANDWIDTH] [-f] [--hq] [--lm] INPUT_FILE OUTPUT_WAV_FILE\n```\nWhen `OUTPUT_WAV_FILE` has the `.wav` extension (as opposed to `.ecdc`), the `encodec`\ncommand will instead compress and immediately decompress without storing the intermediate\n`.ecdc` file.\n\n### Extracting discrete representations\n\nThe EnCodec model can also be used to extract discrete representations from the audio waveform.\n\n```python\nfrom encodec import EncodecModel\nfrom encodec.utils import convert_audio\n\nimport torchaudio\nimport torch\n\n# Instantiate a pretrained EnCodec model\nmodel = EncodecModel.encodec_model_24khz()\n# The number of codebooks used will be determined bythe bandwidth selected.\n# E.g. for a bandwidth of 6kbps, `n_q = 8` codebooks are used.\n# Supported bandwidths are 1.5kbps (n_q = 2), 3 kbps (n_q = 4), 6 kbps (n_q = 8) and 12 kbps (n_q =16) and 24kbps (n_q=32).\n# For the 48 kHz model, only 3, 6, 12, and 24 kbps are supported. The number\n# of codebooks for each is half that of the 24 kHz model as the frame rate is twice as much.\nmodel.set_target_bandwidth(6.0)\n\n# Load and pre-process the audio waveform\nwav, sr = torchaudio.load(\"\u003cPATH_TO_AUDIO_FILE\u003e\")\nwav = convert_audio(wav, sr, model.sample_rate, model.channels)\nwav = wav.unsqueeze(0)\n\n# Extract discrete codes from EnCodec\nwith torch.no_grad():\n    encoded_frames = model.encode(wav)\ncodes = torch.cat([encoded[0] for encoded in encoded_frames], dim=-1)  # [B, n_q, T]\n```\n\nNote that the 48 kHz model processes the audio by chunks of 1 seconds, with an overlap of 1%,\nand renormalizes the audio to have unit scale. For this model, the output of `model.encode(wav)`\nwould a list (for each frame of 1 second) of a tuple `(codes, scale)` with `scale` a scalar tensor.\n\n## Installation for development\n\nThis will install the dependencies and a `encodec` in developer mode (changes to the files\nwill directly reflect), along with the dependencies to run unit tests.\n```\npip install -e '.[dev]'\n```\n\n### Test\n\nYou can run the unit tests with\n```\nmake tests\n```\n\n## FAQ\n\nPlease check this section before opening an issue.\n\n### Out of memory errors with long files\n\nWe do not try to be smart about long files, and we apply the model at once on the entire file. This can lead to a large memory usage\nand result in the process being killed. At the moment we will not support this use case.\n\n### Bad interactions between DistributedDataParallel and the RVQ code\n\nWe do not use DDP, instead we recommend using the routines in `encodec/distrib.py`, in particular `encodec.distrib.sync_buffer` and `encodec.distrib.sync_grad`.\n\n## Citation\n\nIf you use this code or results in your paper, please cite our work as:\n\n```\n@article{defossez2022highfi,\n  title={High Fidelity Neural Audio Compression},\n  author={Défossez, Alexandre and Copet, Jade and Synnaeve, Gabriel and Adi, Yossi},\n  journal={arXiv preprint arXiv:2210.13438},\n  year={2022}\n}\n```\n\n## License\n\nThe code in this repository is released under the MIT license as found in the\n[LICENSE](LICENSE) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fencodec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookresearch%2Fencodec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fencodec/lists"}