{"id":14958076,"url":"https://github.com/pytorch/torchcodec","last_synced_at":"2025-05-14T15:06:21.282Z","repository":{"id":251069797,"uuid":"807642214","full_name":"pytorch/torchcodec","owner":"pytorch","description":"PyTorch video decoding","archived":false,"fork":false,"pushed_at":"2025-05-14T11:36:06.000Z","size":56954,"stargazers_count":545,"open_issues_count":71,"forks_count":35,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-05-14T12:49:50.867Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pytorch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-05-29T13:47:12.000Z","updated_at":"2025-05-13T23:28:10.000Z","dependencies_parsed_at":"2024-08-05T20:13:37.660Z","dependency_job_id":"6a25dd06-8957-41f7-8ff6-ca10c3314c05","html_url":"https://github.com/pytorch/torchcodec","commit_stats":{"total_commits":282,"total_committers":12,"mean_commits":23.5,"dds":0.5992907801418439,"last_synced_commit":"ce99662ba2b6a065aed364beacb5c943aa6be71a"},"previous_names":["pytorch/torchcodec"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorchcodec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorchcodec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorchcodec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorchcodec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pytorch","download_url":"https://codeload.github.com/pytorch/torchcodec/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254169059,"owners_count":22026207,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T13:16:10.607Z","updated_at":"2025-05-14T15:06:21.269Z","avatar_url":"https://github.com/pytorch.png","language":"Python","funding_links":[],"categories":["CV"],"sub_categories":[],"readme":"[**Installation**](#installing-torchcodec) | [**Simple Example**](#using-torchcodec) | [**Detailed Example**](https://pytorch.org/torchcodec/stable/generated_examples/) | [**Documentation**](https://pytorch.org/torchcodec) | [**Contributing**](CONTRIBUTING.md) | [**License**](#license)\n\n# TorchCodec\n\nTorchCodec is a Python library for decoding video and audio data into PyTorch\ntensors, on CPU and CUDA GPU. It aims to be fast, easy to use, and well\nintegrated into the PyTorch ecosystem. If you want to use PyTorch to train ML\nmodels on videos and audio, TorchCodec is how you turn these into data.\n\nWe achieve these capabilities through:\n\n* Pythonic APIs that mirror Python and PyTorch conventions.\n* Relying on [FFmpeg](https://www.ffmpeg.org/) to do the decoding. TorchCodec\n  uses the version of FFmpeg you already have installed. FFmpeg is a mature\n  library with broad coverage available on most systems. It is, however, not\n  easy to use. TorchCodec abstracts FFmpeg's complexity to ensure it is used\n  correctly and efficiently.\n* Returning data as PyTorch tensors, ready to be fed into PyTorch transforms\n  or used directly to train models.\n\n\u003e [!NOTE]\n\u003e ⚠️ TorchCodec is still in development stage and some APIs may be updated\n\u003e in future versions, depending on user feedback.\n\u003e If you have any suggestions or issues, please let us know by\n\u003e [opening an issue](https://github.com/pytorch/torchcodec/issues/new/choose)!\n\n## Using TorchCodec\n\nHere's a condensed summary of what you can do with TorchCodec. For more detailed\nexamples, [check out our\ndocumentation](https://pytorch.org/torchcodec/stable/generated_examples/)!\n\n#### Decoding\n\n```python\nfrom torchcodec.decoders import VideoDecoder\n\ndevice = \"cpu\"  # or e.g. \"cuda\" !\ndecoder = VideoDecoder(\"path/to/video.mp4\", device=device)\n\ndecoder.metadata\n# VideoStreamMetadata:\n#   num_frames: 250\n#   duration_seconds: 10.0\n#   bit_rate: 31315.0\n#   codec: h264\n#   average_fps: 25.0\n#   ... (truncated output)\n\n# Simple Indexing API\ndecoder[0]  # uint8 tensor of shape [C, H, W]\ndecoder[0 : -1 : 20]  # uint8 stacked tensor of shape [N, C, H, W]\n\n# Indexing, with PTS and duration info:\ndecoder.get_frames_at(indices=[2, 100])\n# FrameBatch:\n#   data (shape): torch.Size([2, 3, 270, 480])\n#   pts_seconds: tensor([0.0667, 3.3367], dtype=torch.float64)\n#   duration_seconds: tensor([0.0334, 0.0334], dtype=torch.float64)\n\n# Time-based indexing with PTS and duration info\ndecoder.get_frames_played_at(seconds=[0.5, 10.4])\n# FrameBatch:\n#   data (shape): torch.Size([2, 3, 270, 480])\n#   pts_seconds: tensor([ 0.4671, 10.3770], dtype=torch.float64)\n#   duration_seconds: tensor([0.0334, 0.0334], dtype=torch.float64)\n```\n\n#### Clip sampling\n\n```python\n\nfrom torchcodec.samplers import clips_at_regular_timestamps\n\nclips_at_regular_timestamps(\n    decoder,\n    seconds_between_clip_starts=1.5,\n    num_frames_per_clip=4,\n    seconds_between_frames=0.1\n)\n# FrameBatch:\n#   data (shape): torch.Size([9, 4, 3, 270, 480])\n#   pts_seconds: tensor([[ 0.0000,  0.0667,  0.1668,  0.2669],\n#         [ 1.4681,  1.5682,  1.6683,  1.7684],\n#         [ 2.9696,  3.0697,  3.1698,  3.2699],\n#         ... (truncated), dtype=torch.float64)\n#   duration_seconds: tensor([[0.0334, 0.0334, 0.0334, 0.0334],\n#         [0.0334, 0.0334, 0.0334, 0.0334],\n#         [0.0334, 0.0334, 0.0334, 0.0334],\n#         ... (truncated), dtype=torch.float64)\n```\n\nYou can use the following snippet to generate a video with FFmpeg and tryout\nTorchCodec:\n\n```bash\nfontfile=/usr/share/fonts/dejavu-sans-mono-fonts/DejaVuSansMono-Bold.ttf\noutput_video_file=/tmp/output_video.mp4\n\nffmpeg -f lavfi -i \\\n    color=size=640x400:duration=10:rate=25:color=blue \\\n    -vf \"drawtext=fontfile=${fontfile}:fontsize=30:fontcolor=white:x=(w-text_w)/2:y=(h-text_h)/2:text='Frame %{frame_num}'\" \\\n    ${output_video_file}\n```\n\n## Installing TorchCodec\n### Installing CPU-only TorchCodec\n\n1. Install the latest stable version of PyTorch following the\n   [official instructions](https://pytorch.org/get-started/locally/). For other\n   versions, refer to the table below for compatibility between versions of\n   `torch` and `torchcodec`.\n\n2. Install FFmpeg, if it's not already installed. Linux distributions usually\n   come with FFmpeg pre-installed. TorchCodec supports all major FFmpeg versions\n   in [4, 7].\n\n   If FFmpeg is not already installed, or you need a more recent version, an\n   easy way to install it is to use `conda`:\n\n   ```bash\n   conda install ffmpeg\n   # or\n   conda install ffmpeg -c conda-forge\n   ```\n\n3. Install TorchCodec:\n\n   ```bash\n   pip install torchcodec\n   ```\n\nThe following table indicates the compatibility between versions of\n`torchcodec`, `torch` and Python.\n\n| `torchcodec`       | `torch`            | Python              |\n| ------------------ | ------------------ | ------------------- |\n| `main` / `nightly` | `main` / `nightly` | `\u003e=3.9`, `\u003c=3.13`   |\n| `0.3`              | `2.7`              | `\u003e=3.9`, `\u003c=3.13`   |\n| `0.2`              | `2.6`              | `\u003e=3.9`, `\u003c=3.13`   |\n| `0.1`              | `2.5`              | `\u003e=3.9`, `\u003c=3.12`   |\n| `0.0.3`            | `2.4`              | `\u003e=3.8`, `\u003c=3.12`   |\n\n### Installing CUDA-enabled TorchCodec\n\nFirst, make sure you have a GPU that has NVDEC hardware that can decode the\nformat you want. Refer to Nvidia's GPU support matrix for more details\n[here](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new).\n\n1. Install Pytorch corresponding to your CUDA Toolkit using the\n   [official instructions](https://pytorch.org/get-started/locally/). You'll\n   need the `libnpp` and `libnvrtc` CUDA libraries, which are usually part of\n   the CUDA Toolkit.\n\n2. Install or compile FFmpeg with NVDEC support.\n   TorchCodec with CUDA should work with FFmpeg versions in [4, 7].\n\n   If FFmpeg is not already installed, or you need a more recent version, an\n   easy way to install it is to use `conda`:\n\n   ```bash\n   conda install ffmpeg\n   # or\n   conda install ffmpeg -c conda-forge\n   ```\n\n   If you are building FFmpeg from source you can follow Nvidia's guide to\n   configuring and installing FFmpeg with NVDEC support\n   [here](https://docs.nvidia.com/video-technologies/video-codec-sdk/12.0/ffmpeg-with-nvidia-gpu/index.html).\n\n   After installing FFmpeg make sure it has NVDEC support when you list the supported\n   decoders:\n\n   ```bash\n   ffmpeg -decoders | grep -i nvidia\n   # This should show a line like this:\n   # V..... h264_cuvid           Nvidia CUVID H264 decoder (codec h264)\n   ```\n\n   To check that FFmpeg libraries work with NVDEC correctly you can decode a sample video:\n\n   ```bash\n   ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i test/resources/nasa_13013.mp4 -f null -\n   ```\n\n3. Install TorchCodec by passing in an `--index-url` parameter that corresponds\n   to your CUDA Toolkit version, example:\n\n   ```bash\n   # This corresponds to CUDA Toolkit version 12.6. It should be the same one\n   # you used when you installed PyTorch (If you installed PyTorch with pip).\n   pip install torchcodec --index-url=https://download.pytorch.org/whl/cu126\n   ```\n\n   Note that without passing in the `--index-url` parameter, `pip` installs\n   the CPU-only version of TorchCodec.\n\n## Benchmark Results\n\nThe following was generated by running [our benchmark script](./benchmarks/decoders/generate_readme_data.py) on a lightly loaded 22-core machine with an Nvidia A100 with\n5 [NVDEC decoders](https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvdec-application-note/index.html#).\n\n![benchmark_results](./benchmarks/decoders/benchmark_readme_chart.png)\n\nThe top row is a [Mandelbrot](https://ffmpeg.org/ffmpeg-filters.html#mandelbrot) video\ngenerated from FFmpeg that has a resolution of 1280x720 at 60 fps and is 120 seconds long.\nThe bottom row is [promotional video from NASA](https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4)\nthat has a resolution of 960x540 at 29.7 fps and is 206 seconds long. Both videos were\nencoded with libx264 and yuv420p pixel format. All decoders, except for TorchVision, used FFmpeg 6.1.2. TorchVision used FFmpeg 4.2.2.\n\nFor TorchCodec, the \"approx\" label means that it was using [approximate mode](https://pytorch.org/torchcodec/stable/generated_examples/approximate_mode.html)\nfor seeking.\n\n## Contributing\n\nWe welcome contributions to TorchCodec! Please see our [contributing\nguide](CONTRIBUTING.md) for more details.\n\n## License\n\nTorchCodec is released under the [BSD 3 license](./LICENSE).\n\nHowever, TorchCodec may be used with code not written by Meta which may be\ndistributed under different licenses.\n\nFor example, if you build TorchCodec with ENABLE_CUDA=1 or use the CUDA-enabled\nrelease of torchcodec, please review CUDA's license here:\n[Nvidia licenses](https://docs.nvidia.com/cuda/eula/index.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2Ftorchcodec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpytorch%2Ftorchcodec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2Ftorchcodec/lists"}