PyTorch Consolidates Media I/O Into TorchCodec, Focuses TorchVision and TorchAudio on Transforms

PyTorch has consolidated all media decoding and encoding into TorchCodec, a library that now handles images, video, and audio on CPU and CUDA. TorchVision and TorchAudio are now focused on transforms, with models, datasets, and pipelines no longer under active development. The change addresses the previous fragmentation where decoding and encoding capabilities were scattered across TorchVision and TorchAudio, each with multiple backends and entry points.

The new split

TorchCodec turns media files or encoded bytes into tensors for decoding, and tensors back into media files for encoding. TorchVision and TorchAudio transform the tensors in between. For video, TorchCodec's VideoDecoder returns a FrameBatch, which can be passed to TorchVision's v2 transforms. For audio, AudioDecoder returns AudioSamples that feed into TorchAudio's transforms like MelSpectrogram. Images work similarly, with decode_image returning plain tensors for torchvision.transforms.v2. Encoding goes in the other direction with VideoEncoder, AudioEncoder, JpegEncoder, and PngEncoder.

The consolidation was driven by three factors. First, a single centralized location means users no longer need to figure out which library offers the decoder they need. Second, performance optimizations now go into one library, and TorchCodec is generally more performant than the previous implementations, particularly for CUDA video decoding. Third, maintenance is simplified because media I/O involves many dependencies, including six major FFmpeg versions, NVIDIA's codec SDK, and per-format image libraries, all of which are C/C++ and have complex licensing rules.

ABI stability and migration

All three libraries are now ABI stable, meaning a given version of TorchCodec, TorchVision, or TorchAudio is not tied to a single PyTorch version. They keep working with subsequent PyTorch versions and no longer need to be rebuilt for every release, so their release cadence no longer matches PyTorch's.

The transition was particularly disruptive for TorchAudio, where many APIs were deprecated and eventually removed. Some popular APIs originally slated for removal were kept after community feedback. The migration guide is available for users still on the old decoding and encoding APIs in TorchVision or TorchAudio. A follow-up post will cover what's new in each library and how to get the most performance out of a full decoding and transform pipeline.