Slower than decord

[xiaosu-zhu]

🐛 Describe the bug

I have benchmarked torchcodec with decord and found this suite is slower than that. Here is the benchmark code:

Note: Both of them are using cpu codec.

def torchcodec_pure(video_path):
    decoder = VideoDecoder(video_path, device="cpu")
    picked_frames = np.linspace(0, decoder.metadata.num_frames - 1, 768, dtype=int)

    # warm up
    for _ in range(1):
        decoder.get_frames_at(indices=picked_frames)

    # benchmark
    start = time()
    for _ in trange(10):
        decoder.get_frames_at(indices=picked_frames)
    end = time()
    print("Duration:", end - start)
    return end - start


def decord_pure(video_path):
    vr = VideoReader(video_path)
    picked_frames = np.linspace(0, len(vr) - 1, 768, dtype=int)

    # warm up
    for _ in range(1):
        vr.get_batch(picked_frames)

    # benchmark
    start = time()
    for _ in trange(10):
        vr.get_batch(picked_frames)
    end = time()
    print("Duration:", end - start)
    return end - start

The torchcodec version gives:

Duration (10 iters): 909.5s

The decord version gives:

Duration (10 iters): 279.1s

The video info ffmpeg gives is:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'subPlot_new_all_73.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:10:51.88, start: 0.000000, bitrate: 2625 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1912x806, 2490 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler

Any idea for a speedup?

Versions

PyTorch version: 2.5.1
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Alibaba Cloud Linux release 3 (Soaring Falcon) (x86_64)
GCC version: (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3 2.32)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.32

Python version: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct 4 2024, 13:27:36) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.10.134-007.ali5000.al8.x86_64-x86_64-with-glibc2.32
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: Tesla V100-PCIE-32GB
GPU 1: Tesla V100-PCIE-32GB

Nvidia driver version: 520.61.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 96
On-line CPU(s) list: 0-95
Thread(s) per core: 2
Core(s) per socket: 24
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
Stepping: 4
CPU MHz: 2699.951
CPU max MHz: 3100.0000
CPU min MHz: 1000.0000
BogoMIPS: 5000.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 33792K
NUMA node0 CPU(s): 0-95
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d

Versions of relevant libraries:
[pip3] numpy==2.0.2
[pip3] pytorch-lightning==2.4.0
[pip3] pytorch-metric-learning==2.7.0
[pip3] torch==2.5.1
[pip3] torch-audiomentations==0.11.1
[pip3] torch_pitch_shift==1.2.5
[pip3] torchaudio==2.5.1
[pip3] torchmetrics==1.6.0
[pip3] torchvision==0.20.1
[pip3] triton==3.1.0
[conda] blas 1.0 mkl
[conda] cuda-cudart 11.8.89 0 nvidia
[conda] cuda-cupti 11.8.87 0 nvidia
[conda] cuda-libraries 11.8.0 0 nvidia
[conda] cuda-nvrtc 11.8.89 0 nvidia
[conda] cuda-nvtx 11.8.86 0 nvidia
[conda] cuda-runtime 11.8.0 0 nvidia
[conda] libcublas 11.11.3.6 0 nvidia
[conda] libcufft 10.9.0.58 0 nvidia
[conda] libcurand 10.3.7.77 0 nvidia
[conda] libcusolver 11.4.1.48 0 nvidia
[conda] libcusparse 11.7.5.86 0 nvidia
[conda] libjpeg-turbo 2.0.0 h9bf148f_0 pytorch
[conda] mkl 2023.1.0 h213fc3f_46344
[conda] mkl-service 2.4.0 py312h5eee18b_1
[conda] mkl_fft 1.3.11 py312h5eee18b_0
[conda] mkl_random 1.2.8 py312h526ad5a_0
[conda] numpy 2.0.2 pypi_0 pypi
[conda] pytorch 2.5.1 py3.12_cuda11.8_cudnn9.1.0_0 pytorch
[conda] pytorch-cuda 11.8 h7e8668a_6 pytorch
[conda] pytorch-lightning 2.4.0 pypi_0 pypi
[conda] pytorch-metric-learning 2.7.0 pypi_0 pypi
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torch-audiomentations 0.11.1 pypi_0 pypi
[conda] torch-pitch-shift 1.2.5 pypi_0 pypi
[conda] torchaudio 2.5.1 py312_cu118 pytorch
[conda] torchcodec 0.1.1 pypi_0 pypi
[conda] torchmetrics 1.6.0 pypi_0 pypi
[conda] torchtriton 3.1.0 py312 pytorch
[conda] torchvision 0.20.1 py312_cu118 pytorch

[scotts]

@xiaosu-zhu, thanks for looking into this! We have a benchmark script that is part of our repo that also can compare against Decord. When I run that with the command:

python benchmarks/decoders/benchmark_decoders.py --decoders decord,decord_batch,torchcodec_public --bm_video_speed_min_run_seconds 20

I get the results:

[------------- video=/home/scottas/github/torchcodec/benchmarks/decoders/../../test/resources/nasa_13013.mp4 h264 480x270, 13.013s 29.97002997002997fps -------------]
                           |  decode 10 uniform frames  |  decode 10 random frames  |  first 1 frames  |  first 10 frames  |  first 100 frames  |  create decode first
1 threads: -----------------------------------------------------------------------------------------------------------------------------------------------------------
      TorchCodecPublic     |           413.4            |           375.0           |      118.1       |       140.1       |       404.7        |                     
      DecordAccurateBatch  |           413.2            |           946.4           |       89.8       |       122.2       |       375.9        |                     
      DecordAccurate       |           403.5            |           979.1           |       89.2       |       114.6       |       527.4        |                     
      TorchCodecCore       |                            |                           |                  |                   |                    |          10.4       

Times are in milliseconds (ms).

My system has 56 available cores. The first column, decode 10 uniform frames, is most similar to your benchmark code, and there we have torchcodec and Decord being competitive. However, comparing your code to our benchmark (code is linked above) is that in our benchmark, we instantiate a new decoder object for each new iteration. In your code, you're re-using the same decoder object. That means there will always be a backward seek at the start of the next iteration - that's something we haven't benchmarked yet!

I'd love to dig into this more, so here's some follow-ups that could help us figure out more:

1. Can you run the benchmark from our repo above using the exact command I provide and tell us your results?
2. Can you run the same benchmark and provide your video instead of using our default? You can specify the video path with the --bm_video_paths option. Again, please report the results.
3. In your benchmark code, can you try creating a new decoder for both torchcodec and Decord in each iteration and report the results?
4. Can you provide us a link the video you used? I'd love to reproduce your results locally in my own environment.

Let me know if there's anything above that needs clarification!