Color differences between CPU and GPU decoded frames

[NicolasHug]

From @pjs102793's #424 (comment)

to generate sample video: ffmpeg -f lavfi -i testsrc2=size=1920x1080:rate=30 -t 1 testsrc2.mp4

On main b0de666

Repro:

import torch
from torchcodec.decoders import VideoDecoder

cpudec = VideoDecoder("/home/nicolashug/testsrc2.mp4", device="cpu")
gpudec = VideoDecoder("/home/nicolashug/testsrc2.mp4", device="cuda")

cpu_frames = cpudec[:]
gpu_frames = gpudec[:].cpu()


from torchvision.transforms.v2.functional import to_pil_image
from torchvision.utils import make_grid


for i, (cpu, gpu) in enumerate(zip(cpu_frames, gpu_frames)):
    img = to_pil_image(make_grid([cpu, gpu], nrow=2))
    img.save(f"/home/nicolashug/imgs/{i}.jpg")

torch.testing.assert_close(cpu_frames, gpu_frames.cpu())

We're getting pretty high atol differences:

AssertionError: Tensor-likes are not equal!

Mismatched elements: 165091589 / 186624000 (88.5%)
Greatest absolute difference: 48 at index (6, 0, 189, 236)
Greatest relative difference: inf at index (0, 2, 8, 4)

Looking at the decoded frames, they look very similar but the colors are a bit different. So it seems to be a color issue, not an indexing or seeking issue. Example frames (cpu left, gpu right):

Note that it's unlikely to be related or caused by #413, because I'm observing the exact same differences when reverting that commit.

Thanks again @pjs102793 for the initial report. Do you have any intuition on what might be causing those differences on this specific video? From what I understand, you didn't notice major difference either on other videos, right?

[pjs102793]

Yes, the videos I have, encoded in H.264 with YUV420p, don’t seem to have any issues. I even tested with the Big Buck Bunny video, and it worked fine. The issue appears to occur only with testsrc2. I’m not entirely certain, but it could possibly be related to testsrc2 having its colorspace marked as unknown.

Additionally, when I compared the values between torchaudio on GPU and torchcodec on CPU, the differences were minimal. However, the difference between torchaudio on GPU and torchcodec on GPU was comparable to the difference between torchcodec on CPU and GPU. This suggests that under certain conditions, it’s possible that torchcodec on GPU might produce slightly different colors.