Vulkan Video Decoding: Usage Guide and FAQ

[philipl]

Vulkan Video Decoding: Usage Guide and FAQ

On the 28th of May, we reached the significant milestone of finally merging all the required functionality into ffmpeg, libplacebo, and mpv to do end-to-end Vulkan video decoding and presentation. Although the functionality is now all there, we still have a complex landscape in terms of what is supported on what hardware and with what drivers. This document attempts to lay out all the requirements and limitations, so you have a chance of successfully using the feature.

Why should I even care about Vulkan video decoding?

It's a fair question. Right now, the actual functionality you gain access to is not terribly different from what you would get with existing video decoding APIs that ffmpeg and mpv already support. However, Vulkan video decoding has the potential to be a credible cross-vendor, cross-platform API that is well supported, and can work efficiently with Vulkan based filtering and post-processing. In the short term, there will be rough edges, and only a narrow set of supported codecs (only H.264 and H.265 are standardised today, with AV1 still a work in progress), but in the future, it will hopefully be the most performant and capable way to do hardware decoding and processing.

Software Requirements

We have finally reached the point where all the required components have made official releases. Ensure you are running with the following releases or newer:

• ffmpeg: 6.1
• libplacebo: v6.292.0
• mpv: 0.37.0

These requirements mean you will likely have to compile everything for yourself (especially on Linux), but Shinchiro's windows builds appear to be functional.

Hardware and Driver Requirements

• Intel on Linux
  • Mesa: mesa-23.2.1
  • A GPU supported by the ANV driver
  • An MR is open for support in the hasvk driver
• AMD on Linux
  • Mesa: mesa-23.2.1
  • A GPU supported by the RADV driver
  • The AMDVLK driver is not known to support video decoding
• Nvidia on Linux or Windows
  • Drivers >= 535.xx
  • Any GPU supported by those drivers
• AMD on Windows
  • Drivers >= 23.9.3
  • RDNA 1/2/3 GPUs
• Intel on Windows
  • Unclear and untested
  • There is evidence that drivers from at least 101.4311 expose the video decoding extensions
  • No idea what hardware is supported
• MacOS (for the record)
  • Although Vulkan on MacOS is supported via MoltenVk, this does not include the decoding extensions, and probably never will. Just keep using videotoolbox.

Usage

The basic command line arguments:

mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan

Depending on your system configuration, you may additionally need to force the right gpu-context:

• Windows: --gpu-context=winvk
• Linux X11: --gpu-context=x11vk
• Linux Wayland: --gpu-context=waylandvk

Capabilities

• Global
  • The Vulkan Video Decoding specs only cover H.264, H.265, and AV1. Older codecs will require you to use your traditional hardware decoding API (VAAPI, nvdec, dx11va, etc) for now, and possibly forever - depending on whether effort is put into exposing older codecs.
• Intel
  • You must set ANV_VIDEO_DECODE=1 in your environment to expose video decoding
  • Both H.264 and H.265 are supported
  • Vulkan filters are supported from Mesa 24.1 onward.
• AMD
  • You must set RADV_PERFTEST=video_decode in your environment to expose video decoding
  • Both H.264 and H.265 are supported
  • Vulkan filters are supported
• Nvidia
  • Both H.264 and H.265 are supported
  • AV1 is supported with > 550.54.14 drivers
  • Vulkan filters are supported

[Jules-A]

AMD on Windows appears to be 5000/6000 series according to https://www.amd.com/en/support/kb/release-notes/rn-rad-win-22-11-2-vlk-video-code-decode (possibly 7000 works?). It doesn't work on Polaris: 0.339][e][ffmpeg/video] h264: Device does not support the VK_KHR_video_decode_queue extension!

[llyyr]

After bc28dce, you need to patch your Mesa with this PR to get AV1 decoding on radv.

[aufkrawall]

Reliably crashes driver for me with this sample: https://0x0.st/HbHJ.mkv
Other files seem to work for whatever reason (hwdec beeing used confirmed).
Latest git-master of mpv, libplacebo, ffmpeg and radv on 6700 xt.

[philipl]

Sounds like you probably need to file a mesa issue for that - and even if it was an mpv issue, please open a separate one. This FAQ isn't the place to report it.

[cyanreg]

@aufkrawall AMD currently needs https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227 to avoid crashes, which also fixes quite a lot of bugs

[alakittty]

 (+) Video --vid=1 (*) (hevc 3828x1592 23.976fps)
 (+) Audio --aid=1 --alang=eng (*) (eac3 6ch 48000Hz)
     Subs  --sid=1 --slang=eng (subrip)
     Subs  --sid=2 --slang=eng 'SDH' (subrip)
     Subs  --sid=3 --slang=fre (subrip)
[vo/gpu-next/libplacebo] vk->CreateDevice(vk->physd, &dinfo, PL_VK_ALLOC, &vk->dev): VK_ERROR_OUT_OF_HOST_MEMORY (../../../../../src_packages/libplacebo/src/vulkan/context.c:1308)
[vo/gpu-next/libplacebo] Failed creating logical device!
[vo/gpu-next/libplacebo] Failed initializing vulkan device
[vo/gpu-next] Failed initializing any suitable GPU context!
Error opening/initializing the selected video_out (--vo) device.
Video: no video

what does this error mean, my device has amd and i used the following config
mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk
mpv was working fine with gpu-next and gpu-api=vulkan till yesterdays' shinchiro release. todays release gave me this error
maybe @shinchiro needs to bump libplacebo and ffmpeg version as per required?

[Jules-A]

AMD on Windows
Special 22.11.2 drivers with video decoding support
Hardware supported by those drivers (documented as 5000/6000 series. Apparently not newer 7000 series)

I don't know if that specific driver is required, they may have been included in 23.2.1 where they added other extensions, the driver notes aren't clear. 7000 series wasn't launched when the 22-11-2 Vulkan video drivers were released so that driver doesn't officially support them. If they did include the video extensions in newer drivers then they may support 7000 series but someone with those GPUs will need to test.

[cyanreg]

Yeah, I've reported to nvidia months ago that their descriptor buffer implementation is broken, but they haven't fixed it yet.
If you want filters working, ask nvidia to fix it.

[philipl]

I found this video will white screen when using ffmpeg vulkan decoding, SW decoding or nvpro vk_video_decoder have no problem, I don't know if this is also NVIDIA's problem (also already reported to NVIDIA)
test.mp4

Github claims you attached an audio-only file...

[llyyr]

That file is also broken on anv, it's glitchy and the decode is very slow while utilizing 100% of Render/3D engine.

[philipl]

That file is also broken on anv, it's glitchy and the decode is very slow while utilizing 100% of Render/3D engine.

It's an HEVC file. My experience on ANV is that any HEVC playback pegs the GPU at 100%

[philipl]

Also:

[ffmpeg/video] hevc: First slice in a frame missing.

This is a highly questionable sample.

[lavilao]

Will this work with hasvk in the future? I tried using it through wine (I dont want to have to compile everything) and got this error [ffmpeg/video] h264: Device does not support the VK_KHR_video_decode_queue extension! even though the output of vulkaninfo is:

vulkaninfo | grep video                                                                     ✔  10s  
	VK_KHR_video_decode_h264                    : extension revision 8
	VK_KHR_video_decode_queue                   : extension revision 7
	VK_KHR_video_queue                          : extension revision 8
			videoCodecOperations:
			videoCodecOperations: count = 1

[CharlesMengCA]

NVIDIA 1650 Super, Tested on windows with: mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk
5 videos played OK.

This video has problem: https://4kmedia.org/sony-whale-in-tonga-hdr-uhd-4k-demo/
Switched back to hwdec=nvdec, the problem is gone.

[24fpsDaVinci]

for fedora, you need to edit mesa.spec and rebuild with codecs enabled -D video-codecs=h264dec,h264enc,h265dec,h265enc,vc1dec, codecs is disabled by default.

rpmfusion-infra/fedy#110

https://themaister.net/blog/2023/01/05/vulkan-video-shenanigans-ffmpeg-radv-integration-experiments/

https://forum.endeavouros.com/t/compiling-mesa-22-2-with-codec-support-the-easy-way/30390

[temis01]

I tried changing the repo url ... but I missed the "GIT_TAG av1dec" + "GIT_REMOTE_NAME origin" then it failed at ffmpeg prefix.
Thanks

[temis01]

@cyanreg ffmpeg repo does not work for me

[cyanreg]

Try https://github.com/cyanreg/FFmpeg/tree/av1dec2
I'll have more time to test in a few hours

[temis01]

getting same old

av1dec2.txt

[cyanreg]

AV1 should work on NVIDIA now with git master of ffmpeg.

[temis01]

The patch worked for me.
mpv 0.38
Windows 11 22631.3593
RTX 3050 8Go + drv 552.44

#13375

[wyup]

vulkan renderer working good (except green screen with hwdec=vulkan) on i5-13500 iGPU (UHD 770), Win11 x64, shinchiro build v3, latest intel driver 101.5518, Vulkan api version 1.3.280, conformance 1.3.6.3.
Same timing performance to d3d11 on with gpu-next, gpu-api=vulkan, gpu-context=winvk,tone-mapping=bt.2446a, profile=gpu-hq, deband. It doesn't crash as before. I hope hwdec=vulkan works aswell soon.

mpv N-115189-g0664cbd73 (May 13) libplacebo version: v7.349.0 (v6.338.0-130-gbf1855c-dirty)

Good job!

[tramchamploo]

--vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=waylandvk not working,

[vo/gpu-next/libplacebo] vk->CreateSwapchainKHR(...): VK_ERROR_INITIALIZATION_FAILED (../src/vulkan/swapchain.c:620)
[vo/gpu-next/libplacebo] Failed (re)creating swapchain!

but gpu-context=x11vk is working under both x11 and wayland. Guess xwayland doing its job on wayland.
Laptop linux user with optimus nvidia 3070ti.

[llyyr]

Open an issue after filling out the template. This seems unrelated to vulkan video decoding

[averne]

@cyanreg the attached file fails to display properly on nvidia gpus, with mpv --hwdec=vulkan --gpu-api=vulkan out.mp4.
The luma and chroma planes decode properly, however the yuv2rgb conversion fails.
From my understanding, the chroma plane having a height of 8 leads to the graphics driver misinterpreting the layout of the data output by the decode engine. This is possibly stemming from the fact that the NVDEC hardware engine always outputs tiled surfaces with a GOB-height of 2 (ie. 16 pixel lines), while the more natural layout would be 1 GOB for the chroma plane.
This behavior is not reproduced when using the nvdec hwaccel, because the decoded data will go through a copy step instead of being mapped directly in the 3D engine.

Tested configurations:

• NVIDIA GeForce GTX 950, Arch Linux, nvidia driver 550.78 by myself
• NVIDIA GeForce RTX 3060 Ti, Arch Linux, nvidia driver 535.113 by @po5

out.mp4

[rothn]

This is great! But why do I need to set special flags for this to work?

[CounterPillow]

mpv support for HDR is experimental and only exists in the next-gen implementation. As it becomes more common, this is just what everyone will have to use. VA-API does work, but doesn't produce the correct color mappings in my case and, importantly, does not output an HDR signal to my monitor (only Vulkan can).

This is complete gobbledegook. The colormapping does not depend on the hardware decode API. The metadata signalling to your display also does not depend on the hardware decode API (or the graphics API, as a matter of fact). You can use Vulkan rendering with gpu-next with VA-API hardware decoding. You do not need Vulkan Video decode for HDR.

[rothn]

Well, why didn't you just say so? I did not see that option in the decoder enumeration, but it might be helpful for others who encounter this if you post an output command. That said, lots of the behavior around HDR is fairly counter-intuitive, at least partially because HDR support involves a good amount of reverse-engineering.

[CounterPillow]

Well, why didn't you just say so?

Because you are the first person to have this belief. I know it's hard for you to accept, but you are the outlier here.

[shmerl]

Google, is currently trying to kill this API by mandating all Android platforms do not implement the Vulkan extensions for video decode/encode at all

Is there some place to read more about it? I saw a reference to "there are some problems on Android" here, but there were no details.

[shmerl]

I found it:

• https://issuetracker.google.com/issues/293419320
• https://source.android.com/docs/compatibility/13/android-13-cdd#7142_vulkan

[C-1-11] MUST NOT enumerate support for the VK_KHR_video_queue, VK_KHR_video_decode_queue, or VK_KHR_video_encode_queue extensions.

The Vulkan Video extensions are incompatible with Android’s graphics driver architecture and multimedia security architecture. As some vendors attempt to provide as many extensions as possible, forbidding these prevents compromising security and creating a status-quo dependence on the extensions among 3rd party applications.

The industry-standard Vulkan API has a mechanism for querying which extensions are provided by an implementation (vkEnumerateDeviceExtensionProperties). Extension specifications are hosted in the Khronos Vulkan Registry (https://www.khronos.org/registry/vulkan/) outside of Android API documentation. Android’s

This is a really weird take. Is security claim BS? Or may be they mean it's incompatible with DRM?

[Obegg]

vo=gpu-next
hwdec=vulkan
gpu-api=vulkan
video-sync=display-resample

Is it just me or does this results in a vsync error where video is delayed by a few frames from the audio (the audio comes first and then the video)?
The issue is fixed by removing hwdec=vulkan and gpu-api=vulkan.
I'm on Windows 10 (fresh install from yesterday), NVIDIA GTX 1650 (latest studio driver), tested on this 60fps video and using the latest build (mpv v0.38.0-693-g4e50da3d)

[shmerl]

radv now enabled Vulkan video by default for RDNA 3 (you need latest firmware).

[rothn]

Awesome! Do you know if I will pick this up on Arch through linux-firmware, or do I need to do some manual procedure for this?

[shmerl]

I'm not familiar with Arch's process of updating firmware. You can just use latest from upstream in case it's not fast enough:

https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/log/amdgpu

[llyyr]

What? The firmware has nothing to do with vulkan video, it's exposed by Mesa and it's still behind the RADV_PERFTEST variable.

[shmerl]

You can read it yourself:

• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30837
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30837/diffs?commit_id=7db16e7cdd71d7cafaeca644325bda5ca81be072
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30837/diffs?commit_id=95a980b61fbf4457ce8e2aed260af269ee6334be

With the latest fw in the linux-firmware repo, navi3x passes all the CTS tests.

This was the main missing piece for passing vulkan video CTS as the video firmwares couldn't do proper vulkan events. With new enough firmware this is now possible.

[rankinc]

Vulkan video decoding not working with RDNA2. I am using latest Mesa 24.3-devel:

OpenGL vendor string: AMD
OpenGL renderer string: AMD Radeon RX 6600 (radeonsi, navi23, LLVM 18.1.6, DRM 3.57, 6.10.6)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 24.3.0-devel (git-17f97a69c1)

And I can see the Vulkan video extensions:

$ vulkaninfo | grep video
	VK_KHR_video_decode_av1                       : extension revision 1
	VK_KHR_video_decode_h264                      : extension revision 9
	VK_KHR_video_decode_h265                      : extension revision 8
	VK_KHR_video_decode_queue                     : extension revision 8
	VK_KHR_video_encode_h264                      : extension revision 14
	VK_KHR_video_encode_h265                      : extension revision 14
	VK_KHR_video_encode_queue                     : extension revision 12
	VK_KHR_video_maintenance1                     : extension revision 1
	VK_KHR_video_queue                            : extension revision 8
			videoCodecOperations:
			videoCodecOperations:
			videoCodecOperations: count = 3
			videoCodecOperations: count = 2
			videoCodecOperations:
	videoMaintenance1 = true

However, mpv always drops to using software decoding:

$ mpv --no-config --vo=gpu-next --hwdec=vulkan --gpu-api=vulkan --gpu-context=x11vk ~/Videos/jellyfish-120-mbps-4k-uhd-hevc-10bit.mkv 
 (+) Video --vid=1 (*) (hevc 3840x2160 29.970fps)
VO: [gpu-next] 3840x2160 yuv420p10
V: 00:00:00 / 00:00:30 (0%)../src/vulkan/wsi/wsi_common_x11.c:1912: Swapchain status changed to VK_SUBOPTIMAL_KHR
Exiting... (Quit)

According to --msg-level=all=debug, there is some unspecified error when trying to create the Vulkan device:

...
[vd] Container reported FPS: 29.970000
[vd] Codec list:
[vd]     hevc - HEVC (High Efficiency Video Coding)
[vd]     hevc_qsv (hevc) - HEVC video (Intel Quick Sync Video acceleration)
[vd]     hevc_v4l2m2m (hevc) - V4L2 mem2mem HEVC decoder wrapper
[vd]     hevc_cuvid (hevc) - Nvidia CUVID HEVC decoder
[vd] Opening decoder hevc
[vd] Looking at hwdec hevc-vulkan...
[vo/gpu-next] Loading hwdec drivers for format: 'vulkan'
[vd] Could not create device.
[vd] Using software decoding.
...

[rankinc]

I compiled Mesa 24.3-devel myself to include H.264 and H.265 video codecs, again as evidenced by the vulkaninfo output. Not only that, and for the record:

$ mpv --hwdec=help | grep hevc
  nvdec (hevc-nvdec)
  vaapi (hevc-vaapi)
  vdpau (hevc-vdpau)
  nvdec-copy (hevc-nvdec-copy)
  vaapi-copy (hevc-vaapi-copy)
  vdpau-copy (hevc-vdpau-copy)
  vulkan (hevc-vulkan)
  qsv (hevc_qsv-qsv)
  cuda (hevc_cuvid-cuda)
  vulkan-copy (hevc-vulkan-copy)
  qsv-copy (hevc_qsv-qsv-copy)
  cuda-copy (hevc_cuvid-cuda-copy)
  v4l2m2m-copy (hevc_v4l2m2m-v4l2m2m-copy)

Nor do I get hardware decoding for an AV1 video, and my ffmpeg is from RPM Fusion:

$ dnf info ffmpeg
...
Installed Packages
Name         : ffmpeg
Version      : 6.1.2
Release      : 2.fc40
Architecture : x86_64
Size         : 2.3 M
Source       : ffmpeg-6.1.2-2.fc40.src.rpm
Repository   : @System
From repo    : rpmfusion-free-updates
...

[kasper93]

Does --hwdec=vulkan-copy work then? Or better, does ffmpeg -hide_banner -hwaccel vulkan -hwaccel_output_format vulkan -i <file> -f null - work?

EDIT:

Ok, mpv build has vulkan decoding support disabled. https://kojipkgs.fedoraproject.org/packages/mpv/0.37.0/4.fc40/data/logs/x86_64/build.log

 vulkan-interop: disabled

So it won't work. I'm not sure why the error message is so generic, but this needs to be enabled for Vulkan decoding to work.

[rankinc]

Does --hwdec=vulkan-copy work then? Or better, does ffmpeg -hide_banner -hwaccel vulkan -hwaccel_output_format vulkan -i <file> -f null - work?

You will need to tell me whether this has "worked", because I don't see any reference to hardware decoding here:

$ ffmpeg -hide_banner -hwaccel vulkan -hwaccel_output_format vulkan -i ~/Videos/jellyfish-120-mbps-4k-uhd-hevc-10bit.mkv -f null output.mkv
Input #0, matroska,webm, from '/home/chris/Videos/jellyfish-120-mbps-4k-uhd-hevc-10bit.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso4hvc1iso6
    MAJOR_BRAND     : iso4
    MINOR_VERSION   : 1
    ENCODER         : Lavf56.3.100
  Duration: 00:00:30.10, start: 0.067000, bitrate: 119845 kb/s
  Stream #0:0(und): Video: hevc (Main 10), yuv420p10le(tv), 3840x2160 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn (default)
    Metadata:
      CREATION_TIME   : 2016-02-05 13:00:09
      LANGUAGE        : und
      HANDLER_NAME    : [email protected]
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'output.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso4hvc1iso6
    MAJOR_BRAND     : iso4
    MINOR_VERSION   : 1
    encoder         : Lavf60.16.100
  Stream #0:0(und): Video: wrapped_avframe, vulkan(tv, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn (default)
    Metadata:
      CREATION_TIME   : 2016-02-05 13:00:09
      LANGUAGE        : und
      HANDLER_NAME    : [email protected]
      encoder         : Lavc60.31.102 wrapped_avframe
frame=    0 fps=0.0 q=-0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=  [out#0/null @ 0x55652baf7ac0] video:422kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame=  900 fps=128 q=-0.0 Lsize=N/A time=00:00:29.99 bitrate=N/A speed=4.26x   

[kasper93]

You will need to tell me whether this has "worked"

Looks like it does,

Stream #0:0(und): Video: wrapped_avframe, vulkan(tv, progressive)

it outputs vulkan wrapped frame. If you want more info add -v verbose. Either way it seems ok on this part. I can reproduce Could not create device. when vulkan-interop is disabled. We kinda support Vulkan, but without important part for decoding, so it ends up with device creation failure. This should be improved for better log or even hide vulkan decoding in this case, but I hope we can get rid of this build option all together to resolve those issues.

Currently your only option is to rebuild mpv with this feature enabled.

[mrintrepide]

Fedora mpv build not work properly with vulkan video. Need to use RPMfusion ffmpeg and build mesa (Fedora and RPMfusion doesn't provide vulkan video h264/h265 codec)
I use flatpak version with flatpak mesa-git
FLATPAK_GL_DRIVERS=mesa-git flatpak run --env=RADV_PERFTEST=video_decode,video_encode io.mpv.Mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan

[Shotweb]

[llyyr]

Open a new issue, this thread is for FAQs related to Vulkan Video decoding, not the rendering API.

[dwlsalmeida]

@philipl could you please link to https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31867 for the FOSS implementation for NVIDIA on Linux?

It's still early stages and we only support H.264 so far, but I'd appreciate if more people could test it out. The plan is to support Turing and up on NVK.