This was a great read and finally explained why Vulkan Video is good being "just" a layer ontop of existing HW decode/encode. What's interesting, then, is that Nvidia and others are supporting it. As opposed to rendering where Intel, AMD and Nvidia have their own rendering backend in eg. blender. Nvidia has Optix and CUDA, AMD has HIP, Intel has oneAPI (ironic given this discussion) and Apple has Metal.[0] I guess there isn't much value in being unique for something that's anyways implementing an existing standard (in this case the codecs themselves like h.265).
Video decoding is a massively simpler API surface than a GPU programming framework like CUDA or HIP. Exposing the APIs in Vulkan is not that much extra work.
All of these APIs are very low-level, that’s why the quirks. There’s one system-specific video acceleration API which is not on the list, supports all GPUs, and just works without quirks, that’s IMFMediaEngine interface https://learn.microsoft.com/en-us/windows/win32/api/mfmediae...
You give it a media file, and ask to play. The API delivers frames in DXGI surfaces, and you can do whatever you want with the data.
Ideally, that’s what I would want in all other OSes.
That's the opposite of my experience. Depending on the decoder, lower resolutions may be more efficiently handled by the CPU instead of the hardware decoder (which is a decent chunk of silicon and takes extra power to use). But once you hit a certain resolution, the hardware decoder is pretty much always way more efficient than a software CPU decoder. Some well-implemented hardware decoders even handle low resolutions more efficiently.
My experience is largely focused on battery-powered mobile devices (Android/iOS), though.
To give a concrete example, VAAPI Haswell integrated graphics, peak CPU usage:
mpv hardware H.264 1080p60: 3% CPU usage
mpv software H.264 1080p60: 20% CPU usage
mpv hardware H.264 4K60: 96% CPU usage
mpv software H.264 4K60: 80% CPU usage
There comes a point of diminishing (or in this case, negative) returns. A bit more subjectively, the H.264 software rendering at 4K60 also felt smoother.
I'm using H.264 as it is hardware accelerated on this platform.
I don't think all Haswell chips support 4k H.264. That's one downside of hardware decoders: they have limits on what they support. I suspect the decoder is actually falling back to a software implementation to handle the bitstream since the hardware can't.
I have desktop product that among other things decodes 4K60P using MMF. The decoding part either decodes straight into DirectX texture if hardware based video decoding is supported or does it in software otherwise.
Software based path causes significant CPU load. Using hardware path keeps CPU at about 0%. Since hardware video decoding supported nearly on any modern Laptop, Desktop I would say 4K60 is handled very efficiently.
Low resolution videos (eg. 24 fps non-hd content) are very easy for a modern CPU to decode, so typically other parts of the laptop - ie. the screen, will dominate.
I foresee in the future, as CPU's get even faster, this kind of stuff being decoded by javascript, simply so that 'we can guarantee each platform decodes the video the same'.
[0]https://docs.blender.org/manual/en/latest/render/cycles/gpu_...