GPU Video Decoding Bugs: Rotation, 10-Bit Colour and VFR

FrameQuery decodes footage to pull frames for analysis, and those frames are far smaller than the source. Until version 1.2.9, a hardware decoder handed back full-size frames, FFmpeg downloaded each one to system memory, and the CPU scaled it down. A 3840x2160 frame in NV12 is about 12.4 MB, and we only need a small fraction of it.

So we moved the scaling onto the GPU. The rule for the change was simple: the GPU path had to produce the same frames as the CPU path. Getting there turned up five problems worth knowing about if you build on FFmpeg's hardware decoders.

Portrait phone clips came out sideways

A phone records portrait video as landscape pixels plus a display matrix that says "rotate this 90 degrees". FFmpeg's command-line tool honours that matrix by inserting a transpose filter, which it calls autorotation.

It doesn't do this for hardware frames. With -hwaccel_output_format cuda (or d3d11), the decoded frames live in GPU memory, and FFmpeg skips autorotation for them. A rotated phone clip decoded on NVDEC, D3D11VA or VideoToolbox came out sideways, while the same clip decoded on the CPU came out upright. On a generated test clip the two outputs were 4.7 dB apart.

The fix reads the display matrix from ffprobe and applies the same filter autorotation would, right after the download:

hwdownload,format=nv12,transpose=cclock,<your scale and pad>,format=rgb24

With transpose=cclock after the download, the output matched the CPU decode exactly. Quarter turns and half turns are handled this way. Any other matrix, such as a mirror, goes to the CPU decoder, where FFmpeg's own autorotation applies.

Forcing even sizes moved odd-width sources by a pixel

Our first GPU filter rounded the output size to even numbers, which most GPU scalers want. Sources with odd proportions then landed one pixel off the CPU picture: a 1918x1080 clip scored 26.3 dB against the CPU frames and a 1440x1078 clip 25.7 dB, against 29.3 and 30.3 once fixed.

scale_cuda supports the same force_original_aspect_ratio option and the same size rounding as the CPU scale filter, so on NVIDIA we now let it choose the size:

scale_cuda=w=W:h=H:force_original_aspect_ratio=decrease:format=p016le

scale_d3d11 and scale_vt don't have that option. For those we compute the fitted size in a filter expression, per stream, rounded to even sizes.

Padding in 4:2:0 rounds the offset

Frames that don't fill the target shape are padded with the picture centred. The CPU path pads in RGB. Our GPU path padded the small frame before converting it, while it was still 4:2:0.

In a 4:2:0 frame, chroma is stored at half resolution, so FFmpeg's pad filter rounds the left offset down to the chroma grid. A picture with an odd width landed one pixel left of where the CPU path puts it. Moving the pad after the RGB conversion fixed it.

10-bit footage picked up a colour cast

The GPU scaler's output format decides what reaches the CPU. We used nv12, which is 8-bit, and the scaler truncated 10-bit sources on the way. Against CPU-decoded frames, red and blue shifted by 0.7 to 1.7 levels out of 255. Too small to see in a thumbnail, but the CPU path didn't have it.

For sources deeper than 8 bits, scale_cuda now outputs p016le, which keeps 16 bits per sample, and the CPU's RGB conversion does the rounding exactly as the CPU path does. The cast dropped below 0.1 levels at the same speed, on an RTX 5080 with FFmpeg 8.1, for 10-bit 4:2:0 and 4:2:2 HEVC. 8-bit sources stay nv12, including 4:2:2; trying yuv444p there added a cast of its own, 1.2 to 1.6 levels the other way.

Two related changes landed alongside. D3D11VA, the path for AMD and Intel GPUs on Windows, now decodes 10-bit HEVC into p010 instead of falling back to CPU decode. scale_d3d11 failed on our test machine with FFmpeg 8.1, so a startup probe leaves it off and those machines scale on the CPU after a GPU decode.

Variable frame rate: whose timestamps?

Phones, screen recorders and some drones write variable frame rate, where a clip labelled 30 fps holds stretches at 20. Anything that converts a frame number into a time with index / fps drifts on those clips, so frame times have to come from the stream itself.

Which timestamps you read matters. FFmpeg's default encoder time base is one over the guessed frame rate, so on the encoder side a 20 fps stretch in a clip guessed as 30 fps is snapped to 1/30-second steps, up to 16 ms off. The decoder's own timestamps (a 1/90000 time base on that clip) don't have that rounding. Switching to them brought the worst case in our tests to 0.07 ms.

How we checked it

We compared GPU output against CPU decoding with the bundled FFmpeg on generated clips, including rotated, 10-bit, 4:2:2 and variable-frame-rate ones, on an NVIDIA RTX 5080 under Windows. GPU scaling is probed once per run. If a GPU pass fails where the CPU would have worked, the clip is retried on the CPU and the failure is remembered. We haven't run the VideoToolbox path on a Mac yet.

Earlier GPU work is covered in zero-copy GPU decoding on every platform, and the 1.2.9 release notes list what else shipped with this. Download FrameQuery to try it on your own footage.