Single-Frame H.264 vs JPEG: Shrinking a 6 GB Frame Cache

FrameQuery keeps about one frame per second of every processed video on disk, for face detection and race-number reading. On our test library that cache held 123,554 JPEGs in 6.16 GB. From version 1.2.9 those frames are stored as one H.264 intra frame per file, at 44 to 46% of the JPEG size on our test sets. Packing each video into an MP4, the obvious idea, was not where the saving came from.

What the cache has to do

Each frame is capped at 1920 pixels wide and named after its position in the source. The mean was 51 KB per frame.

Three readers constrain the format:

• Random access by index. Detection, number reading and the search results page all ask for one specific frame.
• Deleting single frames. After a scan, frames that add nothing new are pruned one at a time.
• Hashing the stored bytes. A face match records a hash of the exact bytes it was read from, so every frame needs bytes of its own.

A per-video MP4 breaks all three. Deleting a frame from the middle of a group of pictures means re-encoding it, a single-frame read needs a seeking decoder, and the bytes behind one frame no longer exist on their own.

Frames one second apart barely share anything

We expected inter-frame prediction to carry the saving. It didn't. A GOP of 30 and a GOP of 250 produced identical sizes, and switching from inter to intra-only coding cost between 0 and 16 percentage points depending on the subject. Frames a second apart in real footage are mostly unrelated pictures.

The quality setting mattered far more than the container. Weighted by each subject's share of the cache's bytes, against the original JPEGs:

| Method | Size vs JPEG | | --- | --- | | Lossless JPEG recompression (JPEG XL / Lepton class) | about 81%, byte-exact | | x264, CRF 18 (SSIM 0.99 or better vs the JPEG) | about 68% | | x264, CRF 23 | about 43% | | x265, CRF 23 | about 35%, twice the encode time |

Moving from CRF 18 to CRF 23 saved 25 points. Changing the container saved almost nothing.

Per-frame intra H.264, measured

We then wrote every frame as its own file: one H.264 access unit per frame, intra-only, with the sequence and picture headers repeated in each file so every file decodes on its own. Two subjects bracket the library: B at 153 KB per JPEG frame and D at 43 KB, close to the library mean.

| Variant (intra, -g 1 -bf 0) | B size | D size | SSIM vs JPEG (B / D) | | --- | --- | --- | --- | | High, CRF 22 | 58% | 54% | 0.981 / 0.991 | | High, CRF 24 | 46% | about 44% | 0.972 / about 0.989 | | High, CRF 26 | 36% | not run | 0.962 | | Constrained Baseline, CRF 24 | 52% | not run | 0.965 | | JPEG XL lossy, distance 1.5 / 2.5 | 106% / 84% | not run | 0.992 / 0.987 |

JPEG XL was larger than the source JPEG at transparent quality and about 25 times slower to encode, so it was out. High profile beat Constrained Baseline by roughly 15 points at equal SSIM. We also set psy-rd=0. These frames feed detectors, not eyes, and psychovisual tuning spends bits on what looks good to people.

We chose CRF 24. CRF 26 would reach about 2.8 times smaller at SSIM 0.96, but race numbers are the most compression-sensitive thing we read from these frames. We haven't measured number-reading accuracy at either setting yet, so we stayed closer to the original.

The CRF trap: x264 counts frame duration

The first real encode came out at 99 to 117% of the JPEG size. The test above had fed x264 a 30 fps image sequence. Real extraction hands it frames one second apart.

x264's rate control divides each frame's measured complexity by its duration relative to 1/25 of a second, capped at one second. A frame that lasts a full second looks 25 times simpler than it is, and gets a much finer quantiser. Telling x264 to treat the stream as constant 30 fps fixed it:

-c:v libx264 -preset medium -crf 24 -g 1 -bf 0 -pix_fmt yuv420p -color_range pc \
  -x264-params repeat-headers=1:annexb=1:psy-rd=0.0,0.0:force-cfr=1:fps=30 \
  -frame_pts 1 -f image2 f_%d.h264

| Subject (6 frames) | JPEG bytes | Without force-cfr | With force-cfr | | --- | --- | --- | --- | | D, 1280x720 | 208,117 | 117% | 50.3% | | B, 1920x1080 | 844,670 | 107% | 52.7% | | N, 1920x1080, noisiest | 1,205,656 | 98.8% | 45.9% |

Across all 286 frames of the noisiest video, 69.2 MB of JPEG became 27.0 MB, 39% of the original. The largest single frame was 172 KB, well inside OpenH264's 1 MB limit for one compressed frame.

-f image2 matters too. Without it, the .h264 extension selects FFmpeg's raw H.264 muxer, which writes every frame into one file.

Decoding inside the app

The app decodes frames in process with OpenH264, built from source, which decodes this stream (High profile, I-slices only, no B-frames). We convert YUV to RGB ourselves with the same BT.601 full-range matrix JPEG decoders use, so pixel values line up with the old path. Against the JPEG pipeline on the same source frames, the mean absolute difference was 0.9 to 2.7 levels out of 255, with no systematic offset.

Decoding costs more. In a release build a 1080p frame takes 46 to 51 ms, against 17 ms for the JPEG, and our OpenH264 build doesn't use its assembly routines yet. The frames are read during scans and when you open a match. The disk space is held for as long as the video is in the library.

Everything outside the cache still asks for f_N.jpg. The cache resolves that logical name to an H.264 file, a legacy JPEG or an archived frame, and the protocol that serves frames to the interface transcodes H.264 to JPEG on request. Malformed files, truncated files and files with only headers all return an error instead of crashing the decoder.

Converting existing caches

New videos get H.264 frames straight from the extraction pass. Existing JPEGs are converted by a background job on one thread at idle priority, which stops whenever extraction, detection or anything in the foreground needs the machine. It costs about 78 ms per frame, roughly 2.7 core-hours for our 123,554-frame library.

That conversion is a second lossy generation, from JPEG to H.264, and we accepted that trade for the size. If the bundled FFmpeg has no libx264 encoder, frames stay JPEG and are recompressed losslessly instead. In 1.2.9 each video's frames are also packed into a single SQLite file, which cuts the file count rather than the bytes.

We don't have a library-wide "GB saved" figure from real installs yet. The numbers here come from one library on one machine. For the poster side of the same storage work, see why our thumbnails were secretly lossless, and the 1.2.9 release notes list the other changes that shipped with this one.

Download FrameQuery to try it on your own footage.