Workflows
How Video Scene Detection Breaks Footage Into Searchable Segments
A 30-minute clip might contain 50 shots. Scene detection finds each transition and turns every shot into a searchable segment with its own description, objects and metadata.
A 30-minute interview file might contain 50 shots: wide establishing shot, medium two-shot, close-up of the interviewer, close-up of the subject, cutaway to hands, B-roll insert, and back. A 10-minute highlight reel might contain 80.
If the whole file is indexed as one unit, a search for "close-up of hands" returns the 30-minute file and you scrub for the moment. Scene detection splits the file into its shots. Each shot gets its own description, objects, timestamps and metadata, and search returns the shot.
Interview subject seated against dark background, warm key light from camera left
Conference stage with presenter at podium, blue and white lighting, audience visible
Establishing shot of harbor at golden hour, boats moored along dock, warm light across water
What scene detection identifies
Scene detection compares frames and marks the points where the picture changes.
Hard cuts. Frame 1000 shows an office interior and frame 1001 shows a landscape. The difference between adjacent frames is large, so these are the easiest to detect.
Dissolves and crossfades. One shot blends into another over several frames. Detection looks at the rate of change over a window of frames rather than one adjacent pair.
Camera moves. A static shot that becomes a pan, tilt or dolly, or a move that settles into a hold, can mark a new shot in editorial terms. These are harder to detect and depend on the sensitivity setting.
Fades to black or white. The fade-out and the fade-in are both boundaries.
The target is the set of boundaries a person logging the footage by hand would mark as "new shot here."
Per-scene versus per-file indexing
With per-file indexing, a search for "aerial shot of highway" returns a 45-minute assembly that has a 6-second aerial somewhere in the middle. With per-scene indexing it returns the 6-second segment with a thumbnail, a timestamp and a play button.
Per-scene indexing also improves the descriptions. A 10-minute clip with indoor and outdoor shots described as one unit becomes "mixed indoor and outdoor footage," which matches too many queries. Described shot by shot, the indoor shots get indoor descriptions and a search for "outdoor" returns only the outdoor segments.
Granularity
Granularity is how many segments a file is split into.
Too coarse. Several shots share one segment, descriptions go vague, and results point at ranges you still have to scrub.
Too fine. Camera moves and lighting changes become separate segments, and results fill with near-duplicates of the same shot.
The useful setting is editorial granularity: the shots an editor would count. A 30-minute multicam interview might produce 40 to 60 segments. A 5-minute montage might produce 30 to 50. A locked-off wide with no cuts produces one.
A fast-cut music video needs higher sensitivity than a static interview. A single-take documentary scene needs lower sensitivity than an edited reel.
After detection
Each segment goes through further analysis.
Scene description. A vision model writes a description: "Close-up of a person's hands adjusting a camera lens on a wooden workbench." Indexed for text and semantic search.
Object detection. Objects in the segment are listed: camera, lens, workbench, hands, tools. Indexed as structured data.
Transcript alignment. Speech in the segment is attached to it. A search for a phrase returns the segment where it was said.
Metadata. Timestamps, duration and position in the source file, so a result opens at the right place in the original.
Example
A 20-minute corporate shoot with the camera rolling continuously:
- A 15-second establishing shot of the building exterior
- A 30-second walk-and-talk through the lobby
- A 4-minute interview segment (medium shot)
- A 2-minute cutaway to the product demo
- A return to the interview (close-up, different angle)
- 3 minutes of B-roll: office spaces, team meetings, whiteboard sessions
- A closing wide shot
Without scene detection every search returns the same 20-minute file. With it, the file becomes 15 to 20 segments. The establishing shot is described as "wide shot of a modern office building exterior." The whiteboard B-roll has "whiteboard" in its description and its object list. Each result points to a timecode range.
How FrameQuery runs it
Scene detection is the first step of the processing pipeline. It produces a list of segments with start and end timecodes for each source file.
Each segment then gets a scene description from a cloud-hosted vision model, object detection and transcript alignment. Face and voice recognition run on your device. Results go into a local Tantivy index with BM25 text matching and MiniLM semantic embeddings.
A locked-off interview produces a few long segments. A fast-cut montage produces many short ones. Multicam shoots where each camera ran continuously produce segments for the distinct shots visible from each angle.
Indexing runs at about five minutes per hour of footage. Originals stay on your machine. The index is local, works offline and reads 50+ formats without transcoding.
Download FrameQuery to search your footage at the shot level.