Workflows
Video Scene Analysis: How AI Understands What Is Happening in Your Footage
Scene analysis turns raw footage into searchable content by detecting shot boundaries and generating natural language descriptions of what each shot contains. Here is how the process works and why it matters for editors.
You shot four hours of B-roll across two days. Cityscapes, office interiors, product close-ups, atmospheric transitions. None of it has dialogue. Nobody is speaking on camera. Every clip has a camera-generated filename that tells you nothing about what is inside.
Transcript search cannot help here. There is nothing to transcribe. Manual tagging could work, but four hours of B-roll at three times real-time for logging means 12 hours of work. So the footage sits in a folder, searchable only by filename and your memory of what you shot.
Scene analysis solves this specific problem. It takes footage that has no dialogue, no manual tags, and no useful filename, and makes it searchable by what is visually happening on screen.
What scene analysis actually does
Scene analysis is a two-step process. First, the system identifies distinct shots within a video file. Second, it analyzes each shot and generates a description of what it contains.
The first step is scene detection: finding the boundaries between shots. A single video file might contain one continuous shot or dozens of distinct shots separated by cuts, dissolves, or significant camera movements. Scene detection identifies each transition and segments the file into individual shots.
The second step is scene description: generating a natural language summary of what happens in each shot. This is where a vision model examines the visual content and produces a human-readable description along with structured metadata.
Together, these two steps turn an opaque video file into a collection of described, searchable segments.
Interview subject seated against dark background, warm key light from camera left
Conference stage with presenter at podium, blue and white lighting, audience visible
Establishing shot of harbor at golden hour, boats moored along dock, warm light across water
What scene descriptions contain
A scene description is not a single label. It includes multiple layers of information about the visual content of a shot.
Natural language description. A sentence describing what is happening: "A woman walks through a modern office corridor, passing glass-walled meeting rooms." "Aerial shot pulling back from a construction site, revealing surrounding neighborhood." "Close-up of hands placing a circuit board onto an assembly fixture."
These descriptions are written the way a human would describe a shot. They capture action, setting, and context in plain language.
Shot type. Wide, medium, close-up, extreme close-up. This is extracted as a structured field, so you can filter search results by framing.
Camera angle. Eye level, high angle, low angle, overhead, Dutch angle. Another structured field that supports filtering.
Dominant color. The prevailing color palette of the shot: warm golden, cool blue-grey, neutral, high contrast. Useful for finding footage that matches a particular grade or mood.
Visible elements and objects. Specific items identified in the frame: tables, vehicles, laptops, tools, buildings, trees, food. These overlap with dedicated object detection but are also noted in the scene context.
People count. How many people are visible in the shot. Useful for quickly distinguishing solo interviews from group scenes.
All of this information is indexed and searchable. The natural language description supports free-text queries. The structured fields support filters. Together, they let you search with a combination of description and constraints.
Why this matters for visual-only footage
Every search method has a content type where it excels and a content type where it fails. Transcript search is excellent for dialogue but returns nothing for silent footage. Object detection finds specific items but does not capture context or composition. Face recognition finds people but not empty rooms, landscapes, or product shots.
Scene analysis is the search dimension that covers visual-only footage. It is uniquely valuable for the categories of footage that have no other way of being found:
B-roll. Establishing shots, cutaways, atmospheric footage, transitions. This is the footage editors scrub through most often and the footage that benefits most from being searchable by visual content.
Product and commercial footage. Tabletop shots, product reveals, lifestyle imagery. Scene descriptions capture what the product looks like, where it is placed, and what the surrounding environment looks like.
Stock footage libraries. Whether you maintain a library of self-shot footage or purchased stock, scene descriptions make it searchable by content rather than whatever folder structure or keywords were applied at import.
Archival footage. Old projects, inherited libraries, footage from departed team members. Scene analysis generates descriptions retroactively, making years of footage searchable in the time it takes to process it.
Nature and documentary footage. Wildlife, landscapes, weather, environmental shots. Content that is inherently visual and often has minimal or no accompanying audio.
The role of the vision model
Scene descriptions are generated by a vision model: an AI system trained to understand visual content and describe it in natural language. The model examines representative frames from each detected shot and produces descriptions based on what it sees.
Modern vision models understand composition, not just content. They can distinguish between a close-up and a wide shot of the same subject. They recognize that a "person sitting at a desk" and a "person standing at a podium" are different setups even though both contain a person. They describe spatial relationships, lighting conditions, and camera perspective.
The descriptions are not perfect. Fast motion, unusual subjects, and ambiguous scenes can produce vague or inaccurate results. But for the vast majority of footage, the descriptions are accurate enough to make the right clips surface when you search for them.
How scene analysis fits into FrameQuery
During processing, FrameQuery first runs scene detection to identify shot boundaries within each clip. Each detected shot is then analyzed by a vision model that generates the natural language description and structured metadata.
The generated data is stored in a local search index built on Tantivy, a Rust search engine. Queries run against both BM25 text matching (for exact keyword matches) and MiniLM semantic embeddings (for meaning-based matches). This means a search for "sunset over water" can return a shot described as "orange sky reflected in a calm lake at dusk" because the semantic model understands the relationship.
Processing runs at roughly five minutes per hour of footage. Only lightweight proxies are sent for cloud analysis. Your original files stay on your machine. The resulting index is local, works offline, and has no per-query cost.
From opaque files to described footage
The core problem with video search has always been that video files are opaque. Your operating system sees a filename, a duration, and a file size. It does not see what is inside. Scene analysis opens the box. It turns each shot into a described, searchable unit with structured metadata that persists in your index.
For footage with dialogue, transcript search already solves much of the finding problem. For footage without dialogue, scene analysis is what makes search possible at all.
Join the waitlist to make your visual footage searchable when FrameQuery launches.