Workflows
Video Object Detection Explained: How AI Finds Things in Your Footage
Object detection labels the things in your footage: cars, laptops, dogs, products, furniture. How it works, what it can find, and where it helps an editor searching by what is on screen.
Three days of B-roll for a product launch. Somewhere in those 12 hours there are close-ups of the product on a desk, an unboxing, and a few clips with it on a shelf in the background. Nobody said its name on camera, so it is in no transcript. Nobody tagged it at ingest.
Object detection records that this thing appears in this frame at this timecode.
What object detection does
Object detection is a computer vision model that examines a frame, draws a box around each object it recognises and labels it. One frame might produce "laptop," "coffee cup," "chair," "person," "book" and "plant."
Scene description answers a different question. It writes a summary such as "two people in a meeting room at a whiteboard." Object detection lists the items in the frame without interpreting the situation. Scene description finds footage by situation or mood. Object detection finds footage by the presence of an item.
Product_Shoot_v3.mov
objectProduct tabletop arrangement with laptop and accessories, soft studio lighting
B004_C003_BTS.R3D
objectBehind the scenes camera rig setup, crew preparing equipment on location
Frame sampling
Detection runs on sampled frames rather than every frame. The sample rate is high enough to catch anything on screen for more than a moment.
For each sampled frame the model records every object's label, position and confidence. Detections are merged across frames into time ranges. If a laptop is visible from 00:01:15 to 00:03:42, the index holds one record for that span.
What it can detect
Current models recognise hundreds of common categories.
Everyday items. Laptops, phones, cups, bottles, books, bags, chairs, tables, monitors, keyboards, clocks, pens.
Vehicles. Cars, trucks, buses, motorcycles, bicycles, boats, aircraft.
Animals. Dogs, cats, birds, horses, cows, sheep and other common species.
People. Whole-person detection, plus hands and faces (separate from face recognition).
Outdoor objects. Traffic lights, stop signs, benches, fire hydrants, parking meters, trees.
Food and kitchen items. Plates, bowls, cutlery, ovens, refrigerators, common foods.
The vocabulary depends on the training data. Widely photographed objects are detected reliably. A proprietary product is detected as its general category (electronic device, tool, bottle), not by brand name.
What it cannot detect
Very small objects. Items covering a few pixels, such as a pen on a distant desk or a logo on a far building, are often missed.
Niche items. Specialised equipment, custom products and uncommon items may have no category. A standard chair is detected. A custom ergonomic prototype probably is not.
Obscured objects. An object mostly hidden behind something else may be missed or mislabelled.
Specific instances. The model detects "cup," not your CEO's mug. It detects "car," not the make and model.
How detections become searchable
FrameQuery runs object detection during processing and stores the results in the local index. Each detected object is a search term tied to timecodes in specific clips.
Searching "laptop" returns every clip and timecode where one was detected. Object searches combine with other search types: "laptop" plus a scene description like "office meeting" narrows to laptops in meetings.
B-roll, product shots, establishing shots and lifestyle footage have no dialogue, so transcript search returns nothing for them. Object detection is how that footage gets found.
Examples
Product placement verification. A sponsor wants confirmation their product appeared in specific shots. Search the product category and review every frame it was detected in.
Props across a shoot. The director wants every shot with the red notebook. Search "book" or "notebook" and review the thumbnails.
B-roll assembly. For an office montage, search "laptop," "keyboard," "monitor" and "whiteboard" to pull all the workspace B-roll.
Continuity. If a coffee cup should be on the desk in every shot of a scene, a search shows which shots it is missing from.
Object detection finds the candidates. Choosing between them is still the editor's job.
Download FrameQuery to search your footage by what appears on screen.