How We Built People Matching: Face and Voice Recognition

FrameQuery can now match people across your entire video library by face and voice. Assign a name to someone once, and every video they appear in becomes searchable by who is in it. The find-people guide covers how to search once people are matched; this post covers how the matching works.

Building this required integrating face and voice recognition models, designing an encrypted local biometric database, and making the whole thing compliant with GDPR, CCPA, and BIPA from the start. The privacy constraints shaped most of the architecture.

The Problem

Queries like "CEO speaking" or "Alice B-roll" require knowing who is in the footage. Manual tagging does not scale to a library of thousands of videos, so the matching has to be automatic.

This breaks down into two distinct problems: recognising faces across different videos, lighting conditions, and angles, and identifying speakers across different recordings, microphones, and acoustic environments.

Face Matching

Face recognition runs a two-stage process: detection finds faces in frames and returns bounding boxes, then a recognition model takes each cropped face and produces a mathematical representation (an embedding) that uniquely characterises that face.

The processing flow splits between cloud and desktop:

1. Cloud-side processing runs detection only, finding where faces are in frames and generating thumbnails
2. Thumbnails appear in the desktop app, where you assign names and give consent
3. The desktop downloads the relevant frame regions
4. Recognition runs locally on your machine, generating the face embeddings
5. Embeddings are stored encrypted on your local disk

The split matters for privacy. Detection (finding where faces are) happens in the cloud. Recognition (turning a face into a unique mathematical representation that could identify someone) happens exclusively on your machine.

How matching works

When you identify a person, FrameQuery compares their face embedding against all other detected faces in your library using similarity scoring. Results come back ranked by confidence, and the top matches get linked to the person record in your local database.

The entire face index is encrypted at rest using AES-256-GCM, with encryption keys stored in your OS keychain.

Voice Matching

Voice identification uses a speaker recognition model that takes audio and outputs a compact mathematical representation of the speaker's voice. This works alongside the existing transcript data, which already includes speaker diarization from cloud processing.

The pipeline:

1. For each speaker segment in the transcript, the desktop extracts the longest continuous audio chunk (minimum three seconds)
2. Audio is preprocessed into a format suitable for the voice recognition model
3. The model produces a voice embedding
4. The embedding is stored encrypted alongside face embeddings

All voice processing runs locally on your machine, using a pure Rust audio processing pipeline with no external dependencies.

When you link a voice embedding to a person who also has face embeddings, you can find them by either modality: face in frame, or voice in the audio track.

What You Can Search For

Once people are set up:

• "Alice" resolves the name to all linked biometric data (face and voice), then queries across your library
• "Alice speaking" narrows to voice matches with timestamp ranges
• People panel on a video shows all matched people with their appearance types and confidence scores

You can also merge duplicate person records if you realise two entries are the same individual, and all biometric links transfer to the merged record.

Privacy: Why the Architecture Looks This Way

GDPR, CCPA, and BIPA have enforcement mechanisms and specific requirements around biometric data. Compliance was a design constraint from the start.

Embeddings Never Leave Your Machine

Face and voice embeddings are stored in an encrypted database on your local disk. The encryption uses AES-256-GCM with keys stored in your OS keychain (DPAPI on Windows, Keychain on macOS, secretservice on Linux). Each encryption operation generates a fresh random initialisation vector.

Person names are local-only and never synced to our servers. The cloud side never receives embeddings, appearance logs, confidence scores, or the names you have assigned.

Consent That Satisfies BIPA

Illinois' Biometric Information Privacy Act is one of the strictest biometric privacy laws in the US. It requires informed consent before collection, a published retention schedule, and a plan for destruction. Our consent system tracks:

• Whether consent was given, and when (ISO-8601 timestamp)
• The consent version, so we can handle changes to terms
• Whether the retention period was acknowledged

Every consent action gets logged to a local audit table. This creates the audit trail that BIPA requires.

Before any biometric processing begins, the app presents a setup dialogue with explicit checkboxes. No boxes are pre-checked.

Retention and Deletion

A retention enforcement process runs regularly and checks for records older than the retention period (365 days from first consent by default), purging them automatically and logging every action.

The right-to-delete implementation goes beyond removing database rows. When you delete your biometric data:

1. Cascading deletes run across all related tables
2. Biometric data files are securely overwritten before deletion
3. Index files are removed
4. Thumbnail images are deleted
5. Encryption keys are deleted from the OS keychain

The data and the keys that could have decrypted it are both gone.

Shared Indexes and Biometric Data

Biometric data is never included in shared or exported indexes. When you publish or share an index, all face embeddings, voice embeddings, and person labels are automatically stripped. Only non-biometric metadata (descriptions, transcripts, tags, and timestamps) travels with the shared index.

If a recipient has access to the underlying video files, they can run their own face and voice recognition locally on their own machine and assign their own labels. That processing is independent of your biometric data. It generates fresh embeddings on their device, governed by their own consent.

What We Do Not Collect

FrameQuery does not collect video file contents, search queries, file paths, screenshots, location data, keystrokes, clipboard data, or video filenames. Analytics are opt-in and limited to pseudonymised feature usage events and crash reports.

Trade-offs

Running recognition locally means the models ship with the app. They download on first use, so the bandwidth cost applies only if you enable people matching.

Local-only storage also means you are responsible for backups. If your machine dies and you did not back up the encrypted biometric database, those embeddings are gone. FrameQuery includes encrypted backup and restore.

Download FrameQuery to try people matching.