Technology
From 500 Threads to Sanity: Taming Concurrency in a Rust Video Pipeline
Tokio's spawn_blocking pool grows to 500 threads, which breaks CPU-bound video decoding. We replaced it with a bounded Rayon pool, cancellation tokens, and atomics.
FrameQuery decodes professional video formats (R3D, BRAW, ProRes RAW) on the desktop. These are CPU-heavy operations that run alongside an async Tokio runtime handling UI events, database queries, and network I/O. The decode work and the async runtime have to share the machine without starving each other.
The spawn_blocking trap
Tokio's spawn_blocking is the standard way to run blocking work without stalling the async runtime. You hand it a closure, and Tokio runs it on a dedicated thread pool separate from the async worker threads. For blocking I/O like file reads or synchronous database calls, this works well.
The problem is that Tokio's blocking pool can grow to about 500 threads. Each call to spawn_blocking that cannot be served by an idle thread spawns a new one. For I/O-bound work where threads spend most of their time waiting, this is fine. For CPU-bound video decoding, it oversubscribes the CPU.
Routing the R3D and BRAW decoders through spawn_blocking spun up dozens of decode threads during a large library scan. Each thread was doing real CPU work: debayering raw sensor data, color space conversions, pixel format transforms. Per-thread decode buffers thrashed memory, context switching between compute-bound threads wasted CPU, and latency in the async runtime became unpredictable.
A bounded Rayon decode pool
FrameQuery replaces spawn_blocking with a dedicated Rayon thread pool sized to the machine's core count. The entire module is 48 lines.
static DECODE_POOL: OnceLock<rayon::ThreadPool> = OnceLock::new();
fn decode_pool() -> &'static rayon::ThreadPool {
DECODE_POOL.get_or_init(|| {
let threads = std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(4)
.max(2);
rayon::ThreadPoolBuilder::new()
.num_threads(threads)
.thread_name(|i| format!("fq-decode-{i}"))
.build()
.expect("failed to create decode thread pool")
})
}
The pool is lazily initialised via OnceLock and lives for the lifetime of the process. Thread count matches available_parallelism (which respects cgroup limits and processor affinity), with a floor of 2. Named threads (fq-decode-0, fq-decode-1, ...) show up by name in flamegraphs and logs.
The pool has a fixed size. On an 8-core machine, you get 8 decode threads. If 20 decode tasks are submitted, 8 run concurrently and 12 wait in the queue. Memory usage is bounded by the thread count, and the CPU stays saturated without being oversubscribed.
Bridging Rayon back to Tokio
The decode pool lives outside Tokio. Async code submits work and awaits the result over a oneshot channel.
pub fn spawn_decode<F, T>(f: F) -> tokio::task::JoinHandle<T>
where
F: FnOnce() -> T + Send + 'static,
T: Send + 'static,
{
let (tx, rx) = tokio::sync::oneshot::channel();
decode_pool().spawn(move || {
let _ = tx.send(f());
});
tokio::task::spawn(async move {
rx.await.expect("decode pool task panicked or was cancelled")
})
}
The function returns a JoinHandle<T>, making it a drop-in replacement for spawn_blocking. Callers see the same interface whether the work runs on Rayon or on Tokio's blocking pool. The Rayon thread executes the closure and sends the result over the oneshot channel. A lightweight Tokio task awaits the result and surfaces it to the caller.
We use this in r3d_decoder.rs, braw_decoder.rs, prores_raw_decoder.rs, and prores_raw_vulkan.rs. Swapping in spawn_decode was a one-line change in each file.
If the Rayon pool is saturated, new decode tasks queue rather than getting a fresh thread, so decode latency for any individual frame depends on queue depth. The pipeline applies backpressure through bounded channels (described below), which keeps the queue short.
Structured shutdown with TaskTracker and CancellationToken
Desktop applications need clean shutdown. When the user closes FrameQuery, calling std::process::exit would abandon database writes in flight, encode pipelines flushing buffers, and temporary files that need cleanup.
FrameQuery's task manager wraps tokio_util::task::TaskTracker and tokio_util::sync::CancellationToken.
pub struct AppTasks {
tracker: TaskTracker,
token: CancellationToken,
}
This is a global singleton initialised at app startup. Every background task that should participate in structured shutdown is spawned through it. The spawn_cancellable method gives each task a child cancellation token.
pub fn spawn_cancellable<F, Fut>(&self, f: F)
where
F: FnOnce(CancellationToken) -> Fut,
Fut: Future<Output = ()> + Send + 'static,
{
let token = self.token.child_token();
self.tracker.spawn(f(token));
}
Inside a task, the pattern is always the same: use tokio::select! to race work against cancellation.
tokio::select! {
_ = token.cancelled() => break,
result = do_work() => {
// handle result
}
}
When shutdown is requested, we cancel the root token and wait for all tracked tasks to complete.
pub async fn shutdown(&self) {
self.token.cancel();
self.tracker.close();
self.tracker.wait().await;
}
Cancellation propagates through child tokens automatically. A scan task with ten subtasks cancels all of them when the parent token fires.
Bounded concurrency for file scanning
Library scanning is a different concurrency challenge. FrameQuery walks source directories that can contain thousands of video files. Each file needs metadata extraction: duration, resolution, codec, creation date. This work is I/O-bound (reading file headers), so the decode pool is the wrong tool.
We use a Tokio semaphore to bound concurrency.
let semaphore = Arc::new(Semaphore::new(opts.max_threads));
let total_queued = Arc::new(AtomicU64::new(0));
let discovery_complete = Arc::new(AtomicBool::new(false));
The semaphore limits how many files are being processed simultaneously. The atomic counters track progress without locks. An AtomicBool signals when directory discovery is finished so the progress UI can show a determinate progress indicator instead of an indeterminate spinner.
A heartbeat task fires every 200 milliseconds to push progress updates to the frontend. This avoids flooding the UI with per-file events while keeping the progress display responsive.
Rayon is the wrong pool for this stage. Its work-stealing is designed for CPU-bound parallelism, and metadata extraction spends its time waiting on disk reads, or on the network for NAS-mounted media. Tokio's semaphore with async tasks fits that better.
The concurrency primitive cheat sheet
The codebase uses a consistent set of primitives.
OnceLock<T> for lazy singletons. The database connection pool, HTTP client, and decode pool are all initialised once on first access and live for the process lifetime. OnceLock has been in the standard library since Rust 1.70.
Arc<Mutex<T>> for shared mutable state that needs synchronisation. The database write connection and app configuration use this. We use Tokio's async Mutex only when holding the lock across an .await point. Otherwise, std::sync::Mutex is faster because it avoids the overhead of the async runtime.
Arc<AtomicBool> and Arc<AtomicU64> for lock-free flags and counters. Scan cancellation, online/offline state detection, and progress counters all use atomics. They are cheaper than a mutex for simple values and cannot deadlock.
tokio::sync::mpsc for streaming pipelines. The frame decode pipeline uses a bounded channel with capacity 4, which is enough to keep the encoder fed without buffering too many decoded frames in memory. Scan results flow through a channel with capacity 64. Bounded channels provide backpressure: if the consumer is slow, the producer blocks instead of filling memory.
tokio::sync::oneshot for the Rayon-to-Tokio bridge described above. Each bridge sends one result to one consumer.
CancellationToken and TaskTracker from tokio-util for structured task lifecycle. Every long-running background operation uses these.
Handling child processes on Windows
Child processes are a separate concurrency concern. FrameQuery spawns FFmpeg for certain encode and transcode operations. If the app crashes or is killed, those FFmpeg processes can become orphans consuming CPU and disk I/O indefinitely.
On Windows, we assign child processes to a Job Object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. When the FrameQuery process exits for any reason (clean shutdown, crash, task manager kill), the OS automatically terminates all processes in the job. We also set BELOW_NORMAL_PRIORITY_CLASS on these child processes so background transcoding does not compete with the user's foreground work.
What changed in practice
After these changes, FrameQuery's thread count during a large library scan is bounded: core count for decode, the semaphore limit for file scanning, and the small fixed Tokio worker pool for async I/O. Memory usage is stable, and decode latency is independent of how many other operations are running.
The total code for all of this is modest. The decode pool module is 48 lines. The task manager is 94 lines. The semaphore-based scanner is a small addition to an existing module. Rust's type system enforces most of the invariants at compile time: you cannot accidentally send a non-Send type across threads, you cannot forget to await a JoinHandle, and the borrow checker prevents data races on shared state.
Tokio's defaults are tuned for network services, where threads spend most of their time waiting. Replacing them where compute dominates changed FrameQuery's behaviour under load.
We are building FrameQuery to handle professional video libraries without melting your hardware. Download FrameQuery to try it.