Computer vision and rendering engine for video

01 · What it does

It finds what is in a video and renders the result.

A model finds things in each frame. Rust code then follows them, counts them and decides what to do with them. FFmpeg and OpenCV make the output. Everyframe made these clips with Renderbox.

It finds the face and keeps it in frame.

face detection and tracking

It writes down the words and cuts in the pauses.

speech to text, parakeet-tdt-0.6b-v3

It removes the background.

person found with dfine_l_coco, cut out with sam2-tiny

It cleans the voice.

speech denoising, DPDFNet2 · sound on

After the models

track: Follows each object from frame to frame with ByteTrack. If a face is hidden for a few frames, it keeps a box where it expects the face to be, so the blur does not flicker off.
zone: Keeps only what is inside an area you draw.
count: Counts what it finds, and how long each one was in view.
diarize: Groups speech by who is talking.
redact: Blurs, pixelates or blacks out what was found.
motion-redact: Masks everything that moves in front of a fixed camera.
mti: Cancels a moving camera's own motion, such as a drone's, and reports what still moves.
klv_decode: Reads drone telemetry from the video and lines it up with each frame.

Apps can also ask for an outcome by name. There are 30 ready-made pipelines, such as face redaction, object detection, trimming and slideshows.

02 · The graph

A wrong connection does not compile.

A job is a graph of steps. Each step reads and writes typed streams: Video, Audio, Detection, Pose, Transcript, Depth and ten more. A video step only takes video. Passing it audio is a compile error.

You build the graph in Rust or TypeScript. In Rust a stream is Stream<S>, where S is its type. In TypeScript each stream type has its own interface. The option types are generated from the Rust op registry, the engine's list of operations, so a misspelled option does not compile either.

A model's output is a stream too. Detection, tracking and redaction connect like any other steps. A second model on the same clip reads from the same decode.

import { graph } from '@jtdigital/renderbox-sdk';

const g = graph();
const [v, a] = g.open('street.mp4');        // [VideoStream, AudioStream]

const faces = v
  .detect({ model: 'retinaface_mv2' })      // → DetectionStream
  .track({ algorithm: 'bytetrack' });       // stable ids across frames

g.write('redacted.mp4', v.redact(faces, { mode: 'black' }), a);

a.redact(faces);
  ^^^^^^
error TS2339: Property 'redact' does not exist on type 'AudioStream'.

The graph is from the SDK README. The last line is a mistake, and below it is what tsc says.

03 · The compiler

A job that cannot run is refused before it starts.

Every submitted graph goes through compile_graph. It checks the graph against the op registry and makes sure every slot has a file bound to it. Then it optimizes the graph, compiles it into a plan, seals the plan and encodes it as protobuf. The compiler works on OpNodes and never sees source text.

a validate

Checks that each container accepts its codec, each step gets the right input types, parameters are in range, FFmpeg expressions are valid and frame sizes are even. File paths and endpoints get a separate check. A graph with over 10,000 nodes, or over 500 streams into one combine, is refused.
b optimize

Makes one pass over the graph in dependency order. It writes new nodes and never edits a node in place.
c partition

Tags each node with the runtime that runs it, such as FFmpeg, ONNX or OpenCV. Neighbouring nodes on the same runtime become one segment. Every model that reads frames gets a letterbox or resize step in front of it. A plan with over 1,000 segments is refused.
d wire pipes

Connects the segments. Most of the assembly code is here. Every raw video pipe gets a stated frame size. One decode is split so each step that reads it gets its own feed. Audio model steps get their PCM settings fixed in the plan.
e encode

Writes the plan as protobuf with prost. Each of the 21 pipe formats maps to a wire value in a match with no wildcard. A new format without a mapping does not compile.

some refusals, by code

E028: a model adapter in the submitted graph (only the compiler adds them)
E064: a pixel node in a plan that cannot stream it
E066: a raw video pipe with no known frame size
E069: a model input at the wrong size that no converter can fix
E070: one stdout asked for two formats or two frame rates
E021: an op the registry does not know, such as a deleted one
SEC001–009: path traversal, unsafe characters, filter names, duplicate outputs, URL schemes, path length, reserved keys, endpoint kinds, and input formats such as lavfi

04 · The seal and the receipt

A plan's id is a hash of its contents.

The plan id is the SHA-256 of the plan's protobuf bytes, without the id itself and the source id. One crate of about 360 lines defines it. Three programs share that code. The compiler writes the id. The worker checks it before anything runs. An offline verifier checks a stored plan later.

The worker decodes the plan and encodes it again before it hashes. If the plan has a field the worker's build does not know, the hashes differ and the plan does not run. The verifier hashes the bytes exactly as stored.

// renderbox-seal, in words
canonical = top-level TLV fields of the plan
            minus field 6 (plan_id), field 7 (source_id)
plan_id   = sha256(canonical)              // 32 bytes

// errors
PlanIdMissing · PlanIdMalformed · PlanIdMismatch · MalformedWire

$ everyframe-verify --receipt receipt.json --media out.mp4 --plan plan.pb
signature       skip   no signatures in the manifest yet
media hashes    ok     out.mp4
plan seal       ok     7f3a9c1e…
model digests   ok     retinaface_mv2
timestamp       skip   not in this build
exit 0

everyframe-verify checks a finished job against its receipt. The output layout is an example. The five checks, their order and the two skips are the binary's. It needs no network and no account.

05 · The runtime

When a step fails, it says which one and why.

Before any process starts, the runtime checks the plan against what its build can run. If something is missing, it fails once and names every segment it cannot run. Causes include a missing ONNX provider, a motion renderer version that is not installed, and a precision the provider does not support.

When every segment can stream, frames pass between them through bounded channels. Otherwise the plan runs in batch, one segment at a time. If a segment fails, the runtime cancels the plan. A broken pipe, a SIGPIPE or a cancel counts as a victim of another failure. The report names the first error that is not a victim.

segment  runtime  ended
seg-0    ffmpeg   exit -13       victim · SIGPIPE
seg-1    onnx     error          root cause
seg-2    ffmpeg   cancelled      victim

{ "kind": "onnx", "runtime": "onnx", "exit_code": null,
  "stderr": "…" }

Example data. The ranking and the report's fields are the runtime's own.

handlers: ffmpeg, onnx (CoreML, CUDA or TensorRT, chosen at build time), opencv, native
multi-pass: two-pass encodes, loudnorm and vidstab, each run as one graph of stages
progress: the lowest percent among the plan's final outputs. It never goes down, and holds at 99.0 until the last event.
builds as: a library, a C ABI, and the renderbox-exec CLI

06 · Motion not on the fleet yet

It draws motion graphics on the CPU.

renderbox-motion reads one document that describes a whole composition and writes raw frames. It needs no browser and no GPU. The runtime starts it as a separate binary, at the renderer version (epoch) the plan was sealed with. Before that, the runtime checks the document's hash.

The same document gives identical frames on every machine in the same hash class. The build enforces this. Transcendental functions such as sin and exp go through libm. Nothing calls mul_add. An x86 build with AVX2 or FMA in its baseline does not compile.

The renderer in detail →

rasterizer: vello_cpu, one context per frame
hash classes: fused and unfused, identical within each
golden cases: 46 drawing cases and 150 documents, checked at every SIMD level
also builds for: WebAssembly, behind a small C interface

type-festival · 13.6 s · drawn by renderbox-motion

launch-film · 16.1 s · glass cards, counters, an animated chart

title-storm · 8 s · rendered through today's API

07 · Where it runs

A job goes to the first server that can run it.

The orchestrator lists the servers in a TOML file, built into its binary with include_str!. Changing the list takes a reviewed change and a deploy. Servers are listed from least to most capable, and each job goes to the first one that has everything it needs. If none does, the job is refused when it is submitted. It is not sent to the closest match.

broker

Orchestrator

Runs RabbitMQ. Compiles and seals plans, routes jobs, keeps the ledger of jobs in flight and writes manifests. Five of its tasks are essential. If one ends, the process exits and systemd restarts all of it.

first pick

Encode hosts

CPU servers. They filter and encode video, and get every plain render first.

last pick

GPU hosts

Model work on CUDA, NVENC hardware encodes and batch language model jobs. Plain renders never queue behind them.

Audio model steps run on the CPU, even on a GPU host. The plan records the CPU type they were sealed for, and the runtime refuses to run them on another.

A release installs the motion epoch on the hosts first, then deploys the workers, then the orchestrator. The orchestrator seals plans with parameters the workers must already understand. A motion plan goes only to a host the orchestrator's topology lists for its epoch, so that one line turns motion on. The deploy copies working trees, not tags, so it refuses to start while any crate has uncommitted or unpushed work.

Scripts install the same fonts and the same pinned FFmpeg on every render host. deploy-fleet.sh --check prints what each host runs next to the local checkouts. A machine that has drifted shows up before a render fails.

Made with it

Everyframe

Everyframe is an AI video editor. It uses Renderbox for rendering and computer vision.

How Everyframe uses Renderbox → everyframe.si ↗

One brief, rendered as 27 files. Each file is its own job.

Access is by request.

Renderbox runs JT Digital's own products. There is no sign-up. Write to us if you want to run work on it, or to ask how a part of it works.

A computer vision and rendering engine for video.