The TypeScript SDK: graphs of typed streams

The graph

A job is a graph of typed streams.

You open inputs, call operations on the streams they return, and attach outputs. Each stream has a sort: video, audio, detections, a segmentation mask, depth, pose, a transcript. An operation is a method on the sort it takes, so passing audio to a video operation does not compile.

A model's output is a stream too. The graph decodes once and forks, so every branch reads the same frames and a second model costs inference, not another decode.

clip.ts

import { graph } from '@jtdigital/renderbox-sdk';

const g = graph();
const [v, a] = g.open('clip.mp4');      // [VideoStream, AudioStream]

const matte = v.segment({ model: 'sam2-tiny' });
const depth = v.estimateDepth({ model: 'depth-anything-small' });
const skeleton = v.estimatePose({ model: 'yolov8n-pose' });

const out = v
  .maskBlur(matte, { sigma: 30 })        // blur what the mask covers
  .depthBlur(depth, { max_blur: 12 })   // a shallow depth of field
  .drawSkeleton(skeleton);
g.write('out.mp4', out, a);

The registry

The engine's rules are TypeScript errors.

Operation options are generated from the engine's Rust op registry. A model takes only the catalog models its operation runs on the fleet. An option whose values FFmpeg fixes takes FFmpeg's own list, from the fleet's FFmpeg 8.1.2.

errors.ts

import { graph } from '@jtdigital/renderbox-sdk';

const g = graph();
const [v, a] = g.open('in.mp4');
const faces = v.detect({ model: 'retinaface_mv2' });

v.redact(faces, { mode: 'black', radius: 21 });
v.pad({ w: 1920, h: 1080, align: 'center', x: 0 });
v.xfade(v, { transition: 'slidelft', duration: 0.5, offset: 4 });
a.transcribe({ model: 'whisper-base' });

$ tsc
errors.ts(7,34): Type 'number' is not assignable to type 'undefined'.
errors.ts(8,44): Type 'number' is not assignable to type 'undefined'.
errors.ts(9,14): Type '"slidelft"' is not assignable to type '"custom" | "fade" |  …  | undefined'.
                  Did you mean '"slideleft"'?
errors.ts(10,16): Type '"whisper-base"' is not assignable to type '"parakeet-tdt-0.6b-v3-int8"'.

redact: A black box takes no radius. The radius is the strength of a blur or a pixelation.
pad: align and x/y exclude each other.
xfade: A transition is one of FFmpeg's names.
transcribe: The model is the one the fleet runs for speech.

The package

What it exports.

import from	what it holds
@jtdigital/renderbox-sdk	`graph()`, more than 210 typed operations on their streams, media analysis, and the job, progress, completion and failure messages
…/motion	The composition API, on the Motion tab
…/presets	Audio mastering and dynamics, text styles, video sequencing
…/recipes	`letterbox`, `snip`, `faceRedact`, `redactPasses`, `depthBlurPipeline`, `makeSlideshow`
…/pipelines	Typed arguments for the 30 server-side pipelines
…/protocol	The job message and its input and output specs

Running it

A graph is a description.

g.json() is the graph. g.io() binds each input and output to an S3 key. Both go into a job message, published over AMQP. The orchestrator compiles the graph and sends it to a GPU or CPU worker, which reports progress and the result.

The SDK assigns no identity. The job id and the tenant come from whoever submits. A managed HTTP client is not published yet.

submit.ts shortened

const g = graph();
const [v, a] = g.open('uploads/raw.mp4');   // inputs by S3 key
g.write('renders/out.mp4', v.scale({ w: 1920, h: 1080 }), a);

const io = g.io();
const job = {
  job_id: myApp.newJobId(),     // yours, not the SDK's
  tenant_id: ctx.tenantId,
  graph_ir: g.json(),
  input_files: io.inputs,
  output: { bucket, key: io.outputs[0].s3_key, format: 'mp4' },
};

Access is by request.

There is no sign-up. To run work on Renderbox, or to ask how a part of it works, write to us.

Write to us →

The TypeScript SDK for Renderbox.

A job is a graph of typed streams.

The engine's rules are TypeScript errors.

What it exports.

A graph is a description.

Access is by request.