U.S. Patent No. 11,676,328

Predicting where the picture is about to change.

A method for streaming 3D scenes that renders the parts most likely to matter to the viewer first — using a risk map that learns where change is coming.

Granted June 13, 2023 · Co-invented during work at Miris · Read the full patent →

The problem

Spatial streaming runs out of bandwidth before it runs out of pixels.

When you stream a 3D scene — spatial video, VR, an interactive 3D environment — the renderer has to decide what to draw, in what order, with what fidelity. Bandwidth and compute are finite; the scene is not. A naïve approach renders everything in a fixed traversal order, which means the viewer often sees the least interesting part of the frame come in first while the part their eyes are drawn to lags behind.

The problem gets worse the moment the viewer moves. The view frustum changes, a new set of primitives becomes visible, and the renderer has milliseconds to decide which of those primitives to draw before the human visual system notices anything missing.

The idea

A risk map that predicts which pixels are about to change.

The patent describes a system that, while rendering one frame, also computes a risk map — a per-pixel score quantifying how likely each pixel is to change as the view frustum moves. A pixel showing a stable, distant background gets a low score. A pixel near a fast-moving object, or near the edge of the current view, gets a high score.

When the view actually changes, the risk values from the previous frame are mapped onto the new set of primitives now visible in the next frame. The renderer then performs an out-of-order render: it draws the high-risk primitives first and the low-risk ones last, instead of following a fixed traversal of the scene.

Risk values from frame 1 are mapped to primitives in frame 2 → highest-risk primitives are rendered first.

In four steps

How the system works.

Render the current view

A first image is generated from the primitives that fall within the current view frustum. Standard rasterization, nothing exotic yet.
Generate a risk map

For each pixel, the system computes a probability that the pixel will change as the view changes. The result is a per-pixel scalar field — a heatmap of expected motion or update.
Map risk values onto the next view

When the view frustum shifts, the system aligns risk values from the previous frame onto the new primitives now visible. Each new primitive inherits a priority score.
Render & stream out of order

High-risk primitives are rendered (and, in the streaming case, transmitted) first. Low-risk primitives come later. The viewer perceives a sharper, more stable result with fewer total resources spent.

Why it matters

Most rendering systems decide what to draw based on geometry. This one decides based on attention — what a viewer is most likely to notice next.

For traditional rendering on a single machine with abundant compute, the difference is marginal. For spatial streaming — sending 3D content over a network to a headset, a phone, or a remote screen — it is structural. Bandwidth becomes the bottleneck before compute does, and the order in which bytes arrive determines what the viewer experiences. Prioritizing by predicted change spends those bytes on the part of the image the viewer is most likely to perceive as motion or detail.

The same principle generalizes to other constrained pipelines: progressive rendering for cloud workstations, foveated rendering with predictive falloff, real-time streaming of telepresence and volumetric capture.

Citation

Baize, D. et al. Systems and methods for prioritized rendering and streaming based on risk maps that predict change in a three-dimensional environment. U.S. Patent No. 11,676,328, granted June 13, 2023.