PixVid: Video Generation in Pixel Space

We lift a pretrained latent video model into pixel space.

Overview

Latent video diffusion models work in a VAE latent space that compresses video in both space and time. Doubling the resolution quadruples the token count, conditions apply to groups of frames rather than individual ones, and each response must wait for a whole chunk of actions.

PixVid finetunes a pretrained latent video model to generate every frame directly in pixels. It keeps the diffusion transformer, initializes a pixel patch embedding from a linear fit to the VAE encoder, and replaces the VAE decoder with a causal convolutional detailer head. Predicting clean video rather than velocity, together with a much larger noise shift, is critical to quality.

Finetuned from Wan2.1-14B for 20K iterations, PixVid is competitive with the original latent model, generates 2× higher-resolution video with the same tokens per frame, keeps depth-conditioned editing aligned under fast motion, and responds to each action at the next frame.

Method

PixVid keeps the DiT backbone of a pretrained latent video model and replaces the VAE: a linear patch embedding maps each frame's pixels to tokens, and a causal convolutional detailer head renders the DiT tokens into pixels.

Lifting a latent DiT to pixel space

Use the arrows or click the figure to step through.

An offline linear pixel–latent alignment is fused with the pretrained patch embedding to initialize the pixel patch embedding, and a timestep calibration matches the pretrained signal-to-noise ratio. The DiT is initialized from the latent model, and a detailer head replaces the linear head and VAE decoder to predict clean frames directly.

Higher resolution at the same token count

To adapt from patch size p to 2p, 2×2 average downsampling is fused with the 1×p×p patch embedding, and a new up block L+1 (initialized as nearest-neighbor upsampling) is prepended to the detailer head. With the token grid unchanged, we train at 2× resolution at the same DiT cost.

Token grid at 480p and 960p

480p 832 × 480

30 × 52 tokens

One token: 16 × 16 pixels

Baseline cost

960p, latent model 1664 × 960

60 × 104 tokens

One token: 16 × 16 pixels

4× tokens, 16× attention vs. 480p

960p, PixVid 1664 × 960

30 × 52 tokens

One token: 32 × 32 pixels

Same tokens and DiT cost as 480p

Causal detailer head

All 3D convolutions in the detailer head are causal in time (kt = 3): output frame t reads only frames t−2 to t. In frame-by-frame generation, the head caches features from the previous two frames and decodes each frame as soon as its DiT tokens arrive.

Low-latency autoregressive generation

An autoregressive model generates video in blocks, and a block can start only once the commands for all of its frames have arrived. In a temporally compressed latent model, each block spans at least one multi-frame latent, so new commands wait for the rest of the block; PixVid generates single-frame blocks, so each action takes effect on the next frame.

Control-to-screen latency

With temporal compression (N = 4 here), each generation step commits 4 frames to a single action and queues incoming actions until the block finishes, adding up to 133 ms of delay for a 4-frame block at 30 FPS. PixVid generates one frame per step, so every action is reflected in the next generated frame.

Responsiveness to abrupt camera controls

PixVid on camera paths with abrupt direction changes. The overlay shows the keys pressed on each frame.

Interactive demo

Press Start and steer with the arrow keys or W, A, S, D. The left model uses 4-frame blocks and the right model uses 1-frame blocks, both at 30 FPS.

Press Start, then hold the arrow keys

Latent model (4-frame blocks)

Lag behind input: 0.0 frames, 0 ms

PixVid (1 frame per step)

Lag behind input: 0.0 frames, 0 ms

The latent model samples input only at the start of each 4-frame block and holds that direction for all 4 frames, quantizing every key press to whole blocks. PixVid samples input every frame and applies each change on the next frame.

Results

Text-to-video at 960p

PixVid samples at 1664 × 960 with patch size 32, using the same 30 × 52 tokens per frame as the 832 × 480 model.

Depth-conditioned video editing

Each model re-renders the source clip from its per-frame depth and a caption. Latent (VACE-1.3B) is finetuned with the same control branch, data and settings as PixVid; VACE, Cosmos and Wan-Fun are used without finetuning.

Camera control

Top: one scene under three camera paths of increasing difficulty. Bottom: PixVid and the SolarWM-5B latent student on the same camera path. Both are 4-step causal students distilled from SolarWM-5B.

BibTeX

@misc{pixvid,
  title  = {PixVid: Video Generation in Pixel Space},
  author = {To be added},
  year   = {To be added}
}

Citation details will be updated upon publication.