U-JiT Beats Latent Video Models by Generating Directly in Pixel Space
U-JiT generates video directly in pixel space, skipping the usual pretrained autoencoder stage. Uses a UNet-shaped transformer with variable patch sizes to handle high-dimensional video input end-to-end. Improves FVD by 27% over prior SOTA on Kinetics-600 frames-to-video at 128 resolution.
Takeaways
- U-JiT generates video directly in pixel space, skipping the usual pretrained autoencoder stage.
- Uses a UNet-shaped transformer with variable patch sizes to handle high-dimensional video input end-to-end.
- Improves FVD by 27% over prior SOTA on Kinetics-600 frames-to-video at 128 resolution.
- Lead over latent models grows as spatiotemporal compression increases up to 64x64x8.
- On class-to-video, cuts per-frame FID by 27% vs latent and 61% vs pixel baselines.
- Paper by Vallaeys, Verbeek and Cord ; no public code release announced.
Most leading video generators compress clips with a pretrained autoencoder, then train a diffusion or flow-matching model in the resulting latent space. A paper by Théophane Vallaeys, Jakob Verbeek, and Matthieu Cord presents U-JiT , a model that generates video directly in pixel space and outperforms latent and pixel-space baselines on Kinetics-600.
Video models adopted latent generation because raw clips contain millions of pixel values. Compression can reduce that workload by a factor of 64 or more, making attention and denoising substantially cheaper. The resulting pipeline has two separately optimized components: an autoencoder trained for compression and reconstruction, followed by a generator trained on encoded clips.
That division introduces trade-offs. The autoencoder discards some visual information, and reconstruction quality does not always predict generation quality. Researchers must also choose a compression rate that balances detail, training cost, and the difficulty of modeling the latent representation.
Why pixels tend to blur
Pixel-space image models such as JiT, short for Just image Transformers, have narrowed the gap with latent image generators. Video presents a larger token problem because every frame adds another spatial grid. Keeping the sequence manageable requires larger spatiotemporal patches, which merge fine texture and motion into coarse tokens.
A fixed-width transformer struggles to recover the high-frequency information lost through that patching process. The paper links this limitation to blurred, oversmoothed frames and a gap between the principal-component spectra of generated and real clips. That spectrum measures how visual variation is distributed across the dataset.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
