APIUp to 28% cheaper than official pricesTry the API →
Hermes
Back to News

CMU's HiT Builds Smarter Image Generators With 2.4x Less Training Compute

CMU and Rice researchers released HiT , a pixel-space diffusion transformer that learns to merge related patches into hierarchical groups. HiT-XL/16 reaches 1.48 FID on ImageNet 256×256, compared with 1.86 for the similarly sized JiT-H/16 baseline.

CMU's HiT Builds Smarter Image Generators With 2.4x Less Training Compute
Source
AlphaSignal
Published
Author
AlphaSignal
Read
2 min read

Takeaways

  • CMU and Rice researchers released HiT , a pixel-space diffusion transformer that learns to merge related patches into hierarchical groups.
  • HiT-XL/16 reaches 1.48 FID on ImageNet 256×256, compared with 1.86 for the similarly sized JiT-H/16 baseline.
  • At matched quality (FID ≈ 2.16), HiT needs 2.4× less training compute and 2.3× fewer updates than JiT.
  • Object and part groups, plus word-to-region links, appear without segmentation masks or region annotations.
  • The 250M text-to-image HiT-B/16 edges past UniLiP-3B's five-benchmark average with 6× fewer parameters, but trails on GenEval.
  • Code is MIT-licensed on GitHub , with checkpoints on Hugging Face .

Many high-performing image generators use two stages. A pretrained autoencoder first compresses each image into a small latent grid, then a diffusion model learns to generate within that space. Pixel-space models generate directly from RGB values, removing the separately trained visual bottleneck while taking on more computation and a harder optimization problem. Researchers from Carnegie Mellon University and Rice University propose a fix called HiT, short for Hierarchical Diffusion Transformer, in the HiT preprint .

HiT targets a common Transformer design choice: retaining one token per image patch throughout the network. The model instead merges related tokens, processes the compressed representation, and restores the original patch layout. Its learned groups often correspond to objects, object parts, and background regions, despite receiving no segmentation labels.

Raw pixels carry three costs

  • More computation: A typical latent model reduces each spatial dimension by 8×. A pixel model operates at full image resolution, and self-attention cost grows quadratically with the token count.
  • Slower convergence: The model must learn fine texture and global composition together. A latent model delegates much of the low-level detail to its autoencoder.
  • No visual prior: Latent models begin in a representation that already captures image structure. Some systems also align their representations with pretrained encoders such as DINOv2. Raw RGB values provide no corresponding signal about which pixels belong together.

HiT addresses the missing visual prior through compression. To preserve useful information while reducing the token count, the model must infer which patches represent related content. The resulting hierarchy also cuts the cost of the large central Transformer stack.

Group, process, restore

HiT uses an hourglass architecture with two compression levels. Outer Transformer blocks process every patch token. Two successive grouping stages then produce a smaller set of group tokens, which pass through the central stack. Ungrouping stages restore features to their original image positions, while skip connections recover higher-resolution information that could otherwise disappear through the bottleneck.

Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.

Read original report

Sources

  1. 01arxiv.org
  2. 02github.com
  3. 03huggingface.co
  4. 04nicholas0228.github.io