UIUC Researchers' Multi-Branch Transformer Beats Standard Models While Cutting Cache 32%
The Multi-Branch Transformer runs full Transformer blocks in parallel branches between shared early and late layers. At matched training compute, all 12 final MBTs (0.8B to 2.1B MoE) beat standard Transformers on validation loss.

Takeaways
- The Multi-Branch Transformer runs full Transformer blocks in parallel branches between shared early and late layers.
- At matched training compute, all 12 final MBTs (0.8B to 2.1B MoE) beat standard Transformers on validation loss.
- Fitted curves show standard Transformers need up to 1.26x the training compute to reach the same loss.
- Joint-KV gives all branches at the same depth one shared KV cache, cutting total cache by 19% to 34%.
- With branch grouping and head folding, decoding runs up to 51.4% faster on one H200.
- Training code is on GitHub and checkpoints are on Hugging Face . Tests stop at 2.1B parameters.
Transformer scaling usually makes each block wider or stacks more blocks along one sequential path. Researchers from UIUC, Rice, Michigan, and Cornell evaluate a third layout in the Multi-Branch Transformer paper : several independent stacks of full Transformer blocks process the same hidden states in parallel, then merge their outputs. Across four mixture-of-experts scales ranging from 0.8B to 2.1B total parameters, every final MBT reached lower validation loss than a standard Transformer trained with the same compute. The final architecture also reduced KV-cache use and improved decoding throughput on a single H200 GPU.
Parallelism moves beyond the block
Recent architecture research has changed the components inside Transformer blocks through new attention variants, sparse experts, and richer residual connections. The blocks usually remain arranged in a single chain. PaLM-style designs already execute attention and feed-forward layers in parallel within one block; MBT extends that approach to complete blocks with separate weights and layer sequences.
The design draws on branched vision models such as Inception and ResNeXt, along with recent ParScale work. ParScale runs several passes through one shared backbone, while MBT assigns independent parameters to each branch.
The network branches in the middle
The network first sends every token through shared early blocks called the prelude . The resulting hidden states enter K independently parameterized branches, with every token traversing every branch. Each output passes through RMSNorm, the normalized outputs are concatenated along the hidden dimension, and a bias-free linear layer projects them back to the model width. Shared final blocks called the coda produce the remaining transformations.
At matched training compute, the paper allocates width and depth across branches in three ways:
- Wide-Shallow: Preserves the original block width and divides the middle blocks among branches, making each branch shallower.
- Narrow-Deep: Preserves the original sequential depth within every branch and reduces block width.
- Balanced: Uses intermediate width and depth chosen to match the standard model’s training FLOPs.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report

