IFM Opens K2-Horizon's Full Training Stack Across Six AI Models
IFM released K2-Horizon , a fleet of six open models from 0.9B to 375B-A23B, with a 134-page report. New MoVA attention lets a 36B-A4B model nearly match a 32B dense model using one-seventh the active parameters.

Takeaways
- IFM released K2-Horizon , a fleet of six open models from 0.9B to 375B-A23B, with a 134-page report.
- New MoVA attention lets a 36B-A4B model nearly match a 32B dense model using one-seventh the active parameters.
- Uno adapters give autoregressive models up to 2.2x faster lossless decoding without a draft model.
- Four models share one 22T-token corpus so scaling-law recipes transfer cleanly from small to large.
- Release includes 380 checkpoints, 13TB of data, 58 training runs, and 17 repos on Hugging Face .
- Roughly 10T of the pre-training tokens are synthetic, with the full generation pipeline documented.
The Institute of Foundation Models has published a 134-page technical report documenting the construction of K2-Horizon, a family of six language models ranging from a 0.9-billion-parameter dense network to a 375-billion-parameter mixture-of-experts model.
IFM pairs the model weights with checkpoints, training data, run records, and production code. Four models share a 22-trillion-token corpus and training recipe, giving researchers controlled comparisons across model scale and sparsity instead of isolated benchmark results.
A six-model ladder
The lineup covers dense and mixture-of-experts architectures. In a mixture-of-experts model, a router sends each token through a subset of the available parameters, reducing computation while retaining a larger pool of learned weights.
| Model | Architecture | Total parameters | Active parameters |
|---|---|---|---|
| 375B-A23B | Mixture of experts | 375B | About 23B |
| 36B-A4B | MoE with MoVA | 36B | About 4.6B |
| 32B | Dense | 32B | 32B |
| 7B | Dense | 7B | 7B |
| 3.7B | Dense | 3.7B | 3.7B |
| 0.9B | Dense | 0.9B | 0.9B |
For the sparse models, the “A” label indicates the approximate number of parameters activated for each token. That figure estimates per-token computation. Serving still requires access to the full set of expert weights, whether they reside on one accelerator, across several devices, or in offloaded memory.
IFM designed the family for coding and agentic workloads, including multi-step tasks that call tools and respond to environment output. The authors report leading or competitive results against similarly sized open models, though independent evaluations will be needed to test those claims across different prompts, inference settings, and tool environments.
The training trail is public
The package follows the LLM360 approach of publishing enough training material to support audits, ablations, and reuse. Its main components include:
- Checkpoints: 380 intermediate and final model snapshots.
- Data: 13 TB of pretraining data and pipelines that generated roughly 10 trillion synthetic tokens.
- Runs: Records from 58 training runs.
- Code: 17 repositories, including the production-scale xLLM training framework and the RL360 reinforcement-learning stack.
- Retrieval: Search360, an index of 994 million web documents used to ground synthetic-data generation.
These artifacts connect model behavior to specific data mixtures, architectural choices, and training stages. Intermediate checkpoints also let researchers study when capabilities or failure modes emerged instead of comparing only the initial and final models.
Sparsity reaches attention
A conventional mixture-of-experts transformer applies sparse routing mainly to feedforward layers, which contain most of the model’s parameters. K2-Horizon adds Mixture-of-Value Attention , or MoVA , which routes tokens among different attention value projections.
MoVA gives tokens access to specialized value projections while activating only a subset for each forward pass. The 36B-A4B model combines this design with mixture-of-experts feedforward layers and activates about 4.6 billion parameters per token.
The same model also uses softplus attention gating . Softplus maps gating scores to positive values and lacks the upper bound of a sigmoid gate, giving the network a wider range of attention scaling values. On IFM’s evaluation suite, the 36B-A4B model nearly matches the 32B dense model while using about one-seventh as many active parameters.
Uno changes the decoding path
Uno is an adapter that adds diffusion-style parallel decoding to an autoregressive model. The paper describes it as lossless, indicating that the acceleration is designed to preserve the base model’s decoding behavior, and reports peak speedups of 2.2 times.
Conventional speculative decoding uses a smaller draft model to propose tokens for a larger model to verify. Uno attaches an adapter directly to the base model, removing the operational cost of training, loading, and maintaining a separate draft network.
In the paper’s 1K/8K throughput test, the base models support batches of up to 64. At that batch size, K2-Horizon-7B-Uno runs 1.5 times faster and K2-Horizon-0.9B-Uno runs 1.2 times faster. The 2.2-times result occurs at another operating point, so expected gains will depend on batch size, sequence length, hardware, and serving implementation.
One recipe across four scales
IFM trained the 3.7B, 7B, 32B, and 36B models on the same 22-trillion-token corpus with a shared recipe. After normalization, their loss curves follow nearly the same trajectory, suggesting that experiments on smaller models can predict some training behavior at larger scales.
More than 60 pages of the paper cover data construction and selection, including:
- Generation and filtering methods for roughly 10 trillion synthetic tokens.
- Target-scale score predictions used to choose the final data mix.
- Cases where target-scale projections reversed rankings observed in smaller experiments.
- Corpus assembly and deduplication pipelines.
- Mid-training recipes for tool use and agentic behavior.
Agentic training data includes long, multi-turn trajectories containing tool calls, environment responses, and follow-up actions. IFM generated some trajectories through internal pipelines and drew others from public datasets, then used them during both mid-training and post-training.
Stability through longer contexts
Large training runs can diverge when activations, gradients, or optimization schedules become unstable. K2-Horizon uses Grouped RMSNorm, which divides hidden dimensions into groups and normalizes each group separately. The paper reports improved stability across multiple model sizes.
The training schedule changes as context length increases. During stages two through four, the learning rate remains at a floor equal to 10% of its peak value. IFM also raises the RoPE base parameter from 500,000 as contexts grow, adjusting the rotary positional encoding used to represent token positions.
Post-training for the 375B flagship combines supervised fine-tuning and reinforcement learning. The report documents the production pipeline and tracks repetition failures that can disrupt reasoning and tool-use loops.
Overall reasoning repetition falls to 3.8% by the end of the first supervised fine-tuning phase. At the released checkpoint, an environment returning unchanged terminal output no longer triggers the measured failure in which the model repeats the same command.
Memory and safety limits
The current quantized releases compress model weights. KV-cache compression remains in development and is planned as a separate release. The KV cache stores attention keys and values from earlier tokens, and its memory use grows with context length and batch size. It can dominate serving memory during long agent sessions even when the weights are quantized.
The paper’s safety table gives K2-Horizon a 58.3% rate on the “severity under 2” measure across malicious trajectories. Production evaluations for user-facing agents should add domain-specific red teaming, restricted tool permissions, sandboxing, and monitoring rather than relying on one aggregate threshold.
Artifacts and access
IFM has posted the weights and checkpoints in the K2-Horizon collection on Hugging Face. Separate collections cover quantized builds, GGUF packages for local inference tools, and MLX packages for Apple Silicon. The report is also available as an HTML paper .
Each repository’s license and hardware requirements govern production use, so deployment reviews should verify the terms and resource needs of the selected artifact.
Reproducibility at model scale
The combination of shared data, intermediate checkpoints, run histories, and implementation code gives model builders a traceable record of how K2-Horizon was trained. Researchers can reproduce individual stages, compare dense and sparse variants, test alternative data mixtures, and locate changes in behavior across training.
MoVA, softplus gating, and Uno also provide bounded implementation targets for independent experiments. Their published code and matched-model comparisons make the release useful as a reference for sparse attention, decoding acceleration, scaling studies, and agent-focused post-training.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report

