Edge0 Squeezes a 120B AI Agent Into one Consumer GPU

Takeaways

Edge0 released 120B and 145B latent sparse MoE models trained natively at 4 bits.
Built on Qwen3.8-Flash-Next; 512 experts per layer with 10 routed plus 1 shared active per token.
120B Q3_K hits 144.6 tok/s agent decode and 75.3 terminal-bench 2.1 on an RTX 5090 D.
Designed for 24GB VRAM with the rest streamed from RAM or SSD, enabling consumer-GPU inference.
Ships as GGUF in Q3_K and NVFP4 formats with 256K context and int8 KV cache.
Runs via edge0-infer , llama.cpp, vLLM, Ollama, LM Studio under a restricted community license.

Edge0 is a new model family designed to run a 120-billion-parameter agent on one consumer GPU while streaming additional weights from system RAM or memory-mapped storage. Built from Qwen3.8-Flash-Next, the models use native 4-bit training and ship as GGUF files in 120B and 145B variants. Available formats include Q3_K, a roughly 3-bit GGUF quantization scheme, and NVFP4, Nvidia’s 4-bit floating-point format.

Edge0 reports that the 120B Q3_K model reaches 144.6 output tokens per second in its agent workload, 4,576.5 prompt tokens per second at 16K context, and 75.3 on Terminal-Bench 2.1 using an RTX 5090 D with 24GB of VRAM. The 145B version reportedly scores 86.5 on Terminal-Bench at about 121 output tokens per second on the same card. These are vendor-supplied results that still need independent reproduction.

How 120B parameters fit around 24GB

Each of the model’s 48 layers contains 512 experts in a sparse mixture-of-experts architecture. For every token, the router activates 10 specialized experts and one shared expert, leaving most parameters idle. The 120B model therefore uses 8.64B active parameters per token, which determines much of the computation required during generation.

Attention operates at the full hidden width of 2,560, while the experts work in a compressed latent space. The latent width is 1,280 for the 120B model and 1,792 for the 145B model, with a SwiGLU intermediate width of 640. A down-projection compresses each token before routing, every selected expert applies a latent → 640 → latent transformation, and an up-projection restores the result to width 2,560.

That layout turns each 120B expert into a fixed record of about 1.16MB. The runtime keeps several thousand frequently used experts in VRAM and places the remaining experts in system memory or memory-mapped storage. VRAM acts as a cache, while routing patterns and transfer bandwidth influence generation speed.