APITry the API →
HermesHermes Agent Docs
API
Back to News
paperTraining InfraLlms

Stanford's SOMA Slashes AI Training Noise by Sharding Models Into Expert Shards

A new architecture called SOMA shards models into independent experts, slashing the gradient variance that has blocked zero-order methods from scaling to pretraining.

Stanford's SOMA Slashes AI Training Noise by Sharding Models Into Expert Shards
Source
AlphaSignal
Published
Author
AlphaSignal Newsroom
Read
1 min read

A new architecture called SOMA shards models into independent experts, slashing the gradient variance that has blocked zero-order methods from scaling to pretraining.

Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.

Read original report

Next reads

Zyphra Cuts MoE Training Communication Costs by 2.63x Using Router Patterns
AlphaSignal · paper

Zyphra Cuts MoE Training Communication Costs by 2.63x Using Router Patterns

SmolLM3 Retrieves Perfectly at 64x Its Training Length by Moving Dropout
AlphaSignal · paper

SmolLM3 Retrieves Perfectly at 64x Its Training Length by Moving Dropout

Yousef Saad's Survey Bridges Math and LLMs With One Shared Language
AlphaSignal · paper

Yousef Saad's Survey Bridges Math and LLMs With One Shared Language