KAUST's EdgeCIM Hits 1,000 Tokens per Second Running AI on Edge Devices

Researchers at KAUST, UC Irvine, and Rain Neuromorphics have proposed EdgeCIM, a compute-in-memory accelerator and hardware search framework for decoder-only language models with up to 4 billion parameters. The EdgeCIM paper , accepted for publication in IEEE TCAD, reports simulated results from a 65 nm design. It targets the memory traffic that limits token generation on phones, laptops, and embedded boards.

One token creates a bandwidth problem

Transformer inference divides into prefill and decoding. Prefill processes the prompt with dense matrix-matrix multiplication, or GEMM, which reuses data efficiently and keeps parallel arithmetic units busy.

Autoregressive decoding processes one new token at a time. Each layer applies large weight matrices to a one-token activation vector, producing matrix-vector multiplication, or GEMV, with little data reuse. Attention also reads a growing key-value cache. At batch size one, the device cannot spread those memory reads across several requests, so memory bandwidth and latency dominate execution.

Profiling LLaMA 3.2-1B on a Jetson Orin attributed an average of 96.6% of inference time to decoding across prompt lengths from 64 to 1,024 tokens and generated sequences of up to 512 tokens. That imbalance explains why adding arithmetic capacity alone produces limited gains for interactive edge inference.

SRAM takes over the multiply

Compute-in-memory reduces data movement by performing multiply-accumulate operations against weights stored inside an SRAM array. EdgeCIM uses a bit-serial SRAM macro that supports INT4 and INT8 arithmetic, with processing elements grouped into tiles and clusters.