Vals AI's CUA-Bench Humbles Every Frontier Agent Below 20 Points

CUA-Bench puts frontier agents below 20%

Vals AI’s CUA-Bench evaluates whether AI agents can play six commercial PC games using only the screen, keyboard, and mouse. Among five tested frontier models, the highest average across public and held-out titles was 19.2 points out of 100. The benchmark combines capabilities that text and coding evaluations rarely test together: visual perception, low-latency control, long-horizon planning, and adaptation during a live task.

Six games, one narrow interface

• Game set: Six commercial titles spanning sandbox, first-person action, and sports genres.
• Time limit: Three hours per title.
• Controls: Keyboard and mouse commands based on the game’s visual output.
• Restrictions: No direct access to game state, memory, save files, internal APIs, or network calls to the game.
• Sampling: One run for each model-game combination.

The three public titles are Minecraft, SUPERHOT, and eFootball. Each is paired with an undisclosed game from the same genre, allowing the benchmark to compare title-specific familiarity with transfer to a related environment.

A 100-point progression ladder

Each game uses ordered milestones totaling 100 points. Agents must complete the milestones in sequence, preventing a later achievement from bypassing earlier requirements.

Thirty-four seconds per action

Because the game clock continues while a model reasons, response latency directly affects performance. Across all trials, models completed one decision cycle every 15 to 59 seconds, with a median of 34 seconds. They generated 2 to 19 input events per minute, while a person typically produces several per second during active play.

Astra leads, still below 20

A simple average of each model’s public and held-out means leaves every tested system below 20 points.

Claude Opus 5’s Minecraft run shows how slowly progress accumulated. Acting once every 36 seconds, it collected cobblestone at minute 50 and made a stone pickaxe at minute 97. It also built a furnace, but no tested model reached iron.

GPT-6 Astra cleared five of SUPERHOT’s 25 levels, the best result on that title. In eFootball, Astra completed a decision cycle every 28 seconds and recorded 743 direction-key holds during the trial. It won twice at Beginner difficulty and failed to win at Regular.

Inside the evaluation harness

Each model ran through its vendor’s command-line agent: Codex for OpenAI models, Claude Code for Anthropic models, and Google’s computer-use agent for Gemini. The agent process ran in a separate CPU sandbox and controlled a GPU desktop through Model Context Protocol, or MCP. Agents could use their own shell and Python environment for scratch work based on captured screens.

The access restrictions applied to the agent, while grading used preserved evidence after each run. Four offline games were scored from native world or save-file state. The two online sports games were graded from video sampled at 2 Hz, meaning two frames per second, with each candidate result confirmed against three distinct source frames.

Hidden games test transfer

Because labs know the public titles but not the hidden ones, score divergence can reveal weak transfer or benchmark-specific optimization. Four models scored higher on the public set, with gaps ranging from 2.0 points for GPT-5.6 Sol to 18.3 points for Claude Fable 5.1. Gemini 3.8 Flash reversed the pattern, scoring 5.0 points higher on the held-out set.

Genre familiarity, title-specific training, contamination, and ordinary run variance could all produce these gaps. With one trial per model-game cell, the current results cannot distinguish among those causes. Repeated runs would be required to estimate variance and support stronger conclusions about contamination.

Adaptation is hard to cache

Vals designed the benchmark to track within-session learning, meaning whether an agent improves by incorporating earlier attempts during the same run. In the company’s Kerbal Space Program evaluation, an agent can often summarize a lesson as text and reuse it. Real-time game control depends more heavily on timing, spatial judgment, and interpreting a sequence of visual outcomes.

Progressing from Beginner to Regular in eFootball illustrates that challenge. The agent must infer why a movement or pass failed, adjust its timing, and execute the revision before the next opportunity disappears.

Computer use remains brittle

Computer-use research has also found large performance swings after minor interface changes. An unrelated CUA benchmark reports that the best frontier agent completed 6 of 25 expert-authored tasks in KiCad, an electronics schematic editor. None reliably constructed a schematic from a blank canvas.

Real-time games add continuous motion and deadlines to that interface problem. CUA-Bench rewards a closed control loop in which an agent observes the screen, chooses an action, executes it promptly, inspects the result, and revises its plan over several hours. Desktop automation and robotics depend on the same loop when privileged state and task-specific APIs are unavailable.