Kaggle Arenas Puts AI Models to a Blind Public Vote

Static AI benchmarks face persistent problems with test contamination, narrow grading rubrics, and weak links to production work. A high multiple-choice score reveals little about whether a model can design a readable chart or build a responsive landing page. Kaggle Arenas evaluates those practical outputs through public, head-to-head comparisons.

Each matchup gives two models the same prompt and displays their work side by side without model names. A voter selects the stronger result, and the ballot updates an Elo-style leaderboard. An Elo rating rises with wins, with victories over highly rated opponents carrying more weight. The resulting score measures preference across the sampled prompts, model configurations, and voters.

Six arenas, six deliverables

Kaggle launched separate arenas for tasks with different formats and quality criteria. Several specify output constraints or pass conditions, making their results more useful than a single general-purpose ranking.

• Chart Arena: Models turn a small dataset and brief into a self-contained SVG chart. Tasks cover time series, distributions, funnels, and forecast uncertainty. Every model uses mini-swe-agent 2.4.6 as the agent wrapper.
• Floorplan Arena: Models furnish hand-drawn room sketches, and voters choose the more effective floorplan.
• Landing Page Arena: Models build a one-page website from a small-business brief. Submissions must be self-contained, work on desktop and mobile, and include the requested information.
• ASCII Arena: Models generate ASCII art from the same prompt, with voters selecting the stronger rendering.
• ELI5 Arena: Models answer 100 plain-language explanation prompts across 10 categories. Responses must contain 20 to 300 words.
• Code2Video Arena: Models generate HyperFrames code that renders into short videos, with evaluations focused on adherence to the original prompt.

Code2Video shows how an external organization can bring a specialized evaluation to Kaggle. HyperFrames developed the arena with Kaggle and Google DeepMind. Each matchup presents two videos generated from the same brief. HyperFrames’ existing benchmark uses an LLM judge to score five categories; the arena collects pairwise human preferences for the rendered results.

How the crowd scores creative work

LLM judges provide inexpensive, repeatable scoring, but their ratings inherit the judge model’s preferences and blind spots. Creative tasks expose those limits quickly: a valid chart can remain unreadable, while a constraint-compliant floorplan can waste space. Human reviewers can weigh legibility, composition, usefulness, and prompt adherence together.

Pairwise voting also reduces the burden on each reviewer. Voters choose between two concrete outputs without defining a universal scoring scale. Hiding model names during selection limits brand bias, while the Elo system aggregates many individual comparisons into a running ranking.

The leaderboard still reflects its prompt set and voter population. Longer answers, elaborate formatting, visual polish, and position effects can influence choices independently of correctness. Factual accuracy, accessibility, cost, latency, and production reliability require separate evaluation.

Game Arena supplied the machinery

Kaggle previously used the format in Kaggle Game Arena , a Google DeepMind collaboration that matched AI systems in strategic games such as chess. Formal rules supplied objective outcomes, while an all-play-all schedule and Elo ratings converted match results into rankings. Kaggle’s infrastructure handled the game environments, state management, evaluation harnesses, streams, and replays.

Arenas applies that matchup and ranking infrastructure to open-ended artifacts. Human votes supply the result for tasks where a game engine or test suite cannot fully judge quality.

What developers can learn

Task-specific rankings can help teams create a shortlist for a concrete workload. Broad benchmarks such as MMLU summarize general capabilities, while an arena dedicated to SVG charts or responsive pages measures performance closer to the intended application.

1. Choose by workload. Rankings can reveal that models suited to chart generation differ from those preferred for layout, explanation, or video code.
2. Inspect the harness. Chart Arena’s use of mini-swe-agent 2.4.6 narrows variation from agent scaffolding. Provider settings, system prompts, tool access, and sampling parameters remain relevant when interpreting results.
3. Use rankings as a shortlist. Final selection should include tests on private examples, edge cases, latency, token usage, price, failure recovery, and output consistency.
4. Track rating stability. Elo scores change as votes arrive, so close rankings may reflect limited evidence. Vote counts and confidence intervals are essential context when available.
5. Build domain evaluations. Code2Video demonstrates a route for specialized benchmarks that combine a shared execution environment with public preference data.

Where the signal can bend

Human evaluation scales only when the voter pool remains broad, prompts stay fresh, and the platform limits coordinated voting or repeated exposure. Task constraints can reduce style bias, though they cannot eliminate it. Model updates also complicate comparisons when a provider changes a system behind an unchanged name.

Kaggle Arenas gives developers a live measure of which outputs people prefer under defined tasks and shared prompts. Its leaderboards offer useful comparative evidence when paired with local testing and operational metrics.