VideoZeroBench Exposes Leading Video AI Models Scoring Near Zero on Visual Grounding
VideoZeroBench introduces 500 long-video questions with manually annotated temporal intervals and spatial boxes as evidence. A five-level protocol separates answering from temporal grounding and spatial grounding.
Takeaways
- VideoZeroBench introduces 500 long-video questions with manually annotated temporal intervals and spatial boxes as evidence.
- A five-level protocol separates answering from temporal grounding and spatial grounding.
- Gemini-3-Pro tops standard QA at just 17% accuracy, exposing benchmark saturation elsewhere as illusory.
- No model exceeds 1% when correct answers plus accurate spatio-temporal grounding are both required.
- Agentic thinking-with-video systems like VideoChat-R1.5 score zero at the strict grounding levels.
- Manually zooming to evidence regions lifts Gemini past 40%, pointing to retrieval as the real bottleneck.
VideoZeroBench tests whether video multimodal models can identify the frames and pixels supporting an answer. Its authors evaluated 17 proprietary, open-source, and specialist systems on 500 verified questions from 138 long videos. Gemini-3-Pro led the standard, hint-free Level 3 evaluation with 17.0% accuracy. Under Level 5, which also requires correct temporal intervals and bounding boxes, no model exceeded 1.0%, and many scored zero.
High scores leave evidence untested
Recent video multimodal large language models, or video MLLMs, have exceeded 80% on benchmarks such as VideoMME, Video-MMMU, and LongVideoBench. Their standard protocols grade the final answer without requiring supporting timestamps or image regions.
Answer-only grading cannot determine whether a model used the relevant visual evidence, relied on language priors, inferred from broad scene cues, or guessed. The benchmark paper adds temporal and spatial checks that make those paths easier to distinguish.
Task format and question difficulty limit direct comparison with earlier headline scores. VideoZeroBench uses exact answers and long videos, then adds progressively stricter grounding requirements.
Inside the 500-question test
The manually curated dataset covers long-form material across 13 domains, including sports, gaming, driving, instruction, music, vlogs, and animation.
| Dataset feature | Value |
|---|---|
| Videos | 138 |
| Verified questions | 500 |
| Language split | 220 English, 280 Chinese |
| Average video length | 667 seconds, about 11.1 minutes |
| Questions with temporal evidence | 442 |
| Questions with spatial evidence | 372 |
| Average annotated box area | 6.8% of the frame |
Temporal and spatial annotations overlap because many questions require both. The small average box area also means models often need to locate fine visual details within a large frame.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
