HKUST's LongChang Catches AI Agents Inventing Broken Physics That Never Existed
LongChang tests whether LLMs can notice and explain altered physics with no task or hints given. Only GPT-6-Astra discovered hidden laws, scoring 16.7% while 11 other frontier models scored 0%.
Takeaways
- LongChang tests whether LLMs can notice and explain altered physics with no task or hints given.
- Only GPT-6-Astra discovered hidden laws, scoring 16.7% while 11 other frontier models scored 0%.
- Altered laws included heavier objects falling slower, red balls attracting, and oscillating gravity strength.
- Explorer agents manipulate objects, build custom tools, and write reports judged by separate LLM agents.
- In normal-physics control worlds, about one third of reports falsely claimed something was wrong.
- Code and 540 run records are available at EnVision-Research/LongChang .
Researchers at HKUST (Guangzhou), Alibaba, and Zhejiang University have introduced LongChang , an automated benchmark for open-ended scientific discovery. An explorer model enters a simulated glass room, manipulates objects, receives textual observations, and tries to identify hidden changes to the laws of physics without an assigned task or hints.
Among 12 evaluated language models, the benchmark reports a 16.7% discovery rate for GPT-6-Astra and 0% for the other 11. In control worlds governed by ordinary physics, roughly one-third of reports still claimed an anomaly. The results expose two risks for autonomous agents: overlooking contradictory evidence and inventing explanations for changes that never occurred.
| Measure | Reported result |
|---|---|
| Explorer models | 12 language models |
| Evaluation protocol | Three runs per model per scene |
| Total runs | 540 |
| Highest discovery rate | GPT-6-Astra at 16.7% |
| Other discovery rates | Eleven models at 0% |
| Normal-physics controls | About one-third of reports alleged an anomaly |
| Judge model | GPT-6.1-Sol |
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
