APIUp to 28% cheaper than official pricesTry the API →
Hermes
Back to News

Appear2Meaning Benchmark Exposes AI's Blind Spot on Museum Artifacts

Appear2Meaning benchmarks 9 VLMs on inferring culture, period, origin, and creator from museum images. 750 verified artifacts curated from Getty and the Met across 4 cultural regions and 4 object types. Exact-match accuracy stays between 0.5% and 2.9% for every model, closed or open.

Appear2Meaning Benchmark Exposes AI's Blind Spot on Museum Artifacts
Source
AlphaSignal
Published
Author
AlphaSignal
Read
2 min read

Takeaways

  • Appear2Meaning benchmarks 9 VLMs on inferring culture, period, origin, and creator from museum images.
  • 750 verified artifacts curated from Getty and the Met across 4 cultural regions and 4 object types.
  • Exact-match accuracy stays between 0.5% and 2.9% for every model, closed or open.
  • Qwen3-VL-Flash leads partial-match at 65.8%, beating GPT-4.1-mini, GPT-5.4-mini, and Claude Haiku 4.5.
  • East Asia is easiest (up to 79% culture accuracy); Europe and the Americas are hardest.
  • Dominant errors: cross-cultural misattribution, object-type confusion, period compression, creator memorization.

A museum model may recognize a porcelain vase while missing where, when, and by whom it was made. The Appear2Meaning benchmark tests that deeper attribution task by asking vision-language models to recover structured museum metadata from an image alone. Across nine models, full-record exact-match accuracy ranged from 0.5% to 2.9%.

Five fields from one image

Most heritage captioning datasets, including earlier heritage datasets such as Artpedia, ArtCap, and EmoArt, reward descriptions of an object’s appearance or emotional tone. Appear2Meaning evaluates the controlled fields used in collection records: title, cultural tradition, historical period, geographic origin, and creator, which may be an artist, maker, or workshop.

The benchmark treats those fields as latent attributes inferred from visual evidence. Models may generate captions as an intermediate step, but evaluation scores only the structured outputs. This design measures whether a system can connect visible features to historical and cultural knowledge when no catalog metadata is supplied.

How 750 objects were selected

Researchers drew the dataset from the Getty and Metropolitan Museum of Art collections. Rule-based retrieval produced candidate objects, followed by two stages of human verification. The team sampled 50 artifacts for each included culture-and-object-type combination, producing 750 verified records.

Dataset featureCoverage
SourcesGetty and Metropolitan Museum of Art
Regional groupsEast Asia, Ancient Mediterranean, Europe, and the Americas
Object categoriesCeramics, paintings, metalwork, and sculpture
Sample size750 verified objects
Sampling rule50 objects per included culture-and-type combination
LicensingDataset and code under MIT; heritage images under CC0

Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.

Read original report

Sources

  1. 01arxiv.org
  2. 02arxiv.org
  3. 03getty.edu
  4. 04metmuseum.org