The cheapest AI models from OpenAI, Google and Anthropic can usually tell you that a picture shows the night sky. A new astronomy VQA study shows they often cannot tell you how big that patch of sky is, whether it is a contour plot, or whether the stars in it are real.
The paper was posted to arXiv on 15 October 2026. It comes with a public dataset of 2,001 figures and 47,357 questions, and it suggests that “good enough” AI is not yet good enough to read decades of astronomy papers for us.
Key takeaways
- J. P. Naiman of the University of Illinois Urbana-Champaign tested GPT-5.4-nano, Gemini-3.5-flash-lite and Claude-Haiku-4.5 on astronomical sky figures.
- The models labelled real sky images correctly 80-98% of the time, but spotted fake, statistically generated skies only 4-14% of the time.
- GPT overestimated the angular height of the sky region by more than ten times in 53% of answers on clean figures.
- Simulated page ageing, the yellowing and noise of old scanned journals, made almost no measurable difference.
- Running roughly 6,000 questions per model cost between about $4 and $15.
What is astronomy VQA, and why build a new dataset?
VQA stands for visual question answering. You show a model an image, ask it a question (“What is the width of this field in degrees?”) and score the answer against a known truth. Large multimodal models, or LMMs, are AI systems that take in both text and images.
Astronomy has a specific problem here. More than a century of journal pages hold sky maps, contour plots and telescope images that have never been turned into searchable data. The paper ties into the Reading Time Machine, a NASA-funded project to turn that historical literature into data researchers can query. If a cheap model could read those figures reliably, a huge archive would open up.
To test that, you need figures where the right answer is known exactly. So Naiman generated them. The paper on arXiv describes three equal groups of 667 figures each: synthetic contour plots, real cutouts from NASA’s SkyView virtual observatory, and “GMM sky” images. These are fakes built from a Gaussian mixture model, a statistical recipe that sums many blurry blobs, and they use the same axes and scales as the real images.
What did the models actually get wrong?
The questions climb from simple reading (what colormap is used, what the tick labels say) to estimation (how many degrees across is this field) to inference (was this image made from a real sky or a model). The evaluation used 255 figures, 85 per type, which works out to 6,035 questions per model.
| Task (clean figures) | GPT-5.4-nano | Gemini-3.5-flash-lite | Claude-Haiku-4.5 |
|---|---|---|---|
| Correctly identifies a contour plot | 66% | ~80% | 15% |
| Angular height off by more than 10x | 53% | Not reported | 32% |
| Approximate cost of the run | $8.11 | $4.42 | $15.14 |
The fake-sky result is the most striking. Across models, real sky images were labelled real 80-98% of the time, but GMM skies were called fake only 4-14% of the time. In practice, the models answer “real” to almost everything.
That is not because the task is impossible. A plain gradient-boosted classifier, an old-school machine learning method, separated real from fake skies with 85% accuracy, against a 50% coin-flip baseline. Think of a tourist who calls every painting in a museum an original: they are right about the originals, but only by luck.
Size estimates were also shaky. A tenfold error, one “dex” in astronomer shorthand, is the gap between a patch the size of the full Moon and one ten times taller. GPT made that error in more than half of its answers on clean figures.
Does old, yellowed paper confuse the AI?
Mostly no, and this is good news for anyone digitising archives. Every figure ships in a pristine version and an “archive-light” version that imitates aged, scanned journal pages. Outside a single case, the study found no significant difference between the two.
The odd case went the “wrong” way: GPT labelled aged sky images correctly 72% of the time, against 53% for clean ones. The author adds a caveat. With this sample size, the test can only rule out ageing effects bigger than about 21 percentage points, so smaller effects could still be hiding.
Do synthetic figures predict performance on real papers?
Roughly, yes. Naiman ran Gemini on 104 human-verified figures from historical journals. The median width error was -0.08 dex on the published figures and -0.05 dex on the aged synthetic ones, which is close.
Real figures were messier at the extremes, though. For angular height, 10% of answers on published figures were more than ten times too large, against 2% for synthetic ones. Real papers have cramped labels, odd fonts and multiple panels that a clean generator does not reproduce.
How does this fit with other astronomy AI benchmarks?
It matches a wider pattern. AstroVLBench, released in April 2026 with more than 4,100 expert-verified items, found that six frontier models all trailed specialised astronomy methods. It also found that giving the same data as a numeric table instead of a plot raised accuracy by up to 13 percentage points.
Here is my read (opinion): the low-cost tier is where large archive projects actually operate, because you cannot pay flagship prices to read millions of figures. This paper is useful because it measures that tier instead of the expensive models that make headlines.
What happens next
The paper has been accepted at the 4th Workshop for AI for Scientific Publications (WASP 2026), held alongside the IJCNLP-AACL conference. Naiman plans a more realistic image pipeline, new figure types such as spectra, and a closer look at the models’ open-ended explanations.
Everything is open. The dataset on Hugging Face includes all model responses, and the code on GitHub lets anyone rerun the tests on newer models. The dataset card also flags known issues, including a corrected pixel-scale ground truth and an open-ended question that almost every model fails because of its wording.
FAQ
Which AI models were tested in the astronomy VQA study?
Three low-cost models: GPT-5.4-nano, Gemini-3.5-flash-lite with minimal thinking, and Claude-Haiku-4.5 with thinking turned off.
Can I use the dataset myself?
Yes. The 2,001 figures, 47,357 questions and all model outputs are on Hugging Face under ReadingTimeMachine/astro_sky_image_vqa, released under Apache 2.0. Real SkyView images still carry survey acknowledgement requirements.
Does this mean AI cannot help astronomers?
No. It means cheap general models should not be trusted unsupervised for precise figure measurements yet. They did well on simple reading tasks, and simulated page ageing barely affected them.
If you build tools that read charts or scanned documents, download the dataset and test your own model on the fake-sky questions before trusting it with real data.
1 comment