Semantic search for 100M+ galaxy images using AI-generated captions

Researchers developed AION-Search, an AI-powered semantic search engine that catalogs over 100 million galaxy images using Vision-Language Models to generate captions and create searchable embeddings without manual labeling. The system achieved state-of-the-art performance in discovering rare astronomical phenomena and identified 36 new extragalactic stellar stream candidates, while offering a generalizable approach for making large unlabeled scientific image archives semantically searchable.

This breakthrough demonstrates how Vision-Language Models can unlock value in massive, unlabeled scientific datasets by automating the traditionally bottlenecked process of manual annotation. Rather than requiring astronomers to painstakingly label billions of images, the researchers leveraged VLMs to generate descriptions automatically, then aligned these with a pre-trained astronomy foundation model to produce searchable embeddings at scale. The result is a qualitative leap in astronomical discovery capability.

The technical innovation reflects a broader trend in AI where foundation models trained on diverse data are being adapted for specialized domains. The AION-Search system's zero-shot performance on rare phenomena—without seeing curated examples—suggests that semantic understanding captured during pretraining generalizes effectively to scientific domains. The VLM-based re-ranking method that doubled recall for difficult targets shows iterative refinement can push performance further.

Beyond astronomy, this approach has immediate applicability across scientific fields struggling with data exploration bottlenecks. Earth observation platforms analyzing satellite imagery and biomedical researchers cataloging microscopy data face identical challenges: enormous archives of unlabeled images that contain valuable discoveries but remain inaccessible to traditional search methods. By making code and data publicly available, the researchers enable rapid adoption across disciplines.

The identification of 36 new stellar stream candidates validates that the system surfaces genuine scientific value, not merely plausible-sounding results. This moves semantic search from theoretical capability to practical tool. Future developments may integrate user feedback loops to continuously improve search relevance and explore whether language-based search can surface unexpected phenomena that keyword-based approaches would miss.

• →AION-Search enables semantic searching of 100M+ galaxy images using AI-generated captions instead of manual labeling, dramatically reducing annotation bottlenecks.
• →The system achieved state-of-the-art zero-shot performance on rare astronomical phenomenon detection without requiring curated training data.
• →Researchers identified 36 previously unknown extergalactic stellar stream candidates, demonstrating the tool's genuine scientific discovery capability.
• →The approach is generalizable across scientific domains including Earth observation and microscopy, addressing data exploration challenges broadly.
• →Full code, data, and application are publicly available, enabling rapid adoption and refinement by the research community.