Characterizing Web Search in The Age of Generative AI

Researchers systematically compared generative search systems (Google, OpenAI, Perplexity) with traditional Google search, revealing fundamental differences in retrieval strategies, source diversity, and output stability. Generative search synthesizes web information into coherent responses but exhibits significant variation in reliance on internal knowledge, consistency across executions, and evaluation metrics, necessitating new assessment frameworks.

This research addresses a critical gap in understanding how generative AI is reshaping information retrieval. As LLM-powered search engines gain adoption, the traditional ranking-based search paradigm is being displaced by systems that synthesize information into narrative responses. The study's systematic comparison between Google's organic search and five generative competitors reveals that these new systems operate under fundamentally different operational principles—some rely heavily on internal training data while others prioritize external web sources, creating divergent information landscapes for users.

The findings highlight a transition already underway in the search ecosystem. Generative search systems compress vast amounts of indexed information into single coherent responses, which users find intuitive but which obscures source attribution and introduces consistency challenges. The research demonstrates that outputs vary across time and multiple executions, indicating robustness issues that current evaluation methodologies don't capture.

For the AI industry, this study validates concerns about generative search's reliance on proprietary models versus transparent source attribution. Publishers and content creators face uncertainty about visibility and attribution in a system that synthesizes rather than ranks. The fragmentation among providers—each using different retrieval and synthesis strategies—suggests the generative search landscape remains immature and unstandardized.

Future implications center on standardization. As generative search matures, stakeholders will demand better consistency, source transparency, and reproducibility. The research indicates that existing SEO and search evaluation frameworks are obsolete for generative systems, driving development of new benchmarks focused on retrieval behavior and synthesis quality. This creates both challenges for search quality assurance and opportunities for companies developing evaluation tools.

• →Generative search engines show substantial variation in internal knowledge reliance and source diversity, indicating no standardized approach has emerged.
• →Output instability across time and executions raises robustness concerns that traditional search evaluation metrics fail to capture.
• →Generative systems achieve comparable topical coverage to traditional search but through markedly different retrieval footprints and synthesis strategies.
• →Source attribution and visibility become opaque in generative search, affecting publishers and content creators dependent on search discovery.
• →New evaluation paradigms are needed to assess retrieval behavior, synthesis quality, and stability in generative search systems.