All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application

A production rental-search web application integrated with large language models and multi-market support accumulated 1,553 passing test cases over six weeks, yet defects continued reaching users. Analysis of 252 bug-fix commits revealed that 44% of failures occurred at integration seams—live browser runtime, non-default markets, end-to-end flows, and system-level interactions—that component-level unit tests cannot detect.

This study exposes a critical gap between test coverage metrics and actual software reliability in modern, complex applications. The rental-search assistant represents a common architectural pattern: LLM integration, internationalization across markets, and browser-based front-ends querying external data sources. Despite achieving near-perfect test pass rates, the application shipped defects regularly, revealing that traditional unit testing provides false confidence when applied to systems with multiple integration boundaries.

The four-seam framework identifies where defects escape: the live browser runtime where timing and DOM interactions diverge from test environments, non-default markets where localization assumptions break, end-to-end flows involving multiple system components, and whole-system interactions. These seams represent zones where individual components function correctly in isolation but fail when integrated. The finding that 44% of production bugs originated from these seams underscores how heavily modern applications depend on integration points that unit tests systematically miss.

For development teams and organizations adopting LLM-integrated applications, this research validates a shift toward seam-level testing strategies. The study's practical contribution—identifying which seam carries the most fixes for a specific project—enables teams to allocate limited QA resources more effectively. As AI and internationalization become standard features rather than exceptions, the cost of ignoring integration boundaries only increases. Organizations relying solely on component-level metrics face recurring production incidents despite high reported test coverage.

• →High test coverage does not guarantee production reliability when integration seams are not explicitly tested
• →44% of bugs in an LLM-integrated application escaped component-level tests by crossing integration boundaries
• →Browser runtime, market-specific logic, end-to-end flows, and system-level interactions require separate testing strategies
• →Repeating defects can occur when fixes lack guards at the specific seam where they originated
• →Teams can identify their highest-risk seams by analyzing the distribution of historical bug fixes