Using street view images and visual LLMs to predict heritage values for governance support: Risks, ethics, and policy implications

Swedish authorities are using visual Large Language Models to analyze 154,710 street view images across Sweden to identify buildings with heritage values, supporting the EU's Energy Performance of Buildings Directive implementation. The research addresses Sweden's lack of a comprehensive heritage building register while raising critical concerns about LLM transparency, error detection, and potential misuse in government governance.

Sweden faces a regulatory challenge under the EU's Energy Performance of Buildings Directive, which requires all member states to develop National Building Renovation Plans by 2025-2026. The absence of a comprehensive heritage building register has created a significant gap in understanding which structures require preservation during renovation efforts. Researchers deployed multimodal LLMs to analyze street view imagery across Sweden, processing over 154,000 building images to predict heritage value indicators using zero-shot learning approaches. This novel methodology identified approximately 5.0 million square meters of heated floor area with potential heritage significance.

The application demonstrates AI's capacity to accelerate large-scale government data collection and policy development. However, the research explicitly flags systemic risks that agencies must consider before deploying such systems. Transparency challenges emerge when LLM decision-making processes remain opaque to stakeholders and oversight bodies. Error detection becomes problematic at scale—incorrect heritage classifications could either unnecessarily restrict renovation opportunities or allow damage to culturally significant structures. The authors highlight sycophancy risks, where models may provide confirmatory outputs aligned with implicit user expectations rather than objective assessments.

For policymakers and urban planners, this research illustrates both the efficiency gains and governance pitfalls of AI-driven decision support systems. The findings extend beyond Sweden, affecting how EU member states approach heritage preservation during mandatory building renovations. Organizations implementing LLM-based governance tools must establish robust validation frameworks, audit trails, and human oversight mechanisms. The work underscores that technical capability should not precede institutional readiness for algorithmic governance.

• →Researchers used visual LLMs to analyze 154,710 Swedish street view images for heritage value prediction, supporting EU building renovation compliance.
• →The method identified 5.0 million square meters of buildings with potential heritage significance despite Sweden lacking a formal heritage registry.
• →Critical risks including LLM transparency, error detection at scale, and model sycophancy threaten accurate governance applications.
• →Zero-shot LLM predictions offer speed advantages but require substantial validation frameworks before government deployment.
• →Findings highlight institutional gaps between AI technical capability and governance infrastructure readiness across EU member states.