Automated Analysis of Global AI Safety Initiatives: A Taxonomy-Driven LLM Approach

Researchers developed an automated framework using large language models to compare AI safety policy documents across a shared taxonomy of activities. The study found that model choice significantly affects comparison outcomes, with some document pairs showing high disagreement across different LLMs, though human expert evaluation showed high inter-annotator agreement.

• →An automated crosswalk framework was created to systematically compare AI safety policy documents using LLMs.
• →Model choice substantially affects the outcomes of AI policy document comparisons.
• →Some document pairs yield high disagreements across different language models.
• →Human expert evaluation showed high inter-annotator agreement despite model variations.
• →The framework supports comparative inspection of policy documents for AI safety analysis.