AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems

Researchers have developed an open-source benchmark dataset to evaluate AI systems' compliance with the EU AI Act, specifically focusing on NLP and RAG systems. The dataset enables automated assessment of risk classification, article retrieval, and question-answering tasks, achieving 0.87 and 0.85 F1-scores for prohibited and high-risk scenarios.

• →New open-source dataset created to evaluate AI system compliance with EU AI Act regulations
• →The benchmark addresses the lack of automated tools for regulatory compliance assessment in AI systems
• →Dataset includes tasks for risk-level classification, article retrieval, obligation generation, and question-answering
• →Methodology combines domain knowledge with large language models to generate evaluation scenarios
• →Testing shows promising results with F1-scores of 0.87 and 0.85 for prohibited and high-risk AI scenarios