Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization

Researchers propose a tensor factorization method that combines cheap automated evaluation data with limited human labels to enable fine-grained evaluation of AI generative models. The approach addresses the data bottleneck in model evaluation by using autorater scores to pretrain representations that are then aligned to human preferences with minimal calibration data.

• →New statistical model uses tensor factorization to merge automated ratings with human gold-standard labels for AI model evaluation
• →Method enables fine-grained evaluation at the prompt level rather than collapsed performance metrics
• →Approach is sample-efficient and robust to autorater quality while providing tight confidence intervals
• →Methodology can construct granular leaderboards and estimate model performance from autorater scores alone
• →Solution addresses the costly bottleneck of human annotations in large-scale AI model evaluation