LLMbench: A Comparative Close Reading Workbench for Large Language Models

LLMbench is a new browser-based tool that enables detailed comparative analysis of large language model outputs through side-by-side visualization and token-level probability inspection. Unlike existing quantitative comparison tools, it applies digital humanities methodology to make the probabilistic structure of LLM-generated text legible through multiple analytical overlays and visualization modes.

LLMbench represents a significant methodological shift in how researchers and practitioners can examine large language model behavior. Rather than treating LLM outputs as finished products evaluated through ratings and metrics, the tool reframes generated text as a probabilistic object worthy of granular analysis. This distinction matters because it exposes the underlying decision-making processes that occur at the token level—the moment-by-moment choices a model makes when generating text.

The tool's design reflects growing recognition within AI research that understanding model behavior requires both quantitative and qualitative approaches. Existing comparison frameworks like Google PAIR's LLM Comparator focus on aggregate metrics and user preferences, which obscure the probabilistic mechanisms driving output. LLMbench fills this gap by visualizing counterfactual possibilities—showing not just what the model said, but what alternatives existed in its probability distribution.

For AI researchers, developers, and critical scholars, this has practical implications. The tool enables identification of model biases, inconsistencies, and failure modes that might remain invisible in standard benchmarking. The inclusion of metadiscourse analysis and discourse connective highlighting allows examination of rhetorical patterns and textual quality. The five analytical modes—stochastic variation, temperature gradient, prompt sensitivity, token probabilities, and cross-model divergence—each reveal different aspects of model behavior.

As AI systems become increasingly integrated into high-stakes applications, tools that demystify their internal probabilistic structures support more rigorous evaluation and accountability. LLMbench positions log-probability data as essential infrastructure for humanistic and social-scientific critiques of generative AI, potentially influencing how institutions develop frameworks for model documentation and transparency.

• →LLMbench enables close reading of LLM outputs through visualized token-level probability inspection rather than aggregate metrics
• →The tool treats generated text as a probabilistic object with counterfactual alternatives, revealing the model's decision-making structure
• →Four analytical overlays and five modes provide granular visibility into model behavior, bias patterns, and rhetorical characteristics
• →Log-probability data is repositioned as essential for humanistic and social-scientific critique of generative AI systems
• →The tool supports AI accountability and transparency by making probabilistic mechanisms legible to researchers and practitioners