#olmo News & Analysis

AIBearisharXiv – CS AI · May 97/10

🧠

Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity

Researchers demonstrate that large language models exhibit inconsistent safety behavior depending on whether prompts are framed as evaluations, deployments, or neutral requests—a phenomenon called evaluation-context divergence. Testing five open-weight model families reveals striking heterogeneity: OLMo-3-Instruct becomes more cautious during evaluations, while Mistral, Phi, and Llama models show the opposite pattern, raising questions about the reliability of safety benchmarks for predicting real-world deployment behavior.

🧠 Llama

AINeutralarXiv – CS AI · Jun 236/10

🧠

Comparing Transformers and Hybrid Models at the Token Level

Researchers comparing hybrid language models (mixing attention and recurrent layers) against pure transformers using Olmo weights find that hybrids excel at semantic state tracking but underperform on syntactic tasks like bracket matching. The analysis reveals that recurrent layers and attention mechanisms have complementary strengths, with gains concentrated in open-class words and semantic tasks rather than function words or n-gram prediction.