In-Context Examples Suppress Scientific Knowledge Recall in LLMs

Research shows that in-context examples in large language models suppress recall of scientific knowledge, causing models to shift from knowledge-driven reasoning to empirical pattern fitting even when examples are generated from the same formulas they should reinforce. This finding across 60 tasks and four models suggests practitioners deploying LLMs for scientific work should be cautious about using examples, as they may undermine rather than support domain expertise.

A critical vulnerability has emerged in how large language models approach scientific reasoning. Rather than learning from examples, LLMs appear to abandon their pretrained scientific knowledge when provided with in-context demonstrations, instead relying on surface-level pattern matching. This discovery challenges conventional wisdom about prompt engineering and few-shot learning, techniques widely adopted to improve LLM performance across industries.

The research identifies a fundamental trade-off in LLM cognition. Models trained on scientific literature develop genuine knowledge of formulas, constants, and domain principles. However, when examples are introduced during inference, the models deprioritize this knowledge in favor of extracting patterns from the provided data. This isn't a minor efficiency trade-off—it represents a wholesale shift in computational strategy that undermines the core value proposition of using LLMs for scientific tasks: their ability to apply deep domain understanding to novel problems.

For organizations building AI systems in scientific domains, this has immediate implications. Financial modeling, drug discovery, materials science, and climate prediction systems relying on LLMs may face unexpected accuracy degradation when incorporating best-practice prompting techniques. The consistency of this knowledge displacement across five scientific domains and multiple models suggests it's a systemic feature rather than a quirk of specific architectures. Teams must now choose between leveraging in-context examples for task-specific adaptation or preserving knowledge-driven reasoning by minimizing example provision.

The findings suggest future LLM development should prioritize architectures that maintain knowledge recall under example-driven prompting, preventing the cognitive shortcuts that undermine scientific reasoning.

• →In-context examples shift LLMs from knowledge-based reasoning to pattern-fitting, even when examples derive from the same scientific formulas
• →The knowledge suppression effect is consistent across five scientific domains but has varying accuracy impacts depending on task complexity
• →Practitioners should reconsider best-practice few-shot prompting for scientific tasks, as it may degrade performance rather than improve it
• →Four different LLM architectures showed the same knowledge displacement pattern, indicating a systemic issue rather than model-specific behavior
• →Future LLM development must address this vulnerability to maintain reliability for scientific and technical applications