Scaling In, Not Up? Testing Thick Citation Context Analysis with GPT-5 and Fragile Prompts

Researchers tested GPT-5's ability to perform citation context analysis by examining how different prompt designs affect the model's interpretative readings of academic citations. The study found that while GPT-5 produces consistent surface classifications, prompt scaffolding significantly influences which interpretative frameworks and vocabularies the model emphasizes in deeper analysis.

• →GPT-5 demonstrated high stability in surface-level citation classification tasks, consistently identifying citations as 'supplementary'.
• →Prompt design significantly affects how AI models interpret and analyze academic texts, with different scaffolding leading to varied interpretative outcomes.
• →The model generated 450 distinct hypotheses across 90 reconstructions, showing extensive interpretative capability but also inconsistency.
• →GPT-5 identified the same textual patterns as human analysts but interpreted them differently, favoring lineage and positioning over critical readings.
• →The research highlights both opportunities and risks of using LLMs as co-analysts for interpretative academic work.