Lost in Serialization: Invariance and Generalization of LLM Graph Reasoners

Researchers demonstrate that Large Language Models used for graph reasoning lack robustness to common graph representation variations like node reindexing and edge reordering, producing inconsistent outputs. Fine-tuning worsens sensitivity to structural and formatting changes while failing to improve generalization on unseen tasks, raising concerns about LLM-based graph reasoners' reliability in production environments.

This research identifies a fundamental vulnerability in LLM-based graph reasoning systems that has implications for AI reliability and trustworthiness. The core finding is that these models produce different outputs when presented with mathematically equivalent graphs in different serialization formats—a property known as lack of invariance. This matters because graphs are inherently symmetric structures, and any robust reasoning system should treat equivalent representations identically.

The study addresses a critical gap in AI robustness research. As LLMs increasingly power reasoning tasks across diverse domains—from molecule analysis in chemistry to social network analysis—their inability to handle standard graph transformations represents a significant limitation. The research systematically decomposes serialization into three components (node labeling, edge encoding, syntax) to pinpoint exactly where failures occur, providing a roadmap for understanding the problem.

For practitioners deploying LLM-based graph reasoners in production, the findings suggest serious caution. Larger, non-fine-tuned models perform better, but fine-tuning—typically used to improve performance—actually degrades robustness while failing to enhance generalization. This creates a painful trade-off: improving accuracy on training tasks may reduce reliability on novel problems. Organizations relying on these systems should implement input normalization and ensemble approaches across multiple serialization formats as safeguards.

Moving forward, this research catalyzes development of invariant-preserving methods for LLM graph reasoning. Solutions might include architectural modifications ensuring serialization-independent processing, or training procedures that explicitly enforce invariance. The work establishes that scaling model size alone is insufficient—fundamental algorithmic innovation is required for trustworthy graph reasoning at scale.

• →LLM graph reasoners lack invariance to equivalent graph representations, producing inconsistent outputs across node reorderings and formatting changes
• →Fine-tuning reduces robustness to structural variations while failing to improve generalization on unseen tasks
• →Larger non-fine-tuned models demonstrate superior robustness compared to fine-tuned smaller variants
• →Graph serialization effects can be systematically decomposed into node labeling, edge encoding, and syntactic factors for targeted analysis
• →Production deployment of LLM graph reasoners requires protective measures like input normalization and ensemble approaches across multiple serializations