World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings

Research shows that static word embeddings like GloVe and Word2Vec can recover substantial geographic and temporal information from text co-occurrence patterns alone, challenging assumptions that such capabilities require sophisticated world models in large language models. The study found these simple embeddings could predict city coordinates and historical birth years with high accuracy, suggesting that linear probe recoverability doesn't necessarily indicate advanced internal representations.

• →Static embeddings like GloVe and Word2Vec can recover geographic coordinates with R² values of 0.71-0.87 and temporal data with R² values of 0.48-0.52.
• →The spatial and temporal signals depend heavily on interpretable lexical patterns, especially country names and climate-related vocabulary.
• →Simple text co-occurrence preserves more world-like structure than previously assumed.
• →Linear probe recoverability alone may not prove that language models have developed sophisticated internal world representations.
• →The findings challenge interpretations of LLM capabilities that assume complex internal modeling beyond textual patterns.