Addressing the Ecological Fallacy in Larger LMs with Human Context

Researchers developed a method called HuLM (Human-aware Language Modeling) that improves large language model performance by considering the context of text written by the same author over time. Testing on an 8B Llama model showed that incorporating author context during fine-tuning significantly improves performance across eight downstream tasks.

• →Traditional language model training ignores the fact that multiple texts from the same author are linguistically dependent.
• →HuLM addresses the 'ecological fallacy' by modeling author language context in temporally ordered sequences.
• →Human-aware fine-tuning (HuFT) using QLoRA improved 8B Llama model performance over standard fine-tuning methods.
• →Continued HuLM pre-training created a generalizable human-aware model that performed better across eight downstream tasks.
• →The research demonstrates the importance of modeling language in the context of its original authors rather than treating all text uniformly.