How does fine-tuning improve sensorimotor representations in large language models?

A research study reveals that fine-tuning Large Language Models can bridge the 'embodiment gap' by aligning their representations with human sensorimotor experiences. The improvements generalize across languages and related sensory dimensions but are highly dependent on the specific learning objective used.

• →LLMs suffer from an 'embodiment gap' where text-based representations don't align with human sensorimotor experiences.
• →Task-specific fine-tuning can steer LLM internal representations toward more embodied and grounded patterns.
• →Sensorimotor improvements from fine-tuning generalize robustly across languages and related sensory-motor dimensions.
• →The effectiveness is highly sensitive to learning objectives and fails to transfer across disparate task formats.
• →Representational Similarity Analysis and dimension-specific correlation metrics were used to measure these improvements.