Concept Heterogeneity-aware Representation Steering

Researchers introduce CHaRS (Concept Heterogeneity-aware Representation Steering), a new method for controlling large language model behavior that uses optimal transport theory to create context-dependent steering rather than global directions. The approach models representations as Gaussian mixture models and derives input-dependent steering maps, showing improved behavioral control over existing methods.

• →CHaRS addresses limitations of current LLM steering methods that assume homogeneous representation across embedding spaces.
• →The method uses optimal transport theory to model source and target representations as Gaussian mixture models.
• →Input-dependent steering maps are derived through barycentric projection, creating smooth kernel-weighted combinations of cluster-level shifts.
• →Experimental results demonstrate CHaRS provides more effective behavioral control than global steering approaches.
• →The research advances techniques for fine-grained control of large language model behavior at inference time.