Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models

Researchers present Gate-and-Merge, a zero-shot framework enabling vision-language models to recognize and compose multiple user-defined concepts without requiring co-occurrence training data. The approach uses lightweight LoRA adapters for individual concepts and employs a gating mechanism to merge them intelligently at inference time, maintaining concept integrity while enabling compositional personalization.

Gate-and-Merge addresses a fundamental challenge in machine learning: enabling models to understand and combine multiple personalized concepts without explicit training on their joint appearances. This research advances the field of compositional learning, which remains a critical bottleneck in making AI systems more flexible and user-centric. The framework's zero-shot capability eliminates the need for expensive co-occurrence training data, substantially reducing computational overhead and accelerating deployment timelines.

The technical innovation centers on three mechanisms working in concert. Each user concept becomes a separate, lightweight LoRA adapter—a parameter-efficient fine-tuning technique that adds minimal overhead compared to full model retraining. The gating mechanism then selectively activates only relevant concept modules during inference, preventing interference between unrelated concepts. By merging only the most consistent updates in weight space rather than naively combining all adapters, the framework preserves each concept's individual identity while enabling meaningful composition.

For practitioners developing personalized AI applications, this work has immediate implications. The approach reduces the data and computational requirements for customization, making personalized vision-language capabilities more accessible to resource-constrained organizations. The maintained disentanglement of concepts suggests the framework could scale to dozens or hundreds of user-defined concepts without exponential complexity growth. However, real-world deployment would require validating performance across diverse concept combinations and edge cases where concepts semantically overlap or conflict.

Future development should examine how this compositional framework generalizes beyond vision-language tasks to multimodal and text-only domains, and whether the gating mechanism effectively handles adversarial or deliberately conflicting concept combinations.

• →Gate-and-Merge enables zero-shot compositional personalization of vision-language models without requiring training data on concept co-occurrences.
• →Individual concepts are learned as separate LoRA adapters with a gating mechanism that selectively activates relevant modules during inference.
• →The framework preserves concept identity and prevents interference by merging only mutually consistent updates in weight space.
• →This approach substantially reduces computational and data requirements compared to traditional fine-tuning methods.
• →The technique has broad applications for building scalable, customizable AI systems with multiple user-defined personalization capabilities.