Learning Context-Conditioned Predicate Semantics via Prototype Feedback

Researchers introduce AlignG, a machine learning approach that improves scene graph generation by enabling predicates to adapt their meanings based on image context rather than remaining static. The method uses prototype feedback to recalibrate predicate representations while preventing semantic drift, demonstrating measurable performance improvements on standard benchmarks.

AlignG addresses a fundamental limitation in computer vision systems that understand relationships between objects in images. Scene graphs represent visual relationships using predicates—descriptive words like 'sitting on' or 'wearing'—but these terms have different meanings depending on context. Traditional approaches lock predicates into fixed representations or retrieve similar examples without reorganizing semantics, causing systematic errors when predicates appear ambiguous. This research tackles polysemy, where single words carry multiple meanings that shift based on surrounding evidence.

The innovation centers on dynamic semantic adaptation anchored to global prototypes. Rather than treating predicate meanings as static, AlignG infers context-specific interpretations from relationship candidates within individual images, then uses this adapted understanding to recalibrate how relationships are represented. The learning objective prevents unbounded semantic drift—a key challenge in adaptive systems—while permitting selective reorganization when scene evidence supports it. This balancing act enables the model to specialize when appropriate without accumulating errors over time.

Performance gains on VG-150 and GQA-200 datasets reach +1.4 and +2.7 F@100 points respectively under SGDet evaluation protocol, placing AlignG ahead of existing methods. Visualizations reveal that prototypes coherently merge or separate predicates according to scene content, demonstrating the system learns meaningful reorganizations rather than random adjustments. This advancement matters for applications requiring accurate scene understanding—from visual question answering to image retrieval systems—where relationship confusion degrades reliability.

• →AlignG enables predicates to dynamically adapt their semantic meaning based on image-specific context rather than using fixed representations
• →The method achieves consistent improvements over state-of-the-art baselines with F@100 gains of +1.4 on VG-150 and +2.7 on GQA-200 datasets
• →Prototype feedback mechanism prevents semantic drift while allowing selective reorganization when scene evidence supports predicate reinterpretation
• →Visualizations show prototypes coherently merge or separate relationships based on actual scene content, indicating learned adaptations are semantically meaningful
• →The approach addresses polysemy challenge where predicate meanings shift across contexts, improving accuracy in scene graph generation tasks