Knowledge Graph and Hypergraph Transformers with Repository-Attention and Journey-Based Role Transport

Researchers present a new transformer architecture that jointly trains on natural language and structured data by maintaining separate knowledge and language representations. The model uses a key-value repository system with journey-based role transport to enable cross-attention between linguistic context and structured knowledge graphs.

• →New dual-stream architecture enables joint training on sentences and structured data while keeping representations separable.
• →Journey-based role transport unifies knowledge graph traversal, hyperedge traversal, and sentence structure processing.
• →Model includes hierarchical attention layers spanning instance-local, neighborhood, and global mixing patterns.
• →Multi-task training objectives include masked language modeling, link prediction, and role-consistency denoising.
• →Architecture provides explicit separation between linguistic and structured knowledge with inspectable cross-attention alignment.