Plain Transformers are Surprisingly Powerful Link Predictors
Researchers introduce PENCIL, a plain Transformer model that outperforms Graph Neural Networks at link prediction by using attention over sampled local subgraphs instead of complex structural encodings. The approach demonstrates that simpler architectural choices can achieve superior performance while maintaining scalability and parameter efficiency, challenging the industry's reliance on elaborate engineering techniques.