GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification

GATech researchers compared bidirectional encoders versus causal decoders for Arabic medical text classification across 82 categories, finding that specialized bidirectional encoders like AraBERTv2 significantly outperform large language models. The study demonstrates that causal decoders optimized for next-token prediction produce sequence-biased embeddings less effective for precise categorization tasks.

• →Fine-tuned AraBERTv2 with hybrid pooling strategies outperformed large-scale causal decoders including Llama 3.3 70B for Arabic medical classification.
• →Bidirectional encoders capture global context better than causal decoders for fine-grained categorization tasks.
• →Causal decoders produce sequence-biased embeddings that are less effective for classification compared to bidirectional attention mechanisms.
• →The study involved classification across 82 distinct Arabic medical categories with significant class imbalance and label noise.
• →Specialized encoders demonstrate superior semantic compression for domain-specific Arabic NLP tasks.