LEDOM: Reverse Language Model

Researchers have developed LEDOM, an open-source reverse autoregressive language model that trains right-to-left instead of the traditional left-to-right approach. The model demonstrates unique capabilities like abductive inference and question synthesis, and when combined with forward models through 'Reverse Reward' scoring, achieves significant performance gains of up to 15% on mathematical reasoning tasks.

• →LEDOM is the first large-scale reverse autoregressive language model trained right-to-left with 2B/7B parameters on 435B tokens.
• →Reverse training enables unique reasoning capabilities including abductive inference, question synthesis, and resolution of the reversal curse.
• →The Reverse Reward technique combines forward and reverse models to penalize hallucinated reasoning chains.
• →Performance improvements of up to 6.6% on AIME 2024 and 15% on AMC 2023 mathematical benchmarks were achieved.
• →All models, code, and training data have been released as open-source resources.