Rooted Absorbed Prefix Trajectory Balance with Submodular Replay for GFlowNet Training

Researchers propose RapTB, a new training objective for Generative Flow Networks (GFlowNets) that addresses mode collapse issues in fine-tuning large language models. The method includes a submodular replay strategy (SubM) and demonstrates improved performance in molecule generation tasks while maintaining diversity and validity.

• →GFlowNets suffer from mode collapse issues including prefix collapse and length bias when fine-tuning large language models
• →RapTB provides dense prefix-level learning signals by anchoring supervision at the root and propagating rewards to intermediate prefixes
• →SubM replay strategy promotes both high reward and diversity to mitigate distribution shift
• →The combined approach shows consistent improvements in molecule generation using SMILES strings
• →The method preserves molecular diversity and validity while enhancing optimization performance