K-Gen: A Multimodal Language-Conditioned Approach for Interpretable Keypoint-Guided Trajectory Generation

Researchers propose K-Gen, a new multimodal AI framework that uses Large Language Models to generate realistic driving trajectories for autonomous vehicle simulation. The system combines visual map data with text descriptions to create interpretable keypoints that guide trajectory generation, outperforming existing baselines on major datasets.

• →K-Gen framework combines Multimodal Large Language Models with keypoint-guided trajectory generation for autonomous driving simulation.
• →The system uses both rasterized BEV map inputs and textual scene descriptions instead of relying solely on structured vectorized maps.
• →K-Gen generates interpretable keypoints with reasoning that reflects agent intentions before creating full trajectories.
• →The framework employs T-DAPO, a trajectory-aware reinforcement fine-tuning algorithm to enhance keypoint generation.
• →Experiments on WOMD and nuPlan datasets show K-Gen outperforms existing trajectory generation baselines.