Lagrange: An Open-Vocabulary, Energy-Based Sparse Framework for Generalized End-to-End Driving

Researchers introduce Lagrange, an open-vocabulary autonomous driving framework that combines Vision-Language Models with sparse, energy-based planning to address limitations in existing end-to-end driving systems. The approach balances computational efficiency with generalization capacity for handling out-of-distribution scenarios while maintaining kinematic feasibility.

Lagrange represents a significant advancement in autonomous vehicle perception and planning by bridging a critical technical gap in the field. Traditional end-to-end driving systems face a fundamental tradeoff: dense models like occupancy networks provide geometric robustness but consume substantial computational resources and struggle with semantic reasoning, while sparse, query-based planners are efficient but limited to closed-set object definitions that fail when encountering anomalous scenarios. Recent Vision-Language-Action models offer semantic flexibility through open-vocabulary reasoning, yet their autoregressive token generation conflicts with the continuous, high-frequency control demands of vehicle dynamics.

The Lagrange framework resolves this tension through Masked Latent Fields, which encode class-agnostic object proposals as continuous semantic tokens rather than discrete outputs. By leveraging Vision-Language Models to generate these tokens and applying intent-driven masked cross-attention for temporal filtering, the system creates an implicit energy field over spatial coordinates. The key innovation frames autonomous driving decision-making as a Lagrangian optimization problem, ensuring strict compliance with vehicle kinematics while executing collision avoidance—a mathematically elegant approach that guarantees safety constraints.

For the autonomous driving industry, this work validates that open-world generalization and computational efficiency are not mutually exclusive. Performance validation on both nuScenes and CODA benchmarks demonstrates practical viability. The framework's interpretability and kinematic guarantees address persistent safety concerns that regulators and insurance providers require before widespread deployment. This approach may influence how future autonomous systems are architected, potentially shifting from dense volumetric methods toward energy-based sparse representations.

• →Lagrange combines Vision-Language Models with energy-based optimization to balance computational efficiency and open-world generalization in autonomous driving.
• →The framework uses Masked Latent Fields to encode continuous semantic tokens, avoiding discrete token generation limitations of existing VLA models.
• →Intent-driven masked cross-attention enables temporal filtering of irrelevant entities, improving planning robustness.
• →Lagrangian action minimization enforces strict kinematic compliance and collision avoidance simultaneously.
• →Validation on both standard (nuScenes) and long-tail (CODA) datasets demonstrates stronger performance on out-of-distribution scenarios.