PerceptTwin: Semantic Scene Reconstruction for Iterative LLM Planning and Verification

PerceptTwin is an automated pipeline that generates interactive 3D simulations from robot perception data, enabling LLM-based planners to validate and refine strategies before hardware execution. The system improves plan success rates by approximately 39% and enhances safety through semantic scene reconstruction and LLM verification mechanisms.

PerceptTwin addresses a longstanding bottleneck in robotics: the manual, labor-intensive process of creating simulation environments for testing and validation. By automating scene reconstruction from robot perception outputs, the system eliminates the need for bespoke simulation development for each deployment scenario. This approach leverages open-vocabulary object maps, 3D asset generation, and affordance prediction to create faithful digital twins that reflect real-world conditions.

The integration of LLM-based planning with simulation-based verification represents a significant shift in how autonomous systems approach safety and reliability. Traditional robot planning relies on hand-crafted rules or limited learning from constrained datasets. PerceptTwin's framework allows language models to generate candidate plans, then validates them in simulation before execution. The incorporation of an LLM judge—inspired by AI alignment research—adds an additional verification layer that checks plan correctness and human preference alignment.

The reported 39% improvement in plan success across multiple model variants (GPT5, GPT5Mini, GPT5Nano) suggests the verification loop is genuinely valuable, not merely incremental. The 18% improvement in human plan verification for failure cases indicates the system surfaces meaningful insights about precondition violations that might otherwise go undetected. This has direct implications for reducing costly hardware failures and improving deployment reliability.

Looking forward, the approach establishes a replicable pattern: perception → semantic representation → simulation → planning verification. As robotic perception becomes more sophisticated and 3D asset libraries expand, this pipeline could become foundational infrastructure for autonomous systems across manufacturing, logistics, and service robotics. The open-vocabulary aspect is particularly significant, as it reduces dependency on pre-defined object taxonomies.

• →Automated simulation generation from robot perception eliminates manual environment setup, reducing deployment time and cost
• →LLM-based planning combined with simulation verification achieves 39% improvement in plan success rates across tested models
• →Integration of LLM judges for safety verification demonstrates practical application of AI alignment techniques in robotics
• →System improves human plan verification by 18% on average for detecting unmet skill preconditions
• →Open-vocabulary scene reconstruction enables generalization across diverse real-world environments without retraining