Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs

Researchers propose PIPO (Pair-In, Pair-Out), a novel technique that combines input compression and multi-token prediction to accelerate large language model inference. The method eliminates expensive verification steps while achieving up to 2.64x speedups in first-token latency and demonstrating significant improvements on reasoning benchmarks.

PIPO addresses a fundamental bottleneck in modern LLM deployment: the computational cost of autoregressive decoding during inference. While previous approaches treated input compression and output prediction as separate problems, this work elegantly unifies them by treating a latent compressor and multi-token prediction head as mirror operations. The technical innovation lies in training a lightweight confidence head that determines token acceptance, integrating naturally with On-Policy Distillation to avoid the expensive verifier passes that plague existing multi-token prediction methods.

The approach emerges from the broader context of inference optimization, where reducing per-token latency directly impacts user experience and operational costs. As LLMs become central infrastructure for production systems, inference efficiency has become as critical as model quality. PIPO's symmetric design represents a conceptual shift in how practitioners think about the encoding-decoding pipeline, potentially influencing future architecture design.

The experimental results across multiple benchmarks—AIME 2025, GPQA-Diamond, and LiveCodeBench—demonstrate practical value beyond theoretical elegance. Achieving 2.07x per-token latency improvements with smaller backbones (4B-9B parameters) has immediate implications for edge deployment and cost-sensitive applications. The method's compatibility with On-Policy Distillation means practitioners can integrate it into existing training pipelines without substantial overhead.

Looking ahead, the key question is adoption velocity. If PIPO's speedups prove reproducible across diverse model architectures and use cases, it could become standard practice in production deployments. The research also opens questions about whether similar mirror-image principles apply to other encoder-decoder architectures, potentially spawning follow-up work in efficient multimodal models.

• →PIPO unifies input compression and output prediction through symmetric latent operations, eliminating expensive verifier passes in multi-token prediction.
• →Achieves up to 2.64x first-token-latency and 2.07x per-token-latency speedups while improving reasoning benchmark performance by up to 7.15 points.
• →Lightweight confidence head integrates naturally with On-Policy Distillation, enabling efficient training without significant computational overhead.
• →Method demonstrates practical value on smaller model backbones (4B-9B), making it relevant for resource-constrained production deployments.
• →Represents a conceptual shift in encoder-decoder design that may influence future LLM architecture approaches across the industry.