Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method

Researchers present an adaptive two-phase semantic filtering method that improves LLM-based document classification efficiency by 1.6-2.0x compared to existing approaches. The method combines model-free clustering with online proxy training using soft labels and adaptive calibration, achieving 90% accuracy targets while reducing expensive LLM oracle calls.

This research addresses a fundamental computational bottleneck in LLM-based data processing: the prohibitive cost of running language models on every document in large corpora. Semantic filtering—determining whether documents satisfy natural language predicates—is essential for data processing pipelines, yet naive approaches waste computational resources. The paper identifies and solves four distinct limitations in cascade-based filtering systems that currently dominate the field.

The innovation lies in treating cascade methods as composable building blocks rather than rigid architectures. By deploying model-free clustering first and introducing online proxies only when necessary, the system achieves better resource allocation. The shift from binary training labels to soft confidence scores from the oracle LLM represents a significant methodological improvement, enabling the proxy to learn from the LLM's uncertainty rather than discarding valuable information at decision boundaries.

For the AI infrastructure industry, this work has immediate implications. Production systems processing large document corpora—such as content moderation platforms, legal discovery systems, and research data pipelines—currently waste substantial compute on redundant LLM calls. A 1.6-2.0x speedup translates directly to reduced operational costs and lower latency. The paper's theoretical lower bound analysis, suggesting 4-20x additional optimization headroom, indicates the field is still in early stages of efficiency optimization.

The research demonstrates that careful architectural design and training methodology can substantially improve inference efficiency without sacrificing accuracy. As LLM inference costs remain a significant operational expense, these optimization techniques will likely influence how production systems deploy language models at scale.

• →Adaptive two-phase filtering combining clustering and online proxies achieves 1.6-2.0x speedup over existing methods on 10K-document corpora
• →Training proxies with oracle confidence scores as soft labels improves learning from boundary cases where uncertainty matters most
• →Selective calibration that targets sparse regions reduces unnecessary safety margins and oracle call overhead
• →Theoretical analysis indicates 4-20x additional optimization potential, suggesting current cascade methods remain far from optimal efficiency
• →Method applicable to production systems requiring semantic filtering across large document collections with accuracy constraints