Toward General Semantic Chunking: A Discriminative Framework for Ultra-Long Documents

Researchers developed a new discriminative AI model based on Qwen3-0.6B that can efficiently segment ultra-long documents up to 13k tokens for better information retrieval. The model achieves superior performance compared to generative alternatives while delivering two orders of magnitude faster inference on the Wikipedia WIKI-727K dataset.

• →New discriminative segmentation model based on Qwen3-0.6B addresses limitations of existing methods for ultra-long document processing.
• →The model supports single-pass inputs of up to 13k tokens using cross-window context fusion and overlapping sliding-window strategy.
• →Achieves better macro-averaged F1 scores than three generative models while being 100x faster in inference.
• →Includes vector fusion method with scalar correction to compress ultra-long segments without semantic loss.
• →Demonstrates significant improvements in practicality and scalability for long-document processing applications.