Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining

Researchers investigate whether using a single HTML-to-text extractor for web-scale LLM pretraining datasets leads to suboptimal data utilization. The study reveals that different extractors can result in substantially different pages surviving filtering pipelines, despite similar model performance on standard language tasks.

• →Current open-source datasets predominantly use a single fixed extractor for all webpages despite diverse web content
• →Different extractors may achieve similar model performance on standard language understanding tasks
• →Pages that survive fixed filtering pipelines can differ substantially between extractors
• →This suggests potential suboptimal coverage and utilization of Internet data for LLM training
• →The research challenges current preprocessing practices for web-scale LLM datasets