Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering

Researchers benchmarked 22 embedding models on patent data, finding that optimal fine-tuning strategies vary by task and that single-landscape fine-tuning degrades cross-domain performance. The study reveals significant gaps between in-domain and out-of-domain retrieval that cannot be closed with hybrid approaches, challenging assumptions about universal embedding solutions.

This research addresses a critical gap in machine learning practice: whether a single fine-tuning approach can serve multiple downstream applications and different data domains. The study's scope—evaluating 22 models ranging from 22M to 12B parameters across retrieval, classification, and clustering tasks—provides practitioners with concrete evidence that optimization strategies must be task-specific. Cross-sectional alignment excels at retrieval (+7.1% nDCG@10), while combined signal approaches better serve classification and clustering, suggesting that embedding quality depends on alignment with specific objectives rather than general-purpose improvements.

The cross-landscape finding carries deeper implications. Fine-tuning on one patent domain actually harms zero-shot models' performance on other domains, indicating that over-specialization reduces generalization capacity. This contradicts common industry practice where organizations often fine-tune on available data without considering domain transfer effects. The consistent within-family model scaling (Qwen, Llama-Nemotron) contrasted against erratic cross-family performance suggests that architecture families have learned distinct representations that don't transfer uniformly.

For organizations developing patent search systems, trademark databases, or technical document retrieval platforms, these findings demand methodological reconsideration. The persistent 55-65% performance gap between in-domain and out-of-domain retrieval—unresolved even with hybrid BM25-dense fusion—indicates fundamental limitations in current embedding approaches. The finding that Title+Abstract+Claims consistently outperforms other text representations provides immediate actionable guidance for data preparation. The public availability of code and evaluation framework enables broader validation and refinement of these conclusions across different patent and technical document contexts.

• →Optimal fine-tuning recipes vary significantly by downstream task, requiring task-specific optimization rather than universal approaches
• →Single-landscape fine-tuning degrades cross-domain retrieval performance for stronger zero-shot models, reducing their generalization capacity
• →A substantial 55-65% performance gap persists between in-domain and out-of-domain patent retrieval that hybrid fusion methods cannot close
• →Within-family model scaling is consistent while cross-family scaling shows erratic performance, suggesting architecture-dependent knowledge representations
• →Title+Abstract+Claims text representation universally outperforms alternative document views for patent embeddings