#masked-language-modeling News & Analysis

2 articles tagged with #masked-language-modeling. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 96/10

🧠

MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models

Researchers introduce MC-PDD, a black-box method to detect whether specific datasets were used to pretrain large language models by analyzing prediction patterns on masked text. The technique works through standard API access without requiring model probability distributions, enabling practical auditing of closed-source LLMs and addressing transparency concerns around proprietary training data.

AIBullisharXiv – CS AI · May 286/10

🧠

Entropy-aware Masking for Masked Language Modeling

Researchers propose entropy-aware masking for masked language modeling, which selectively masks tokens based on prediction uncertainty rather than random selection. The approach achieves 5% improvement in GLUE scores and performs best when combined with knowledge distillation, offering a more efficient pretraining strategy for encoder-based language models.