Researchers propose a formal threat model framework for evaluating distillation defenses against black-box LLM attacks, arguing that existing output perturbation defenses lack clear specifications about attacker capabilities. The work demonstrates that defense effectiveness depends heavily on assumed threat parameters, raising concerns about false security claims in deployed systems.
The paper addresses a critical gap in AI security research: the absence of standardized threat models for evaluating defenses against model distillation attacks. When attackers can query proprietary language models via APIs and train student models on the outputs, companies deploy output perturbation defenses to protect intellectual property. However, without explicit threat model specifications, these defenses resist evaluation and comparison, potentially creating dangerous assumptions about their robustness.
The research emerges from growing tensions between AI safety and security. As LLMs become valuable proprietary assets, protecting them from extraction becomes essential. Yet the distillation attack surface remains poorly mapped—attackers vary in query budgets, available training data, and API interaction patterns. The authors propose measuring defenses across three dimensions: query budget, data budget, and interface profile. Their case study with antidistillation sampling reveals that the same defense can appear effective under one threat model but fail catastrophically under another.
This directly impacts AI companies and regulators. Organizations implementing these defenses to justify compliance or protect IP may unknowingly deploy weak protections. Policy frameworks built on unclear threat models risk legitimizing inadequate safeguards, creating liability concerns. The absence of standardized evaluation also complicates technical comparison—teams cannot reliably assess whether combining defenses increases security or introduces conflicts.
Future distillation defense development must prioritize explicit threat model specification and stress-testing. Governance bodies considering these defenses for regulatory purposes should demand clear capability assumptions. The research highlights how technical underspecification in emerging security areas can propagate into policy, potentially creating false compliance without genuine protection.
- →Existing distillation defenses lack standardized threat models, making their effectiveness difficult to evaluate and compare
- →Defense robustness depends critically on three attacker dimensions: query budget, data budget, and API interface profile
- →False sense of security arises when companies deploy defenses without explicit threat model specifications
- →Policy frameworks and regulatory compliance built on vague defense assumptions risk legitimizing inadequate protections
- →Future AI security work must explicitly specify and stress-test threat models before deployment or governance adoption