7 articles tagged with #llm-auditing. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers introduce LoRA-MINT, a methodology for detecting whether specific data samples were used to train fine-tuned large language models, achieving 77-92% precision. This auditing tool addresses growing concerns about intellectual property protection and sensitive data exposure in adapted AI models, with implications for responsible AI deployment.
Researchers introduce LLM-FACETS, an open-source framework designed to make LLM auditing accessible to non-technical practitioners while preserving data privacy. The system addresses regulatory compliance needs outlined in the EU AI Act and NIST frameworks by providing browser-based evaluation tools that keep sensitive data on self-hosted servers rather than transmitting it to external services.
Researchers introduce KBF, a black-box auditing protocol that detects fraudulent LLM API substitutions by analyzing model behavior at knowledge boundaries. Testing across 16 production endpoints revealed all economically relevant model swaps without false positives, and identified inconsistencies in 7 of 27 model cells across major AI platforms, particularly affecting Claude premium endpoints.
Researchers introduce Chaintrix, an LLM-augmented smart-contract auditing framework that combines AI analysis with deterministic structural verification to reduce false positives. The system achieves 71.7% recall on high-severity vulnerabilities, outperforming existing AI and static analysis tools by 26 percentage points on OpenAI's EVMbench benchmark.
Researchers introduce IMMACULATE, a framework that audits commercial large language model API services to detect fraud like model substitution and token overbilling without requiring access to internal systems. The system uses verifiable computation to audit a small fraction of requests, achieving strong detection guarantees with less than 1% throughput overhead.
Researchers introduce LLMSurgeon, a framework that reverse-engineers the pretraining data composition of Large Language Models by analyzing their generated text, addressing the opacity surrounding how foundation models are trained. The method estimates domain-level distributions across a predefined taxonomy without requiring access to actual training datasets, offering a practical auditing tool for understanding model behavior and capabilities.
Researchers benchmarked 43 large language models used for academic scholar recommendations, revealing that prompt design significantly affects recommendation quality and diversity. The study found that model choice, persona prompting (language, location, role), and context variables independently shape which scholars are recommended, with geographic location prompts producing the most variation in factuality and representativeness across disciplines.
