CitePrism: Human-in-the-Loop AI for Citation Auditing and Editorial Integrity

CitePrism introduces a human-in-the-loop AI framework designed to assist editors and reviewers in auditing manuscript citations for relevance, accuracy, and ethical appropriateness. The system combines large language models, semantic similarity analysis, and metadata verification to flag potentially problematic citations, achieving moderate agreement with human reviewers in preliminary testing on a pavement engineering manuscript.

CitePrism addresses a fundamental challenge in academic publishing: the labor-intensive and inconsistent manual review of citations across thousands of submitted manuscripts. As research output accelerates and citation networks grow increasingly complex, editors and peer reviewers struggle to verify whether cited works genuinely support the claims they purportedly back. This creates vulnerabilities to citation manipulation, obsolete references, and poor bibliographic practice that can undermine research integrity at scale.

The tool represents a practical application of LLM technology to editorial workflows, combining multiple verification layers—contextual reasoning, embedding-based similarity matching, metadata validation, and self-citation pattern detection. Rather than attempting fully autonomous decision-making, CitePrism deliberately positions itself as a triage and screening layer that surfaces problematic citations for human expert judgment. This design choice reflects emerging best practices in AI deployment for high-stakes domains where transparency and human oversight remain essential.

The preliminary validation results reveal both promise and limitations. Perfect recall for irrelevant citations demonstrates potential value in conservative screening, yet the presence of false positives confirms that algorithmic scoring alone cannot replace editorial expertise. The modest Cohen's kappa of 0.429 indicates substantial disagreement with human labelers, likely reflecting the nuanced, context-dependent nature of citation appropriateness. The authors appropriately acknowledge these constraints and explicitly reject claims of broader applicability beyond their single case study.

For the academic publishing ecosystem, CitePrism signals growing institutional investment in AI-assisted quality assurance. Widespread adoption would require validation across diverse disciplines, training on larger annotation datasets, and integration with editorial management systems. The framework's emphasis on interpretability and configurable thresholds positions it favorably for institutional deployment where explainability matters to stakeholders.

• →CitePrism combines LLM reasoning, semantic similarity, and metadata verification to assist editors in detecting problematic manuscript citations.
• →Preliminary testing on 104 references achieved perfect recall for irrelevant citations but generated false positives requiring human analyst review.
• →The system explicitly positions itself as decision-support rather than autonomous misconduct detection, reflecting responsible AI design for editorial workflows.
• →Moderate Cohen's kappa agreement (0.429) with human reviewers indicates citation appropriateness remains a context-dependent judgment requiring human expertise.
• →Broader validation across multiple domains, disciplines, and annotators remains required before operational deployment in academic publishing.