PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference

PoQ-Judge introduces a reference-free quality evaluation framework for decentralized LLM inference networks using lightweight judge models trained on UltraFeedback and GPT-labeled data. The framework achieves 0.747 Pearson correlation with ground-truth benchmarks while reducing evaluation costs by 72.7% through cascade evaluation, addressing a critical infrastructure need for decentralized AI systems.

PoQ-Judge tackles a fundamental infrastructure challenge in decentralized LLM inference: efficiently validating output quality without centralized reference data or ground-truth answers. This matters because Proof-of-Quality mechanisms are essential for trustless networks where participants must verify work without relying on a central authority. The framework's three-architecture approach—TextCNN, MiniLM cross-encoder, and DeBERTa—acknowledges that decentralized systems operate under varied computational constraints, requiring both high-accuracy and lightweight options.

The research builds on growing recognition that decentralized inference networks need scalable quality assurance. Traditional reference-based evaluation requires maintaining authoritative answer sets, creating bottlenecks and centralization risks. PoQ-Judge's reference-free design aligns with decentralized principles while achieving performance parity with reference-based methods through sophisticated training strategies.

The cascade evaluation finding—reducing costs by 72.7% with modest quality trade-offs—has direct implications for network economics. Lower validation costs translate to better margins for inference providers and lower overhead for network operators. However, the framework's stronger performance on QA versus summarization suggests limitations that could affect heterogeneous workloads.

For the emerging decentralized AI infrastructure sector, this represents progress toward economically viable Proof-of-Quality systems. As projects like Akash, Gensyn, and others build decentralized inference networks, efficient quality validation becomes competitive differentiation. The research points toward production-ready evaluation mechanisms, though broader architectural questions about PoQ integration remain open.

• →Reference-free judge models achieve 0.747 Pearson correlation with ground-truth on held-out test sets, matching or exceeding reference-based evaluators.
• →Cascade evaluation reduces computational costs by 72.7% while maintaining acceptable quality benchmarks, improving network economics.
• →Multi-architecture approach (TextCNN, MiniLM, DeBERTa) enables quality-cost tradeoffs suitable for heterogeneous decentralized infrastructure.
• →Framework shows significantly stronger performance on QA tasks than summarization, indicating domain-specific limitations.
• →Two-stage training on UltraFeedback plus GPT-labeled in-domain data enables effective reference-free quality assessment without centralized ground truth.