PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis

A research paper demonstrates that parameter-efficient fine-tuning of small language models (3B parameters) using LoRA achieves competitive performance for telecommunications customer support while consuming significantly less energy than larger models. Critically, the study reveals that traditional validation loss metrics poorly predict real-world conversational quality, with the lowest-loss model ranking 6th-7th in human-aligned evaluation while the worst-loss model ranked first.

This research addresses a fundamental disconnect in AI model evaluation that has significant implications for enterprise deployment. The study systematically tests 16 LoRA configurations on a 3B parameter model fine-tuned for telecom customer support, using 30,000 synthetically generated training examples grounded in domain-specific terminology. The core finding—that validation loss inversely correlates with human preference in conversational tasks—challenges conventional machine learning wisdom and reflects a broader reality in large language model deployment.

The work emerges from practical constraints facing telecommunications companies: data sovereignty concerns, regulatory compliance requirements, and the need to handle sensitive customer information without relying on externally hosted models. Small language models offer a viable alternative to billion-parameter foundation models, particularly when optimized through efficient fine-tuning techniques. This trend reflects industry recognition that model size alone does not guarantee utility.

For enterprise AI practitioners and infrastructure providers, this research validates the viability of smaller, locally-deployed models while introducing energy consumption as a critical evaluation metric alongside traditional performance indicators. Organizations deploying conversational AI systems now have evidence that they should prioritize human-aligned evaluation frameworks using LLM-as-judge approaches rather than relying exclusively on perplexity or validation loss metrics. The synthetic data generation methodology combining glossaries with generative pipelines offers reproducible approaches for domain adaptation.

Looking forward, similar validation-performance divergences will likely emerge in other specialized domains. This creates opportunities for companies developing lightweight, energy-efficient fine-tuning infrastructure and evaluation frameworks that better capture conversational quality than traditional metrics.

• →Validation loss is an insufficient metric for selecting fine-tuning configurations in conversational AI systems.
• →Small 3B parameter language models with LoRA fine-tuning achieve competitive performance with significantly lower energy consumption than larger alternatives.
• →LLM-as-judge evaluation frameworks using multiple judges reveal preferences that contradict traditional loss-based model selection.
• →Synthetic data generation from domain glossaries can produce 30,000 diverse training examples across 1,560 problem scenarios for specialized applications.
• →Target module selection in LoRA injection substantially impacts both model performance and energy efficiency trade-offs.