Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models

Researchers propose Talaria, a new confidential inference framework that protects client data privacy when using cloud-hosted Large Language Models. The system partitions LLM operations between client-controlled environments and cloud GPUs, reducing token reconstruction attacks from 97.5% to 1.34% accuracy while maintaining model performance.

• →Talaria framework addresses privacy concerns in cloud-based LLM inference while preserving model performance and computational efficiency.
• →The system uses Reversible Masked Outsourcing (ReMO) protocol to secure interactions between client environments and cloud GPUs.
• →Token reconstruction attack accuracy drops from over 97.5% to average 1.34% with Talaria's protection.
• →The framework is lossless, guaranteeing identical output to original models without significant efficiency loss.
• →This represents the first solution to simultaneously protect client prompts, responses, and model intellectual property.