Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models

Researchers propose Talaria, a new confidential inference framework that protects client data privacy when using cloud-hosted Large Language Models. The system partitions LLM operations between client-controlled environments and cloud GPUs, reducing token reconstruction attacks from 97.5% to 1.34% accuracy while maintaining model performance.