With Nvidia Groq 3, the Era of AI Inference Is (Probably) Here

Nvidia announced the Groq 3 LPU at GTC 2024, its first chip specifically designed for AI inference rather than training, incorporating technology licensed from startup Groq for $20 billion. The chip uses SRAM memory integrated within the processor to achieve 7x faster memory bandwidth than traditional GPUs, optimizing for the low latency required for real-time AI inference applications.

• →Nvidia released its first inference-specific chip, the Groq 3 LPU, using technology licensed from Groq for $20 billion.
• →The chip uses integrated SRAM memory instead of external HBM to achieve 150 TB/s memory bandwidth, 7x faster than the Rubin GPU.
• →AI workloads are shifting from training to inference as models are deployed at scale, creating demand for specialized low-latency chips.
• →Multiple startups are competing in the inference chip space with different architectural approaches including neuromorphic and analog computing.
• →The move signals Nvidia's recognition that inference and training require fundamentally different chip architectures and optimizations.