Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey

A comprehensive survey examines how data efficiency, memory constraints, and compute budgets interact as coupled bottlenecks in LLM training. The research reveals that optimal training strategies are resource-dependent rather than universal, with GPU memory often being the primary limiting factor rather than raw computational power.

This survey addresses a fundamental challenge in modern AI development: resource constraints have become the defining factor in what can realistically be trained and deployed. Rather than treating efficiency improvements as isolated technical problems, the research adopts a systems-level perspective that recognizes how data selection, memory management, and compute allocation interact as interdependent constraints. This approach reflects the reality that optimization in one dimension often creates bottlenecks in another.

The finding that GPU memory, not raw compute, dominates fine-tuning bottlenecks challenges conventional scaling assumptions and suggests that infrastructure investments should prioritize memory efficiency over additional processing cores. The research demonstrates that effective training requires simultaneous optimization of weight storage, optimizer states, and activation memory—a systems-engineering problem rather than a purely algorithmic one.

For the AI industry, these insights have significant implications. Organizations attempting to fine-tune large models must reconsider their hardware configurations and training strategies in light of memory constraints. The evidence that optimal data subsets vary by task objective and resource budget undermines the notion of universal "best" training datasets, requiring more sophisticated selection mechanisms tailored to specific constraints.

The framework unifying compute-aware data selection with scaling laws and adaptive inference provides practitioners with a coherent decision-making model for resource-constrained environments. As LLM deployment becomes increasingly competitive and resource-intensive, understanding these coupled constraints becomes essential for competitive advantage. Future research should focus on developing automated methods for jointly optimizing across all three dimensions simultaneously.

• →GPU memory constraints, not raw compute power, typically limit LLM fine-tuning performance in resource-constrained scenarios.
• →Optimal training data subsets are resource-dependent and task-specific rather than universally applicable across different budgets.
• →Effective scaling requires simultaneous optimization of weight storage, optimizer states, and activation memory rather than isolated component improvements.
• →Compute-optimal allocation strategies should halt training when marginal performance gains fall below budget-dependent thresholds.
• →Unified resource-conditioned decision-making across data selection, scaling, and inference improves overall training efficiency.