3 articles tagged with #theoretical-bounds. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers extend the bounded attention prefix oracle (BAPO) model to establish theoretical lower bounds on chain-of-thought reasoning tokens required by LLMs, proving that canonical tasks require Ω(n) tokens as input size n grows. Experiments with frontier models confirm linear scaling behavior, revealing fundamental computational bottlenecks in inference-time scaling.
Researchers introduce the NOVA framework, which models AI knowledge discovery as an adaptive sampling process and identifies fundamental scaling limitations. The analysis reveals a contamination trap where false positives accumulate faster than genuine discoveries as knowledge becomes scarce, with cumulative generation costs following a Zipf-distributed scaling law demonstrating asymptotic diminishing returns.
Researchers establish information-theoretic lower bounds for bit-constrained stochastic optimization, proving that B-bit quantized gradients require communication overhead of TB = Omega(d) and statistical complexity of T = Omega(sigma^2 d / eps^2 * max{1, d/B}). The work provides the first rigorous characterization of what's theoretically possible in low-precision pretraining, contrasting with existing empirical studies of FP8 and MXFP4 systems.
