ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks

Researchers introduce ThinkSwitch, a method that distills reasoning capabilities from large language models into smaller, more efficient models using LoRA and weight interpolation. The technique improves performance on mathematical and scientific reasoning tasks while maintaining low computational costs, doubling accuracy on AIME problems at minimal expense.

ThinkSwitch addresses a fundamental tension in modern AI deployment: reasoning-capable models require substantial inference-time computation, creating latency and cost penalties that limit practical adoption. The paper demonstrates that explicit reasoning traces—typically generated during inference through chain-of-thought prompting—can be partially distilled into model weights through iterative training loops, preserving reasoning benefits while reducing runtime overhead.

The approach builds on established techniques including QLoRA for efficient fine-tuning and spherical weight interpolation, but applies them in a novel training loop where a thinking model generates solutions, answers are extracted and distilled into an instruction-following model, then thinking weights are reconstructed. This self-supervised cycle eliminates reliance on human labels, reducing annotation burden. The experimental results are striking for the resource investment: on AIME 2026 problems, the instruct model improves from 33% to 67% accuracy, while the thinking model reaches 73%, all for $2.86 on consumer hardware.

For the broader AI ecosystem, this suggests that inference-time compute benefits aren't locked into larger models—carefully structured distillation can transfer reasoning capabilities into parameter-efficient fine-tuning. This has immediate implications for edge deployment, mobile inference, and cost-sensitive applications. The approach validates a middle path between static weights and expensive test-time scaling, potentially enabling reasoning-capable models to scale without proportional cost increases. Future work should validate results on larger problem sets and diverse domains, but the framework suggests distillation-based reasoning enhancement could become standard practice for specialized models.

• →ThinkSwitch doubles AIME reasoning accuracy from 33% to 67% using low-cost distillation, demonstrating reasoning benefits can transfer into model weights
• →The method combines QLoRA fine-tuning with weight interpolation to co-train instruct and thinking models without human labels
• →Complete experiment costs only $2.86 on single RTX 3070, suggesting scalable path to reasoning-capable deployment on consumer hardware
• →Approach validates partial transfer of inference-time reasoning into weights, potentially reducing deployment latency and token costs
• →Results remain small-scale but indicate distillation loops can preserve explicit reasoning modes while improving efficiency