Apple Research Shows LLMs Can Level Up via Self-Distillation.
Apple Research Unlocks "SSD": Boosting LLM Performance Through Simple Self-Distillation A research team at Apple has unveiled a breakthrough in Large Language Model (LLM) training known as Simple Self-Distillation (SSD) . This technique allows a model to improve its own performance by training on its own generated outputs, effectively removing the need for high-quality data from larger "teacher" models or complex, supervised feedback loops. The SSD Methodology The researchers tested this concept using Qwen3-4B and Qwen3-30B models. The process involved: Generation: The models attempted 10,000 problems from the rSTARcoder dataset. Filtering: A basic "common sense" filter was applied to remove obviously flawed outputs (e.g., extremely short or empty responses). Refinement: The remaining outputs were fed back into the model for self-training. The results, measured against the LiveCodeBench v6 benchmark, showed significant gains. Notably, Qwen3-30B-Ins...
