{"id":609,"date":"2026-03-25T02:49:40","date_gmt":"2026-03-24T18:49:40","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=609"},"modified":"2026-03-25T02:49:40","modified_gmt":"2026-03-24T18:49:40","slug":"this-ai-paper-introduces-tinylora-a-13-parameter-fine-tuning-method-that-reaches-91-8-percent-gsm8k-on-qwen2-5-7b","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=609","title":{"rendered":"This AI Paper Introduces TinyLoRA, A 13-Parameter Fine-Tuning Method That Reaches 91.8 Percent GSM8K on Qwen2.5-7B"},"content":{"rendered":"

Researchers from FAIR at Meta<\/strong>, Cornell University<\/strong>, and Carnegie Mellon University<\/strong> have demonstrated that large language models (LLMs) can learn to reason using a remarkably small number of trained parameters. The research team introduces TinyLoRA<\/strong>, a parameterization that can scale down to a single trainable parameter under extreme sharing settings. Using this method on a Qwen2.5-7B-Instruct<\/strong> backbone, the research team achieved 91.8% accuracy<\/strong> on the GSM8K benchmark with only 13 parameters<\/strong>, totaling just 26 bytes in bf16.<\/p>\n

Overcoming the Constraints of Standard LoRA<\/strong><\/h3>\n

Standard Low-Rank Adaptation (LoRA) adapts a frozen linear layer W \u2208 Rd<\/em><\/sup><\/strong>x<\/em><\/sup>k<\/em><\/sup> <\/strong>using trainable matrices A \u2208 Rd<\/em><\/sup><\/strong>x<\/em><\/sup>r<\/sup><\/strong> and B \u2208 Rr<\/em><\/sup><\/strong>x<\/em><\/sup>k<\/em><\/sup><\/strong>. The trainable parameter count in standard LoRA still scales with layer width and rank, which leaves a nontrivial lower bound even at rank 1. For a model like Llama3-8B, this minimum update size is approximately 3 million parameters<\/strong>.<\/p>\n

TinyLoRA circumvents this by building upon LoRA-XS<\/strong>, which utilizes the truncated Singular Value Decomposition (SVD)<\/strong> of frozen weights. While LoRA-XS typically requires at least one parameter per adapted module, TinyLoRA replaces the trainable matrix with a low-dimensional trainable vector \ud835\udf10 \u2208 Ru<\/em><\/sup><\/strong> projected through a fixed random tensor P<\/em><\/strong> \u2208 Ru<\/sup><\/em><\/strong>x<\/sup><\/em>r<\/sup><\/em><\/strong>x<\/sup><\/em>r<\/sup><\/em><\/strong>.<\/p>\n

The update rule is defined as:<\/strong><\/p>\n

$$W\u2019 = W + USigma(sum_{i=1}^{u}v_{i}P_{i})V^{top}$$<\/div>\n

By applying a weight tying factor (n<\/em><\/strong>tie<\/strong><\/sub>), the total trainable parameters scale as O<\/strong><\/em>(nmu\/ntie<\/sub><\/em><\/strong>), allowing updates to scale down to a single parameter when all modules across all layers share the same vector.<\/p>\n

Reinforcement Learning: The Catalyst for Tiny Updates<\/strong><\/h3>\n

A core finding of the research is that Reinforcement Learning (RL)<\/strong> is fundamentally more efficient than Supervised Finetuning (SFT)<\/strong> at extremely low parameter counts. The research team reports that models trained via SFT require updates 100 to 1,000 times larger<\/strong> to reach the same performance as those trained with RL.<\/p>\n

This gap is attributed to the \u2018information density\u2019 of the training signal. SFT forces a model to absorb many bits of information\u2014including stylistic noise and irrelevant structures of human demonstrations\u2014because its objective treats all tokens as equally informative. In contrast, RL (specifically Group Relative Policy Optimization<\/strong> or GRPO<\/strong>) provides a sparser but cleaner signal. Because rewards are binary (e.g., exact match for a math answer), reward-relevant features correlate with the signal while irrelevant variations cancel out through resampling.<\/p>\n

Optimization Guidelines for Devs<\/strong><\/h3>\n

The research team isolated several strategies to maximize the efficiency of tiny updates:<\/strong><\/p>\n