{"id":729,"date":"2026-04-16T16:30:30","date_gmt":"2026-04-16T08:30:30","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=729"},"modified":"2026-04-16T16:30:30","modified_gmt":"2026-04-16T08:30:30","slug":"ucsd-and-together-ai-research-introduces-parcae-a-stable-architecture-for-looped-language-models-that-achieves-the-quality-of-a-transformer-twice-the-size","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=729","title":{"rendered":"UCSD and Together AI Research Introduces Parcae: A Stable Architecture for\u00a0Looped Language Models That Achieves the Quality of a Transformer\u00a0Twice the Size"},"content":{"rendered":"

The dominant recipe for building better language models has not changed much since the Chinchilla era: spend more FLOPs, add more parameters, train on more tokens. But as inference deployments consume an ever-growing share of compute and model deployments push toward the edge, researchers are increasingly asking a harder question \u2014 can you scale quality<\/em> without scaling memory footprint<\/em>?<\/p>\n

A team of researchers from UC San Diego and Together AI have introduced Parcae<\/strong>, a stable looped transformer architecture that outperforms prior looped models and beats fixed-depth Transformer baselines at every scale tested \u2014 all while using the same parameter count and the same training data budget<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2604.12946<\/figcaption><\/figure>\n<\/div>\n

What is a Looped Language Model?<\/strong><\/h3>\n

In a standard Transformer, activations flow through a fixed stack of layers exactly once. A looped architecture<\/strong> instead routes activations through a block of layers T<\/em> times in a loop, multiplying effective compute without adding parameters. Think of it as running the same group of transformer blocks repeatedly rather than building a taller model.<\/p>\n

Parcae specifically uses a middle-looped<\/strong> design, partitioning the architecture into three functional blocks: a prelude (P)<\/strong> that embeds the input sequence into a latent state e<\/em>; a recurrent block (R)<\/strong> that iteratively updates a hidden state h<\/em>t<\/em> <\/sub>for T<\/em> loops, with e<\/em> injected at each iteration to maintain the input\u2019s influence; and a coda (C)<\/strong> that processes the final h<\/em>T<\/em> <\/sub>to produce the output. This structure keeps the model compact in memory, a valuable property for on-device deployment, while enabling significantly more compute per forward pass.<\/p>\n

Past works on looped transformers, including Recurrent Depth Models (RDMs), showed early promise but were quite difficult to train. They suffered from residual state explosion<\/strong> \u2014 where the hidden state vector grows uncontrollably across loop iterations \u2014 and frequent loss spikes<\/strong>. Sensitive hyperparameter tuning was required just to achieve convergence.<\/p>\n

The Root Cause: An Unconstrained Residual System<\/strong><\/h3>\n

The research team behind Parcae\u2019s key insight is to recast the looped model\u2019s forward pass as a nonlinear time-variant dynamical system<\/strong> over the residual stream:<\/p>\n

ht+1<\/sub> = \u0100 ht<\/sub> + B\u0304 e + R\u0304(ht<\/sub>, e),<\/code><\/pre>\n
Here, \u0100<\/em> controls the balance between prior and current residual states, B\u0304<\/em> injects the input signal, and R\u0304<\/em> is the nonlinear contribution of the transformer blocks (attention and MLPs). Dropping R\u0304<\/em> yields a discrete linear time-invariant (LTI) system<\/strong>, and classical control theory immediately gives you the stability condition: the system is stable when the spectral norm \u03c1(\u0100) < 1<\/strong>, marginally stable when \u03c1(\u0100) = 1, and unstable when \u03c1(\u0100) > 1.<\/p>\n
Examining prior methods under this framework reveals the problem precisely. Addition-based input injection sets \u0100 = I<\/em> (the identity matrix), meaning \u03c1(\u0100) = 1 \u2014 marginally stable<\/em>. The concatenation-with-projection approach used by RDMs leaves \u0100<\/em> entirely unconstrained, making \u03c1(\u0100) potentially far greater than 1 \u2014 unstable<\/em>. Empirical training curves confirm this directly: divergent training runs learn \u03c1(\u0100) \u2265 1, while the few convergent runs maintain \u03c1(\u0100) < 1.<\/p>\n
How Parcae Enforces Stability by Design<\/strong><\/h3>\n
Rather than parameterizing \u0100<\/em> directly, Parcae works in continuous form and discretizes using zero-order hold (ZOH) and Euler schemes<\/strong> \u2014 borrowing a standard technique from state space models like Mamba and S4 \u2014 with a learned step size \u0394 \u2208 \u211ddh<\/sub><\/sup>, giving \u0100 = exp(\u0394A) and B\u0304 = \u0394B. To guarantee \u03c1(\u0100) < 1, the continuous matrix A is constrained as a negative diagonal matrix<\/strong>: A := Diag(\u2212exp(logA<\/sub>))<\/code>, where logA<\/sub> \u2208 \u211ddh<\/sub><\/sup> is a learnable vector. Because diagonal entries are always negative before exponentiation, the spectral norm constraint is satisfied at all times by construction.<\/p>\n
Results: Outperforming Models Twice the Size<\/strong><\/h3>\n
Against parameter- and data-matched RDMs trained on the Huginn dataset, Parcae reduces validation perplexity by up to 6.3%<\/strong> \u2014 a figure that peaks at 350M scale (improving from 10.76 to 10.09 PPL) versus a 4.5% gain at 100M scale (14.23 to 13.59 PPL). WikiText perplexity improves by up to 9.1%<\/strong> at 350M scale. Average downstream zero-shot benchmark accuracy improves by up to 1.8 points.<\/p>\n
Against standard fixed-depth Transformer baselines trained with a nanochat-inspired setup on FineWeb-Edu, Parcae outperforms at every scale. At 1.3B parameters trained on 104B tokens<\/strong>, Parcae beats the parameter-matched Transformer by 2.99 points on Core<\/strong> and 1.18 points on Core-Extended<\/strong>. The 770M Parcae model<\/strong> (25.07 Core) reaches quality comparable to the 1.3B Transformer (25.45 Core) \u2014 roughly half the parameters for equivalent capability. The research team quantifies Parcae\u2019s parameter efficiency as achieving up to 87.5% of the quality of a Transformer twice its size<\/strong>, measured against the quality gap to the next larger model.<\/p>\n
The First Scaling Laws for Looping<\/strong><\/h3>\n
The second major contribution of this research is establishing the first predictable scaling laws for layer looping<\/strong>. Using isoFLOP experiments at 140M and 370M scales, the research team shows that compute-optimal training increases mean recurrence \u00b5rec<\/sub> and training tokens D<\/em> in tandem, following power laws with consistent exponents across both scales: optimal \u00b5rec<\/sub> scales as C0.40<\/sup> and optimal tokens scale as C0.78<\/sup>, where C is the training FLOP budget.<\/p>\n
When looped Parcae models trained at their optimal \u00b5rec<\/sub> are compared against fixed-depth Parcae models (\u00b5rec<\/sub> = 1) under identical FLOP and parameter budgets, looping achieves a strictly lower validation loss \u2014 translating into 1.2 to 2.0 points higher Core scores<\/strong> depending on the FLOP budget. Looping is a genuinely orthogonal axis for scaling compute, not a free lunch from weight sharing.<\/p>\n
At test time, increasing loop count T<\/em> beyond training depth follows a saturating exponential decay<\/strong>: L(T) = L\u221e<\/sub> + Z\u00b7e\u2212z\u00b7T<\/sup>, where L\u221e<\/sub> is an irreducible floor determined by training depth. Gains plateau near \u00b5rec<\/sub> \u2014 the mean recurrence used during training \u2014 meaning training depth sets a hard ceiling on test-time scaling. These dynamics unify into a single parametric law that predicts held-out model loss within 0.85\u20131.31% average error<\/strong>.<\/p>\n
Key Takeaways<\/strong><\/h3>\n