{"id":989,"date":"2026-05-28T08:51:53","date_gmt":"2026-05-28T00:51:53","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=989"},"modified":"2026-05-28T08:51:53","modified_gmt":"2026-05-28T00:51:53","slug":"sakana-ai-proposes-diffusionblocks-a-block-wise-training-framework-that-converts-residual-networks-into-independently-trainable-denoising-modules","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=989","title":{"rendered":"Sakana AI Proposes DiffusionBlocks: a Block-wise Training Framework That Converts Residual Networks into Independently Trainable Denoising Modules"},"content":{"rendered":"

Researchers from Sakana AI and the University of Tokyo propose DiffusionBlocks. It trains transformer-based networks one block at a time. Training memory is reduced by a factor of B, where B is the number of blocks. Performance is maintained across diverse architectures.<\/p>\n

The Memory Problem in Neural Network Training<\/strong><\/h2>\n

End-to-end backpropagation requires storing intermediate activations across every layer. Memory consumption grows linearly with network depth. As models grow deeper, this becomes a significant training bottleneck.<\/p>\n

One existing technique, activation checkpointing, reduces activation memory by recomputing activations on demand. However, it does not reduce memory for parameters, gradients, or optimizer states. With the Adam optimizer, each layer requires memory for parameters, gradients, and two optimizer states (momentum and variance). This totals 4 times the parameter size per layer, unchanged by activation checkpointing.<\/p>\n

Block-wise training offers a different approach. Partitioning a network into B blocks and training each independently reduces memory to roughly 1\/B. The reduction is proportional to the number of blocks. The challenge is defining a principled local objective for each block that still produces a globally coherent model.<\/p>\n

Prior approaches like Hinton\u2019s Forward-Forward algorithm and greedy layer-wise training rely on ad-hoc local objectives. They consistently underperform end-to-end training and are largely limited to classification tasks.<\/p>\n

DiffusionBlocks addresses both the theoretical gap and the limited applicability of prior methods.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2506.14202<\/figcaption><\/figure>\n<\/div>\n

The Core Idea: Residual Connections as Euler Steps<\/strong><\/h2>\n

The key insight builds on an established connection in the literature. Residual networks update each layer input via z<\/mi>\u2113<\/mi>=<\/mo>z<\/mi>\u2113<\/mi>\u2212<\/mo>1<\/mn>+<\/mo>f<\/mi>\u03b8<\/mi>\u2113<\/mi>(<\/mo>z<\/mi>\u2113<\/mi>\u2212<\/mo>1<\/mn>)<\/mo><\/mrow>z\u2113 = z\u2113\u22121 + f\u03b8\u2113 (z\u2113\u22121) <\/annotation><\/semantics><\/math>. This corresponds to Euler discretization of ordinary differential equations.<\/p>\n

The research team show these updates correspond specifically to the probability flow ODE in score-based diffusion models. In the Variance Exploding (VE) formulation, the reverse diffusion process follows:<\/p>\n

<\/mrow>d<\/mi><\/mrow>\ud835\udc33<\/mi>\u03c3<\/mi><\/msub><\/mfrac><\/mrow><\/mrow>d<\/mi><\/mrow>\u03c3<\/mi>=<\/mo>\u2212<\/mo>\u03c3<\/mi>\u2207<\/mo>\ud835\udc33<\/mi><\/msub>log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>\u03c3<\/mi><\/msub>(<\/mo>\ud835\udc33<\/mi>\u03c3<\/mi><\/msub>)<\/mo> frac{mathrm{d}mathbf{z}_sigma}{mathrm{d}sigma} = -sigma nabla_{mathbf{z}} log p_sigma(mathbf{z}_sigma) <\/annotation><\/semantics><\/math><\/p>\n

Applying Euler discretization to this equation produces an update rule that structurally matches the residual connection update. A stack of residual blocks can be interpreted as discretized denoising steps. The steps span a noise level range <\/code>[\ud835\udf82min<\/sub>, \ud835\udf82max<\/sub>]<\/code>.<\/p>\n

In score-based diffusion models, the score matching objective can be optimized independently at each noise level. This means each block can be trained independently, using only its own local objective. No inter-block communication is needed during training.<\/p>\n

Converting a Network: Three Steps<\/strong><\/h2>\n

Converting a standard residual network to DiffusionBlocks requires three modifications<\/strong>:<\/p>\n