{"id":77,"date":"2025-12-08T12:59:18","date_gmt":"2025-12-08T04:59:18","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=77"},"modified":"2025-12-08T12:59:18","modified_gmt":"2025-12-08T04:59:18","slug":"from-transformers-to-associative-memory-how-titans-and-miras-rethink-long-context-modeling","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=77&lang=en","title":{"rendered":"From Transformers to Associative Memory, How Titans and MIRAS Rethink Long Context Modeling"},"content":{"rendered":"

What comes after Transformers?<\/strong> Google Research is proposing a new way to give sequence models usable long term memory with Titans and MIRAS, while keeping training parallel and inference close to linear.<\/p>\n

Titans is a concrete architecture that adds a deep neural memory to a Transformer style backbone. MIRAS is a general framework that views most modern sequence models as instances of online optimization over an associative memory.<\/p>\n

Why Titans and MIRAS?<\/strong><\/h3>\n

Standard Transformers use attention over a key value cache. This gives strong in context learning, but cost grows quadratically with context length, so practical context is limited even with FlashAttention and other kernel tricks.<\/p>\n

Efficient linear recurrent neural networks and state space models such as Mamba-2 compress the history into a fixed size state, so cost is linear in sequence length. However, this compression loses information in very long sequences, which hurts tasks such as genomic modeling and extreme long context retrieval.<\/p>\n

Titans and MIRAS combine these ideas. Attention acts as a precise short term memory on the current window. A separate neural module provides long term memory, learns at test time, and is trained so that its dynamics are parallelizable on accelerators.<\/p>\n

\n
\"\"
https:\/\/research.google\/blog\/titans-miras-helping-ai-have-long-term-memory\/<\/figcaption><\/figure>\n<\/div>\n

Titans, a neural long term memory that learns at test time<\/strong><\/h3>\n

The Titans research paper<\/a> introduces a neural long term memory module that is itself a deep multi layer perceptron rather than a vector or matrix state. Attention is interpreted as short term memory, since it only sees a limited window, while the neural memory acts as persistent long term memory.<\/p>\n

For each token, Titans defines an associative memory loss<\/p>\n

\u2113(M\u209c\u208b\u2081; k\u209c, v\u209c) = \u2016M\u209c\u208b\u2081(k\u209c) \u2212 v\u209c\u2016\u00b2<\/p>\n

where M\u209c\u208b\u2081 is the current memory, k\u209c is the key and v\u209c is the value. The gradient of this loss with respect to the memory parameters is the \u201csurprise metric\u201d. Large gradients correspond to surprising tokens that should be stored, small gradients correspond to expected tokens that can be mostly ignored.<\/p>\n

The memory parameters are updated at test time by gradient descent with momentum and weight decay, which together act as a retention gate and forgetting mechanism.To keep this online optimization efficient, the research paper shows how to compute these updates with batched matrix multiplications over sequence chunks, which preserves parallel training across long sequences.<\/p>\n

Architecturally, Titans uses three memory branches in the backbone, often instanced in the Titans MAC variant:<\/strong><\/p>\n