{"id":982,"date":"2026-05-27T13:24:33","date_gmt":"2026-05-27T05:24:33","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=982"},"modified":"2026-05-27T13:24:33","modified_gmt":"2026-05-27T05:24:33","slug":"memo-a-modular-framework-for-training-a-dedicated-memory-model-on-new-knowledge-without-modifying-llm-parameters","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=982","title":{"rendered":"MEMO: A Modular Framework for Training a Dedicated Memory Model on New Knowledge Without Modifying LLM Parameters"},"content":{"rendered":"

Large language models become static after pretraining. Their knowledge does not update as the world changes. Retraining a full LLM is too expensive at modern scales. Fine-tuning risks degrading previously learned knowledge. Retrieval-augmented generation (RAG) struggles when answers require reasoning across many documents.<\/p>\n

A team of researchers from the National University of Singapore, MIT CSAIL, A*STAR, and the Singapore-MIT Alliance for Research and Technology (SMART) proposes a new approach called MEMO (Memory as a Model)<\/strong>.<\/p>\n

What Problem Does MEMO Solve?<\/strong><\/h2>\n

Existing methods for integrating new knowledge into LLMs fall into three categories. Non-parametric methods like RAG retrieve documents at inference time. They are sensitive to retrieval noise and struggle with cross-document reasoning. Parametric methods such as continual pretraining or supervised fine-tuning internalize knowledge into model weights. They are computationally expensive and cause catastrophic forgetting<\/strong>, where new training degrades previously acquired knowledge. Latent memory methods compress knowledge into soft tokens. These representations are tightly bound to the model that produced them \u2014 a limitation the research team calls representation coupling<\/strong> which limits transferability across LLMs.<\/p>\n

<\/strong>MEMORY<\/strong> as a Separate Model<\/h2>\n

MEMO separates memory from reasoning. The MEMORY model<\/strong> is a small, dedicated language model trained to internalize knowledge from a target corpus. The EXECUTIVE model<\/strong> is the main LLM \u2014 frozen and queried only through its standard input-output interface.<\/p>\n

In experiments, the MEMORY model is Qwen2.5-14B-Instruct. The EXECUTIVE model is either Qwen2.5-32B-Instruct or Gemini-3-Flash, a proprietary closed-source model. Because MEMO treats the EXECUTIVE model as a black box, it does not require weight access or output logits.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2605.15156<\/figcaption><\/figure>\n<\/div>\n

How the MEMORY Model is Trained<\/strong><\/h2>\n

Training begins with a five-step data synthesis pipeline<\/strong> guided by a GENERATOR model<\/strong> \u2014 Qwen2.5-32B-Instruct in experiments. The pipeline converts a raw document corpus into a reflection QA dataset<\/strong>: question-answer pairs that represent corpus knowledge under diverse query variations.<\/p>\n

The five steps are:<\/strong><\/p>\n

    \n
  1. Fact extraction<\/strong> \u2014 direct extraction of explicitly stated facts, and indirect extraction of inferred information, run in parallel per document chunk.<\/li>\n
  2. Consolidation<\/strong> \u2014 QA pairs sharing a common context (entity, time period, relationship) are merged into multi-fact pairs.<\/li>\n
  3. Verification and rewriting<\/strong> \u2014 each QA pair is checked for self-containment. Pairs with unresolved pronouns or implicit references are rewritten using the source chunk or discarded.<\/li>\n
  4. Entity surfacing<\/strong> \u2014 QA pairs are generated where questions encode entity attributes and relationships, and answers reveal entity identities. This targets the reversal curse<\/strong>, where models trained on \u201cA is B\u201d fail to infer \u201cB is A.\u201d<\/li>\n
  5. Cross-document synthesis<\/strong> \u2014 the GENERATOR model constructs QA pairs spanning multiple documents. It identifies two types of cross-document connections: converging clues (multiple documents about the same entity) and parallel properties (different entities sharing a common attribute or role).<\/li>\n<\/ol>\n

    Step-5 is the most critical component. A leave-one-out ablation shows that removing it drops accuracy from 24.00% to 6.37% on NarrativeQA. It is also the dominant source of training pairs in the final dataset.<\/p>\n

    The MEMORY model is then trained via supervised fine-tuning (SFT)<\/strong>. The loss is computed over answer tokens only. Source documents are never provided at inference. The model must answer from internalized parametric knowledge.<\/p>\n

    Inference: The Structured Multi-Turn Protocol<\/strong><\/h2>\n

    At inference, the EXECUTIVE model queries the MEMORY model through a structured multi-turn protocol<\/strong> with three sequential stages.<\/p>\n

    Stage 1: Grounding.<\/strong> The EXECUTIVE model decomposes the query into atomic sub-questions. Each targets a single identifying constraint. The MEMORY model answers each independently.<\/p>\n

    Stage 2: Entity identification.<\/strong> Using the grounding responses, the EXECUTIVE model issues targeted follow-up sub-queries. It iteratively narrows down candidate entities until one is confirmed or the stage budget runs out.<\/p>\n

    Stage 3: Answer seeking and synthesis.<\/strong> Conditioned on the identified entity, the EXECUTIVE model queries the MEMORY model for supporting facts. It then synthesizes all retrieved responses into a final answer.<\/p>\n

    The MEMORY model\u2019s responses are compact natural-language snippets. Their length is independent of corpus size, so retrieval cost does not scale with the number of documents. This contrasts with RAG, where inference cost grows with the corpus.<\/p>\n

    Experimental Results<\/strong><\/h2>\n

    MEMO is evaluated on three benchmarks: BrowseComp-Plus<\/strong> (multi-hop deep-research), NarrativeQA<\/strong> (discourse understanding over books and movie scripts), and MuSiQue<\/strong> (2\u20134 hop reasoning over Wikipedia paragraphs). Baselines include BM25, NV-Embed-V2, HippoRAG2, and Cartridges. Cartridges requires white-box access to the EXECUTIVE model and scored 0.00% on BrowseComp-Plus and 3.75% on NarrativeQA.<\/p>\n

    On NarrativeQA with Gemini-3-Flash, MEMO achieves 53.58%<\/strong>. HippoRAG2 reaches 23.21% on the same setup. On MuSiQue, MEMO achieves 60.20%<\/strong> against HippoRAG2\u2019s 57.00%. On BrowseComp-Plus, MEMO achieves 66.67%<\/strong> against HippoRAG2\u2019s 66.33%.<\/p>\n

    With Qwen2.5-32B-Instruct as EXECUTIVE model, MEMO achieves 54.22% on BrowseComp-Plus and 48.30% on MuSiQue. Switching to Gemini-3-Flash yields gains of 12.45%, 26.73%, and 11.90% on the three benchmarks. The MEMORY model is not retrained when the EXECUTIVE model changes.<\/p>\n

    Robustness to retrieval noise<\/strong>: The research team evaluates performance when distractor documents are added to the corpus. NV-Embed-V2 and HippoRAG2 drop by up to 6.22% on BrowseComp-Plus when one negative document is added per evidence document. MEMO\u2019s accuracy on the same benchmark changes by +0.55% \u2014 within one standard deviation.<\/p>\n

    MEMORY model architecture robustness<\/strong>: The research team also tests three MEMORY model families at similar parameter scale: Qwen2.5-1.5B-Instruct, Gemma3-1B-IT, and LFM2.5-1.2B-Instruct (a hybrid state-space and transformer architecture). Performance is largely consistent across all three, indicating the framework is not sensitive to the specific pretraining lineage of MEMORY model.<\/p>\n

    Continual Knowledge Integration via Model Merging<\/strong><\/h2>\n

    MEMO supports incremental knowledge updates through model merging<\/strong>. When a new corpus arrives, a separate MEMORY model is trained on it independently. Its task vector \u2014 the parameter difference from the base model \u2014 is then merged with the existing MEMORY model in parameter space.<\/p>\n

    The research team test this on NarrativeQA using TIES merging<\/strong> (\u03c1=0.3). For K=2 corpora, merging accumulates 48 GPU-hours versus 72 GPU-hours for full retraining \u2014 a 33% reduction. At K=10, merging scales as \u0398(K) while full retraining scales as \u0398(K\u00b2), yielding a 5.5\u00d7 saving (240 vs. 1,320 GPU-hours).<\/p>\n

    The merged MEMORY model trails full retraining by 11.04% under Qwen2.5-32B-Instruct (15.81% vs. 26.85%). It trails by 19.11% under Gemini-3-Flash (34.47% vs. 53.58%). Despite this gap, it outperforms all retrieval baselines on NarrativeQA.<\/p>\n

    Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n
    \n
    \n Marktechpost \u2014 Research Explainer<\/span>
    \n MEMO: Memory as a Model<\/span>\n <\/div>\n
    \n
    \n
    \n 01 \/ 06 \u2014 The Problem<\/span>
    \n LLMs Freeze After Pretraining<\/span>
    \n Their knowledge becomes outdated as the world evolves.<\/span>\n

    Large language models are static once pretraining ends. For applications requiring up-to-date or domain-specific knowledge, three approaches exist \u2014 and each has a critical flaw.<\/p>\n

    <\/div>\n
    \n
    \ud83d\udd0d<\/span>RAG<\/span>Sensitive to retrieval noise. Struggles when answers span multiple documents.<\/span><\/div>\n
    \u2699<\/span>Fine-Tuning<\/span>Causes catastrophic forgetting. Expensive. Cannot be used on proprietary LLMs.<\/span><\/div>\n
    \ud83d\udcbe<\/span>Latent Memory<\/span>Representations are tightly coupled to one specific model architecture only.<\/span><\/div>\n<\/div>\n

    MEMO \u2014 Memory as a Model \u2014 from researchers at NUS, MIT CSAIL, and A*STAR addresses all three limitations simultaneously.<\/p>\n<\/div>\n

    \n 02 \/ 06 \u2014 The Concept<\/span>
    \n Memory Separated From Reasoning<\/span>
    \n Two models. One frozen. One trained on new knowledge.<\/span>\n

    MEMO introduces two distinct model roles that operate together.<\/p>\n

    \n
    \u25c6 MEMORY Model<\/span>A small, dedicated language model trained to internalize knowledge from a target corpus. It stores facts and cross-document relationships in its parameters. It never sees source documents at inference \u2014 it answers only from what it has learned.<\/span><\/div>\n
    \u25c7 EXECUTIVE Model<\/span>The main LLM \u2014 frozen and unchanged throughout. It queries the MEMORY model through targeted sub-questions, reasons over retrieved responses, and produces the final answer. Works with any LLM, including closed-source APIs.<\/span><\/div>\n<\/div>\n
    <\/div>\n

    In experiments: Qwen2.5-14B-Instruct<\/strong> as MEMORY model. Qwen2.5-32B-Instruct<\/strong> or Gemini-3-Flash<\/strong> as EXECUTIVE model. Only black-box API access required \u2014 no weights, no logits.<\/p>\n<\/div>\n

    \n 03 \/ 06 \u2014 Training<\/span>
    \n How the MEMORY Model Is Built<\/span>
    \n A five-step pipeline converts raw documents into a reflection QA dataset.<\/span>\n
    \n Fact Extraction<\/span>\u2192<\/span>
    \n Consolidation<\/span>\u2192<\/span>
    \n Verification<\/span>\u2192<\/span>
    \n Entity Surfacing<\/span>\u2192<\/span>
    \n Cross-Doc Synthesis<\/span>\n <\/div>\n
    \n
    \n
    01<\/div>\n
    Fact Extraction<\/span>Direct extraction of stated facts and indirect extraction of inferred information run in parallel per document chunk.<\/span><\/div>\n<\/div>\n
    \n
    02<\/div>\n
    Consolidation<\/span>QA pairs sharing a common entity, time period, or relationship are merged into multi-fact pairs.<\/span><\/div>\n<\/div>\n
    \n
    03<\/div>\n
    Verification & Rewriting<\/span>Each pair is checked for self-containment. Pairs with unresolved pronouns or implicit references are rewritten or discarded.<\/span><\/div>\n<\/div>\n
    \n
    04<\/div>\n
    Entity Surfacing<\/span>QA pairs are generated where questions encode entity attributes and answers reveal identities, targeting the reversal curse.<\/span><\/div>\n<\/div>\n
    \n
    05<\/div>\n
    Cross-Document Synthesis<\/span>The most critical step. Removing it drops NarrativeQA accuracy from 24.00% to 6.37%. Constructs QA pairs spanning multiple documents via converging clues and parallel properties.<\/span><\/div>\n<\/div><\/div>\n

    MEMORY model trained via supervised fine-tuning (SFT)<\/strong> \u2014 loss over answer tokens only. Source documents never provided at inference.<\/p>\n<\/div>\n

    \n 04 \/ 06 \u2014 Inference<\/span>
    \n Three-Stage Query Protocol<\/span>
    \n The EXECUTIVE model queries the MEMORY model through structured sub-questions.<\/span>\n

    Complex user queries are decomposed across three sequential stages. No documents are retrieved \u2014 all answers come from internalized parametric knowledge.<\/p>\n

    \n
    \n
    S1<\/div>\n
    Grounding \u2014 Budget: 1 interaction<\/span>The user query is decomposed into atomic sub-questions, each targeting one identifying constraint. MEMORY model answers each independently.<\/span><\/div>\n<\/div>\n
    \n
    S2<\/div>\n
    Entity Identification \u2014 Budget: 7 interactions<\/span>Using grounding responses, the EXECUTIVE model issues follow-up sub-queries to iteratively narrow candidate entities until one is confirmed.<\/span><\/div>\n<\/div>\n
    \n
    S3<\/div>\n
    Answer Seeking & Synthesis \u2014 Budget: 8 interactions<\/span>Conditioned on the confirmed entity, the EXECUTIVE model gathers supporting facts then synthesizes all retrieved responses into a final answer.<\/span><\/div>\n<\/div><\/div>\n
    <\/div>\n

    MEMORY model responses are compact natural-language snippets. Retrieval cost is fixed and does not scale with corpus size<\/strong> \u2014 unlike RAG.<\/p>\n<\/div>\n

    \n 05 \/ 06 \u2014 Advantages<\/span>
    \n What MEMO Does Differently<\/span>
    \n Compared to RAG, fine-tuning, and latent memory methods.<\/span>\n
    \n
    \n Other Methods<\/span>
    \n Retrieval noise significantly degrades RAG accuracy<\/span>
    \n Fine-tuning causes catastrophic forgetting in the LLM<\/span>
    \n Latent memory tied to one specific model architecture<\/span>
    \n Retrieval cost grows with corpus size at inference<\/span>
    \n Cannot be used with proprietary closed-source LLMs<\/span>
    \n Adding new knowledge requires full retraining<\/span>\n <\/div>\n
    \n MEMO<\/span>
    \n Accuracy changes \u00b11.77% under added distractor documents<\/span>
    \n Main LLM stays frozen; no catastrophic forgetting possible<\/span>
    \n Works across Qwen, Gemma, and LFM2.5 architectures<\/span>
    \n Fixed-size responses; cost independent of corpus size<\/span>
    \n Black-box compatible \u2014 works with any LLM including APIs<\/span>
    \n New corpora merged via model merging without full retraining<\/span>\n <\/div>\n<\/div>\n

    TIES merging (\u03c1=0.3) cuts compute by 33% at K=2 corpora<\/strong> and 5.5\u00d7 at K=10 corpora<\/strong> vs full retraining.<\/p>\n<\/div>\n

    \n 06 \/ 06 \u2014 Results<\/span>
    \n Benchmark Performance<\/span>
    \n Qwen2.5-14B-Instruct as MEMORY model. Gemini-3-Flash as EXECUTIVE model.<\/span>\n
    \n
    53.58%<\/span>NarrativeQA<\/span>vs HippoRAG2: 23.21%<\/span><\/div>\n
    60.20%<\/span>MuSiQue<\/span>vs HippoRAG2: 57.00%<\/span><\/div>\n
    66.67%<\/span>BrowseComp-Plus<\/span>vs HippoRAG2: 66.33%<\/span><\/div>\n<\/div>\n
    <\/div>\n

    Switching EXECUTIVE model from Qwen2.5-32B-Instruct to Gemini-3-Flash yields gains of +12.45%<\/strong>, +26.73%<\/strong>, and +11.90%<\/strong> across the three benchmarks \u2014 without retraining the MEMORY model.<\/p>\n

    Under retrieval noise, HippoRAG2 drops 6.22% on BrowseComp-Plus. MEMO changes by +0.55%<\/strong> on the same benchmark \u2014 within one standard deviation.<\/p>\n

    <\/div>\n

    Source: arXiv 2605.15156 \u2014 Quek, Lee, Leong, Verma et al., NUS \/ MIT CSAIL \/ A*STAR \/ SMART, May 2026.<\/p>\n<\/div>\n<\/div>\n<\/div>\n

    \n