Back to AI guides

Memorilla: Learned Memory Beyond the Context Window

Stanley Ulili
Updated on November 16, 2026

An assistant can retain an entire conversation and still fail to recover the detail that matters. Expanding its context window makes more text available, but also increases the work required to process it. Retrieving a few passages reduces that work, provided the retrieval system selects the right evidence.

Memorilla explores a different approach: learn a compact representation of stored documents and let the language model answer from it. The important tradeoff is between a small decoder input and the precision of the information that survives compression. A faster answer is useful only if the memory preserves what the question requires.

This guide explains how learned memory differs from retrieval, why its timing numbers need careful interpretation, and what small demonstrations reveal about exact factual recall.

Why longer context does not settle the memory problem

A conventional assistant can append earlier messages to its next request. The model then receives both the current question and the history needed to answer it. This is a straightforward design when the history fits within the supported context window and the extra processing is affordable.

The difficulty grows with the history. A user may discuss a preference once and return to it months later. Repeatedly sending every message increases input length, while trimming older messages can remove the only relevant evidence. A larger context window changes the capacity limit; it does not decide which facts deserve attention.

Retrieval-augmented generation, or RAG, separates storage from the answer context. Documents are indexed, the question selects a subset, and the selected text reaches the language model. This keeps the prompt smaller and can retain the exact wording of evidence. However, retrieval becomes another place where an answer can fail: the right passage may exist without ranking among the selected results.

Document history encoded for a compact learned-memory path

Learned memory places a representation of the history between stored documents and the decoder.

These approaches solve different resource problems. Full context spends decoder capacity on broad access. RAG spends effort selecting source passages. Learned memory spends effort representing information in a form that a decoder can use without reading all the original text.

What Memorilla stores and reads

Memorilla has a document-writing path and a question-answering path. Documents first pass through an embedding encoder. Each becomes a stored vector, which can be reused for later questions instead of encoding the same document repeatedly.

At answer time, a trained memory module reads those vectors in relation to the question. It produces a fixed set of sixteen continuous memory vectors for the frozen language model. Those vectors occupy positions in its input representation, but they are not sixteen ordinary words or a human-readable summary. The research paper describes this as learned memory that connects an embedding store to a frozen decoder.

Memory module attending over document embeddings to create question-dependent memory vectors

The question affects what the memory module reads from the stored embeddings.

That distinction explains both the appeal and the risk. The module can combine information from many documents without inserting all their text into the decoder. Conversely, a name, date, exception, or qualification may not remain recoverable in the compact representation. Nothing about a fixed-size decoder input implies lossless preservation of arbitrarily many independent facts.

The memory vectors also depend on the question. Memorilla is not merely generating one permanent sixteen-vector summary and reusing it for every request. It stores document representations and performs a learned read for each question. The decoder receives a small result, while the memory system still has work to do over its collection.

Training determines what compression preserves

A memory system must learn which distinctions will matter later. Memorilla's training progresses through document reconstruction, broader question-answering and summarization objectives, and adaptation for downstream tasks. The encoder and decoder can remain frozen while the intervening memory module learns how to connect them.

Three stages of memory training, from document reconstruction to task adaptation

Training gives the memory module a reason to preserve information that supports its target tasks.

Reconstruction encourages the representation to carry document content. Question-answering objectives encourage useful selection and combination. Task adaptation then aligns the memory with a particular evaluation setting. This is a more specific capability than adding a general-purpose memory database to an arbitrary assistant.

A checkpoint that performs well on its intended tasks may behave differently on new chat histories, fictional preferences, or long-form fiction. The question is whether it preserves the kinds of distinctions the application needs. Remembering a broad dietary preference and recovering an exact birthday are related operations, but they place different demands on the representation.

This is why a successful example is best treated as a starting point for evaluation. It shows that the pipeline can operate and produce a plausible result. It does not establish the range of facts that the memory can recover reliably.

A working example needs an evidence check

One demonstration reproduces the project's example using 107 conversations and roughly 32,000 tokens of history. Asked for breakfast ideas, the memory-backed model produces a personalized answer rather than a generic list.

Personalized breakfast answer returned from a collection of earlier conversations

A personalized answer demonstrates that stored context can influence generation, but each claimed personal fact still needs support.

There is a subtle issue in the supporting history: asking whether maternal family history influences a condition is not the same as stating that the condition occurs in one's family. A model can turn the topic of a question into an assumed biographical fact. An answer can therefore look impressively personalized while overstating its evidence.

This is a useful evaluation principle beyond Memorilla. Check not only whether an answer mentions the expected subject, but also whether it preserves the relationship expressed in the source. A question, a hypothetical example, and an explicit personal statement should not become interchangeable after compression.

An application that stores user preferences needs this distinction. Otherwise, a tentative discussion can become a persistent assertion, and future answers may repeatedly reinforce a fact that the user never supplied.

Exact recall is a separate challenge

The Buried Clue demonstration constructs 150 fictional conversations containing a few scattered preferences and an explicit factual control. It compares four paths using the same language-model family: no memory, top-five retrieval, full context, and Memorilla.

Four answer panels comparing a factual control across memory and retrieval methods

The comparison exposes differences between finding a topic and recovering an exact fact.

Memorilla misses all four questions in this small test. Retrieval answers the explicit control correctly, while none of the methods recovers the three hidden preferences. Full context also struggles, including refusals around personal information. The published demonstration findings make clear that these samples describe behavior rather than statistically reliable accuracy estimates.

The errors matter qualitatively. A wrong answer that invents an allergy or substitutes a birthday is different from an honest statement that the evidence could not be recovered. An evaluation should distinguish unsupported assertions, abstentions, and correct answers instead of reducing all three to a single impression of helpfulness.

A follow-up isolates the conversation containing a sister's name and birthday, then adds more conversations. The model can recover the name in the smaller setting while still getting the date wrong; broader context introduces additional failure.

Small and larger conversation collections used to probe exact name and birthday recall

Reducing the collection can help isolate a failure, without proving that collection size is its only cause.

Possible explanations include information loss in embeddings, the memory module's learned preferences, and a mismatch between training and the new task. These are hypotheses, not diagnoses established by the example. Separating the encoder, memory reader, and decoder would require more targeted experiments.

Small answer inputs do not eliminate ingestion costs

Timing needs the same care as accuracy. Memorilla separates document encoding from answering, so the measured operation determines the headline. Reusing existing document embeddings is different from encoding the whole collection immediately before a question.

In the H100 demonstration, answer generation with the compact memory input remains near 0.12 seconds as the collection grows to 512 documents. Including document encoding raises the total to about 3.25 seconds at that size. The full-context path takes about 19.33 seconds in the same demonstration. Those are specific measurements from that setup, not latency guarantees for other hardware or workloads.

Latency curves separating full-context processing, compact-memory answering, and document encoding

A short answer path and a low total first-query cost are different measurements.

The difference has practical consequences. A stable document collection serving many questions can amortize its writing cost. A rapidly changing collection may repeatedly pay to encode new material. A single question over entirely new documents has a different cost profile again.

The fixed decoder input also does not make storage or the memory read independent of collection size. More documents mean more stored vectors, and reading their representations still consumes resources. The benefit is that the decoder avoids processing all original document tokens for every answer.

For an evaluation, report ingestion time, answer time, and total first-query time separately. Record the number of questions sharing the same embeddings. Include memory use and any cache assumptions. Otherwise, two valid measurements can appear contradictory simply because they time different boundaries.

Long documents test both capacity and precision

A second demonstration uses The Hound of the Baskervilles, approximately 72,000 tokens, and eight evidence-checked questions. Memorilla answers one correctly, retrieval four, and full context five. The full-context baseline is constrained by the native window and truncates the text, which limits what that comparison establishes.

Long-book evaluation showing the portion retained by the full-context baseline

A truncated baseline cannot answer questions using material it never received.

This is an important limitation rather than a reason to discard the experiment. It shows how an application behaves under a realistic capacity constraint. It does not compare Memorilla with an unconstrained model that receives the entire book.

The eight questions are also too few to rank all learned-memory and retrieval systems. Their value is diagnostic: exact details from a long narrative can remain difficult even when the memory path is fast. A result that preserves the broad theme of a book may still be unsuitable for answering evidence-sensitive questions about it.

Side-by-side answers to a precise question about a letter in the novel

Precise questions expose whether a method retains the evidence needed to distinguish similar answers.

Evaluating learned memory for an application

A useful evaluation begins with the application's facts, not its desired compression ratio. Build questions around names, dates, negation, changed preferences, and multiple documents that discuss similar subjects. Include questions whose answers are deliberately absent. A memory system should not earn credit for confidently completing a plausible story.

Keep the decoder, answer budget, and question wording consistent across baselines where possible. Give retrieval an appropriate index and record how many passages it returns. State whether full context receives all documents or a truncated subset. These choices can influence an outcome as much as the memory mechanism itself.

Evaluate updates as well as static recall. If a user changes a preference, ask whether the system favors the newer statement and whether it can explain the supporting evidence. A compact representation that combines old and new information may need a separate strategy for freshness and provenance.

One possible design is to use learned memory for broad orientation while retaining source retrieval for exact claims. That is an architectural option, not a capability demonstrated automatically by installing this checkpoint. It still requires decisions about when to retrieve, how to cite evidence, and how to handle disagreement.

Finally, assess latency at the expected operating point. The relevant question may be the cost of one newly ingested collection, hundreds of questions over stable documents, or frequent updates to a growing history. A single compression figure cannot describe all three.

Final thoughts

Memorilla demonstrates an interesting way to move work out of a language model's long text input and into a trained memory reader. Its compact decoder input can make answering substantially faster in a suitable setup.

That efficiency does not establish reliable factual recall. The small demonstrations highlight unsupported personalization, missed details, and the importance of separating encoding time from answer time. Learned memory is most useful when evaluated against the exact information an application must preserve, with retrieval and full-context baselines measured on clearly stated terms.

Got an article suggestion? Let us know
Licensed under CC-BY-NC-SA

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.