Building memory for a personal assistant, I learned to ask what it actually saw

Development log · About 8 min read · September 9, 2026

After reading Google’s E-E-A-T framework a while ago, I felt it resembled the situation of a personal assistant: faced with a lot of information, it has to decide what is worth trusting, keeping, and using right now.

But after going through the logs, tracing the call chain, and running several controlled comparisons, my attention moved to something more basic.

How many times does information get lost between a memory entering from the screen and being sent to the model that answers a question? Why does one failure run again? When a fact has a citation, does that citation actually support it?

The most valuable discoveries today were hidden in those questions.

First, look at where the money goes

In logs through 19:44 that day, there were 1,253 model calls and about 3.68 million tokens recorded. Memory extraction accounted for 65.2%, graph organization 20.7%; together they made up 85.9%.

This distribution changed the order of optimization. Users directly feel Cue and Q&A, but most of the cost is in the background.

Tracing further showed that some extraction entry points did not share the same failure backoff. The model had already failed, but another scheduler could still send the work back in. Successful batches had receipt protection, while retry management for failed batches was incomplete.

This made me realize that before discussing caching and prompt compression, we must first check whether the system is repeatedly paying for failed work.

We fixed the task members, recorded successful progress, content failures, and provider failures separately, and let different entry points share scheduling constraints. In a controlled scenario that failed continuously, running the same 100 rounds reduced extraction calls from 100 to 7.

That 93% drop belongs only to this experiment and cannot be treated as a reduction in a real daily bill. It does prove one concrete thing: failure does not have to be paid for again with every sampling round.

A personal memory system needs to remember not just the user, but what it has already done, where it failed, and when it is worth trying again.

The clearest accuracy gap appeared at the last step

The most surprising discovery today was that the Q&A path had already retrieved the complete citation, but when assembling the final prompt it kept only the memory summary and source coordinates; the original text was discarded.

The model knew “this memory has a source,” but could not see what the source said.

Suppose the summary is “the user is free on Friday,” while the original says “if the review ends early, I can join Friday afternoon.” Once the condition disappears from the prompt, even a cautious answering model can hardly recover the qualification.

We fixed this explicit Q&A path so the complete citation travels with the memory, then ran a real-model comparison on 12 public synthetic cases. The cases deliberately included conditions, speakers, or states omitted by summaries; they do not represent the natural distribution of everyday questions.

The result:

Improved accuracy came at a cost: tokens for this Q&A set increased by 32.6%.

I think this cost is worth keeping. A lower bill achieved by compressing away the condition that determines a sentence’s meaning is not reliable.

The ablation that followed was also interesting: changing only the instructions without adding the original text stayed at 6/22; adding only the original text while keeping the old instructions reached 21/22.

In this set of cases, most of the gain came from the model finally seeing evidence it should have seen in the first place.

“Provenance and context closure” is becoming concrete

I now prefer to understand provenance and context closure this way:

When using a memory, the necessary evidence that makes it valid and correctly understood should arrive with it.

At least four things are involved:

• The source still exists and is still allowed to be used.
• Necessary context—speaker, time, conditions, draft status, and so on—has not been lost.
• The citation actually supports the memory, rather than merely containing a few matching keywords.
• The material is sufficient for the current question; when it is not, the system can acknowledge the gap.

These guarantees cannot substitute for one another.

Today’s source inspection found that a citation can have perfectly valid byte offsets and still omit the speaker information that came before it. Even when both original passages are included, the model may still mistake a draft for a confirmed decision.

So “the citation exists” is a useful engineering guarantee, but it is still a distance from “the meaning is correct.”

This also means complete closure cannot come from a stricter prompt alone. Every layer—capture, slicing, extraction, retrieval, and answering—needs to preserve necessary structure. When a source is forgotten or expires, memories that depend on it also need to be re-evaluated.

There is still plenty of design and validation work to do on these deeper dependency and failure mechanisms. What we completed today was one explicit gap: making the evidence already retrieved actually reach the model that uses it.

One experiment that did not pass

We also tried adding stricter citation-faithfulness requirements to the extraction model.

Across 16 public cases, unsupported claims fell from 3 to 2, but complete coverage of necessary facts fell from 13/21 with one additional case undecided to 11/21; total tokens also increased.

We did not enable this approach.

It reminded me that a memory system can easily produce a superficial improvement: extracting less may mean fewer mistakes. But users need valuable information to be remembered reliably; omitting complete commitments, conditions, and arrangements also damages the experience.

Therefore evaluation must look at errors and omissions together. We cannot show only the falling error count while hiding the facts that were lost.

A lock in the background can also change the experience of “intelligence”

The hands-on test after completing key authorization exposed another issue unrelated to model capability.

A simple question was eventually answered correctly, but it took 59.8 seconds. The provider spent about 1.3 seconds actually handling the Q&A; most of the rest was waiting for background extraction to release a shared lock.

It turned out the model adapter held a mutex for the entire network request. A single slow background request could block foreground Q&A.

After the fix, each request briefly takes a consistent snapshot of model configuration, then performs the network call independently. In the final measurement, the same question returned the correct answer in 1.28 seconds even while background work was still unfinished.

These are two field observations, not a complete latency benchmark. But the call chain and controlled concurrency tests confirmed the cause of the blockage.

Users do not distinguish a slow model, a slow queue, or a lock waiting. They feel only whether the assistant responds in time.

For a close-at-hand assistant, scheduling and concurrency are also product capabilities.

What I am taking away today

Looking back, today did not produce a new architecture worthy of claiming “a decisive overall lead,” nor did we measure a daily savings figure comparable under natural use. Background extraction validation failures and model timeouts still need work.

But I am clearer about what to do next.

First reduce repeated work with no value, then decide how to compress each input. First check whether evidence was lost in the chain, then ask the model to be smarter. When evaluating accuracy, record complete coverage, unsupported assertions, and omissions together, rather than keeping only one flattering score.

I want to make HaroCue an assistant that understands people better over time. Today made me feel that this “understanding” has to be grounded in concrete details: it knows who said a sentence, under what conditions, whether it later changed, and which parts it is still uncertain about.

The architecture work ahead should continue to revolve around those details.

Back to the blog →