The Field Guide to AI Memory Field Guide
V — The landscape · 15 min

How MemoryPlugin Works

A teach-first case study of one production memory system, how MemoryPlugin applies explicit facts, a chat-history recall pipeline, curation, and cross-tool injection, what we benchmarked to build it, and the one feature we built and parked.

The rest of this guide argues in the abstract: extraction, hybrid retrieval, conflict resolution, hierarchical summaries, portability. This page picks one system that had to commit to real answers and shows what those choices look like in production, including where they bite. The system is MemoryPlugin, which this wiki's authors build. Read it as a worked example, not a sales pitch. The interesting parts are the tradeoffs, and there is one behind every feature.

The bet is narrow, and worth saying out loud. Every big assistant now ships its own memory, and every one is a walled garden: ChatGPT remembers you inside ChatGPT, Claude inside Claude. MemoryPlugin's wager is that the memory should be yours, not the app's, and that it should travel between them. Two layers carry the bet, and the product teaches them as the notebook and the archive. The notebook is discrete, editable facts, short statements you can actually read and fix, built like the old ChatGPT memory rather than the newer summary blob nobody can see into. The archive is stranger and more useful: your chat history itself becomes the memory, synced and searchable, on the theory that you can never guess in advance which conversation will matter six weeks from now. One rule holds across both: the AI suggests and retrieves, it never silently writes. Nothing lands in your store without you, and you can see, edit, version, export, and delete all of it.

#The notebook: facts you can see and edit

#Explicit memories

Start with the oldest complaint in the field: the assistant forgets you the second the tab closes, so you introduce yourself again, and again. A memory here is a plain short fact ("works at MemoryPlugin", "prefers TypeScript"), kept in Postgres and embedded into a Zilliz vector collection with Voyage's voyage-3.5-lite. Two choices there are the embeddings page's lessons made concrete. First, you rarely need full-width vectors: these are truncated Matryoshka-style from 1024 down to 512 dimensions, which roughly halves storage and search cost for little quality loss. Second, dense vectors are bad at exact strings, so a BM25 keyword index rides alongside them to catch the names and IDs that a pure embedding blurs together. The dense side is indexed HNSW (an approximate-nearest-neighbour graph) under cosine. You add a fact by telling the assistant to remember it, or by accepting one it offers, and the write goes through a tool call, not a hidden text command. Delete or merge one and it leaves a soft-delete trail pointing at what it became, so nothing simply disappears.

The catch is the deliberate flip side of "no silent writes": capture is not automatic. Somebody has to decide a fact is worth keeping, even if that somebody is the model when you prompt it (rag vs memory and smart extraction pull that apart). And write-time dedup is blunt on purpose, it catches only byte-identical text in the same bucket. The near-duplicates that say the same thing in different words are the curator's problem, two sections down.

#Buckets

One global pile of facts goes bad fast. Work leaks into personal, project A smears into project B, and shoving all of it into every prompt burns tokens for nothing. Buckets are folders that keep memories apart. Every account gets an undeletable General bucket, and the bucket is the unit everything advanced runs on: suggestions, Smart Memory, and the knowledge graph all work per bucket, never across your whole store at once. The price: the tidying is on you, and a memory lives in exactly one bucket.

#Suggestions: the curator

Ask anyone who has run AI memory for a month what goes wrong and the answer is always the same. It rots. Duplicates breed, facts start contradicting each other, junk creeps in and drags down recall. The fix is an offline curator that runs per bucket. For each memory it pulls the nearest neighbours into a small cluster, hands them to an LLM (a Gemini Flash model), and asks for one of three moves: bin the junk, fold near-duplicates together (keeping the oldest as the anchor), or reconcile a fact that has changed ("works at Google" becoming "works at MemoryPlugin", carried as a "was X, now Y" progression instead of a quiet overwrite).

Two guardrails make it safe to point a model at your memory. Nothing is ever auto-applied: a suggestion is inert until you accept it, and you can edit the text first. And every ID the model hands back is checked against the input set, then re-checked for ownership, before a single write, because models invent IDs with total confidence. The full design is in memory suggestions, the deeper contradiction question in updates and conflicts. The limit worth naming: this is a batch you review, not live cleanup, and it leans toward keeping things, so some near-duplicates survive on purpose.

#Smart Memory: summaries first, load on demand

Even a spotless bucket has a ceiling. Inject all of it on every message and you bloat the context window, worse as the bucket grows. Smart Memory is the hierarchical dodge (hierarchical memory). Per bucket, an LLM sorts the memories into a handful of categories, writes a dense summary for each, and notes what detail is parked behind it and when fetching it is worth the trouble. At use time you get the summaries plus the most recent memories, and the assistant pulls a full category only when the conversation actually turns that way. Roughly 5,000 tokens of raw memory standing in as about 500. The cost: it needs enough memories to earn the overhead (the gate sits around 30), it skips very large buckets, the categorising is a manual action, and the categories set once and stay, so reorganising means starting over.

#The knowledge graph

Past a few hundred memories you lose the plot of your own bucket. The per-bucket knowledge graph draws the map: an LLM pipeline extracts entities, dedupes them deterministically by name, reviews and merges the variants, then pulls out directed relationships, with a deliberately stronger model (a Claude Sonnet) on the final review because nothing downstream cleans up after it. That last choice is a rule the whole system runs on. Use a fast, cheap model where a later stage will catch its mistakes; pay for the good one only where its output is the final word. What the graph is not, which knowledge graphs develops: it is something you look at, not something recall runs through, so it does not directly sharpen answers, and it is a second store to keep in sync, not a cure for facts that change over time.

#The archive: past conversations as memory

This layer is the part that makes MemoryPlugin different. You cannot predict which conversation will matter later, so do not try. Rather than distil facts up front, keep the raw history and search it on demand. (Across tools makes the portability case in full.)

Sync. Two ways in: upload an export for a fast first import, or let the browser extension trickle conversations in from supported platforms as you go. Each one is parsed and chunked (around 256 tokens, light overlap), with a single chunking rule applied on purpose: role and timestamp prefixes live as structured fields and are stripped from the embedded text, because baking them in just adds noise. It is strictly opt-in. And the scale forces the design: a real export can run to 20 million tokens, so "just paste it into the context window" was never on the table. Chunk, index, retrieve, or nothing.

#Recall, end to end

Indexing thousands of chats is pointless unless the right fragments surface on their own. Recall runs the full pipeline fresh on every query, with no cached answers, because the question is never quite the same twice. The stages:

  1. Expand the query. An LLM rewrites your question into a few variants, biased toward what you probably said back then rather than search-engine queries for the answer, and generates time filters when the question implies them. The verbatim original is re-prepended so a rare name or identifier does not get washed out by the rewrites.
  2. Search, hybrid. Each variant runs dense plus BM25 over the vector store, and the lists are fused by Reciprocal Rank Fusion. Why fuse by rank instead of summing raw scores is the entire argument of hybrid retrieval.
  3. Rerank. A cross-encoder (Voyage rerank-2.5-lite) reorders the merged pool against the intent.
  4. Assess relevance, per chunk. An LLM reads each surviving chunk and decides whether it actually answers the query, dropping the ones that merely look similar.
  5. Expand context. For the keepers, pull the neighbouring messages around each hit so the model sees the exchange, not a sentence ripped out of it.
  6. Summarize to a budget. Fold the survivors into a token budget (default around 2,000) with source citations back to the original conversation and date.

One stage dominates, and it is worth being precise about which. Query expansion and embedding are tens of milliseconds. The hybrid search is a hundred-odd milliseconds warm, more on a cold cache. Rerank and context expansion are a few hundred each. Then step 4, the per-chunk relevance LLM, reading every candidate one by one, runs to low single-digit seconds and swamps everything else combined. That step is the floor. Every speed decision in the system is really a decision about it: run it on a fast model, run the chunks in parallel, or, in the speed-tuned mode, make it lighter.

The reason any of this matters is a number from the sibling product. AskLibrary's first recall took the better part of 30 seconds, which is fine for "research this book" and useless for "remember this while I chat." Rebuilding it as a live memory layer meant landing the same answer in roughly two seconds, and almost all of that work was attacking step 4. What that does not fix: recall still costs you a few seconds, it leans on how well the host model wields tools, and flat chunks do not model how a belief evolves. Struggle with a topic in March and master it by June, and recall may surface the wrong era of you (temporal memory).

#Pick the parts on benchmarks, not vibes

A two-second budget decides components in a way leaderboards do not. The embedder has to be fast and steady under load, so the choice came down to measured latency: voyage-3.5-lite landed around 327 ms a call, while a cheaper open alternative (a Qwen embedder on DeepInfra) came in near 738 ms, which on its own would blow the budget before search even started. Voyage won on latency and consistency. That it was also cheaper, roughly $0.02 per million tokens against Cohere's $0.12, was a bonus, not the deciding factor. And the embedder swap itself (from BGE-M3 to voyage-3.5-lite) shipped as a 50/50 A/B judged on signals that actually mean something, whether people opened the cited source and whether they asked a follow-up, not an offline score that may not match how people actually read. The LLM steps run on fast open models (Groq-hosted) for the same reason the embedder does: when the job is "does this chunk answer the question," tokens per second buys more than a few points of reasoning.

#Summarize just in time, not ahead of time

The obvious way to make a giant history cheap to read is to summarize it in advance: roll each conversation into a digest once, store the digest, serve that. MemoryPlugin does not, and the reason is the bet again. A summary written ahead of time is written without the question, so it keeps what is generically important and drops whatever you turn out to want. Ask about one specific detail later and it is already gone. So summarization happens at recall time, shaped by the query that triggered it: the same conversation condenses differently depending on what you asked, because the goal is to answer you, not to file the conversation neatly. You pay for that on every read instead of once, which is the trade you make on purpose.

For conversations large enough that a single pass is neither affordable nor possible, the ones that run to a million tokens, it folds them down progressively: summarize a chunk, carry that running summary into the next chunk, repeat, keeping the working set near the size of one chunk instead of the whole. Cheap models do the folding; the budget is spent where it counts, on the final synthesis you actually read.

#Life Context

Raw history is too big to read, so a background job periodically renders it into a structured profile (work, personal, what is top of mind). The clever bit is what it refuses to believe: a topic has to turn up across several distinct conversations to count, and anything about other people, or material you were only helping with or critiquing, gets thrown out. It is the closest thing here to extracting facts from chat (smart extraction), done as a gated profile rather than silent per-message writes. The tradeoff: it is a snapshot, so it lags reality, and a genuinely new but rarely-mentioned fact is treated, by design, as probable noise.

#The one we parked: the timeline tool

The feature I was most excited to build is not live, and the reason it is dark is the most useful thing on this page.

The timeline tool answered a question the rest of the system cannot. Not "what do I know about X" but "tell me the story of X over time." Point it at a side project and it fanned your history into time buckets, from the last seven days back through the last year and earlier, pulled what you said in each window, and wrote the arc: where the thing started, what changed, where it stands now, with the milestones in order. The output was the good part. It read like something a thoughtful person who had watched the whole thing would write.

What killed it was arithmetic. The first build handed the whole relevant slice of your store to the model in one pass, and "the whole slice of a year" is enormous: a single timeline could ingest well over a million tokens and cost real money per call. Dollars, not fractions of a cent. You cannot put dollars-per-press behind a button people tap on a whim, and charging per timeline would have taught people not to touch the best thing in the product. So it is off.

It is parked, not abandoned, and the fix is unglamorous: the same map-reduce summarization the live recall path already uses. Fan the buckets out, summarize each one cheaply and in parallel, then stitch the summaries into the narrative, which turns one ruinous prompt into many trivial ones. The idea was never the problem. The bill was. When the cost comes down to the price of a normal recall, it comes back.

#The connective layer: MCP and injection

All of it is worthless trapped in one place, so it reaches out through a couple of channels. Where the assistant can call a tool, it does: a Model Context Protocol (MCP) server (Claude, ChatGPT's connectors, Cursor and other MCP clients), local or hosted-and-remote over OAuth, plus a Custom GPT with Actions on ChatGPT. But every one of those connectors is off until the user adds and enables it, and a default chat has none wired up, so a browser extension injects memory and history straight into the page, inside ChatGPT, Claude, Gemini and other web apps. That second path is harder than it sounds, and memory without tool calling is the page on why: injecting your own memory can read to a model like an attack, and there is a right way to do it. The tradeoffs in brief: the tool route only fires if the model decides to call it (a nudge in your instructions helps), and the extension rides each platform's page structure, so a vendor redesign can break it until it is patched.

#Images, files, and Ask

Three smaller pieces round it out, all premium and all sitting on top of the two core layers rather than inventing a third idea. Image memories let a text query surface a stored screenshot through a multimodal embedding (approximate, threshold-gated, in its own vector space). File buckets make uploaded PDFs, Word, and Markdown queryable with page-level citations (no scanned-image PDFs, 10 MB a file). And Ask is the one dashboard spot to query memories, history, or files and get cited answers back.

#Where it fits, and where it doesn't

It fits when you use several AI tools and want one memory across all of them, when you would rather see and correct what is stored than trust a black box, and when your best context is buried in chats you will never sit down and hand-curate. It fits badly if you need cryptographic privacy. The system is not end-to-end encrypted, and that is a deliberate call, not an oversight: the value comes from server-side search, summarisation, and synthesis, and real end-to-end encryption would kill all three. What you get instead is transparency and control: export, delete, audit, revoke. A tradeoff to weigh with your eyes open, not a guarantee. And it will not, today, reliably tell you which of two contradicting facts from your history is the current one. Nobody has solved that yet. It is the field's open problem, not a checkbox.

If you want to poke at the live version, it is at memoryplugin.com.

#References