Memory: working vs long-term, embeddings, RAG

Working01The taxonomy: working, episodic, semantic (and the Mnemosyne report, decoded)

Cognitive scientists spent decades sorting human memory into types, and the agent literature (via CoALA, the “Cognitive Architectures for Language Agents” framework) imported the taxonomy wholesale. Four types matter:

CoALA’s precise definitions (arXiv:2309.02427): episodic memory stores “experience from earlier decision cycles” — instance-specific, order-preserving; semantic memory stores the generalized facts you can derive from many episodes; procedural memory stores skills. Working memory is not a store at all — it’s the stage where the agent’s current decision loop operates.

The distinction between episodic and semantic is the one to internalize, because it’s the one every tool gets wrong:

Decoding the Mnemosyne report you saw (audit 2026-08-23). The memory-keeper cron reported tiers as canonical / recalled / episodic, and the terms were never explained. Here is what they mean in the tool you actually run:

• Working memory — Mnemosyne’s hot tier. Auto-injected into the prompt before LLM calls, session-scoped by default, evicted by TTL (time-to-live). This is what the report meant by “recalled”: it’s the tier that gets pulled up into working context.
• Episodic memory — the long-term store: full events and facts with vector + full-text search. This is the “episodic” tier.
• Canonical facts / knowledge triples — the consolidated semantic store: durable facts (“source=user-profile-backup, importance 9”) and a temporal knowledge graph of subject–relation–object triples. This is the “canonical” tier — the encyclopedia.
• (A fourth export key, legacy_memories, holds older entries that consolidation moved out of the hot tier — the archive.)

EPISODIC  ──consolidation──>  CANONICAL  ──retrieval──>  RECALLED
the diary                     the encyclopedia          the desk
raw events, in order          distilled facts+triples   hot, auto-injected
                                     |
                              LEGACY_MEMORIES  <- the archive

Semantic02Embeddings: how text becomes numbers

An embedding is a list of floating-point numbers — a vector — that represents a piece of text. “The cat sat on the mat” becomes a list of, say, 768 numbers. That transformation is the entire foundation of semantic search, RAG, and every “similarity” feature in your stack, so it’s worth building from first principles.

What a vector is. A vector is just a point in N-dimensional space — an address. With 2 numbers you get a point on a flat map; with 3, a point in a room; with 768, a point in a space you can’t picture but whose geometry still works. The trick of embeddings: the address encodes the meaning. Texts with similar meanings land near each other; unrelated texts land far apart.

Where the numbers come from. Inside every LLM there is an embedding matrix — literally the first pile of weights. 3Blue1Brown’s transformer explainer walks the arithmetic: for GPT-3, the vocabulary is ~50,000 tokens and the embedding dimension is 12,288, so that first matrix holds ~617 million numbers — one 12,288-dimensional vector per token. The model learned those vectors during training, by the distributional hypothesis: “you shall know a word by the company it keeps” (Firth, 1957). Words that appear in similar contexts end up with similar vectors. “cat” and “dog” co-occur with the same neighborhoods — “the ___ sat”, “my ___ is sick” — so their vectors drift together. This is why the famous word2vec arithmetic works: king − man + woman ≈ queen. The geometry of the space encodes relationships between meanings.

Why not just use one-hot vectors? The naive way to encode a word is a vector of zeros with a single 1 at the word’s index — a 50,000-dim vector where “cat” and “dog” share zero overlap. One-hot encodes identity, nothing about meaning. Dense embeddings compress meaning into a few hundred dimensions where distance itself is information. Every embedding model is a small neural network trained so that semantically-related texts produce nearby vectors — sentence-level embeddings (like nomic-embed-text) do this for whole phrases, not just words.

curl -X POST http://localhost:11434/api/embed \
  -d '{"model": "nomic-embed-text",
       "input": "The sky is blue because of Rayleigh scattering"}'

# => {"embeddings": [[ ...768 floats... ]]}  — L2-normalized to unit length

Semantic03Cosine similarity: the distance that means “meaning”

Once texts are vectors, “are these related?” becomes “are these points near each other?” The standard measure is cosine similarity: the cosine of the angle between two vectors, ranging from −1 to +1 (+1 = same direction, 0 = unrelated/orthogonal, −1 = opposite).

Why cosine and not plain distance? Because raw vectors vary in length as well as direction, and length usually encodes irrelevant stuff (how long the text is, how common the words are). Cosine throws away length and keeps only direction — “which way does this meaning point?” — which is almost always what you care about. And when vectors are L2-normalized (unit length, as Ollama returns), cosine similarity collapses to the simple dot product: a single multiplication-and-sum per pair, fast enough to do millions of comparisons.

Why semantic search works. Keyword search fails on paraphrase: “how do I prune my context window?” shares zero words with the stored note “Hermes compacts at 50% of the window” — yet they mean the same thing. Embedding both sentences puts them near each other, because both contain “context/compaction”-shaped meaning. Semantic search = embed the query, embed everything, sort by cosine, take the top-k. That’s it. No magic — just a learned space where meaning is geometry.

Store04RAG: retrieve-then-generate

RAG (Retrieval-Augmented Generation) is the pattern that connects long-term memory back to the working window. The original paper — Lewis et al., NeurIPS 2020, arXiv:2005.11401 — framed it as combining two kinds of memory in one model:

• Parametric memory: knowledge stored inside the LLM’s weights, learned at training time.
• Non-parametric memory: knowledge stored outside the model, in a searchable index, retrieved on demand.

The paper’s claim, still the pitch today: retrieval-based generation produces “more specific, diverse, and factual” output than generation from the frozen weights alone — because the model doesn’t have to guess what it can look up.

The pipeline, end to end:

1. Chunk — split your documents into pieces (a few hundred to a couple thousand tokens each).
2. Embed — run every chunk through an embedding model; store the vectors.
3. Index — put them in a vector store: an index optimized for nearest-neighbor search (approximate nearest neighbor, ANN — e.g., HNSW graph indexes) rather than exact scans. “Vector database” is marketing; underneath it’s a fast index over vectors, usually on top of Postgres (pgvector), SQLite (sqlite-vec), or a dedicated store.
4. Retrieve — at query time, embed the user’s question, find the top-k nearest chunks by cosine.
5. Generate — stuff the retrieved chunks into the prompt above the question (“Here is context from the docs… now answer”), and let the model answer with the facts in front of it.

RETRIEVE  ──>  STUFF  ──>  GENERATE

RETRIEVE   embed the question · cosine top-k against the vector index
STUFF      place the retrieved chunks ABOVE the question in the prompt
GENERATE   the model answers with those chunks in the window
           ^ the model is unchanged. RAG only changes what goes in.

Where RAG fits vs the 101-02 machinery. Three different ways out-of-window knowledge gets back in, and they must not be confused:

• Compaction summarizes old working memory into a lossy handoff — good for “where were we?”, bad for detail.
• RAG fetches specific long-term content on demand — good for “what does the docs/notes say about X?”, bad for reconstructing a conversation.
• Reset + re-state drops everything and lets you start clean — the maximal, zero-baggage option.

A well-built agent uses all three: compact mid-session, RAG into the window when a task needs stored knowledge, and reset when the past is baggage.

All tiers05The memory hierarchy in your stack

You run this taxonomy daily. Here is where each piece sits:

• Hermes (built-in) MEMORY.md / USER.md are the semantic tier made working: tiny, always-injected every turn, costing tokens every message. That’s the L0 discipline: hard rules and identity only, everything else pushed down to the long-term tier (Mnemosyne) and pulled back up by retrieval. Hermes also has an external memory provider slot (memory.provider: mnemosyne). Per the Hermes docs, when a provider is active the framework: injects provider context into the system prompt, prefetches relevant memories before each turn (background, non-blocking), syncs conversation turns to the provider after each response, extracts memories at session end, and mirrors built-in memory writes. That prefetch step is RAG-style retrieval running invisibly — the provider picks what you’re likely to need and places it in the working window before you ask.
• Mnemosyne The concrete three-tier system from section 01: working (hot, auto-injected, TTL), episodic (long-term, hybrid vector+FTS+importance search), triples (temporal knowledge graph). Its auto-consolidation — “sleep cycles” that summarize old working memories into episodic summaries — is the episodic→semantic consolidation move from CoALA and the Generative Agents reflection loop (arXiv:2304.03442), automated.
• MCP memory server The official Model Context Protocol memory server (@modelcontextprotocol/server-memory) is a knowledge graph, not a vector store: entities, relations, observations, persisted across chats. create_entities("John Smith", person, ["Speaks fluent Spanish"]), create_relations(John, works_at, Acme), search_nodes. This is semantic memory as graph: precise, queryable relations (“which hosts run the design daemon?”) instead of fuzzy similarity. The trade-off that matters: vectors answer “what’s most related?”, graphs answer “what exactly connects to what?” — serious stacks keep both and route the query. Mnemosyne’s triples tier is the graph side.
• Ollama (local embeddings) The retrieval engine on your own hardware: nomic-embed-text (137M params, 768-dim, 274 MB) or mxbai-embed-large (1024-dim). A fully local RAG loop — chunk a directory, embed with Ollama, store in SQLite with sqlite-vec, retrieve by cosine, generate with a local model — runs with no cloud dependency, which is how you’d teach yourself the whole pipeline in an afternoon.
• Open Design The design-agent platform’s DESIGN.md / BRIEF.md in the working directory is project-level semantic memory: durable facts survive individual runs because they live in files, not in the window. The principle from 101-02 repeats: what matters goes in a file or a store; the window is just the stage.

mnemosyne store   "Mnemosyne is the L1 memory provider"  # importance weight 1-10
mnemosyne recall  "user profile backup"                  # semantic search
mnemosyne stats
mnemosyne export                                         # tier keys below

working_memory · episodic_memory · legacy_memories · canonical_facts · triples

Working06Why it matters: “forgot” is almost never erasure

Put the whole lecture together and one practical conclusion drops out: when an agent “forgets”, it almost never lost the information — it failed to retrieve it into the working window. The fact is in episodic or canonical storage; the retrieval policy (or the lack of one) didn’t page it up.

That reframes your debugging:

• If a fact was never stored → a capture problem (fix the write path: mnemosyne store, auto-capture, nightly compaction).
• If a fact was stored but not found → a retrieval problem (fix the query: top-k too small, wrong embedding model, keyword missing for identifiers, importance weighting too low).
• If a fact was found but not used → a window problem (it got retrieved but pushed out by other content, or buried mid-context — the 101-02 “lost in the middle” effect).

And it reframes your architecture: don’t grow the always-injected tier when the agent forgets — that’s the wrong fix and it costs tokens on every message. Store to long-term, and make retrieval cheap and eager. The skill the whole course has been building toward is knowing, for any given fact: which tier does it live in, and what moves it up?

Working07Key takeaway

08Go deeper

• [1]

  CoALA — Sumers, Yao, Narasimhan, Griffiths. “Cognitive Architectures for Language Agents” (HTML v3, TMLR)

  arxiv.org/abs/2309.02427 · arxiv.org/html/2309.02427v3

  The canonical taxonomy: working vs episodic vs semantic vs procedural memory for agents, and how existing agent systems slot into it. This is the paper the whole lecture’s vocabulary comes from.

• [2]

  3Blue1Brown — “Transformers, the tech behind LLMs” (Deep Learning Chapter 5) (lesson page)

  youtube.com/watch?v=wjZofJX0v4M · 3blue1brown.com/lessons/gpt

  The embedding matrix made visual: GPT-3’s 50,257-token vocabulary, 12,288-dimension embedding, and exactly how text becomes numbers before attention does anything. The single best 90 minutes for section 02.

• [3]

  Lewis et al. — “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (NeurIPS 2020)

  arxiv.org/abs/2005.11401

  The original RAG paper: parametric vs non-parametric memory, the retrieve-then-generate framing, and the “more specific, diverse, and factual” result.

• ·

  MemGPT — “Towards LLMs as Operating Systems”

  arxiv.org/abs/2310.08560

  Main context as RAM, external context as disk, an agent-side memory controller. The OS metaphor made concrete.

• ·

  Generative Agents — Park et al. (Stanford)

  arxiv.org/abs/2304.03442

  The memory stream + retrieval + reflection loop; the canonical implementation of episodic→semantic consolidation.

• ·

  Stanford CS224N — Lecture 1, Intro & Word Vectors (2024 playlist, course page)

  youtube.com/playlist?list=PLoROMvodv4rOaMFbaqxPDoLWjDaRAdP9D · stanford.edu/class/cs224n

  Where word embeddings come from, taught properly, free.

• ·

  Jurafsky & Martin, Speech and Language Processing (3rd ed. draft)

  web.stanford.edu/~jurafsky/slp3/

  The “Vector Semantics and Embeddings” chapters: the distributional hypothesis, cosine similarity, and evaluation of embeddings in textbook form.

• ·

  Wikipedia — Word embedding, Cosine similarity, Retrieval-augmented generation, Episodic memory, Semantic memory, Working memory

  en.wikipedia.org/wiki/Word_embedding · Cosine_similarity · Retrieval-augmented_generation · Episodic_memory · Semantic_memory · Working_memory

  The cognitive-science definitions behind the taxonomy, all free.

• ·

  Hermes — Memory Providers

  hermes-agent.nousresearch.com/docs/user-guide/features/memory-providers

  The provider slot, the six automatic behaviors (inject context, prefetch, sync turns, extract on session end, mirror writes, expose tools), and the provider comparison table.

• ·

  Mnemosyne

  github.com/AxDSan/mnemosyne

  The three-tier BEAM architecture (working/episodic/scratchpad + triples), hybrid 50% vector / 30% FTS / 20% importance scoring, auto-consolidation sleep cycles, and the Hermes integration guide.

• ·

  MCP Memory Server

  github.com/modelcontextprotocol/servers/tree/main/src/memory

  The knowledge-graph memory server: entities, relations, observations; create_entities, create_relations, search_nodes.

• ·

  Ollama — Embeddings (nomic-embed-text)

  docs.ollama.com/capabilities/embeddings · ollama.com/library/nomic-embed-text

  /api/embed, L2-normalized output, and running embedding models locally for a zero-cloud RAG loop.