Memory: working vs long-term, embeddings, RAG

Working01The taxonomy: working, episodic, semantic (and the Mnemosyne report, decoded)

Four memory types matter (CoALA, “Cognitive Architectures for Language Agents”):

CoALA (arXiv:2309.02427): episodic memory is “experience from earlier decision cycles”, order-preserving; semantic memory holds facts derived from episodes; procedural memory holds skills. Working memory is no store.

Every tool gets the episodic/semantic split wrong:

Decoding the Mnemosyne report (audit 2026-08-23). The memory-keeper cron reported tiers as canonical / recalled / episodic, unexplained. In the tool you run:

• Working memory — Mnemosyne’s hot tier: auto-injected before LLM calls, session-scoped by default, evicted by TTL (time-to-live). The report’s “recalled”, pulled into working context.
• Episodic memory — the long-term store: events and facts with vector + full-text search. This is “episodic”.
• Canonical facts / knowledge triples — the consolidated semantic store: durable facts (“source=user-profile-backup, importance 9”) plus a temporal knowledge graph of subject–relation–object triples. This is “canonical”, the encyclopedia.
• (A fourth export key, legacy_memories, holds older entries consolidation moved out of the hot tier.)

EPISODIC  ──consolidation──>  CANONICAL  ──retrieval──>  RECALLED
the diary                     the encyclopedia          the desk
raw events, in order          distilled facts+triples   hot, auto-injected
                                     |
                              LEGACY_MEMORIES  <- the archive

Semantic02Embeddings: how text becomes numbers

An embedding is a vector: floating-point numbers representing text. “The cat sat on the mat” becomes, say, 768 numbers. It underlies semantic search, RAG, and every “similarity” feature in your stack.

What a vector is. A point in N-dimensional space: an address. Two numbers plot a flat map; three, a room; 768, a space you can’t picture but whose geometry works. The address encodes meaning: similar meanings land near each other, unrelated texts far apart.

Where the numbers come from. Every LLM has an embedding matrix, its first pile of weights. 3Blue1Brown’s transformer explainer walks the arithmetic: GPT-3’s vocabulary is ~50,000 tokens and its embedding dimension 12,288, so the matrix holds ~617 million numbers, one 12,288-dimensional vector per token.

The model learned those vectors by the distributional hypothesis: “you shall know a word by the company it keeps” (Firth, 1957). Words in similar contexts get similar vectors, so “cat” and “dog” drift together, co-occurring with “the ___ sat” and “my ___ is sick”. This is why word2vec works: king − man + woman ≈ queen.

Why not one-hot vectors? One-hot is the naive encoding: zeros with a single 1 at the word’s index. In 50,000 dimensions, “cat” and “dog” share zero overlap; it encodes identity, not meaning. Dense embeddings compress meaning into a few hundred dimensions where distance is information. Embedding models are small networks trained so related texts land nearby; nomic-embed-text does this for phrases.

curl -X POST http://localhost:11434/api/embed \
  -d '{"model": "nomic-embed-text",
       "input": "The sky is blue because of Rayleigh scattering"}'

# => {"embeddings": [[ ...768 floats... ]]}  — L2-normalized to unit length

Semantic03Cosine similarity: the distance that means “meaning”

Once texts are vectors, “are these related?” becomes “are these points near each other?” The measure is cosine similarity: the cosine of the angle between two vectors, from −1 to +1. +1 same direction, 0 unrelated, −1 opposite.

Raw vectors vary in length as well as direction, and length usually encodes irrelevant things (text length, word frequency). Cosine keeps direction, answering “which way does this meaning point?” When vectors are L2-normalized (as Ollama returns), cosine collapses to the dot product: one multiplication-and-sum per pair, fast enough for millions of comparisons.

Why semantic search works. Keyword search fails on paraphrase: “how do I prune my context window?” shares zero words with “Hermes compacts at 50% of the window”, yet means the same. Both embed near each other. Semantic search: embed the query, embed everything, sort by cosine, take the top-k. Meaning becomes geometry.

Store04RAG: retrieve-then-generate

RAG (Retrieval-Augmented Generation) connects long-term memory to the working window. The original paper (Lewis et al., NeurIPS 2020, arXiv:2005.11401) framed it as two memories in one model:

• Parametric memory: knowledge inside the LLM’s weights, learned at training time.
• Non-parametric memory: knowledge outside the model, in a searchable index, retrieved on demand.

The claim, still the pitch: retrieval-based generation produces “more specific, diverse, and factual” output than frozen weights alone, because the model needn’t guess what it can look up.

The pipeline, end to end:

1. Chunk — split documents into pieces (a few hundred to a couple thousand tokens each).
2. Embed — run every chunk through an embedding model; store the vectors.
3. Index — put them in a vector store: an index optimized for nearest-neighbor search (approximate nearest neighbor, ANN — e.g., HNSW graph indexes), not exact scans. “Vector database” is marketing; underneath it is a fast vector index, usually Postgres (pgvector), SQLite (sqlite-vec), or a dedicated store.
4. Retrieve — embed the question, find the top-k nearest chunks by cosine.
5. Generate — stuff the retrieved chunks into the prompt above the question (“Here is context from the docs… now answer”); the model answers with them.

RETRIEVE  ──>  STUFF  ──>  GENERATE

RETRIEVE   embed the question · cosine top-k against the vector index
STUFF      place the retrieved chunks ABOVE the question in the prompt
GENERATE   the model answers with those chunks in the window
           ^ the model is unchanged. RAG only changes what goes in.

Where RAG fits vs the 101-02 machinery. Three ways out-of-window knowledge returns:

• Compaction summarizes old working memory into a lossy handoff — good for “where were we?”, bad for detail.
• RAG fetches specific long-term content on demand — good for “what do the notes say about X?”, bad for reconstructing a conversation.
• Reset + re-state drops everything and starts clean — the zero-baggage option.

A well-built agent uses all three: compact mid-session, RAG when a task needs stored knowledge, reset when the past is baggage.

All tiers05The memory hierarchy in your stack

You run this taxonomy daily:

• Hermes (built-in) MEMORY.md / USER.md are the semantic tier made working: tiny, injected every turn, costing tokens every message. L0 discipline: hard rules and identity only; the rest lives in Mnemosyne, pulled back by retrieval. Hermes also has an external memory provider slot (memory.provider: mnemosyne). Per the docs, an active provider injects context into the system prompt, prefetches relevant memories before each turn (background, non-blocking), syncs turns after each response, extracts memories at session end, and mirrors writes. That prefetch is invisible RAG-style retrieval.
• Mnemosyne The three-tier system from section 1: working (hot, auto-injected, TTL), episodic (long-term, hybrid vector+FTS+importance search), triples (temporal knowledge graph). Auto-consolidation (“sleep cycles” summarizing old working memories) automates the CoALA episodic→semantic move and the Generative Agents reflection loop (arXiv:2304.03442).
• MCP memory server The official Model Context Protocol memory server (@modelcontextprotocol/server-memory) is a knowledge graph, not a vector store: entities, relations, observations, persisted across chats. create_entities("John Smith", person, ["Speaks fluent Spanish"]), create_relations(John, works_at, Acme), search_nodes. That is semantic memory as graph: precise relations (“which hosts run the design daemon?”) instead of fuzzy similarity. Vectors answer “what’s most related?”, graphs answer “what exactly connects to what?” Serious stacks keep both; Mnemosyne’s triples tier is the graph side.
• Ollama (local embeddings) Local retrieval on your hardware: nomic-embed-text (137M params, 768-dim, 274 MB) or mxbai-embed-large (1024-dim). A fully local RAG loop (chunk a directory, embed, store in SQLite with sqlite-vec, retrieve by cosine, generate locally) runs with no cloud dependency.
• Open Design The platform’s DESIGN.md / BRIEF.md in the working directory: project-level semantic memory, durable because it lives in files, not the window. The 101-02 principle repeats: what matters goes in a file or a store; the window is just the stage.

mnemosyne store   "Mnemosyne is the L1 memory provider"  # importance weight 1-10
mnemosyne recall  "user profile backup"                  # semantic search
mnemosyne stats
mnemosyne export                                         # tier keys below

working_memory · episodic_memory · legacy_memories · canonical_facts · triples

Working06Why it matters: “forgot” is almost never erasure

When an agent “forgets”, it almost never lost the information. It failed to retrieve it into the working window. The fact sits in episodic or canonical storage; the retrieval policy didn’t page it up.

That reframes your debugging:

• If a fact was never stored → a capture problem (fix the write path: mnemosyne store, auto-capture, nightly compaction).
• If a fact was stored but not found → a retrieval problem (fix the query: top-k too small, wrong embedding model, keyword missing for identifiers, importance weighting too low).
• If a fact was found but not used → a window problem (retrieved but pushed out, or buried mid-context — the 101-02 “lost in the middle” effect).

It also reframes your architecture: don’t grow the always-injected tier when the agent forgets; that costs tokens every message and fixes nothing. Store to long-term, and make retrieval cheap and eager. The skill the course builds toward: for any fact, which tier holds it and what moves it up?

Working07Key takeaway

08Go deeper

• [1]

  CoALA — Sumers, Yao, Narasimhan, Griffiths. “Cognitive Architectures for Language Agents” (HTML v3, TMLR)

  arxiv.org/abs/2309.02427 · arxiv.org/html/2309.02427v3

  The canonical agent memory taxonomy, and the source of this lecture’s vocabulary.

• [2]

  3Blue1Brown — “Transformers, the tech behind LLMs” (Deep Learning Chapter 5) (lesson page)

  youtube.com/watch?v=wjZofJX0v4M · 3blue1brown.com/lessons/gpt

  The embedding matrix made visual: GPT-3’s 50,257-token vocabulary, 12,288-dimension embedding, and how text becomes numbers before attention. The best 90 minutes for section 2.

• [3]

  Lewis et al. — “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (NeurIPS 2020)

  arxiv.org/abs/2005.11401

  The original RAG paper: parametric vs non-parametric memory, retrieve-then-generate, and the “more specific, diverse, and factual” result.

• ·

  MemGPT — “Towards LLMs as Operating Systems”

  arxiv.org/abs/2310.08560

  Main context as RAM, external context as disk, an agent-side memory controller.

• ·

  Generative Agents — Park et al. (Stanford)

  arxiv.org/abs/2304.03442

  The memory stream + retrieval + reflection loop; the canonical episodic→semantic consolidation.

• ·

  Stanford CS224N — Lecture 1, Intro & Word Vectors (2024 playlist, course page)

  youtube.com/playlist?list=PLoROMvodv4rOaMFbaqxPDoLWjDaRAdP9D · stanford.edu/class/cs224n

  Where word embeddings come from, taught properly, free.

• ·

  Jurafsky & Martin, Speech and Language Processing (3rd ed. draft)

  web.stanford.edu/~jurafsky/slp3/

  The “Vector Semantics and Embeddings” chapters: distributional hypothesis, cosine similarity, embedding evaluation.

• ·

  Wikipedia — Word embedding, Cosine similarity, Retrieval-augmented generation, Episodic memory, Semantic memory, Working memory

  en.wikipedia.org/wiki/Word_embedding · Cosine_similarity · Retrieval-augmented_generation · Episodic_memory · Semantic_memory · Working_memory

  The cognitive-science definitions behind the taxonomy.

• ·

  Hermes — Memory Providers

  hermes-agent.nousresearch.com/docs/user-guide/features/memory-providers

  The provider slot, the six automatic behaviors (inject context, prefetch, sync turns, extract on session end, mirror writes, expose tools), and the comparison table.

• ·

  Mnemosyne

  github.com/AxDSan/mnemosyne

  The three-tier BEAM architecture (working/episodic/scratchpad + triples), hybrid 50% vector / 30% FTS / 20% importance scoring, auto-consolidation sleep cycles, and the Hermes integration guide.

• ·

  MCP Memory Server

  github.com/modelcontextprotocol/servers/tree/main/src/memory

  The knowledge-graph server: entities, relations, observations; create_entities, create_relations, search_nodes.

• ·

  Ollama — Embeddings (nomic-embed-text)

  docs.ollama.com/capabilities/embeddings · ollama.com/library/nomic-embed-text

  /api/embed, L2-normalized output, and local embedding models for a zero-cloud RAG loop.