dllama.tech · Knowledge Course KC 101-02
KC 101 · Foundations — Unit 02

Context windows & what fits

~12 min read · Written 2026-08-23 · Research: Mara · Slug context-compaction
Summary

Every LLM reads a fixed amount of text at once — its context window, measured in tokens, not words. That window is the model’s entire working memory: whatever isn’t inside it might as well not exist. Long conversations fill the window, so agent frameworks do context compaction: they summarize the older turns into a compact handoff and keep going. Compaction is lossy — it keeps the gist, the decisions, and the unfinished work, and discards most of the raw detail. That is why an agent in a very long session can “feel dumber”: it is working from a summary of the past instead of the past itself, and research shows long contexts degrade attention even before the window runs out. Knowing the window size, the compaction trigger, and the difference between “summarized” and “gone” is the difference between an agent that drifts and one that finishes.

AudienceSomeone who runs an agent stack daily and wants the layer under it.

KC 101 · Foundations · Status: published

01The context window is the model’s working memory

A token is the smallest unit a model reads — roughly a chunk of a word. A rule of thumb: an English word is about 1.3–1.5 tokens, so a 4,000-token window holds roughly 3,000 words, and a 128K window holds roughly a long novel.

Key idea

The context window is the maximum number of tokens the model can see at one time when generating a response. Everything in that window is “directly available”; anything outside it is not available at all unless it is summarized, retrieved, or provided again (Wikipedia’s exact phrasing — read it again, it is the whole lecture).

And the window isn’t just your chat history. It also holds:

  • the system prompt (the agent’s identity and rules),
  • every tool definition (function schemas are thousands of tokens each),
  • your messages and the agent’s replies,
  • tool results (file contents, terminal output, search hits),
  • attached documents, and
  • space for the reply itself (the model can’t output more than the remaining budget).

The model must attend to all of this at once. The more tokens pile in, the harder it is for the model to hold the thread — that is not a vibe, it’s the architecture: in a transformer, every token can attend to every other token, which grows quadratically, and models were mostly trained on shorter sequences than they’re asked to reason over at test time.

Karpathy’s mental model: he compares the context window to RAM in an LLM “operating system” — “your finite precious resource of your working memory.” The model can page in new information (tools, search, files), but it cannot hold more than RAM at once. Disk (long-term memory) is a different thing, and we’ll meet it later in the course (101-03, 201-03).

Jargon translation
Token budget — every request is priced and constrained by tokens. The context window is the ceiling on one request; your API bill is the total of all the tokens you’ve pushed through it. Bigger window ≠ better, it just means more can fit before something has to give.

02How a long session approaches the limit

A single question fits easily. But an agent session is a loop: you ask → the agent calls tools → tools return output → the agent reasons → it calls more tools → it replies. Every one of those turns is appended to the window. File reads, terminal dumps, and search results are the big eaters — a single cat of a large file can be tens of thousands of tokens.

On a 200K-token model, a real working session with tool output reaches ~50% of the window (100K tokens) in maybe an hour of real work. That is not a design flaw; it’s the physics of the loop. The session has three ways forward once it approaches the ceiling:

  1. Truncation — drop the oldest turns. Crude, silent, and it loses exactly the stuff you may still need (“what was that file path again?”).
  2. Compaction — summarize the old turns into a compact handoff and keep the recent ones verbatim. This is what well-built agents (Hermes, Claude Code, and others) do.
  3. Hard reset — start a fresh session with nothing but what the user re-states (or what was persisted to long-term memory beforehand).

Real agent frameworks use a combination: compact in the middle, and reset only when you deliberately want a clean slate.

In practice — Ollama

If you run local models, the default context is easy to get wrong. Ollama defaults num_ctx to a VRAM-dependent value (4,096 tokens on small machines, 32K with 24–48 GB VRAM, up to 256K on big GPUs) — and its FAQ historically said 4,096. A model that can handle 128K tokens will still truncate to 4K unless you set the context explicitly (/set parameter num_ctx 32768 or OLLAMA_CONTEXT_LENGTH). Hermes refuses to run agent mode below a 64K window precisely because system prompt + tool schemas + working state need room.

03Compaction — what the handoff actually is

Here is the mechanism, and it’s worth reading slowly because it generalizes to every agent you’ll use.

When the session crosses the compaction threshold, the framework takes the older turns (the “middle”), hands them to a summarizer model, and replaces them with a structured summary. The recent turns — the “tail” — stay verbatim. The next request the model sees is:

Hermes · compaction handoff — what the next request looks like 
[system prompt]                    ← unchanged, still authoritative
[user: the original first ask]     ← first exchange is protected
[CONTEXT COMPACTION — REFERENCE ONLY]
  ## Goal
  What the user is trying to accomplish
  ## Progress
  ### Done — specific files, commands, results
  ### In Progress — what's half-finished right now
  ### Blocked — blockers and issues
  ## Key Decisions — and why
  ## Relevant Files — what was read/created
  ## Next Steps
  ## Critical Context — error strings, config values, exact numbers
[recent messages, verbatim]        ← the tail

That is not a description of a hypothetical system — that is Hermes’s actual compression template, and the [CONTEXT COMPACTION — REFERENCE ONLY] banner you may have seen inside long sessions is exactly this handoff. Two details are important:

Gotcha — what survives, what’s lost

What survives: goals, decisions, file paths, commands run, error messages, what’s done, what’s in progress, what’s blocked, next steps. The summary is designed to answer “where were we?” after the middle is gone.

What’s lost: the raw tool outputs, the intermediate failed attempts, the exact wording of older exchanges, and anything the summarizer judged (or failed to judge) as unimportant. Old tool results are the first thing pruned — once a tool has been called deep in history, why carry the full output? — and they’re replaced with a stub like [Old tool output cleared to save context space].

The summarizer has a budget: typically ~20% of the compressed content, capped (Hermes: content_tokens × 0.20, min 2,000, max min(context × 0.05, 12,000)). A 95K-token session compacts to roughly 45K. On later compactions, the framework updates the previous summary rather than starting from scratch, so the “In Progress” from last time becomes “Done” this time.

The single most important sentence in the handoff: the summary is reference only. The agent is explicitly told it is background, not active instructions — respond only to the latest user message, don’t resurrect old tasks. This exists because agents reliably confuse “this was discussed earlier” with “I should do this now.” If you’ve ever watched an agent re-start a task you already told it to stop — that’s the failure mode compaction handoffs are built to prevent.

In practice — Hermes

Hermes runs a dual system. The agent-side compressor fires at 50% of the window with real API-reported token counts; a gateway “session hygiene” pass fires at 85% as a safety net for sessions that grew between turns (overnight Telegram/Discord accumulation). Compaction is in-place by default: the session keeps one durable ID, the pre-compaction turns are soft-archived (still searchable via session_search, never deleted), and /compress lets you trigger it manually. That “compaction is non-destructive — archived, not deleted” property is the reason a summary session can still find a lost detail: the framework recovers it by searching the archive instead of expecting the summary to have kept it.

In practice — Claude Code / Anthropic

Anthropic calls the same move “compaction” and describes the art precisely: “the art of compaction lies in the selection of what to keep versus what to discard, as overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later.” Their guidance for building compaction systems: tune for recall first (capture everything relevant), then tighten for precision.

04Why a long chat agent “feels dumber”

Three separate effects stack up, and they’re worth keeping apart.

  1. Compaction is lossy by design. The agent is now reasoning from a summary of the past. Summaries keep the gist and lose texture. A decision made in passing in turn 3 (“we’re using FastAPI, not Flask, because the user said so”) can be missing from the turn-3 summary, and the turn-60 agent confidently re-litigates it. The agent didn’t get worse — its inputs got thinner.
  2. “Lost in the middle” — long contexts degrade attention even before the limit. The famous result here is Liu et al. (Stanford), Lost in the Middle (TACL 2024, arXiv:2307.03172): when the relevant fact is placed in the middle of a long context, model performance drops significantly compared to the same fact near the start or end. Models don’t robustly use all the tokens they can hold. A bigger window doesn’t mean better use — several studies show performance can degrade as context fills, even under the cap. Anthropic calls this “context rot” and treats context as a finite resource with diminishing marginal returns: “Every new token introduced depletes [the attention] budget by some amount.”
  3. Instruction dilution and stale premises. A 100K-token window where 90K is old tool noise gives the latest instruction less relative weight. And if a fact the agent believed at turn 2 was contradicted at turn 50, the turn-2 version may still be present, contradicting the summary. This is why the compaction handoff says “the latest user message WINS.”

So the “dumber” feeling is real and diagnosable. When an agent in a long session starts repeating itself, ignoring recent instructions, or re-asking questions it already knew — check the context: it’s probably deep in a summary, not the live conversation.

05When a fresh session beats a summary

Compaction is not always the right move. A hard reset (Hermes: /new or /reset; a new chat thread; hermes chat fresh) wins in three situations:

  1. The past is a liability. If the conversation is mostly failed attempts, wrong paths, and superseded decisions, the summary will carry ghosts of them. Anthropic’s engineering team says it plainly: rewinding to a clean context preserves only correct implementations and reduces goal drift — you avoid carrying forward the sections where the agent deviated.
  2. The task is done and the next one is unrelated. “Summarize the past 3 hours of Python work” and “design a landing page” do not belong in one window. The compaction handoff itself tells the agent to treat the old task as reference-only — which is a polite way of saying start a fresh session instead.
  3. You need the agent to be maximal, not convenient. A fresh session with a tight system prompt, the key facts re-stated, and no 60 turns of baggage is the highest-performance configuration an agent can have. The user message after a reset carries the full attention budget.

When compaction wins: the work is continuous (a long migration, a multi-hour research sweep), the recent tail is still relevant, and losing the thread would be worse than losing the detail. Compaction maintains conversational flow; a reset changes identity.

Key idea

The practical rule: if you wouldn’t want the agent to act on something from an hour ago, it shouldn’t be in the window at all — reset, and re-state only what matters. If you would want it acted on, make sure it’s in the protected tail or the summary, not buried mid-context.

06It shows up in your stack

You run these tools daily; here’s what each does with context.

  • HermesDual compaction (50% agent-side, 85% gateway safety net), in-place session IDs, soft-archived pre-compaction turns searchable via session_search, manual /compress, session reset policies (none/idle/daily), and the [CONTEXT COMPACTION — REFERENCE ONLY] handoff you’ve seen in long sessions. session_search is the designed recovery path: when the summary misses a detail, Hermes searches the archived transcript instead of asking you to repeat it.
  • MnemosyneThe long-term semantic memory layer: it auto-captures every turn as embeddings outside the context window. This is exactly the “structured note-taking / agentic memory” pattern Anthropic describes: notes persisted out-of-window, pulled back in when needed. It’s the reason a fresh session isn’t amnesia — the durable facts survive in memory, not in the window.
  • Open DesignThe design-agent platform works the same way at the project level: a DESIGN.md + BRIEF.md in the working directory is the persistent context that survives individual run sessions, so each run starts with the important context pinned and doesn’t have to re-learn it. Same principle: put what matters in a file, not in the chat.
  • OllamaLocal inference defaults: 4K–256K context depending on VRAM, and truncation when you exceed it. Check ollama ps; if your local agent “forgets” mid-task, the most likely cause is a 4K window silently chopping your history.

The through-line: the window is for what’s happening now; the filesystem, memory store, and retrieval are for what happened before. Everything you move out of the window is either summarized (lossy), archived (recoverable), or persisted to memory (durable). Knowing which of those three happened to any given fact is the skill.

07Key takeaway

Key idea

The context window is a finite working memory, not a file cabinet. Long sessions must either summarize, truncate, or reset — and every summarization loses detail, which is why long-chat agents drift. Run your stack with intent: keep the important context in the protected tail or in persistent memory, compact when the work is continuous, reset when the past is baggage, and treat every compaction as a lossy handoff you can recover from — never as the full record.

08Go deeper

Top references (free)
  1. [1]

    Karpathy — “Intro to Large Language Models” (YouTube, 1 hr)

    https://www.youtube.com/watch?v=zjkBMFhNj_g

    The “context window = RAM” analogy around minute ~28, plus the LLM-as-OS framing. The best 60-minute mental model of what a window is.

  2. [2]

    Wikipedia — “Context window”

    https://en.wikipedia.org/wiki/Context_window

    The canonical definition (“anything outside that window is not directly available unless it is summarized, retrieved, or provided again”), tokenization, and how window sizes grew from 512 tokens (2018) to millions.

  3. [3]

    Liu et al. — “Lost in the Middle: How Language Models Use Long Contexts” (arXiv:2307.03172, TACL 2024)

    https://arxiv.org/abs/2307.03172

    The evidence that bigger windows aren’t better-used windows: position matters, and performance degrades when the relevant info sits mid-context.

  4. [4]

    Anthropic — “Effective context engineering for AI agents” (engineering post, free)

    https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

    Compaction defined, the keep-vs-discard tradeoff, context rot, and the three long-horizon strategies (compaction, note-taking, sub-agents).

Tool-specific docs
  • —

    Hermes — Context Compression and Caching (developer guide)

    https://hermes-agent.nousresearch.com/docs/developer-guide/context-compression-and-caching

    The actual algorithm: 50%/85% thresholds, the summary template, tail protection, in-place compaction, before/after examples with real token counts.

  • —

    Hermes — Sessions (user guide)

    https://hermes-agent.nousresearch.com/docs/user-guide/sessions

    Session reset policies, session_search, continuity after crashes, recap on resume.

  • —

    Ollama — FAQ / Context length (faq, context-length)

    https://docs.ollama.com/faq · https://docs.ollama.com/context-length

    num_ctx defaults, OLLAMA_CONTEXT_LENGTH, VRAM-based sizing.

Research: MaraDesign: Kyra + Open DesignPowered by Hermes Agent