KC 101-02 · Context windows & what fits

01The context window is the model’s working memory

A token is the smallest unit a model reads, roughly a word chunk. An English word is about 1.3–1.5 tokens, so a 4,000-token window holds roughly 3,000 words; a 128K window holds roughly a long novel.

Key idea

The context window is the maximum number of tokens the model can see at one time when generating a response. Everything inside is “directly available”; anything outside is unavailable unless it is summarized, retrieved, or provided again (Wikipedia’s exact phrasing).

It also holds:

the system prompt (identity and rules),
every tool definition (schemas run thousands of tokens each),
your messages and the agent’s replies,
tool results (file contents, terminal output, search hits),
attached documents, and
space for the reply itself (output can’t exceed the remaining budget).

More tokens make attention harder: every token attends to every other, which grows quadratically. Models were trained on shorter sequences than they reason over at test time.

Karpathy’s mental model: the window is RAM in an LLM “operating system”, “your finite precious resource of your working memory.” It can page in data (tools, search, files) but never exceed RAM. Disk (long-term memory) is different; more later (101-03, 201-03).

Jargon translation

Token budget — every request is priced and constrained by tokens. The window is the ceiling on one request; your bill totals every token pushed through it. Bigger window ≠ better; more just fits before something gives.

02How a long session approaches the limit

An agent session is a loop: ask → tools run → output returns → the agent reasons → more tools → a reply. File reads, terminal dumps, and search results are the big eaters; one cat of a large file can be tens of thousands of tokens.

On a 200K-token model, a real session with tool output hits ~50% of the window (100K tokens) in maybe an hour of real work. Three ways forward:

Truncation — drop the oldest turns. Crude and silent; it loses what you may still need.
Compaction — summarize old turns into a compact handoff, keeping recent ones verbatim (Hermes, Claude Code, others).
Hard reset — start fresh with only what the user re-states (or what was persisted to long-term memory).

Frameworks combine them: compact in the middle, reset for a clean slate.

In practice — Ollama

num_ctx defaults to a VRAM-dependent value (4,096 tokens on small machines, 32K with 24–48 GB VRAM, up to 256K on big GPUs), and its FAQ historically said 4,096. A model that can handle 128K still truncates to 4K unless you set context (/set parameter num_ctx 32768 or OLLAMA_CONTEXT_LENGTH). Hermes refuses agent mode below a 64K window; system prompt + tool schemas + working state need room.

03Compaction — what the handoff actually is

At the threshold, the framework takes the older turns (the “middle”) and summarizes them into a structured summary. The recent turns (the “tail”) stay verbatim.

The next request:

Hermes · compaction handoff — what the next request looks like

[system prompt]                    ← unchanged, still authoritative
[user: the original first ask]     ← first exchange is protected
[CONTEXT COMPACTION — REFERENCE ONLY]
  ## Goal
  What the user is trying to accomplish
  ## Progress
  ### Done — specific files, commands, results
  ### In Progress — what's half-finished right now
  ### Blocked — blockers and issues
  ## Key Decisions — and why
  ## Relevant Files — what was read/created
  ## Next Steps
  ## Critical Context — error strings, config values, exact numbers
[recent messages, verbatim]        ← the tail

This is Hermes’s actual compression template.

Gotcha — what survives, what’s lost

What survives: goals, decisions, file paths, commands run, error messages, status (done, in progress, blocked), and next steps.

What’s lost: raw tool outputs, intermediate failed attempts, older exchanges’ exact wording, and whatever the summarizer judged unimportant. Old tool results are pruned first and replaced with a stub like [Old tool output cleared to save context space].

The summarizer’s budget is ~20% of the compressed content, capped (Hermes: content_tokens × 0.20, min 2,000, max min(context × 0.05, 12,000)). A 95K-token session compacts to roughly 45K. Later compactions update the previous summary, so “In Progress” becomes “Done”.

The handoff’s most important line: the summary is reference only. The agent must respond to the latest user message and not resurrect old tasks.

In practice — Hermes

Hermes runs a dual system. The agent-side compressor fires at 50% of the window on real API-reported token counts. A gateway “session hygiene” pass fires at 85% as a safety net for sessions that grew between turns (overnight Telegram/Discord accumulation). Compaction is in-place: one durable session ID, pre-compaction turns soft-archived (searchable via session_search, never deleted), and /compress triggers it manually. A lost detail is still recoverable from the archive.

In practice — Claude Code / Anthropic

Anthropic calls the same move “compaction” and names the art: “the art of compaction lies in the selection of what to keep versus what to discard, as overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later.” Guidance: tune for recall first, then tighten for precision.

04Why a long chat agent “feels dumber”

Three effects stack up:

Compaction is lossy by design. The agent reasons from a summary, which keeps the gist and drops texture. A decision made in passing at turn 3 (“we’re using FastAPI, not Flask, because the user said so”) can be missing from that summary. The turn-60 agent then re-litigates it.
“Lost in the middle” — long contexts degrade attention before the limit. Liu et al. (Stanford), Lost in the Middle (TACL 2024, arXiv:2307.03172): performance drops sharply when the relevant fact sits mid-context versus near the start or end. Models don’t robustly use all the tokens they can hold; performance can degrade as context fills, even under the cap. Anthropic calls this “context rot”: context is finite with diminishing returns, and “Every new token introduced depletes [the attention] budget by some amount.”
Instruction dilution and stale premises. In a 100K-token window where 90K is old tool noise, the latest instruction gets less weight. If a fact from turn 2 was contradicted at turn 50, the turn-2 version may still contradict the summary. Hence the rule: “the latest user message WINS.”

The “dumber” feeling is diagnosable: an agent that repeats itself or ignores recent instructions is deep in a summary, not the live conversation.

05When a fresh session beats a summary

Compaction is not always right. A hard reset (Hermes: /new or /reset; a new chat thread; hermes chat fresh) wins in three cases:

The past is a liability. If the conversation is mostly failed attempts, wrong paths, and superseded decisions, the summary carries their ghosts. Anthropic’s engineering team says rewinding to a clean context preserves only correct implementations and reduces goal drift.
The task is done and the next is unrelated. “Summarize the past 3 hours of Python work” and “design a landing page” do not belong in one window. The handoff calls the old task reference-only, a polite start fresh.
You need the agent maximal, not convenient. A fresh session with a tight system prompt, the key facts re-stated, and no 60 turns of baggage is the highest-performance configuration. The message after a reset carries the full attention budget.

When compaction wins: the work is continuous (a long migration, a multi-hour research sweep) and the tail is still relevant. Losing the thread hurts more than losing the detail.

Key idea

The practical rule: if you would not want the agent to act on something from an hour ago, it should not be in the window. Reset, and re-state only what matters. Otherwise put it in the protected tail or summary, not buried mid-context.

06It shows up in your stack

What each tool does with context:

HermesDual compaction (50% agent-side, 85% gateway safety net), in-place session IDs, soft-archived pre-compaction turns searchable via session_search, manual /compress, reset policies (none/idle/daily), and the [CONTEXT COMPACTION — REFERENCE ONLY] handoff. session_search recovers a missed detail from the archive, not by asking you to repeat it.
MnemosyneLong-term semantic memory: it auto-captures every turn as embeddings outside the window, exactly Anthropic’s “structured note-taking / agentic memory” pattern. It is why a fresh session isn’t amnesia; durable facts live in memory, not the window.
Open DesignThe design-agent platform does this per project: a DESIGN.md + BRIEF.md in the working directory is persistent context that survives individual run sessions, so each run starts pinned.
OllamaLocal defaults: 4K–256K context by VRAM, with truncation past it. Check ollama ps; when a local agent “forgets” mid-task, a 4K window silently chopping history is the likely cause.

The through-line: the window is for what’s happening now; the filesystem, memory store, and retrieval are for what happened before. Everything you move out is summarized (lossy), archived (recoverable), or persisted to memory (durable).

07Key takeaway

Key idea

The context window is a finite working memory, not a file cabinet. Long sessions summarize, truncate, or reset, and every summarization loses detail — which is why long-chat agents drift. Run your stack with intent: keep important context in the tail or memory, compact when work is continuous, reset when the past is baggage. Treat every compaction as a lossy handoff you can recover from, never the full record.

08Go deeper

Top references (free)

[1]

Karpathy — “Intro to Large Language Models” (YouTube, 1 hr)
https://www.youtube.com/watch?v=zjkBMFhNj_g
The “context window = RAM” analogy (~minute 28) and the LLM-as-OS framing.
[2]

Wikipedia — “Context window”
https://en.wikipedia.org/wiki/Context_window
The canonical definition, tokenization, and growth from 512 tokens (2018) to millions.
[3]

Liu et al. — “Lost in the Middle: How Language Models Use Long Contexts” (arXiv:2307.03172, TACL 2024)
https://arxiv.org/abs/2307.03172
Bigger windows aren’t better-used: position matters.
[4]

Anthropic — “Effective context engineering for AI agents” (engineering post, free)
https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
Compaction, the keep-vs-discard tradeoff, context rot, and the three long-horizon strategies.

Tool-specific docs

—

Hermes — Context Compression and Caching (developer guide)
https://hermes-agent.nousresearch.com/docs/developer-guide/context-compression-and-caching
50%/85% thresholds, the summary template, tail protection, in-place compaction.
—

Hermes — Sessions (user guide)
https://hermes-agent.nousresearch.com/docs/user-guide/sessions
Reset policies, session_search, continuity after crashes.
—

Ollama — FAQ / Context length (faq, context-length)
https://docs.ollama.com/faq · https://docs.ollama.com/context-length
num_ctx defaults, OLLAMA_CONTEXT_LENGTH, VRAM sizing.