dllama.tech · Knowledge Course KC 301-02
KC 301 · Hard Engineering — Unit 02

Caching:
prompt caching, KV cache

~31 min read · Written · Research: Mara · Slug prompt-caching-kv-cache
Summary

The Sep 13 audit carried two gaps that look unrelated but are one object: prompt caching (why an API charges 1/50th as much for input it has seen before) and memory bandwidth (why a 4B model does ~9 tokens/second on a phone). The object is the KV cache: a model’s saved attention state for tokens it already processed. On a hosted API the provider keeps that state on disk and rents it back; a prefix match is a cache hit, a fraction of a cache miss. On your own hardware you keep the cache and pay in memory bandwidth and RAM, because each generated token re-reads the weights and the cache from memory. The habits: freeze your prefix (SOUL prompt, MEMORY.md, skills, tool list), keep volatile bits last, don’t swap models or credentials mid-run, and treat context length as the memory setting it is.

AudienceSomeone who runs an agent stack daily and wants the layer under it.

KC 301 · Hard Engineering — Unit 02 · Status: published
Fig. 1 — One cache, two readings
A · The bill — what the cache costs on the API prefix units →
TURN 1 written at the miss price
TURN 2 prefix hit · ~1/50 the price
TURN 3 prefix hit · ~1/50 the price
TURN 4 model switch → miss — the whole prompt
already in the cache before this turn · read at the cache-read price
written this turn · billed at the miss price
a changed byte at position 4,000 makes position 400,001 uncacheable
prefixes only · exact match · best-effort
One object · two readings
B · The wall — what the same cache costs per token on a device
Phone / tablet SoC ~50–90 GB/s ▲
Consumer GPU 716.8 GB/s
Datacenter GPU ~2–3 TB/s
▲4B · Q4 · ~9 tok/s — the tier the number is stuck on
bars: linear, one scale · ranges drawn as ranges
tokens/sec ≈ memory bandwidth ÷ bytes read per token
One key-value cache is a billing lever on the API and a memory wall on the device. A cache hit is a read you skip. A decode step is a read you cannot skip.
SOURCES — ~1/50 hit-to-miss and the 4,000 / 400,001 rule: live DeepSeek price page and §07, checked · ~50–90 GB/s and ~2–3 TB/s: the on-device-LLMs survey (v-chandra.github.io) · 716.8 GB/s: Wikipedia, GeForce RTX 40 series · 4B · Q4 · ~9 tok/s: llama-bench pp512 / tg128 on a Pixel-class device, audit .

01One object, two bills

Five words first.

Jargon translation
Token — the unit a language model reads and writes. Roughly 3–4 English characters, one Chinese/Japanese/Korean character. The unit of both pricing (dollars per million tokens) and work (one token per forward pass).
Jargon translation
Cache — a store for expensive work’s result, so you don’t redo it. A website caches a rendered page; an LLM caches the attention state for tokens already read.
Jargon translation
KV cache — the key/value cache. In attention each token becomes a query (Q), a key (K) and a value (V). A new token looks back at earlier tokens’ K and V, which never change on append, so compute each once and store it. That store is the KV cache. KV = “keys and values”.
Jargon translation
Prefix — the start of a request. A request runs system prompt, tools, memory, history, then the new user message; everything before the new message is the prefix. Caching works only on prefixes, and only on exact matches (section 3).
Jargon translation
Prefill and decode — a request’s two phases. Prefill processes the whole prompt at once (parallel, compute-heavy); a cache hit saves money and time-to-first-token here. Decode emits one token at a time (sequential, memory-bandwidth-heavy) and caching does not help: those tokens are new.

With those five words, the whole unit collapses into one table:

The two bills
Hosted API (DeepSeek, Anthropic, OpenAI)Your own box / phone (llama.cpp, Ollama)
Who holds the KV cacheThe provider, on their disk and GPU memoryYou, in VRAM or system RAM
What you payDollars per token: a hit (reused prefix) vs a miss (recomputed prefix)Memory bandwidth per generated token + RAM capacity for the cache
What limits youThe hit rate you can achieveGB/s of memory bandwidth; how much context fits
What breaks itAny change in the prefix, a model switch, expiry (TTL)Longer context, bigger KV dtype, an OS that reclaims your RAM

Gap 1 is the left column: an accounting question with a 50× price difference. Gap 2 is the right: a physics question about how fast bytes move. Same cache, two bills.

PrefixPrefix02Why a cache exists at all

The plain explanation

A transformer emits one token at a time. Token 500 needs attention over tokens 1–499; recomputing all of it every step costs on the order of 500 × 500 operations, the quadratic blow-up. Earlier tokens’ K and V vectors never change, so compute each token’s K and V once, keep them, and let attention read the whole table.

Jargon translation
Attention — how each token decides how much to “look at” every other token. Practically, a weighted average of past value vectors, weighted by the new token’s query against each past key.

The HF Transformers docs: KV scores “are calculated every time because the model predicts one token at a time... A KV cache stores these calculations so they can be reused without recomputing them.”

It is not free — it is RAM that grows with your conversation

The cache copies part of your state and grows linearly with context length. Meta’s 2026 on-device survey[4]: it “grows linearly with sequence length and can dominate memory usage during long-context inference, often exceeding the model weights themselves.”

The arithmetic, from Qwen3-4B. Its published config.json says 36 layers, 8 key/value heads (grouped-query attention), head dimension 128:

Calc — KV bytes per token 
KV cache bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_element
                         = 2 × 36 × 8 × 128 × 2 bytes  (f16)
                         ≈ 144 KiB per token
Cache-only size, Qwen3-4B
Context lengthf16 KV cacheq8_0 KV cache
4,096 tokens~0.60 GB~0.30 GB
32,768 tokens~4.8 GB~2.4 GB

Cache-only numbers. On top sit the weights (2.5 GB for Qwen3-4B at Q4_K_M) and runtime overhead. Hence Apple’s 4,096-token on-device window (section 6), and why your phone’s “15.5 GB of RAM” behaves like 5 GB (section 6).

Where you meet it in practice

  • llama.cpp: -c, --ctx-size N is literally “how big is the KV cache you’re willing to allocate”. Default 0 means “take it from the model”. -ctk / --cache-type-k and -ctv / --cache-type-v choose the cache’s data type; f16 is the default, and q8_0, q4_0 and friends are quantization formats for the cache itself.
  • Ollama: same idea, different names. Ollama’s default context window is 4096 tokens (OLLAMA_CONTEXT_LENGTH overrides it, /set parameter num_ctx per run). OLLAMA_KV_CACHE_TYPE selects f16 (default), q8_0 (~1/2 the memory, “very small loss in precision”) or q4_0 (~1/4 the memory, more noticeable quality loss at high context).
  • Hugging Face Transformers: the cache is an explicit object you choose among — DynamicCache (default, grows), StaticCache (pre-allocated, for torch.compile), QuantizedCache (smaller), and offloaded variants that keep only the current layer’s cache on the GPU. The doc’s framing: the KV cache “can occupy a significant portion of memory and become a bottleneck for long-context generation.”

Why it matters: the KV cache ties together three things you usually treat separately: your API bill, your time-to-first-token, and your maximum usable context. Tuning one touches the others.

BillBill03Prompt caching on the API: hits, misses and “cache prefix units”

The plain explanation

A hosted model is stateless: you resend the whole conversation each turn. The provider is not. It keeps your last request’s KV cache on disk; if the next request starts with identical tokens it reads that state instead of recomputing. Identical prefix → cache hit. Any difference → cache miss (full price, full prefill).

Three load-bearing properties, from DeepSeek’s Context Caching guide[1]:

  1. Prefix only. Caching covers the beginning of the request, not the middle or the end.
  2. Exact match. A hit requires that the request fully matches a persisted “cache prefix unit”. Not “mostly matches”. Not “semantically similar”. The same tokens, in the same order, byte for byte.
  3. Best effort. “The cache system works on a ‘best-effort’ basis and does not guarantee a 100% cache hit rate.” Nor is it instant: “Cache construction takes seconds”, and an unused cache clears “usually within a few hours to a few days.”

What counts as a cacheable unit

DeepSeek persists cache prefix units at:

  1. Request boundaries — the end of your input and the end of the model’s output. In A + B (round 1) then A + B + C (round 2), round 2 fully matches A + B and hits.
  2. Detected common prefixes — if round 1 is A + B and round 2 is A + C, round 2 does not hit (A + C never matches A + B). The system notices the shared A, persists it as its own unit, and a third request A + D hits on A. Hits take a request or two to “warm up”.
  3. Fixed token intervals — for long inputs and outputs, units are carved out at intervals so a very long prefix isn't uncacheable forever just because it never reaches an end position.

Reading the hit rate off the response

DeepSeek exposes the accounting in the usage object:

Usage — DeepSeek response 
{
  "id": "chatcmpl-...",
  "model": "deepseek-flash",
  "usage": {
    "prompt_tokens": 61248,
    "completion_tokens": 1842,
    "total_tokens": 63090,
    "prompt_cache_hit_tokens": 58752,
    "prompt_cache_miss_tokens": 2496,
    "prompt_tokens_details": { "cached_tokens": 58752 }
  }
}

Read it like a bill: prompt_cache_hit_tokens is input served from the provider’s disk cache (cheap tier); prompt_cache_miss_tokens is what it computed (full price). Here 96% of a 61K-token prompt was a hit. Others name the pair differently: Anthropic’s cache_creation_input_tokens / cache_read_input_tokens; OpenAI’s usage.input_tokens_details.cached_tokens. Whichever field you read is your cost lever’s dashboard.

The 50× lever, verified

DeepSeek’s live price page[2] (fetched ; the model is now deepseek-flash, i.e. DeepSeek-V4.1-Flash):

Price — per 1M input tokens
Input tier (per 1M tokens)Off-peakPeak
Cache hit$0.003$0.006
Cache miss$0.15$0.30
Output$0.60$1.20
Key idea

That is a 50× spread between hit and miss on input, in both windows (off-peak is half of peak; peak is 01:00–04:00 and 06:00–10:00 UTC, Mon–Fri). The audit’s anchor — “a cache hit costs roughly 1/50th of a miss ($0.003 vs $0.15 per 1M tokens)” — reproduces exactly.

Two caveats, since prices move fastest here:

  • DeepSeek retired and repriced on with V4.1; some mirrors still serve the old tier (V4-Flash at $0.007 hit / $0.22 miss, ~31× spread). The lesson holds either way; the numbers must come from the live page.
  • The spread is so large the multiplier barely matters: 50× or 31× on input is “budget for it” vs “don’t think about it”.

Why so cheap? A hit is a disk read, not a GPU forward pass. DeepSeek’s V4.1 announcement[3]: “Compared with the previous generation, V4.1-Flash’s KV cache needs just 1/4 the HBM and 1/8 the SSD storage. Cache-hit charges often account for a large share of agent costs. Compressing the cache cuts those costs significantly.” Pushing the KV cache down the memory hierarchy (GPU memory → host RAM → SSD) is how agent work gets affordable: section 5’s physics, monetized.

The same mechanism, three vendors

The same lever, three vendors
DeepSeekAnthropic (Claude)OpenAI
How you opt inAutomatic, on by defaultcache_control (automatic or explicit breakpoints)Automatic (prompt_cache_options to tune)
Unit of cachingPrefix; persisted “cache prefix units”Prefix; cache breakpoints, hierarchy tools → system → messagesPrefix; implicit or explicit breakpoints (prompt_cache_key)
Write cost(bundled — no separate write tier)1.25× base input for 5m TTL; 2× for 1h1.25× base input
Read (hit) cost1/50 of miss on input0.1× base input (0.025× on the newest flagships)0.1× base input
LifetimeHours to days, best effort5 min default, refreshed on use; 1h optional30 min default (ttl: "30m"), refreshed on use
Minimum cacheable prefixNot documented as a token floor; short prefixes just may not persistModel-specific minimum; check your model's row1,024 tokens (GPT-5.6+), 2,048 for older
Where the numbers show upprompt_cache_hit_tokens / prompt_cache_miss_tokenscache_creation_input_tokens / cache_read_input_tokensinput_tokens_details.cached_tokens

Worked example. A 60K-token agent turn (system prompt + memory + skills + tools + history) with a 2K-token answer, on deepseek-flash off-peak:

  • Miss on the whole input: 0.06M × $0.15 = $0.00900 input + 0.002M × $0.60 = $0.00120 output → $0.0102
  • Hit on the whole input: 0.06M × $0.003 = $0.00018 input + $0.00120 output → $0.00138

Per turn that is 7.4× cheaper; over 50 turns, $0.51 vs $0.069. It isn’t 50× because output tokens are never cached; they are new work. Hence the next two sections: caching discounts reading the past, never producing the future.

Bill04Your stack's prefix: what is stable, what is volatile

Now the part that changes your behaviour. Hermes builds its system prompt in three ordered tiers — stable → context → volatile — with the stated intent: “This ordering matters... This separation keeps the stable prefix stable for caching.”

Walking the layers in the order the API sees them:

The prompt layers, front to back
Prompt layerSourceStabilityCache consequence
Agent identitySOUL.mdStable across turnsPart of the cacheable prefix — edit it between sessions, not mid-run
Tool-aware behaviour guidance, skills indexcode + skills/Stable if the skill set is stableSkills index is inside the cached prompt; loading a different set of skills mid-run changes it
Project context.hermes.md, AGENTS.md, CLAUDE.mdStable per projectRebuilding it (new file, changed file) invalidates from that point
Memory snapshotMEMORY.mdFrozen for the sessionMid-session writes update disk but “do not mutate the already-built cached system prompt until a rebuild path runs”
User profileUSER.mdFrozen for the sessionSame as above
API-call-time layersephemeral_system_prompt, pre_llm_call plugin contextChanges every callDeliberately kept out of the cached prompt — appended to the current user message instead

The last row is the design lesson: Hermes keeps volatile context out of the prefix by placing it after the stable content. Do the same by hand in your prompts, cron job bodies, and skill text.

In practice

Hermes also manages the provider cache. Prompt caching is automatic for Anthropic-family models (native API or via OpenRouter); the TTL is set in config.yaml (prompt_caching.cache_ttl: “5m” or “1h”), and the CLI announces it at startup: Prompt caching: ENABLED (Claude via OpenRouter, 5m TTL). If Claude sessions feel expensive per turn, check that TTL line: a stop-start rhythm and a 5-minute cache is the common silent miss.

What the tool definitions cost you

The cache hierarchy is ordered tools → system → messages, and Anthropic’s invalidation table is brutal at the top. Changing tool definitions (names, descriptions, parameters) invalidates the entire cache: tools, system, messages. Changing tool_choice invalidates only messages; the system prompt invalidates system and messages.

Your stack maps onto this: a large tool surface behind progressive disclosure (tool_search / tool_describe / tool_call, 298 tools not all resident). Progressive disclosure is cheaper because a smaller, more stable tool array keeps the top of the cache hierarchy constant. Reshuffling loaded tools mid-run — enabling a skill, attaching an MCP server, switching tool sets — is a cache event, not just a context-length event.

The hit/miss table for your daily actions

Action level — hit or miss
ActionHit or miss?Why
Another turn in the same session, same model, nothing editedHit on everything before the new messageThe prefix is byte-identical; the reused part is served from cache
Editing MEMORY.md mid-sessionHit for this session, miss next session/rebuildThe snapshot is frozen until the prompt is rebuilt — you pay at the rebuild, not immediately
/model switch, provider fallback, or credential-pool rotationMiss — the whole promptHermes' own docs warn this: “means the next request gets zero cache hits and re-reads the full conversation at undiscounted input price. This is inherent to how provider caches work”
Loading different skills mid-runMiss from the skills-index layer downSkills index sits in the cached system prompt
Attaching an MCP server / changing the tool setMiss — entire cacheTool definitions are first in the hierarchy; a change there invalidates everything below
Context compaction fires (default at 50% of the window)Miss once, then re-warmCompaction rewrites the live message list; OpenAI says it plainly: “the first request after compaction may reuse less of the previous cache even when the conversation is logically the same”
A cron job with a fixed promptHit across runs while the cache survivesIdentical prefix → identical unit; DeepSeek's cache lives hours-to-days
A cron job that interpolates a timestamp at the top of its promptMiss, every runYou changed the prefix before anything stable
Retrieved chunks (RAG / Mnemosyne lookup) injected in the middleMiss from the insertion pointCaching is prefix-based; material inserted mid-prompt truncates the reusable region
A 5-minute gap and a short turn, then the next callHit (Anthropic TTL refreshed on use)But a 5-minute TTL with a long gap goes cold — a 4-minute streaming response already eats most of it

Two habits fall out of it:

  1. Stable content first, volatile content last. Identity, tools, memory, skills, project context → front; today’s date, current query, retrieved snippets, per-run IDs → back. Inject volatile context at the end of the user message, as Hermes’ API-call-time layer does.
  2. Don’t change the middle of a run. Model switches, skill churn, tool-set changes, memory rewrites are fine — do them between runs. The cache is prefix-exact, so everything after a change is a miss.
Jargon translation
TTL — “time to live”: how long a cached entry survives unused. DeepSeek: hours to days, best effort. Anthropic: 5 minutes default, refreshed free on every read; 1 hour for a 2× write. OpenAI: 30 minutes. A TTL is why “the same prompt, run tomorrow” is not automatically a hit.

WallWall05Memory bandwidth: why decode is not a compute problem

Switch columns from section 1. Locally nobody charges per token, so why is a 4B model on a phone stuck at single-digit tokens per second?

The plain explanation

Generating one token runs the whole model once: each layer’s weights are read from memory, plus the KV cache for the current context, then a tiny amount of arithmetic. That ratio of arithmetic to bytes moved, or arithmetic intensity, is low, so the chip waits on memory, not computing.

So TOPS (trillions of operations per second) is the wrong number to judge a device by. Meta’s on-device survey[4]: mobile NPUs “now deliver serious TOPS, getting close to the capability of data-center GPUs in 2017 (for example, V100 is 125 TOPS)”: Apple’s A19 Pro Neural Engine ~35 TOPS, Snapdragon 8 Elite Gen 5 ~60 TOPS, Dimensity 9400+ ~50 TOPS. “But TOPS alone doesn’t tell you much.” Then the decisive sentence:

Quoted source
“The deeper constraint is memory bandwidth. Mobile devices have 50–90 GB/s; data center GPUs have 2–3 TB/s. That's a 30–50x gap. For LLM inference, this gap is decisive because decode is memory-bound: you load the entire model weights for each token generated, so the compute units sit idle waiting for memory.”
Chandra & Krishnamoorthi — On-Device LLMs: State of the Union, 2026 [4]
Jargon translation
Memory bandwidth — how many bytes per second the processor can pull from RAM. Measured in GB/s (gigabytes) or TB/s (terabytes). Not to be confused with storage speed (SSD) or capacity (how many GB you have).

The napkin formula

The lever 
tokens per second  ≈  memory bandwidth ÷ bytes read per token

“Bytes read per token” is dominated by the model weights, plus the KV cache you carry. Two real datapoints.

Datacenter (official llama-bench README[5]). Qwen2.5-7B-Instruct-Q4_K_M, 4.677 GB of weights, on an RTX 4080:

  • 4080 memory bandwidth: 716.8 GB/s (Wikipedia's RTX 40-series desktop table)
  • Ceiling: 716.8 ÷ 4.677 ≈ 153 tokens/s
  • Measured: tg128 = 120.6 tokens/s → 79% of the theoretical ceiling

Phone (the Sep 13 audit’s own bench). Qwen3-4B-Q4_K_M, ~2.5 GB of weights, Termux + llama.cpp on a Pixel-class device:

  • Measured: llama-bench pp512 / tg128 = 9.28 tokens/s for generation
  • Implied achieved bandwidth: 2.5 GB × 9.28 ≈ 23 GB/s
  • Against the survey's 50–90 GB/s range, that is 26–46% of the ceiling — while the 4080 sits at 79%

Same formula both times. The phone is further from its ceiling for reasons visible in section 6: KV-cache reads on top of weights, CPU overhead in Termux, thermal throttling, and an OS that reclaims memory behind your back.

Bandwidth by device class
Class of deviceTypical memory bandwidthWhat that means for a 2.5 GB Q4 model
Phone / tablet SoC~50–90 GB/s20–36 tok/s ceiling; realistically single digits to low teens
Datacenter GPU (current)~2–3 TB/s800–1,200 tok/s ceiling on the same bytes
Consumer GPU (RTX 4080, 2022)716.8 GB/s~153 tok/s ceiling
Jargon translation
Quantization — storing model weights (and, optionally, KV cache entries) in fewer bits. Q4_K_M ≈ 4 bits per weight; f16 = 16 bits. Quantization speeds up generation not because the math is cheaper, but because fewer bytes cross the memory bus per token, the actual bottleneck.

Hence 16-bit to 4-bit weights is roughly a 4× throughput win, not a 4× size-only win. And speculative decoding (a small draft model proposes tokens; the big model verifies them in one pass) is nearly “free”: one memory-bound pass is amortized over several accepted tokens.

Where it shows up in practice

llama-bench measures this on your own hardware. From the official README:

Bench — llama-bench on your own hardware 
# prompt processing (-p) vs text generation (-n); -ngl = layers offloaded to GPU
./llama-bench -m qwen3-4b-q4_k_m.gguf -ngl 99 -p 512 -n 128

# how much does context depth cost you? prefill the KV cache to 0, 512, 4096 tokens
./llama-bench -m qwen3-4b-q4_k_m.gguf -d 0,512,4096 -p 512 -n 128

# how much does the KV cache data type cost? f16 vs q8_0
./llama-bench -m qwen3-4b-q4_k_m.gguf -ctk q8_0 -ctv q8_0 -n 128

Reading the output (the README's own example, retitled for clarity):

Bench — reading the output 
| model  | size     | params | backend | ngl | test  | t/s            |
| qwen2 7B Q4_K - Medium | 4.36 GiB | 7.62 B | CUDA | -1 | pp512 | 7340.20 ± 23.45 |
| qwen2 7B Q4_K - Medium | 4.36 GiB | 7.62 B | CUDA | -1 | tg128 |  120.60 ± 0.59  |
  • pp512 = prefill of 512 tokens: thousands of tokens/s, parallel and compute-bound.
  • tg128 = generation of 128 tokens: tens of tokens/s, sequential and memory-bandwidth-bound. The number a phone lives and dies by.
  • -d N pre-fills the KV cache to N tokens, so you see context length costing speed instead of just believing it.
  • The README notes the measurements exclude tokenization and sampling: real-world numbers run a little lower.
If this fails

One more trap, from llama.cpp’s completion docs. When the window fills, “some of the earlier tokens (half of the tokens after --keep) will be discarded. The context must then be re-evaluated before generation can resume. On large models and/or large context windows, this will result in a significant pause in output.” That pause is a cache miss felt through your chair, not on an invoice. The local “freeze your prefix” is --keep N (or --keep -1 to retain the initial prompt when the window rolls).

Wall06The phone case, concretely

The Sep 13 audit benched a Pixel-class Android device: Termux, llama.cpp, Qwen3-4B-Q4_K_M (~2.5 GB). The numbers, and what each says:

Bench observations — what each number says
ObservationReading
llama-bench pp512 / tg128 = 9.28 tok/s generation~23 GB/s achieved from a 50–90 GB/s memory system. Slow decode is a bandwidth/overhead story, not a “small chip” story
-ngl 99 (offload all layers to the Adreno GPU via OpenCL) crashed on SET_ROWS with a q8_0 KV cacheQuantized KV cache is not universally supported by accelerator backends. Falling back to f16 KV fixed the crash — and doubled the cache size. Speed and correctness trade against each other at the backend's mercy
MemAvailable 5.2 GB against 15.5 GB headline RAMWeights (2.5 GB) + KV cache (grows with context) + Android's own services + the GPU driver's allocations. “16 GB of RAM” never means 16 GB for your model
Context length was the knob that matteredPer section 2's arithmetic: f16 KV at 32K context is ~4.8 GB for a 4B model. Long context is not free locally — it is a RAM bill paid up front
Jargon translation
SET_ROWS — a low-level array-writing operation in the compute backend. When a backend (here, the Adreno OpenCL path) hasn’t implemented an operation the quantized KV-cache path needs, the process doesn’t slow down — it crashes. Backend coverage, not raw FLOPS, decides which flags work on a phone.
If this fails

-ngl 99 with -ctk q8_0 -ctv q8_0 crashing on SET_ROWS is a backend-coverage failure, not a memory failure. Fall back to f16 KV and accept the doubled cache, or drop the offload. Read a backend’s op coverage before its TOPS.

Apple's 4,096 tokens is a memory decision

Apple’s on-device Foundation Models framework[6] documents a 4096-token context window per session. Exceed it and it “throws a LanguageModelError.contextSizeExceeded error and the session stops responding.” The prescribed fixes are memory engineering, not “be less ambitious”:

  • Split large tasks across multiple sessions, chaining summaries forward (their worked example summarizes a long article in chunks).
  • Start a new session with a summary plus the first and last transcript entries: “The first transcript entry often contains important instructions and the last entry contains the most recent context.”
  • Limit tool use: three to five tools per request, short descriptions — because “tool definitions and their input and output” consume the same context budget as your prompt.
  • Keep prompts and instructions concise, and ask for shorter answers.
  • Use contextSize (the model's actual maximum) and tokenCount(for:) instead of hardcoding 4096.

Read that list as a KV-cache budget in English. A 4,096-token window is small because each token costs KV memory, on a device with 50–90 GB/s shared with the whole OS. Same 50–90 GB/s as section 5, same 144 KiB/token as section 2.

The local knobs, gathered

In practice — the local knobs
Goalllama.cppOllama
Cap context (cap the KV cache)-c, --ctx-size Nnum_ctx option / OLLAMA_CONTEXT_LENGTH (default 4096)
Shrink the cache per token-ctk q8_0 -ctv q8_0 (f16 default)OLLAMA_KV_CACHE_TYPE=q8_0 or q4_0 (needs Flash Attention)
Enable Flash Attention (needed for quantized KV)-fa onOLLAMA_FLASH_ATTENTION=1
Keep the cache on the CPU, not the GPU-kvo / --no-kv-offload— (managed automatically)
Keep the initial prefix when the window rolls--keep N (-1 = keep all)—
Cache prompt state to a file across runs--prompt-cache FNAME (plus --prompt-cache-all, --prompt-cache-ro)— (server keeps state while loaded)
Keep the model (and its KV cache) resident--sleep-idle-seconds controls when it stops being residentkeep_alive (default ~5 min; ollama ps shows the countdown)
Restore saved cache state--slot-save-path—
Gotcha

Two notes that save a debugging session. In Ollama the KV-cache quantization type is global — “all models will run with the specified quantization type” — and quantizing the cache “can harm latency if the context length is short and there is enough GPU memory”, so the win is context-length-dependent. And llama-server’s --sleep-idle-seconds unloads “the model and its associated memory (including the KV cache)” after idleness, so a warm local cache expires on your hardware’s clock.

CrossoverCrossover07The two halves are the same object

Side by side, the spine appears:

The join
Prompt caching (API)KV cache on your hardware
What is storedThe KV cache for a prefix of your requestThe KV cache for your whole context
Who pays to store itThe provider (their HBM, their SSD)You (your VRAM, your RAM)
What a “hit” costs~1/50 of a miss on input tokensA read of N bytes per token at 50–90 GB/s
What a “miss” costsFull input price + slower time-to-first-tokenRe-prefill: the “significant pause” llama.cpp warns about
What invalidates itPrefix edit, model change, tool-set change, TTL expiryContext growth, cache dtype, OS memory pressure, backend limits
What you can controlPrompt layout, model stability, TTL strategy, watching hit-token countersctx-size, KV dtype, Flash Attention, quantization, context discipline
What stays constantOutput tokens are never cachedDecode is always memory-bound

The join is the sentence to take away:

Key idea

Prompt caching is renting somebody else's KV cache. Running a model locally is owning yours — and paying the bill in bandwidth and RAM instead of dollars.

Both obey the same two rules:

  1. The cache is prefix-exact. On the API, a changed byte at position 4,000 makes position 400,001 uncacheable. On your device, a cache dtype your backend can’t execute makes the whole thing nonexistent. Cacheability is a property of your input’s beginning and your hardware’s compatibility, not your intentions.
  2. You only ever discount the past. Prefill (reading the prompt) is what caching helps; decode (producing the answer) is what memory bandwidth limits, and it is never cached, not by DeepSeek’s disk tier or your phone’s LPDDR. Every cost or latency optimization must answer: how much of this request is re-reading, and how much is new work?

08Playbook

  1. Freeze your prefix. Put identity (SOUL/system prompt), tool definitions, memory, skills and project context at the front, in a stable order. Nothing volatile above them.
  2. Push volatility to the end. Current time, session IDs, retrieved chunks, per-run variables → end of the user message. Use an ephemeral-context hook rather than editing the system prompt.
  3. Stabilize the tool array. Keep the resident tool set (including progressive-disclosure/tool_search names and descriptions) constant within a run. Tool definition changes invalidate the entire cache.
  4. Don’t swap models, providers, or credentials mid-session. Hermes’ docs: this means zero cache hits on the next request. If a fallback is needed, expect a full-price turn.
  5. Edit memory between runs, not during them. A frozen MEMORY.md snapshot is a feature: you keep the session’s cache while writes land on disk for the next one.
  6. Measure the hit rate, don’t assume it. Log prompt_cache_hit_tokens / prompt_cache_miss_tokens (DeepSeek), cache_read_input_tokens (Anthropic), or cached_tokens (OpenAI). A hit rate is a metric; a feeling is not.
  7. Design cron prompts with a stable body. Fixed instructions first; interpolated values (dates, run ids) last. An identical prefix across runs is free money while the cache lives.
  8. Right-size context on local hardware. ctx-size × KV dtype is a RAM allocation. Start at the smallest workable window, then reach for q8_0 KV (≈ half) or q4_0 KV (≈ a quarter) if supported.
  9. Benchmark before optimizing. llama-bench -p 512 -n 128 shows prefill vs decode; add -d 0,512,4096 to price context depth; add -ctk q8_0 -ctv q8_0 to price cache quantization. Change one thing at a time.
  10. Expect a prefill pause when the window rolls. In llama.cpp that is the --keep/context-shift path; on a phone it’s the gap between a responsive assistant and a visible stall. Budget context accordingly.

09Key takeaway

One KV cache, two ways to meet it. On a hosted API it is the provider’s, kept on disk and charged by whether your request’s prefix matches. On DeepSeek today a hit costs $0.003 per 1M input tokens against $0.15 for a miss: a 50× lever, measurable (prompt_cache_hit_tokens / prompt_cache_miss_tokens) and controllable through prompt layout — stable first, volatile last, no mid-run model or tool-set changes, memory edits between sessions. On your own hardware the cache is yours, billed as memory bandwidth and RAM, because decode is memory-bound: tokens/s ≈ bandwidth ÷ bytes-read-per-token. That explains why a phone (50–90 GB/s) does ~9 tok/s on a 2.5 GB model while a datacenter GPU (2–3 TB/s) does hundreds, and why Apple’s 4,096-token window, Ollama’s 4,096-token default, and llama.cpp’s -c/-ctk/-ctv flags are one decision in different clothes. In both columns caching discounts only the past; the next tokens are new work. Freeze the prefix, keep volatile parts last, count your context.

10Go deeper

Gap 1 — prompt caching (the 1–3 best free references)
  • [1]

    DeepSeek — Context Caching

    api-docs.deepseek.com/guides/kv_cache

    The primary source: request-boundary persistence, common-prefix detection, fixed token intervals, the “fully matches a cache prefix unit” rule, best-effort semantics, the prompt_cache_hit_tokens / prompt_cache_miss_tokens fields, and the A+B / A+C / A+D examples. Read this one first.

    Cited in §03
  • [2]

    DeepSeek — Models & Pricing

    api-docs.deepseek.com/quick_start/pricing

    The live hit/miss price table, the off-peak/peak split, and the model-rename footnote. Check it before quoting any number here.

    Cited in §03
  • [3]

    Introducing DeepSeek-V4.1-Flash

    www.deepseek.com/en/news/deepseek-v4-1-flash

    The vendor’s statement that “cache-hit charges often account for a large share of agent costs”, with the 1/4 HBM and 1/8 SSD figures. Pairs with the technical report PDF and the model card.

    huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf · huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash Cited in §03
Gap 1 — the other vendors’ versions of the same lever:
  • ·

    Anthropic — Prompt caching

    docs.anthropic.com/en/docs/build-with-claude/prompt-caching

    cache_control, the tools → system → messages hierarchy, 1.25×/2× write multipliers, 0.1× reads, the 20-block lookback window, and the single most useful table in the discipline: what invalidates the cache. Tool definition changes kill everything; tool_choice changes kill only messages.

    Cited in §03, §04
  • ·

    OpenAI — Prompt caching

    platform.openai.com/docs/guides/prompt-caching

    Minimum cacheable prefix (1,024 tokens on GPT-5.6+, 2,048 before), implicit vs explicit breakpoints, prompt_cache_key, and an instructive gotcha about implicit caching writing through dynamic content.

    Cited in §03, §04
Gap 2 — memory bandwidth and KV-cache sizing
  • [4]

    Vikas Chandra & Raghuraman Krishnamoorthi (Meta) — “On-Device LLMs: State of the Union, 2026”

    v-chandra.github.io/on-device-llms

    The 50–90 GB/s vs 2–3 TB/s comparison, “decode is memory-bound: you load the entire model weights for each token generated”, the mobile-NPU TOPS table, and a “KV Cache Management” section (KV quantization to ~3 bits, StreamingLLM, DuoAttention, ChunkKV, EvolKV).

    Cited in §02, §05
  • [5]

    llama.cpp — llama-bench (tools/llama-bench README)

    github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench

    The measurement tool, with real tables for pp/tg, threads, GPU layers and prefilled context depth. Paired with the server README (-ctk, -ctv, -kv-offload, --sleep-idle-seconds) and the completion README (--prompt-cache, --keep).

    github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md · github.com/ggml-org/llama.cpp/blob/master/tools/completion/README.md Cited in §05
  • [6]

    Apple — Managing the context window

    developer.apple.com/documentation/foundationmodels/managing-the-context-window

    The 4096-token window, contextSizeExceeded, tokenCount(for:), and the prescribed strategies (split tasks, fresh sessions, ≤3–5 tools) that are really a memory budget. Coverage: InfoQ.

    www.infoq.com/news/2026/03/apple-foundation-models-context Cited in §06
Supporting references (free, verified ) Watch (free) Next lecture: KC 301-03 — Quantization & VRAM math (the other half of the memory equation: how many bytes your weights cost, and why Q4 is the default), with KC 301-04 — Serving: prefill vs decode (where this lecture's pp/tg split becomes a deployment decision).
Research: MaraDesign: Kyra + Open DesignPowered by Hermes Agent