Core model terms
9 terms
| Term | Plain English | In Mo’s stack |
|---|---|---|
| Token | The smallest unit of text a model reads or writes — roughly ¾ of an English word (“tokenization” chops text into these). | Everything is billed and counted in tokens: the DeepSeek API invoice, Hermes’s token-count logs at compaction, and Ollama’s num_ctx (measured in tokens, not words). |
| Context window | The maximum number of tokens a model can see in one request — its entire working memory for that moment. | Hermes refuses agent mode below a 64K window; Ollama on Mizuki defaults low unless you set num_ctx / OLLAMA_CONTEXT_LENGTH; DeepSeek’s window size is the ceiling on a single request. See 101-02. |
| Parameter | A learned number (weight) inside the model; the count (“7B”, “671B”) is the rough measure of model size and capability floor. | “Qwen 7B on Mizuki” = 7 billion parameters, which is why it fits on your 4080 while the DeepSeek V4 Flash model you call via API is far larger and lives in someone else’s datacenter. |
| Temperature | A knob that flattens or sharpens the probability distribution — higher = more random, lower = more deterministic. | A temperature setting in Hermes model config and in the DeepSeek API temperature param; Ollama exposes it as temperature in a Modelfile or OLLAMA_TEMPERATURE. |
| Top-p | “Nucleus sampling” cutoff: only sample from the smallest set of tokens whose combined probability reaches p (e.g. 0.9) — an alternative randomness knob. | Sits next to temperature in Hermes provider config, Claude Code model settings, and Ollama’s top_p option; many people tune p down instead of temperature for cleaner output. |
| Logit | The raw, pre-softmax score the model assigns to each possible next token — its unnormalized preference before probabilities exist. | Invisible in daily use, but it’s the thing temperature and top-p actually operate on; it’s what a “logit bias” API parameter (DeepSeek, OpenAI-compatible endpoints) lets you poke directly. |
| Sampling | The final step that actually picks the next token from the probability distribution — the model doesn’t “choose”, it rolls weighted dice. | Happens on every streamed token from Ollama on Mizuki and every DeepSeek API response; num_predict (how many tokens to sample) and stop sequences are sampling-side controls. |
| Attention | The transformer mechanism where every token looks at every other token to decide what matters; cost grows quadratically with context length. | The reason your KV cache grows, the reason long contexts are expensive, and the reason Mizuki’s VRAM fills up as sessions get long. |
| Hallucination | Confident output that isn’t grounded in the input or reality — the model fluently inventing. | Why your fact-check-gate skill exists, why the stack prefers tool-grounded answers over memory, and why RAG (below) exists at all: pull the fact in, don’t trust the model to recall it. |
Training terms
6 terms
| Term | Plain English | In Mo’s stack |
|---|---|---|
| Pretraining | The giant first stage: predict the next token over trillions of words of internet text until the model learns language and world structure. | You never do this — you rent the result. DeepSeek V4 Flash and the Qwen models on Mizuki are pretrained “foundation” models; everything after this is why they’re usable. |
| SFT(supervised fine-tuning) | The “instruction tuning” stage: training the pretrained model on human-written question-answer pairs so it follows instructions instead of just completing text. | The difference between a raw base model and the -instruct/chat variants you actually ollama pull (e.g. qwen2.5:7b-instruct) — that suffix is SFT. |
| RLHF(reinforcement learning from human feedback) | Alignment stage: a reward model learned from human preferences guides further training so the model is helpful and refuses harm. | Why the DeepSeek API model feels cooperative and safe rather than raw-text-completing; the invisible politeness-and-safety layer on top of pretraining. |
| Fine-tuning | Continuing training on a smaller, specialized dataset to change behavior or style of an already-trained model. | Your character/model work: face-lora-training and the FAL hosted LoRA training pipeline are fine-tuning in the broad sense — teaching a model a specific face or style it didn’t know. |
| Quantization | Shrinking weights from high precision (fp16) to low (int8/4-bit) so models fit in less VRAM, at a small quality cost. | The whole reason Mizuki can run anything: GGUF quant files like qwen2.5:7b-q4_K_M — the q4 is 4-bit quantization, chosen by VRAM budget in the local-llm-checker workflow. |
| LoRA(low-rank adaptation) | A fine-tuning shortcut: instead of updating all parameters, train tiny low-rank matrices that slot onto the frozen model — cheap, fast, swappable. | Your miya-character-lora and face-lora-training skills; FAL hosted LoRA training uploads a zip and returns a LoRA file, not a whole model — that’s the point. |
Agent terms
11 terms
| Term | Plain English | In Mo’s stack |
|---|---|---|
| Agent | A model wrapped in a loop: reason → act (call a tool) → observe the result → repeat until the task is done, instead of one-shot Q&A. | Hermes itself, Claude Code headless, and every subagent they spawn — the thing KC 101 is about. |
| Tool call | The model emitting a structured request to run a function (read a file, run a command, search the web) instead of plain text. | Every read_file, terminal, web_search, patch entry you see in a Hermes session transcript is a tool call; Claude Code shows the same as “Tool Use” blocks. |
| Function calling | The API-level mechanism behind tool calls: you declare JSON schemas, the model picks one and fills in the arguments. | The DeepSeek API’s tools parameter; Ollama’s tools support for local models on Mizuki; every Hermes tool schema is a function-calling declaration. |
| MCP(Model Context Protocol) | An open standard (JSON-RPC) for exposing tools and data to agents — one server, any client; the USB-C of agent tooling. | Hermes’s native MCP client (mcp config in config.yaml), the mcp-server-setup skill, and servers you actually run: open-design, fmp (market data), color-scheme, design-inspiration. |
| Orchestrator | The top-level agent that plans, delegates, and integrates — it decides who does what and assembles the result. | Your Hermes main agent, and Claude Code’s top-level process with its Task/Agent tool; the kanban workflow routes specialist work through the orchestrator. |
| Subagent | A child agent spawned for a scoped job with its own fresh context; it works and reports a summary back to the parent. | Hermes delegate_task — this very lecture was written by a subagent (Mara) under a parent; the subagent-driven-development and kanban-delivery skills run on this. |
| System prompt | The persistent instruction block at the top of the context that defines identity, rules, and behavior — always present, always authoritative. | Hermes’s system prompt plus your SOUL.md and profile wiring; Claude Code’s CLAUDE.md; compaction explicitly preserves it because it outranks everything after it. |
| RAG(retrieval-augmented generation) | Before answering, retrieve relevant documents and stuff them into the context so the model answers from evidence, not memory. | The subject of 201-03 (memory-embeddings-rag); Mnemosyne pulling stored notes back into a session; the search-then-answer pattern behind your research skills. |
| Embeddings | Vector (list-of-numbers) representations of text where similar meaning ≈ nearby vectors — the model’s “semantic coordinates”. | Mnemosyne’s auto-capture turns every session turn into embeddings; embedding models run locally on Mizuki via Ollama (ollama pull nomic-embed-text) or through an API. |
| Vector DB | A store that holds embeddings and answers “what’s most similar to this?” via nearest-neighbor search. | Mnemosyne’s underlying store; the memory-architecture skills; the Obsidian-vault retrieval layer. The DB is the disk behind RAG. |
| Memory tiers | The layered memory of an agent stack: context window (RAM) → archived sessions (disk) → semantic memory (Mnemosyne) → filesystem (source of truth). | Hermes session_search reads the archived tier, Mnemosyne auto-captures the semantic tier, Open Design pins context in DESIGN.md/BRIEF.md files — each tier has a different retention and cost. |
Serving terms
8 terms
| Term | Plain English | In Mo’s stack |
|---|---|---|
| Inference | Running a trained model to produce output — the opposite of training; what you pay for per request. | Every DeepSeek API call and every ollama run on Mizuki is inference; the serving-llms-vllm and dedicated-gpu-llm-server skills exist to make it fast and cheap. |
| VRAM | GPU memory where the weights and the KV cache must live during inference — the hard budget for what fits locally. | Mizuki’s 16 GB RTX 4080 decides which quants fit (q4 yes, fp16 no) and how long contexts can get before an out-of-memory kill. |
| KV cache | The stored attention keys/values for every token seen so far — the model’s scratchpad; grows with context length and eats VRAM linearly. | Why a 128K context on a 16 GB card is tight even with a quantized model; why Ollama’s default num_ctx is VRAM-dependent; the thing prompt caching reuses. |
| Prefill | The first pass of inference: process the whole input in parallel, computing and storing the KV cache — sets up the state for generation. | Visible as “time to first token”: longer input (big file, big prompt) means longer prefill — it scales with what you stuff into context, not with the reply. |
| Decode | The second pass: generate the answer token by token, each step reading the KV cache and appending to it. | The visible stream of tokens in the Hermes TUI and Claude Code; why long outputs are slow (one token per step) and why num_predict caps decode time in Ollama. |
| Prompt caching | Reusing a stored KV cache when a new request shares a prefix with an old one — same system prompt, same tools: don’t recompute them. | DeepSeek’s API prices cache hits at a discount (watch “cache hit” line items); Hermes caches your system prompt and tool schemas so every turn after the first is cheaper. |
| vLLM | A high-throughput inference server: PagedAttention (VRAM-efficient KV cache), continuous batching, OpenAI-compatible API — what serious self-hosting runs. | The serving-llms-vllm skill; under the hood of most hosted providers; the thing you’d deploy on Mizuki if multiple people (or agents) should share one local model. |
| Ollama | The friendly local model runner: pulls GGUF models, manages context, exposes an OpenAI-compatible API, serves your GPU box. | Mizuki, full stop — ollama pull/run/ps, OLLAMA_CONTEXT_LENGTH, the local-llm-checker skill’s Qwen 7B audit, and embedding models for Mnemosyne. |
The map
Here is how the terms actually chain together in the system you run:
- Training makes the model. Pretraining → SFT → RLHF produce the weights (parameters). Quantization and LoRA are the two knobs that let you reshape or shrink those weights to fit your hardware. You interact with this layer once, at
ollama pulltime, and never think about it again. - Inference spends the model. Your text becomes tokens inside a context window. The server runs prefill (all input tokens at once, filling the KV cache) then decode (one token at a time, sampling from the logits under temperature and top-p). Weights + KV cache must fit in VRAM — that’s why quantization matters locally and why your API bill grows with input length.
- Serving manages the cost. Prompt caching reuses the KV cache for your stable prefix (system prompt, tool schemas) so repeated turns are cheap; vLLM and Ollama are the two engines that implement all of the above, one for scale, one for your desk.
- Agents live on top. The model is just the token machine. A system prompt + tool calls + the observe-reason loop make it an agent; an orchestrator with subagents scales it; function calling and MCP are the two plumbing standards for tools; RAG (embeddings → vector DB) and memory tiers give the loop a long-term memory so it isn’t reborn every session.
tokens fill the context, attention fills the KV cache, the KV cache fills VRAM (or your bill), and everything an agent does is a loop of spending that budget on tool calls.
Key takeaway
Key idea
Every term in this sheet is a knob or a budget you already touch daily — you just didn’t always have the name for it. When something breaks, classify it by layer: a bad answer is sampling (temperature/top-p) or grounding (RAG/hallucination); a slow answer is serving (prefill/decode, KV cache, VRAM); a forgetful agent is memory (context window, compaction, memory tiers).
Name the layer and the fix is usually one setting away.
Go deeper
Each term above is a one-line definition by design. Where a term has a whole unit behind it, the sheet says so — those units are here.
Elsewhere in the course-
[1]
KC 101-02 · Context windows & what fits
/knowledge-course/lectures/101-02-context-compaction.htmlCited by the Context window row: window size, compaction handoffs,
num_ctxandOLLAMA_CONTEXT_LENGTHin full. -
[2]
KC 201-03 · Memory: working vs long-term, embeddings, RAG
/knowledge-course/lectures/201-03-memory-embeddings-rag.htmlCited by the RAG, Embeddings, Vector DB and Memory tiers rows — the whole retrieval layer, one unit deep.
-
[3]
KC 101 → KC 401 · Course index
/knowledge-course/Every unit these terms belong to, by layer: Foundations, Building with Agents, Hard Engineering, Systems & Frontier.