Core model terms
9 terms
| Term | Plain English | In Mo’s stack |
|---|---|---|
| Token | The smallest unit of text a model reads or writes, roughly ¾ of an English word; “tokenization” chops text into these. | Billed in tokens: the DeepSeek API invoice, Hermes compaction logs, Ollama’s num_ctx (tokens, not words). |
| Context window | The most tokens a model can see in one request: its entire working memory for that moment. | Hermes refuses agent mode below 64K; Ollama/Mizuki defaults low unless you set num_ctx / OLLAMA_CONTEXT_LENGTH; DeepSeek’s window caps one request. See 101-02. |
| Parameter | A learned number (weight) inside the model. The count (“7B”, “671B”) roughly measures model size and capability floor. | “Qwen 7B on Mizuki” = 7 billion parameters, why it fits your 4080. DeepSeek V4 Flash (via API) is far larger, in someone else’s datacenter. |
| Temperature | A knob that flattens or sharpens the probability distribution: higher = more random, lower = more deterministic. | Hermes model config and the DeepSeek API temperature param; in Ollama, temperature in a Modelfile or OLLAMA_TEMPERATURE. |
| Top-p | “Nucleus sampling” cutoff: sample from the smallest set of tokens whose combined probability reaches p (e.g. 0.9). Another randomness knob. | Next to temperature in Hermes provider config, Claude Code model settings, and Ollama’s top_p option. |
| Logit | The raw, pre-softmax score the model gives each possible next token: its unnormalized preference before probabilities. | Invisible daily, but it’s what temperature and top-p act on; a “logit bias” API parameter (DeepSeek, OpenAI-compatible endpoints) pokes it directly. |
| Sampling | The step that picks the next token from the probability distribution: the model doesn’t “choose”, it rolls weighted dice. | Runs on every streamed token from Ollama on Mizuki and every DeepSeek response; num_predict (how many to sample) and stop sequences are sampling-side controls. |
| Attention | The transformer mechanism where every token weighs every other token; cost grows quadratically with context length. | Why long contexts cost more, and Mizuki’s VRAM fills as sessions lengthen. |
| Hallucination | Confident output that isn’t grounded in the input or reality: the model fluently inventing. | Why your fact-check-gate skill exists, and why the stack prefers tool-grounded answers to memory. |
Training terms
6 terms
| Term | Plain English | In Mo’s stack |
|---|---|---|
| Pretraining | The first stage: predict the next token over trillions of words of internet text until the model learns language and world structure. | You never do this; you rent the result (DeepSeek V4 Flash, the Qwen models on Mizuki). |
| SFT(supervised fine-tuning) | The “instruction tuning” stage: train the pretrained model on human-written question-answer pairs so it follows instructions instead of just completing text. | The difference between a raw base model and the -instruct/chat variants you ollama pull (e.g. qwen2.5:7b-instruct). |
| RLHF(reinforcement learning from human feedback) | Alignment stage: a reward model learned from human preferences guides further training so the model is helpful and refuses harm. | Why the DeepSeek API model feels cooperative and safe rather than raw-text-completing. |
| Fine-tuning | Continuing training on a smaller, specialized dataset to change behavior or style of an already-trained model. | Your character work: face-lora-training and the FAL hosted LoRA pipeline, teaching a model a face or style it didn’t know. |
| Quantization | Shrinking weights from high precision (fp16) to low (int8/4-bit) so models fit in less VRAM, at a small quality cost. | Why Mizuki can run anything: GGUF quants like qwen2.5:7b-q4_K_M, where q4 is 4-bit, chosen by VRAM budget (local-llm-checker). |
| LoRA(low-rank adaptation) | A fine-tuning shortcut: instead of updating all parameters, train tiny low-rank matrices that slot onto the frozen model. Cheap, fast, swappable. | Your miya-character-lora and face-lora-training skills; FAL hosted LoRA training returns a LoRA file, not a whole model. |
Agent terms
11 terms
| Term | Plain English | In Mo’s stack |
|---|---|---|
| Agent | A model wrapped in a loop: reason → act (call a tool) → observe the result → repeat until the task is done, not one-shot Q&A. | Hermes itself, Claude Code headless, and every subagent they spawn. |
| Tool call | The model emitting a structured request to run a function (read a file, run a command, search the web), not plain text. | Every read_file, terminal, web_search, patch entry in a Hermes transcript; Claude Code shows the same as “Tool Use” blocks. |
| Function calling | The API-level mechanism behind tool calls: you declare JSON schemas, the model picks one and fills in the arguments. | The DeepSeek API’s tools parameter; Ollama’s tools support on Mizuki; every Hermes tool schema is a function-calling declaration. |
| MCP(Model Context Protocol) | An open standard (JSON-RPC) for exposing tools and data to agents: one server, any client. The USB-C of agent tooling. | Hermes’s native MCP client (mcp in config.yaml), the mcp-server-setup skill, and servers you run: open-design, fmp (market data), color-scheme, design-inspiration. |
| Orchestrator | The top-level agent that plans, delegates, and integrates. It decides who does what and assembles the result. | Your Hermes main agent, and Claude Code’s top-level process with its Task/Agent tool; kanban routes specialist work through it. |
| Subagent | A child agent spawned for a scoped job with its own fresh context; it reports a summary back to the parent. | Hermes delegate_task: this lecture was written by a subagent (Mara) under a parent; the subagent-driven-development and kanban-delivery skills run on this. |
| System prompt | The persistent instruction block at the top of the context that defines identity, rules, and behavior. Always present, always authoritative. | Hermes’s system prompt plus your SOUL.md and profile wiring; Claude Code’s CLAUDE.md. Compaction preserves it because it outranks everything after. |
| RAG(retrieval-augmented generation) | Before answering, retrieve relevant documents and stuff them into the context so the model answers from evidence, not memory. | The subject of 201-03 (memory-embeddings-rag); Mnemosyne pulling stored notes back into a session. |
| Embeddings | Vector (list-of-numbers) representations of text where similar meaning ≈ nearby vectors: the model’s “semantic coordinates”. | Mnemosyne’s auto-capture turns every session turn into embeddings; embedding models run via Ollama (ollama pull nomic-embed-text) or an API. |
| Vector DB | A store that holds embeddings and answers “what’s most similar to this?” via nearest-neighbor search. | Mnemosyne’s store; the memory-architecture skills; the Obsidian-vault retrieval layer. The DB is the disk behind RAG. |
| Memory tiers | The layered memory of an agent stack: context window (RAM) → archived sessions (disk) → semantic memory (Mnemosyne) → filesystem (source of truth). | Hermes session_search reads the archived tier, Mnemosyne auto-captures the semantic tier, Open Design pins context in DESIGN.md/BRIEF.md; each tier retains and costs differently. |
Serving terms
8 terms
| Term | Plain English | In Mo’s stack |
|---|---|---|
| Inference | Running a trained model to produce output: the opposite of training, and what you pay for per request. | Every DeepSeek API call and every ollama run on Mizuki is inference; the serving-llms-vllm and dedicated-gpu-llm-server skills make it fast and cheap. |
| VRAM | GPU memory where the weights and the KV cache must live during inference: the hard budget for what fits locally. | Mizuki’s 16 GB RTX 4080 decides which quants fit (q4 yes, fp16 no) and how long contexts get before an out-of-memory kill. |
| KV cache | The stored attention keys/values for every token so far: the model’s scratchpad. Grows with context length and eats VRAM linearly. | Why a 128K context on a 16 GB card is tight; why Ollama’s default num_ctx is VRAM-dependent; what prompt caching reuses. |
| Prefill | The first pass of inference: process the whole input in parallel, filling the KV cache. Sets up the state for generation. | Visible as “time to first token”: longer input means longer prefill; it scales with input size, not the reply. |
| Decode | The second pass: generate the answer token by token, each step reading the KV cache and appending to it. | The token stream in the Hermes TUI and Claude Code; long outputs are slow (one token per step), and num_predict caps decode time. |
| Prompt caching | Reusing a stored KV cache when a new request shares a prefix with an old one. Same system prompt, same tools: don’t recompute them. | DeepSeek’s API prices cache hits at a discount (watch “cache hit” line items); Hermes caches the system prompt and tool schemas, so later turns are cheaper. |
| vLLM | A high-throughput inference server: PagedAttention (VRAM-efficient KV cache), continuous batching, OpenAI-compatible API. What serious self-hosting runs. | The serving-llms-vllm skill; under most hosted providers; deploy on Mizuki when several people (or agents) share one local model. |
| Ollama | The friendly local model runner: pulls GGUF models, manages context, exposes an OpenAI-compatible API, serves your GPU box. | Mizuki, full stop: ollama pull/run/ps, OLLAMA_CONTEXT_LENGTH, the local-llm-checker Qwen 7B audit, and embedding models for Mnemosyne. |
The map
How the terms chain together in the system you run:
- Training makes the model. Pretraining → SFT → RLHF produce the weights (parameters). Quantization and LoRA let you reshape or shrink those weights to fit your hardware; you touch this layer once, at
ollama pulltime. - Inference spends the model. Your text becomes tokens inside a context window. The server runs prefill (all input tokens at once, filling the KV cache), then decode (one token at a time, sampling logits under temperature and top-p). Weights + KV cache must fit in VRAM: that’s why quantization matters locally and your API bill grows with input length.
- Serving manages the cost. Prompt caching reuses the KV cache for your stable prefix (system prompt, tool schemas), so repeated turns are cheap. vLLM and Ollama implement all of the above: one for scale, one for your desk.
- Agents live on top. The model is just the token machine. A system prompt + tool calls + the observe-reason loop make it an agent; an orchestrator with subagents scales it. Function calling and MCP are the two plumbing standards for tools, and RAG (embeddings → vector DB) plus memory tiers give the loop long-term memory.
tokens fill the context, attention fills the KV cache, the KV cache fills VRAM (or your bill), and everything an agent does is a loop of spending that budget on tool calls.
Key takeaway
Key idea
Every term here is a knob or budget you already touch daily; you just didn’t have the name for it. When something breaks, classify it by layer: a bad answer is sampling (temperature/top-p) or grounding (RAG/hallucination); a slow answer is serving (prefill/decode, KV cache, VRAM); a forgetful agent is memory (context window, compaction, memory tiers).
Name the layer and the fix is usually one setting away.
Go deeper
Terms with a whole unit behind them are listed here.
Elsewhere in the course-
[1]
KC 101-02 · Context windows & what fits
/knowledge-course/lectures/101-02-context-compaction.htmlCited by the Context window row: window size, compaction handoffs,
num_ctxandOLLAMA_CONTEXT_LENGTHin full. -
[2]
KC 201-03 · Memory: working vs long-term, embeddings, RAG
/knowledge-course/lectures/201-03-memory-embeddings-rag.htmlCited by the RAG, Embeddings, Vector DB and Memory tiers rows: the whole retrieval layer, one unit deep.
-
[3]
KC 101 → KC 401 · Course index
/knowledge-course/Every unit these terms belong to, by layer: Foundations, Building with Agents, Hard Engineering, Systems & Frontier.