Glossary
A
- Acceptable use policy (AUP, acceptable use, prohibited use policy)
a list of uses a model licence forbids, usually incorporated into the licence by reference so that breaking it breaks the licence.
- Agent instructions file (instructions file)
a markdown file in a repository that a harness reads into the context at the start of every session, holding build commands, conventions, and standing instructions.
- Agentic loop (agent loop)
the cycle of calling the model, running the tools it asks for, appending the results to the conversation, and calling again until the model asks for nothing.
- Audit log (audit logging, audit trail)
the record of which principal called which tool, with which arguments, against which server, and what came back. No single MCP server can produce one, because a server never learns which human is behind the client.
C
- Cache prefix
the run of bytes from the start of a request that must match a stored request exactly for a cached prompt to be reused. The first difference ends the match.
- Capability negotiation
the declaration of which optional features each side supports, so neither a client nor a server uses one the other lacks.
- Chain of thought (chain-of-thought, CoT)
written-out intermediate steps between a question and an answer. It improves accuracy on multi-step problems, and it is not a faithful record of how the model reached the answer.
- Checkpointing (checkpoint, checkpoints)
a harness snapshotting the working tree as a run proceeds, so the files can be rolled back to an earlier point in the conversation.
- Chunking (chunk, chunks)
cutting documents into pieces before embedding them. The chunk size and the overlap between chunks decide what can ever be retrieved together.
- Closed model (closed models, proprietary model)
a model reachable only as a hosted service, whose weights are never released. The vendor can change or retire it under the name you call.
- Cold start (cold starts)
the delay while a weights file is loaded into memory before the model can answer anything. It is the reason a model is usually kept resident.
- Compaction (compact, compacting)
replacing a stretch of conversation history with a generated summary of it, done by a harness at a threshold rather than at the point of failure.
- Context assembly
the harness's work of building the exact text sent on each call, from the system prompt and tool definitions through the history to the user's question.
- Context budget
the plan for how many tokens each part of the context is allowed, and what gets dropped or summarised when the total will not fit.
- Context management
deciding what stays in the context window, what leaves, and what replaces it, as a conversation grows past what the window holds.
- Context window (context windows, window)
the maximum number of tokens a model can hold for one call. It is a single budget covering the whole request and the generated answer together, not an upload limit.
- Continuous batching (batching, in-flight batching)
adding new requests to a batch that is already generating, as slots free up, so a card is rarely producing tokens for a single user.
D
- Decode (decoding)
the phase that generates output tokens one at a time, each one depending on the last. It is serial by construction and accounts for most of the wall-clock time of a long answer.
E
- Embedding (embeddings, vector)
a fixed-length list of numbers standing for a piece of text, positioned by an embedding model so that similar meanings land near each other.
F
- Fallback chain (fallback, fallbacks)
an ordered list of models to try, where the next one gets the request when the one before it errors, refuses, or times out. A fallback to a weaker model succeeds quietly and changes the answer.
- Fine-tuning (fine-tune, fine tuning)
continuing to train a published model on your own examples so that it behaves differently. It changes the weights without revealing how they were made.
- Foundation model (foundation models)
a model trained once, at great expense, on a broad corpus, then adapted or prompted for many different tasks rather than trained again for each one.
- Function calling
another name for tool calling, used by several vendors for the same mechanism of emitting a structured call for the caller to execute.
G
- Generative AI (generative artificial intelligence)
systems that produce new text, images, audio, or code from a prompt, rather than classifying or scoring something that already exists.
- Greedy decoding (greedy sampling)
always taking the highest-probability token. It removes the sampler's randomness but not the provider's, so a hosted call is still not guaranteed to repeat.
H
- Hallucination (hallucinations, hallucinate)
a fluent, confident output that is not true. The model produces likely-looking text and has no mechanism for telling what it does not know.
- Harness (harnesses)
the program between you and the model API. It assembles the context, executes the tools the model asks for, enforces permissions, and keeps calling the model until the job is done.
- Headless mode (headless, print mode)
running a harness non-interactively. One prompt in, a result out, an exit code set, and nothing to confirm. It is what makes a harness usable in a pipeline.
- Hook (hooks)
a script the harness runs when an event fires, such as after a file edit, with no model involvement in the decision to run it.
- Hybrid search
combining keyword matching and vector search over the same corpus and merging the two result lists, so exact identifiers and paraphrases both work.
I
- Inference (inference call)
running a trained model to produce an output. The weights do not change and no learning happens.
- Inference server (serving stack, model server)
the software that keeps weights resident, exposes an API, manages the per-request caches, and batches many requests into one pass over the weights.
- Input tokens (input token, prompt tokens)
everything sent to the model in a request, including the system prompt, the tool definitions and every earlier turn of the conversation. Billed at a lower rate than output.
K
- Knowledge base
the documents a retrieval system searches, their embeddings, the store holding them, and whatever keeps the three in step.
L
- Large language model (large language models, LLM, LLMs)
a model trained to predict the next chunk of text, and big enough that predicting text well requires it to encode a great deal about the world.
- LLM gateway (LLM gateways, AI gateway, model gateway)
a service inside an organisation that sits between applications and model vendors. It holds the vendor credentials and decides which model gets each call, what the call costs, and whether the caller is allowed to make it.
- Local inference (local, self-hosted inference, on-premise inference)
running a model on hardware you control, so that no prompt or document leaves your network.
- Lost in the middle
the tendency of a model to use material placed in the middle of a long context less reliably than the same material placed at the start or the end.
M
- MCP client
a connector inside the host that speaks the protocol to exactly one server. A host with three servers holds three clients.
- MCP gateway (MCP gateways)
a service that presents many MCP servers to a client as one. It holds the downstream credentials, decides which servers and tools each client may reach, and records what was done.
- MCP host (host)
the application the user interacts with, which holds the model connection, owns the conversation, and decides what is allowed to run.
- MCP prompt (MCP prompts)
a template a server publishes for the user to invoke, often surfaced by a host as a slash command.
- MCP resource (MCP resources, resource)
data a server exposes at a URI for the client to read. The application decides what to attach, rather than the model fetching it.
- MCP server (MCP servers)
a separate program exposing tools, resources, and prompts over the protocol. It never sees the conversation and never talks to the model.
- Memory
text written outside the model and put back into the context on a later call, so that something appears to carry over. The model itself retains nothing between calls.
- Memory write
an update to a memory store during a session, decided by the model, by the user, or by a rule in the application.
- Message (messages)
one entry in the conversation history, carrying a role such as user, assistant or tool result, and its content. The whole list is resent on every call.
- Model Context Protocol (MCP)
an open protocol, using JSON-RPC, for connecting AI applications to external tools and data, so each integration is written once rather than once per application.
- Model coupling
how tightly a harness is bound to one vendor's models, and whether you can point it at another endpoint. It decides whether an LLM gateway can sit in front of it.
- Model licence (model license, community licence)
the terms attached to a weight release. Often bespoke rather than Apache-2.0 or MIT, and often carrying conditions an open source licence could not.
- Model routing (routing, router)
choosing which model handles a request. The choice can be made by alias, by task class, by cost, by observed latency, or by which models are currently healthy.
N
- Next-token distribution (next token distribution, logits)
the probability a model assigns to every token in its vocabulary as the next one, produced by applying softmax to the raw scores from one forward pass.
O
- Open-weights model (open weights, open model)
a model whose weights are published for download, under whatever licence the publisher chose. Downloadable does not mean open source.
- Output tokens (output token, completion tokens)
everything the model generates, produced one token at a time and billed at a higher rate than input, commonly three to five times higher. Thinking tokens count here too.
P
- Parallel tool calls (parallel tool use, parallel function calling)
several tool-call blocks in one reply, which the caller may run concurrently. Every one of them needs a matching result.
- Permission model (permission modes)
the rules a harness applies to decide which tool calls run without asking, which stop and wait for your confirmation, and which are refused outright.
- Persistent memory
a small store that survives the end of a session and is read back into the context at the start of the next one.
- Plugin (plugins)
a distributable, versioned bundle that can hold skills, slash commands, subagent definitions, hooks, and MCP server configuration together.
- Policy enforcement (policy)
the gateway deciding, per request, whether this principal may call this tool on this server. The answer is allow, deny, or held until a named human approves it.
- Prefill (prefilling)
the phase that processes the whole prompt in one parallel pass, before any output token exists. Its cost grows with prompt length, which is why a large prompt is slow to start.
- Progressive disclosure
keeping only a name and a one-line description in the context, and loading the body only when it is needed, so many skills cost little until used.
- Prompt (prompts)
the text sent to a model for it to continue, including everything the harness added that you never typed.
- Prompt caching (prompt cache, context caching)
a provider feature that stores the processed form of a request prefix for a short time and charges a reduced rate to reuse it on a later call.
Q
- Quantization (quantized, quantised, quantisation)
storing each weight in fewer bits, such as 4-bit instead of 16-bit, to cut memory and raise speed. The cost is accuracy, and the damage is uneven.
R
- Rate limiting (rate limit, rate limits)
capping or queueing requests, both to stay inside a vendor's own limits and to stop one team consuming the whole organisation's quota.
- Reasoning effort (thinking budget, thinking level)
the dial that sets how much working a reasoning model does, expressed either as a coarse level such as low or high, or as a target token budget.
- Reasoning model (reasoning models, thinking model)
a model trained to produce a long private working before its answer, and to check and correct itself while doing so, rather than one prompted into showing its steps.
- Recall
reading a memory store back into the context, usually at the start of a session and usually with no query to match it against.
- Reranking (reranker, rerank)
scoring retrieved candidates with a second, small model that reads the query and the chunk together, then keeping only the best few.
- Retrieval-augmented generation (RAG)
retrieving relevant text from a corpus and putting it in the context window before asking the model to answer, so the answer rests on documents the model was never trained on.
S
- Sampling (sampler, samplers)
the step outside the model that picks one token from the next-token distribution. It lives in the serving stack rather than in the weights, and it is where the visible randomness comes from.
- Sandbox (sandboxing, sandboxed)
an enforced boundary around tool execution, covering which files a tool may read or write and which hosts it may reach. It holds whatever the model has been persuaded to try.
- Semantic cache (semantic caching)
a cache that matches an incoming prompt against stored prompts by embedding distance rather than by exact bytes. It hits far more often than an exact-match cache and can answer a question that was not asked.
- Semantic search
finding text by distance between embeddings rather than by matching words, so a paraphrase with no shared vocabulary can still be found.
- Server registry (registry, approved server list)
the reviewed list of MCP servers an organisation has approved, with an owner and a pinned version for each. It turns adding a server from a pasted config line into an act with a name on it.
- Session memory
the conversation history, resent in full on every turn and discarded when the session ends.
- Skill (skills, agent skill, agent skills)
a named set of instructions, packaged as markdown, that a harness loads into the context when it looks relevant. It adds no new capability.
- Slash command (slash commands)
an instruction the user invokes by name, rather than one the model chooses to load on its own.
- Spend cap (spend caps, budget)
a budget enforced on the request path, which refuses calls once it is exhausted. A cap anywhere off the path can only report an overrun after the money is gone.
- stdio transport (stdio)
a server run as a local subprocess, exchanging newline-delimited JSON-RPC over its standard input and output, with credentials taken from its environment.
- Stop condition (stop conditions)
a limit that ends an agentic run whether or not the job is finished, such as a turn cap, a token budget, a wall clock timeout, or a user interrupt.
- Streamable HTTP transport (streamable HTTP)
a server reached over HTTP, where each message is a POST to one endpoint and the reply is a JSON object or a request-scoped stream.
- Streaming
sending each output token to the caller as it is decoded, rather than holding the whole response. Total generation time is unchanged; the wait is just spent reading.
- Structured output (structured outputs, JSON mode)
forcing the model's reply to match a JSON Schema, with no tool and no execution involved. The same schema machinery pointed at the answer rather than at an action.
- Subagent (subagents)
a nested agentic loop with its own context window, given one task and returning only its result, so a large search does not fill the main conversation.
- System prompt (system prompts, system instruction)
the block of instructions an application or harness puts in front of the conversation, setting the model's role, its rules and the shape of its output. The reader usually never sees it.
T
- Temperature
a sampler setting that flattens or sharpens the whole next-token distribution before a token is picked. Below 1 the leading candidate pulls further ahead; above 1 the also-rans get a real share.
- Test-time compute (inference-time compute)
spending more computation at the moment of answering rather than at training time, as a way of buying accuracy. A smaller model thinking hard can beat a larger one answering immediately.
- Thinking tokens (thinking token, reasoning tokens)
the tokens of a reasoning model's private working. They are billed at the output rate and occupy the context window whether or not the provider shows them to you.
- Time to first token (TTFT)
the delay before the first output token reaches the caller, driven mostly by prompt length and queueing rather than by how long the answer will be.
- Token (tokens, tokenisation, tokenization)
the unit a model reads and writes, a chunk of a few characters drawn from a fixed vocabulary. Common English words are often one token; rare names, long numbers and punctuation-dense text cost several.
- Tokenizer (tokenizers, tokeniser)
the software that splits text into tokens, using a vocabulary learned once from a corpus and frozen before training. Each model family has its own, so a token count is only meaningful against a named model.
- Tokens per second (throughput, tok/s)
throughput. It means either what one request feels like as text appears, or what the machine produces across every request at once, and the two differ a lot.
- Tool (tools)
a capability the caller is willing to run on the model's behalf, offered to the model as a name, a description, and a schema.
- Tool call (tool calls, tool use)
a structured block the model emits naming a tool and its arguments. It executes nothing; the caller decides whether to run it.
- Tool definition (tool definitions)
the name, description, and JSON Schema sent in the context so the model knows a tool exists. Every definition costs tokens on every call.
- Tool namespacing (namespacing)
prefixing each tool with the server it came from, so two servers exposing a tool of the same name can both be presented to one client. The prefix is text the model reads, so it changes behaviour as well as resolving the collision.
- Tool result (tool results)
a tool's output, appended to the conversation as a message and sent back to the model. It is text, and it costs context like any other text.
- Top-k (top k)
how many nearest chunks a search returns. A small k keeps the window clean; a large k refills it with noise.
- Top-p (nucleus sampling, top_p)
a sampler setting, also called nucleus sampling, that keeps only the smallest set of candidates whose probabilities sum to p and discards the rest before picking.
- Trace (traces, tracing)
the stored record of one request through a gateway, holding the prompt, the routing decision, every attempt including the failed ones, the tokens counted, and the response.
- Truncation (truncate)
dropping the oldest messages until a request fits, with no summary and no record of what was removed.
- Turn (turns)
one pass through the agentic loop. One model call, plus whatever the harness does with the answer before the next call.
V
- Vector store (vector stores, vector database)
a database that holds embeddings beside the text they came from and answers nearest-neighbour queries against them, usually through an index rather than a full scan.
- Virtual key (virtual keys)
a credential the gateway issues and controls, held by an application and never seen by the model vendor. It can be revoked, capped or attributed without touching the vendor's own key.
W
- Weights (model weights, parameters)
the file of numbers a training run produced, holding everything the model learned. Useless without code to run them, and the part that cost the money to make.