What fills a context window

When you send a single line of text to a modern coding assistant, the model rarely receives just that single line. Behind the scenes, the application packages your query inside an extensive payload that can easily exceed thirty thousand —of which your prompt accounts for less than one percent. Everything else is assembled dynamically by the client software and injected into the request without ever appearing on your screen.

This invisible payload is where most practical questions about cost, latency, and model behavior originate. When a brief prompt unexpectedly costs forty cents, or when an assistant suddenly forgets a constraint you provided ten minutes earlier, the explanation almost always lives in this assembled context.

Anatomy of a prompt payload

To understand where tokens go, it helps to examine what a client environment bundles into each API request. While individual APIs may sequence these components differently on the wire—for instance, placing tool definitions ahead of system instructions for prompt-caching efficiency—the overall contents remain remarkably consistent across harnesses:

• The system prompt: A foundational block of instructions injected by the application or . It establishes the model's persona, permissible behaviors, output formatting rules, and safety boundaries. While a casual consumer chatbot might use a system prompt of a few hundred tokens, an engineering harness outlining complex code-editing schemas, environment restrictions, and terminal formatting rules often consumes several thousand tokens before conversation even starts.
• Tool definitions: Whenever a model is granted the ability to call external functions, every available must be described within the request payload using its name, human-readable description, and a complete JSON schema for its arguments. A single well-documented tool definition typically requires between 100 and 800 tokens depending on parameter complexity. If you connect multiple MCP servers exposing dozens of capabilities, tool definitions alone can easily consume 15,000 to 20,000 tokens on every single turn, regardless of whether any tools are invoked.
• Project instructions and guidelines: Many developer tools automatically discover and ingest repository configuration files from disk—such as AGENTS.md instructions, modular , style guidelines, or architecture runbooks. Placing a detailed 1,000-line documentation file in the repository root means that entire file gets injected into every single API call.
• Conversation history: The chronological transcript of previous interactions, containing every sent by the user, every assistant completion, every tool invocation requested by the model, and the raw text of every tool execution result. Because model APIs are stateless and retain no memory between calls, the client harness must resend the complete cumulative transcript on every subsequent turn.
• Retrieved reference material: Knowledge snippets and documentation pulled from a vector database or search index and inserted into the prompt to ground the model's answer, as detailed in the retrieval guide.
• Read file contents and tool outputs: Whenever an assistant reads a file from disk or runs a shell command, the resulting data is stored as a tool result in the conversation history. Reading a 2,000-line source file can inject 25,000 tokens into the history, where it remains for all future turns unless evicted.
• Multimodal assets: Attached screenshots, UI mockups, and diagrams. Image tokens can add up quickly depending on image resolution and the provider's multimodal tokenization scheme.

The orchestration software's primary responsibility is —gathering, formatting, and ordering these varied elements into a valid API payload on every interaction. The architectural plan that determines how much space each component may occupy is the .

Context arithmetic in practice

Consider a typical coding session using a 200,000-token context window. On turn 3, the token breakdown might look like this:

Component	Tokens
System prompt	2,400
Active tool definitions	11,000
Repository instructions	3,600
Prior conversation history	4,800
Retrieved codebase documentation	6,200
User query	180
Total sent in request	28,180

At this stage, the user's question represents just 0.6% of the total request, while standing tool definitions and background configuration account for the overwhelming majority of the payload.

Fast-forward to turn 20 of the same session. The system prompt, tool definitions, and repository guidelines remain steady, but the conversation history has now accumulated nineteen conversational exchanges, multiple file inspections, and dozens of terminal outputs. The history alone might now occupy 128,000 tokens. As a result, turn 20 costs nearly five times more than turn 3 for a question of identical length, and response latency increases accordingly.

Both bars represent the same context window. Between turn 3 and turn 20, the conversation history expanded to occupy two-thirds of the available capacity.

Attention distribution: lost in the middle

A does not receive uniform attention across its entire span. Transformer attention mechanisms attend to different positions with varying intensity, meaning that identical instructions can be followed or overlooked depending on where they are placed within the prompt.

This phenomenon is commonly known as the effect. Empirical studies have demonstrated that models recall information and follow constraints significantly better when relevant content appears near the very beginning or the very end of a prompt, whereas information buried deep in the middle suffers from reduced retrieval fidelity. While newer architectures have improved long-context reasoning, the effect remains noticeable in production.

Content placed at the beginning (such as the system prompt) benefits from early positional prominence and specialized alignment training. Content positioned at the very end—right before the model begins generating—benefits from recency bias. Consequently, high-priority instructions belong either in the system prompt or immediately adjacent to the user query. Placing critical rules in the middle of a 100,000-token historical transcript makes them much more likely to be ignored.

This also explains why long-running agents sometimes exhibit behavioral drift after dozens of turns: decisions or instructions established on turn 2 become buried under thousands of lines of subsequent tool logs.

Troubleshooting unexpected model behavior

When a model provides an incorrect or unhelpful answer, the natural first impulse is to rewrite the prompt. In many cases, however, the prompt was clear, but the context assembly was flawed.

Before tweaking prompt phrasing, inspect the exact payload sent to the API. Check whether the necessary file was actually included or merely referenced by name. Verify whether an essential instruction was buried in the middle of a massive message history, whether an unneeded tool distracted the model, or whether outdated tool outputs from earlier turns contradicted your current request.

Before you rewrite the question

Check the input before you change the wording.

1. Confirm the file or fact is in the context. Do not assume a tool read it.
2. Move the instruction you need obeyed into the system prompt or the last message.
3. Remove tool definitions the task does not need.
4. Start a new session when the history is longer than the work requires.
5. Rewrite the question only after the input is right.

The rest of this section follows from the same arithmetic. Context management is what happens when the parts stop fitting. Choosing which documents earn a place is retrieval, and memory is how anything survives to the next session at all.

Terms introduced

• System prompt: the block of instructions an application or harness puts in front of the conversation, setting the model's role, rules and output format.
• Message: one entry in the conversation history, carrying a role such as user, assistant or tool result, and its content.
• Context assembly: the harness's work of building the exact text sent on each call, from the system prompt through to the question.
• Context budget: the plan for how many tokens each part of the context is allowed before something has to be dropped or summarised.
• Lost in the middle: the tendency of a model to use material in the middle of a long context less reliably than the same material at the start or the end.

---

How providers do it

The three APIs agree on what goes into a window and disagree on where the system prompt is put. Two treat it as a field beside the messages; one treats it as a message.

Concept	Anthropic	OpenAI	Google
System prompt	Top-level system field. No system role exists	A developer message in messages, or instructions on Responses	Top-level system_instruction field, text only
Conversation history	messages, roles user and assistant	messages on Chat Completions, input on Responses	contents
Tool definitions	tools, schema in input_schema	tools, schema in parameters	tools, function declarations
Tool schemas billed as input	Yes, stated in the docs	Yes, stated in the docs	Yes, plus a reported total_tool_use_tokens
Hidden tool overhead	An automatic tool-use system prompt, with a published per-model token cost	Not documented as a separate figure	Not documented as a separate figure
Count a request before sending	POST /v1/messages/count_tokens, free	Not checked	models.<model>:countTokens

Every row above is confirmed except the OpenAI count-before-sending row. Window sizes, image token costs and per-model figures are left open in the tabs.

The row that changes a design is the last one. Where a free counting endpoint exists, a context budget can be measured before a call rather than inferred from the bill afterwards.

Anthropic

What this maps to: the Messages API request body. The parts this page describes are separate fields on the request: system for the system prompt, tools for the tool definitions, and messages for the history.

Question	Answer	Status
How is the system prompt passed?	A top-level system parameter. The docs state plainly that there is no "system" role for input messages	confirmed
What can system hold?	A string, or an array of text content blocks	confirmed
What roles do messages take?	user and assistant. Tool results are content blocks inside a user message	confirmed
Are tool definitions billed?	Yes. The pricing note counts "the total number of input tokens sent to the model (including in the tools parameter)", naming tool names, descriptions and schemas	confirmed
Is there overhead beyond the schemas?	Yes. Enabling any tool injects an automatic tool-use system prompt with a published per-model token cost. The table gives 286 tokens for Opus 5 with tool_choice of auto or none, and 406 with any or tool	confirmed
Can the total be counted before sending?	Yes. POST /v1/messages/count_tokens takes the same body, including system, tools, images and PDFs, and returns input_tokens. It is free and has its own rate limits	confirmed
Does the count endpoint account for caching?	No. The docs say it does not use caching logic, so the number is the uncached size	confirmed
What does an image cost?	Not checked. The count endpoint accepts images, so measure rather than estimate	unconfirmed
What is the window size per model?	Not checked today. Read it off the model page rather than assuming	unconfirmed

Their vocabulary

Standard term	Their term
System prompt	System prompt, passed as the top-level system parameter
Tool definition schema	input_schema
Tool that runs on the caller's machine	Client tool
Tool that runs on the provider's infrastructure	Server tool
Deferring tool definitions out of the context	defer_loading, with the tool search tool returning a tool_reference

Where to look

The usage object on every response breaks the call into input and output tokens. For a figure before sending, call /v1/messages/count_tokens with the exact body you intend to send, including tools.

Last verified: 2026-09-09 against the Messages API reference (https://platform.claude.com/docs/en/api/messages), the tool use overview (https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview) and the token counting guide (https://platform.claude.com/docs/en/build-with-claude/token-counting).

OpenAI

What this maps to: two APIs with different shapes. Chat Completions puts everything in one messages array. The Responses API adds an instructions parameter that sits outside the input.

Question	Answer	Status
How is the system prompt passed in Chat Completions?	As a message with a role. There is no separate field	confirmed
Which role?	developer. The docs say that with o1 models and newer, developer messages replace the previous system messages. system is still accepted	confirmed
How is it passed in the Responses API?	An instructions parameter, described as a system or developer message inserted into the model's context. It takes priority over a prompt in input	confirmed
Does instructions carry across a chained turn?	No. Used with previous_response_id, the previous response's instructions are not carried over	confirmed
What is the instruction hierarchy?	Developer, then user, then assistant	confirmed
Are tool definitions billed?	Yes, and the docs are explicit: functions are injected into the system message, "count against the model's context limit and are billed as input tokens"	confirmed
What does the vendor suggest about tool bloat?	Limit the functions loaded up front, shorten descriptions, or use tool search for deferred loading	confirmed
Is there a free endpoint that counts a request before sending?	Not checked today	unconfirmed
What does an image cost?	Not checked today	unconfirmed

Their vocabulary

Standard term	Their term
System prompt	Developer message, or the instructions parameter on Responses
Tool definition schema	parameters, with strict for schema adherence
Conversation history	input on Responses, messages on Chat Completions

Where to look

The usage object on a response splits input, output and cached input tokens. On the Responses API, previous_response_id chaining still bills every earlier input token again, so read usage.input_tokens per turn rather than assuming the server-held state is free.

Last verified: 2026-09-09 against the text generation guide (https://developers.openai.com/api/docs/guides/text), the function calling guide (https://developers.openai.com/api/docs/guides/function-calling) and the Responses create reference (https://developers.openai.com/api/reference/cli/resources/responses/methods/create).

Google

What this maps to: the Gemini API request. The docs now front an Interactions API and label the older generateContent surface as legacy, and both accept the same three pieces: a system instruction, a contents list, and tools.

Question	Answer	Status
How is the system prompt passed?	A separate top-level field, system_instruction in the guides and systemInstruction in the REST reference. Not a role in the message list	confirmed
What can it hold?	A Content object, documented as "Currently, text only"	confirmed
What roles does contents allow?	The examples show user and model. No page seen today enumerates the allowed values	unconfirmed
Are tool definitions billed?	Yes. The token guide states that tools, including functions, code execution and Google Search, are counted, and usage reports a separate total_tool_use_tokens	confirmed
Are system instructions billed?	Yes, counted as part of the input tokens	confirmed
Can the total be counted before sending?	Yes. POST /v1beta/models/<model>:countTokens returns the total number of tokens in the input	confirmed
What is the window size?	Stated per model rather than in one table. Gemini 3.8 Flash and Gemini 2.5 Pro both list an input limit of 1,048,576 tokens and an output limit of 65,536	confirmed
Can the limit be read at runtime?	Yes, models.get returns input_token_limit and output_token_limit	confirmed
Do the limits hold for every other model?	Only two model pages were read today	unconfirmed

Their vocabulary

Standard term	Their term
System prompt	System instruction (system_instruction)
Conversation history	contents
Tool definition	Function declaration (tools[].function_declarations[] on the legacy surface)
Tokens spent on tool machinery	total_tool_use_tokens

Where to look

countTokens before the call, and the usage block after it. The per-model pages carry the window sizes, and models.get returns the same numbers if you would rather not hardcode them.

Last verified: 2026-09-09 against the token counting guide (https://ai.google.dev/gemini-api/docs/tokens), the generateContent reference (https://ai.google.dev/api/generate-content), the text generation guide (https://ai.google.dev/gemini-api/docs/text-generation) and the Gemini 3.8 Flash model page (https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash).

---

Check your understanding

0 of 4 answered

1. On a typical coding-assistant call, which part of the context did the user write?

   The question, which is usually a fraction of a percent of the tokens sent.The system prompt and the question.The tool definitions, since the user chose which tools to connect.All of it. The harness only forwards what was typed.
2. A harness has twelve MCP servers connected, each exposing about twenty tools. The user asks a question that needs no tool at all. What does that cost?

   Every tool definition is still sent and still billed, because the model has to see a tool described before it can decide not to use it.Nothing. Definitions are only sent when the model calls a tool.Nothing, because the provider caches tool definitions permanently after the first call.A small fixed fee per connected server, independent of how many tools it exposes.
3. An agent gives a wrong answer about a file it was asked to change. What is the first thing to check?

   Whether the file's contents were ever in the context, rather than just its name.Whether the temperature is set too high.Whether the question was phrased ambiguously.Whether a larger model would do better.
4. A long session has 140,000 tokens of history. A requirement stated at turn 6 is being ignored. Where should it be moved?

   Into the system prompt, or into the most recent message.Roughly halfway through the context, so the model reads it twice.Nowhere. If it is in the window at all, the model weights it equally.Into a tool description, so it is loaded before the conversation.