Tokens and the context window
When you send text to a large language model, it doesn't parse individual characters or complete words directly. Instead, it reads and writes in Tokenthe unit a model reads and writes, a chunk of a few characters drawn from a fixed vocabulary. Common English words are often one token; rare names, long numbers and punctuation-dense text cost several.Full glossary entryIntroduced in Tokens and the context window—discrete chunks of characters drawn from a fixed vocabulary established during training. Tokenization is the translation step that chops raw text into this sequence of numerical identifiers before computation ever begins.
In everyday English, common words usually map to a single token. Simple words like The, cat, and sat are each individual tokens, where the leading whitespace is typically attached directly to the word. Less common words, however, get split into smaller sub-word fragments. An unusual family name, technical jargon, or an unfamiliar acronym might take three or four tokens to represent. Numeric sequences get split as well, with multi-digit numbers chunked into groups of two or three digits.
Understanding tokenization is essential because every constraint and cost in working with language models is denominated in tokens. API billing, context window limits, and response latency all trace back to token counts.
The tokenizer is a fixed vocabulary
The Tokenizerthe software that splits text into tokens, using a vocabulary learned once from a corpus and frozen before training. Each model family has its own, so a token count is only meaningful against a named model.Full glossary entryIntroduced in Tokens and the context window is the software component responsible for breaking text into these chunks. It operates against a pre-compiled vocabulary—typically ranging from 30,000 to over 200,000 token entries—constructed through algorithms like Byte Pair Encoding (BPE) on a massive training corpus before model training starts. The process begins with raw bytes and iteratively merges the most frequently occurring character pairs into unified vocabulary entries until reaching a target vocabulary size.
Once a tokenizer's vocabulary is created, it is permanently frozen. Because every internal weight in the neural network is mathematically tied to the specific token IDs in that vocabulary, you cannot modify the token dictionary after training without invalidating the model. Any new slang, contemporary names, or brand-new terms coined after that point must be pieced together from existing sub-word fragments.
Tokenizers also vary significantly across model families. Different providers train their tokenizers on different corpora with distinct vocabulary sizes and optimization goals, meaning the exact same paragraph will produce different token counts depending on the model you use. Asking "how many tokens is this document?" has no universal answer until you know which model's tokenizer is processing it. A prompt that takes 3,800 tokens on one model might require 4,500 on another—a discrepancy that can easily push a tightly sized prompt over its limits.
The rule of thumb, and where it stops working
For standard English prose, a common estimation is that one token represents roughly four characters, or about 750 words per 1,000 tokens. This ratio works well for rough back-of-the-envelope planning, but relying on it for precise architectural limits will cause unexpected failures.
Several common types of input shift this ratio dramatically:
- Code and structured data: JSON, YAML, and source code feature dense punctuation and formatting characters that rarely merge into larger tokens. A brief payload like
{"user_id": 10482, "ok": true}is only 30 characters long, yet costs more than twice as many tokens as an equivalent sentence in plain English because braces, quotes, colons, and commas frequently become standalone tokens. Minified code and deeply nested data structures are even less efficient. - Non-Latin alphabets and international text: Most foundation model tokenizers are trained predominantly on English text. Consequently, languages using scripts such as Cyrillic, Arabic, Devanagari, Japanese, or Chinese are split into much smaller fragments. A passage in Japanese can easily require two to three times as many tokens as its English translation, making identical workflows significantly more expensive and context-heavy in other languages.
- Long numbers and timestamps: Financial account numbers, ISO timestamps, and mathematical values get fragmented into small digit clusters. A single timestamp like
2026-09-09T14:32:07Zrequires multiple tokens rather than one. - Unique identifiers and hashes: High-entropy strings like UUIDs, Base64-encoded blobs, and cryptographic hashes represent the worst case for token efficiency, often consuming nearly one token for every character or pair of characters.
Pasting a 400-line CSV export of transaction IDs into a prompt will consume far more tokens than word count metrics suggest. An application sized and tested exclusively on clean English text can easily hit context limits when faced with production stack traces or raw database dumps.
Count tokens with the tokenizer of the model you will call. Do not estimate from a character count.
- Use the provider's token-counting endpoint, or the tokenizer library that provider publishes.
- Count the full request. Include the system prompt, the tool definitions and the whole message history.
- Add the maximum output length you allow to the input count.
- Compare that total against the model's context window before you send the request.
- Repeat the count when you change model. Token counts do not carry across model families.
Everything in the call is tokens
The text typed by the end user is usually only a modest fraction of what actually gets tokenized on a given API call. The System promptthe block of instructions an application or harness puts in front of the conversation, setting the model's role, its rules and the shape of its output. The reader usually never sees it.Full glossary entryIntroduced in What fills a context window supplied by the application or platform is tokenized as well, along with the full history of preceding messages. Because model APIs are stateless and hold no memory between requests, the client must resend the entire conversation on every single turn.
Active Toola capability the caller is willing to run on the model's behalf, offered to the model as a name, a description, and a schema.Full glossary entryIntroduced in Tool calls definitions contribute substantial token overhead too. If a harness provides twenty tools to a model, every request packages twenty tool names, descriptive summaries, and complete JSON schemas for their arguments. This catalog alone often adds several thousand tokens to the payload before a user even enters a prompt—an overhead that frequently gets overlooked during token budgeting.
The model's generated response consists of tokens as well, produced sequentially and billed upon generation.
Cloud providers divide usage into two distinct categories: Input tokenseverything sent to the model in a request, including the system prompt, the tool definitions and every earlier turn of the conversation. Billed at a lower rate than output.Full glossary entryIntroduced in Tokens and the context window represent everything sent in the request, while Output tokenseverything the model generates, produced one token at a time and billed at a higher rate than input, commonly three to five times higher. Thinking tokens count here too.Full glossary entryIntroduced in Tokens and the context window cover what the model generates. Output tokens are typically priced three to five times higher than input tokens. They are also fundamentally slower to process: whereas input tokens are digested in parallel across GPU cores, output tokens must be generated one after another in a sequential loop. A 4,000-token response takes substantially longer in wall-clock time and costs far more than a 200-token reply, as explored in the inference chapter.
Encouraging concise outputs yields tangible cost and latency savings. At the same time, long conversations quietly accumulate heavy input costs over time because the entire history must be resent with each interaction. Techniques like Prompt cachinga provider feature that stores the processed form of a request prefix for a short time and charges a reduced rate to reuse it on a later call.Full glossary entryIntroduced in Context management and compaction help mitigate this expense, and the what fills the window guide details how tokens distribute across real requests.
The window is one budget
The Context windowthe maximum number of tokens a model can hold for one call. It is a single budget covering the whole request and the generated answer together, not an upload limit.Full glossary entryIntroduced in Tokens and the context window defines the maximum total number of tokens a model can hold in memory during a single inference call. This limit is an architectural property of the model itself, not an account quota. Crucially, the input prompt and the generated response must both fit within this single allocation.
A common pitfall is treating the context window solely as an input limit. If a model offers a 200,000-token window and an application sends 199,000 input tokens, only 1,000 tokens remain available for the output. Providers generally require setting an explicit cap on the maximum output tokens allowed, and that reserved capacity subtracts directly from the available window budget.
What happens at the edge
When conversations expand toward the context boundary, how the system handles the limit depends on where the boundary is caught. Understanding the difference is vital for debugging conversational applications.
If the incoming request exceeds the context window, the provider API will reject it outright with an error specifying the limit and the received token count. While disruptive, this explicit failure is straightforward to diagnose and trace.
In contrast, when an intermediate Harnessthe program between you and the model API. It assembles the context, executes the tools the model asks for, enforces permissions, and keeps calling the model until the job is done.Full glossary entryIntroduced in The agentic loop manages the conversation, it often attempts to prevent API errors by silently shedding context. It may summarize older messages, evict earlier file contents, or truncate tool results. This process—known as Compactionreplacing a stretch of conversation history with a generated summary of it, done by a harness at a threshold rather than at the point of failure.Full glossary entryIntroduced in Context management and compaction—happens behind the scenes. The symptom you observe is not an overt error, but a model that suddenly forgets constraints established at the beginning of the chat or re-reads code it already inspected. The context management chapter explores these compaction strategies and their associated tradeoffs.
A bigger window is not free
Context windows have expanded dramatically, with multi-million-token windows readily available across leading providers. However, having a large context window available does not mean you should fill it indiscriminately.
Cost scales linearly with volume: processing a million input tokens costs real money on every call, and repeating that in an agent loop quickly escalates cloud spending.
Latency also climbs with prompt size. Because the model must process every input token in the prompt before generating the very first token of the response, large inputs introduce noticeable delays before users see any output.
Finally, retrieval accuracy and attention fidelity decline in overstuffed prompts. A critical constraint or fact positioned in the middle of a massive context window is attended to far less reliably than when placed in a focused prompt—a dynamic detailed in what fills the window. Relying on oversized prompts often leads to paying higher bills for slower, less accurate answers.
Terms introduced
- Token: the unit a model reads and writes, a chunk of a few characters drawn from a fixed vocabulary.
- Tokenizer: the software that splits text into tokens, using a vocabulary learned once and frozen before training.
- Context window: the maximum number of tokens a model can hold for one call, covering the request and the answer together.
- Input tokens: everything sent to the model in a request, including the system prompt, the history and the tool definitions.
- Output tokens: everything the model generates, billed at a higher rate than input and produced one at a time.
How providers do it
All three sell a million-token window and all three now run a token-counting endpoint. The row that changes a design is the last one, because the three answers are not the same shape.
| Question | Anthropic | OpenAI | |
|---|---|---|---|
| Tokenizer published? | No file found; counting endpoint instead | Yes, tiktoken | No file found; SDK tokenizer on Vertex |
| Counting endpoint | POST /v1/messages/count_tokens | POST /v1/responses/input_tokens | models/<id>:countTokens |
| Characters per token, their figure | About 3.5 | Not stated on a page read today | About 4 |
| Largest window read today | 1M (Opus 5, Sonnet 5, Fable 5.1) | 1,050,000, with input capped at 922,000 | 1,048,576 |
| Maximum output | 128K (64K on Haiku 4.5) | 128K | 65,536 |
| Cheapest current model, per 1M in / out | Haiku 4.5, $1 / $5 | gpt-5-nano, $0.05 / $0.40 | gemini-3.5-flash-lite, $0.30 / $2.50 |
| Price varies with prompt size? | No | No | Yes, above 200K tokens on Pro-class models |
| Behaviour at the edge | 400 "prompt is too long" on oversized input; stop_reason: model_context_window_exceeded if generation runs out of room | status: "incomplete" with reason: max_output_tokens, billed but possibly empty | finishReason: MAX_TOKENS on output; input overflow not documented |
Every row above is confirmed except Google's input-overflow behaviour, OpenAI's encoding for its newest models, and OpenAI's characters-per-token figure, which are open in the tabs below.
Two things travel badly. A token count from one vendor does not transfer to another, and OpenAI's window figure is not the amount you may send, because the input cap sits well below it.
- Anthropic
- OpenAI
What this maps to: the Messages API. Tokens are counted under the tokenizer of the exact model you name, and there is a free endpoint that does the counting for you rather than a published tokenizer file.
| Question | Answer | Status |
|---|---|---|
| Which tokenizer? | Model-family specific. Claude Opus 4.7 and later use a newer tokenizer that produces about 30% more tokens for the same text than Sonnet 4.6 and earlier | confirmed |
| Is the tokenizer published? | No public tokenizer file or vocabulary was found on the docs site. The docs point at the counting endpoint instead | unconfirmed; no vendor page states this either way |
| Characters per token | "For Claude, a token approximately represents 3.5 English characters" | confirmed |
| How do you count tokens? | POST /v1/messages/count_tokens, returning {"input_tokens": N}. Free, rate-limited by tier, and documented as an estimate | confirmed |
| Published context window | 1M tokens on Fable 5.1, Opus 5, Sonnet 5, Opus 4.6 through 4.8 and Sonnet 4.6. 200K on Haiku 4.5 and on Sonnet 4.5 and older | confirmed |
| Maximum output | 128K on the 1M-window models, 64K on Haiku 4.5. Batch API can reach 300K output with a beta header | confirmed |
| Is 1M gated? | No. "For every model with a 1M-token context window, 1M is the default: you don't need a beta header, and long-context requests are billed at standard pricing" | confirmed |
| What counts against the window? | System prompt, every message, tool definitions, and the output including extended thinking. Cached tokens still occupy the window | confirmed |
| Input and output prices | Per million tokens: Opus 5 $5 in / $25 out. Sonnet 5 $2 / $10. Haiku 4.5 $1 / $5. Fable 5.1 $10 / $50. Batch API is half price both directions | confirmed |
| Cache pricing | Opus 5: $6.25 per million for a 5-minute cache write, $0.50 to read. Sonnet 5: $2.50 write, $0.20 read | confirmed |
| Tool definition overhead | Tool use adds a per-model system prompt overhead. Opus 5: 286 tokens for tool_choice auto or none, 406 for any or tool | confirmed |
| What happens when the input is too big? | 400 invalid_request_error, "prompt is too long", on every model | confirmed |
What if input plus max_tokens overflows? | On Claude 4.5 and newer the request is accepted, and generation stops with stop_reason: "model_context_window_exceeded" if it reaches the limit. Older models need a beta header | confirmed |
Their vocabulary
| Standard term | Their term |
|---|---|
| Context window | Context window |
| Input tokens | Input tokens; cache write and cache read tokens are priced separately |
| Output tokens | Output tokens, which include thinking tokens |
| Maximum output length | max_tokens, a required field on every request |
Where to look
usage on every Messages API response carries input_tokens, output_tokens, cache_creation_input_tokens and cache_read_input_tokens. Call count_tokens with the same body before sending if you need the input figure in advance.
Last verified: 2026-09-09 against the Anthropic platform docs: token counting, models overview, pricing, context windows, and Create a Message (platform.claude.com/docs/en/build-with-claude/token-counting, /models/overview, /about-claude/pricing, /build-with-claude/context-windows, /api/messages/create).
What this maps to: the Responses API, with the older Chat Completions API alongside it. OpenAI publishes its tokenizer as a library, tiktoken, and also runs a counting endpoint that the docs prefer for anything beyond plain text.
| Question | Answer | Status |
|---|---|---|
| Which tokenizer? | tiktoken, an open library. o200k_base is the encoding for GPT-4o, GPT-4.1, GPT-4.5, GPT-5 and the o-series | confirmed |
| Encoding for the newest models | Not listed in tiktoken's model map, and no vendor page states it. Do not assume o200k_base for gpt-6-astra or the gpt-5.6-* models | unconfirmed |
| Is there a counting endpoint? | Yes. POST /v1/responses/input_tokens takes the same input as the Responses API and returns input_tokens | confirmed |
| Endpoint or library? | The docs recommend the endpoint, because local tokenizers "don't support images/files, struggle with tools and schemas, and can't account for model-specific tokenization changes" | confirmed |
| Published context window | 1,050,000 tokens on gpt-6-astra, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna and gpt-5.5 | confirmed |
| A separate input cap | Yes, and it is lower than the window. Those models cap input at 922,000 tokens against a 1,050,000-token window | confirmed |
| Maximum output | 128,000 tokens on the models above | confirmed |
| Windows on older families | Not checked today for gpt-4.1, o3 or o4-mini | unconfirmed |
| What counts against the window? | "This max tokens number includes input, output, and reasoning tokens" | confirmed |
| Input and output prices | Per million tokens: gpt-6-astra $10 in / $50 out, cached input $1. gpt-5.6-terra $2 / $12. gpt-5.6-luna $0.20 / $1.20. gpt-5.1 and gpt-5 $1.25 / $10 | confirmed |
| Cached input | Priced at roughly a tenth of input on current models. The -pro tiers publish no cached-input price | confirmed |
| What happens at the edge? | Overflow of the output budget is not an error. The response comes back with status: "incomplete" and incomplete_details.reason set to max_output_tokens, possibly with no visible output at all while input and reasoning are still billed | confirmed |
The classic context_length_exceeded 400 | Not present in the current error-codes guide | unconfirmed; could not find it on any current vendor page |
Their vocabulary
| Standard term | Their term |
|---|---|
| Context window | Context window, quoted per model alongside a separate max input tokens figure |
| Maximum output length | max_output_tokens on Responses; max_completion_tokens on Chat Completions |
| Cached input tokens | Cached input, billed as its own line |
| Tokenizer | tiktoken, and its encodings such as o200k_base |
Where to look
usage on a Responses result carries input_tokens, input_tokens_details.cached_tokens, output_tokens and total_tokens. Call /v1/responses/input_tokens for a count before sending.
Last verified: 2026-09-09 against the OpenAI developer docs: token counting guide, pricing, the per-model pages for gpt-6-astra and the gpt-5.6 family, conversation state, and the tiktoken repository (developers.openai.com/api/docs/guides/token-counting, /api/docs/pricing, /api/docs/models/gpt-6-astra, /api/docs/guides/conversation-state, github.com/openai/tiktoken).
What this maps to: the Gemini API. Note that the docs now run in two trees, a current Interactions API and a legacy generateContent tree, and the field names for the same idea differ between them.
| Question | Answer | Status |
|---|---|---|
| Characters per token | "For Gemini models, a token is equivalent to about 4 characters. 100 tokens is equal to about 60-80 English words" | confirmed |
| Is the tokenizer published? | No page on the Gemini docs states that it is. Vertex recommends the Agent Platform SDK's integrated tokenizer for local counting | unconfirmed; the widely repeated claim that it is the Gemma SentencePiece vocabulary appears only in third-party sources |
| How do you count tokens? | POST /v1beta/models/<model>:countTokens, SDK count_tokens or countTokens. Vertex has its own path under publishers/google/models/<id>:countTokens | confirmed |
| Non-text token rates | An image up to 384px on both sides costs 258 tokens. Larger images are tiled into 768x768 tiles at 258 tokens each. Video is 263 tokens per second, audio 32 | confirmed |
| Published context window | 1,048,576 input tokens on gemini-3.8-flash, gemini-3.5-flash, gemini-3.5-flash-lite, gemini-3.1-pro-preview, gemini-2.5-pro and gemini-2.5-flash | confirmed |
| Maximum output | 65,536 tokens on all of the models above | confirmed |
| Is the window shared? | Yes. "The context window defines the combined limit of input and output tokens" | confirmed |
| Input and output prices | Per million tokens: gemini-3.8-flash $0.75 in / $3.75 out through 31 December 2026, doubling to $1.50 / $7.50 on 1 January 2027. gemini-3.5-flash-lite $0.30 / $2.50. Batch is roughly half list | confirmed |
| Tiered pricing above a prompt size | Yes, on the Pro-class models, at a 200,000-token prompt threshold. gemini-2.5-pro is $1.25 in / $10 out up to 200K and $2.50 / $15 above it. The Flash models checked are flat-rate | confirmed |
| Context caching prices | gemini-3.8-flash $0.075 per million cached tokens, doubling on 1 January 2027. gemini-2.5-pro $0.125 up to 200K, plus $4.50 per million tokens per hour of storage | confirmed |
| Minimum for a cache hit | 4,096 tokens on the Gemini 3.x Flash models and 3.1 Pro Preview; 2,048 on Gemini 2.5 Flash and Pro. The Interactions API supports implicit caching only | confirmed |
| What happens when the output limit is reached? | finishReason: MAX_TOKENS | confirmed |
| What error when the input window is exceeded? | Not documented on the errors or troubleshooting pages read today | unconfirmed |
gemini-3.5-flash input price | Two fetches of the pricing page gave conflicting readings | unconfirmed; re-check before quoting |
Their vocabulary
| Standard term | Their term |
|---|---|
| Context window | Context window, described as the model's short-term memory |
| Maximum output length | maxOutputTokens in generationConfig |
| Input tokens | total_input_tokens (Interactions) or promptTokenCount (legacy) |
| Output tokens | total_output_tokens or candidatesTokenCount, with thinking counted separately |
Where to look
interaction.usage carries total_input_tokens, total_output_tokens, total_thought_tokens, total_cached_tokens and total_tool_use_tokens. The legacy tree calls the same things usageMetadata.promptTokenCount, candidatesTokenCount and thoughtsTokenCount.
Last verified: 2026-09-09 against the Gemini API docs: tokens, pricing, the per-model pages for gemini-3.8-flash and gemini-2.5-pro, long context, caching, and API errors (ai.google.dev/gemini-api/docs/tokens, /pricing, /models/gemini-3.8-flash, /long-context, /caching, /api-errors).