Context management and compaction

As an interactive session continues, the message history steadily expands until it threatens to exceed the model's context window. The underlying model cannot decide what to keep or discard when this happens. That responsibility falls entirely to the client harness orchestrating the conversation. How that software manages information at the context boundary determines whether an agent successfully finishes a long task or quietly forgets its original instructions halfway through.

encompasses the policies, algorithms, and architectural strategies used to govern what stays in the active window, what gets pruned, and how critical information is preserved across extended workflows.

How systems fail at the context limit

When a conversation exceeds maximum token capacity, systems typically handle the boundary in one of two ways:

The first is an explicit API rejection: if a payload contains more than the model supports, the cloud provider returns an HTTP 400 error and halts generation. While an abrupt error interrupts execution, it is completely transparent, giving engineers a clear failure signal to handle.

The more deceptive failure mode occurs when intermediate client libraries silently drop older turns to keep requests under the token ceiling. The API call succeeds and returns a completion, but the model has lost visibility into early constraints or requirements established at the start of the session. The user only discovers the eviction when the model contradicts an agreed-upon design decision or starts asking questions that were answered an hour earlier.

Three foundational mitigation strategies

Practitioners rely on three primary techniques to handle expanding context windows:

• FIFO Truncation: simply drops the oldest conversational turns from the transcript until the payload fits within limits. It is computationally free and requires no additional model calls. The critical downside is that the beginning of a conversation usually contains the project goals, user preferences, and operational constraints. Evicting the earliest messages strips away the very instructions the model needs most.
• Context Summarization: Rather than deleting historical turns entirely, the harness synthesizes earlier messages into a compact text overview. This preserves the overarching narrative and major milestones at a fraction of the original token footprint. The tradeoff is that generating the summary requires an extra model invocation, adds several seconds of latency, and risks introducing hallucinations or omitting crucial technical subtleties.
• Externalization with Pointers: Instead of storing verbose file contents or long database outputs directly in the message transcript, the harness writes large artifacts to disk or external storage and leaves concise reference pointers in the prompt. When the model needs specific details, it retrieves them on demand using dedicated tools. This approach prevents permanent data loss, though it introduces tool-call round trips and relies on the model recognizing when to query external references.

Structured compaction

is the proactive, deliberate application of summarization. Rather than waiting for an emergency context overflow, a well-engineered harness monitors token consumption and triggers compaction once the context window reaches a predetermined threshold (often 70% to 80% of capacity). The harness initiates a background model call instructed to distill the session history into a structured summary, replacing the middle block of messages while leaving the initial instructions and the most recent turns intact.

The prefix is untouched, the recent turns survive word for word, and 138,000 tokens of the middle come back as one 2,800 token summary.

A robust compaction prompt directs the model to extract four key elements:

1. The overarching objective of the current task.
2. Architectural decisions and constraints confirmed so far.
3. Specific files, schemas, or resources touched or modified.
4. Current technical state, unresolved errors, and immediate next steps.

Even a well-crafted compaction summary unavoidably loses three kinds of information:

• Exact literal syntax: Specific error messages, compiler warnings, schema column names, and code snippets get condensed. A summary noting that "the database migration error was resolved" strips out the exact SQL error code.
• Rationale behind prior decisions: Summaries generally record what was decided rather than the detailed debate explaining why. When a team asks why a timeout was set to 500ms, that reasoning may be lost.
• Seemingly minor contextual details: A passing detail mentioned on turn 8 that seemed irrelevant at the time may become pivotal on turn 35. The summarization model has no way of anticipating future relevance.

This loss of granularity explains why agents frequently repeat previous work immediately following a compaction event. If an agent inspected a file before compaction and decided against modifying it, a summarized note merely stating "reviewed config.py" may cause the agent to re-read the file post-compaction, spending another 20,000 tokens to reach the identical conclusion. When troubleshooting an agent caught in repetitive loops, check whether the behavior began right after a compaction boundary.

Leveraging prompt caching

Repeatedly transmitting identical 20,000-token system prompts and tool schemas on every conversational turn is both expensive and slow. allows model providers to store the compiled KV representations of static prompt prefixes across calls, significantly discounting the cost of reusing that cached prefix. In practice, cached tokens are often billed at a steep discount (frequently up to 90% cheaper than regular input tokens) while substantially cutting time to first token.

However, prompt caches rely on strict prefix matching. A matches sequentially from byte zero of the request. The provider compares the incoming prompt token-by-token against cached blocks, reusing existing representations until encountering the very first difference. Every token after that divergence must be parsed and processed at standard input rates.

This prefix sensitivity means minor ordering errors can completely destroy cache efficiency. Inserting a dynamic timestamp, a unique session ID, or an oscillating user variable at the beginning of the system prompt guarantees a 0% cache hit rate on every turn. Similarly, reordering tool schemas or altering instruction headers invalidates everything downstream in the payload.

To maximize cache hits, harnesses structure prompts into distinct tiers from most stable to most volatile:

Order a prompt so the cache can be reused

Put the most stable content first. Put the most volatile content last.

1. Place tool definitions at the very start of the request.
2. Place the system prompt next.
3. Place loaded instruction files next.
4. Place the conversation history next, oldest first.
5. Place retrieved documents and the user's question last.
6. Do not put a timestamp, a session id, or a random value near the front.
7. Do not reorder or edit tool definitions between turns.
8. Append to the history. Do not rewrite earlier messages.

Note that compaction and prompt caching have conflicting dynamics. When compaction replaces a historical block of messages with a fresh summary, it alters the prompt prefix, invalidating the cache from that point onward. The request immediately following compaction pays full price for the updated prompt. While this cost is worthwhile when performed periodically, triggering compaction too aggressively will eliminate the economic benefits of caching.

Externalizing state to persistent files

The most reliable approach to managing long-running agent state is to stop relying on the context window as persistent storage. The context window is an ephemeral working scratchpad for the current inference pass; anything that must reliably survive multi-hour tasks belongs in persistent files.

In production engineering workflows, autonomous agents maintain dedicated working documents: a markdown progress log, an architectural task plan, or a verification checklist stored on disk. The agent updates this file as work proceeds, while the active context window holds only a lightweight pointer to the file's path. If the conversation history is subsequently compacted or truncated, the agent's core task state remains completely uncorrupted on disk.

This externalized pattern requires minimal overhead—just occasional tool calls to read or append notes—while making tasks resilient against context loss. The same philosophy extends to multi-session continuity in memory and vast document archives in retrieval.

Terms introduced

• Context management: deciding what stays in the context window, what leaves, and what replaces it, as a conversation grows past the limit.
• Compaction: replacing a stretch of conversation history with a generated summary of it, done by a harness at a threshold rather than at the point of failure.
• Truncation: dropping the oldest messages until the request fits, with no summary and no record of what was removed.
• Prompt caching: a provider feature that stores the processed form of a request prefix and charges a reduced rate to reuse it on a later call.
• Cache prefix: the run of bytes from the start of a request that must match exactly for a cached prompt to be reused. The first difference ends it.

---

How providers do it

Both vendors sell the same three ideas under different names: cache the stable prefix, drop or summarise the old turns, and let the platform hold the history. The defaults differ, and the defaults are what bite.

Concept	Anthropic	OpenAI
Caching on by default	No. A cache_control breakpoint is set by the caller	Yes, with automatic breakpoints
Minimum cacheable prompt	512 to 4,096 tokens, by model	1,024 tokens
Cache lifetime	5 minutes, or 1 hour	30 minutes by default, or 24 hours on older models
Cache write cost	1.25x base input, or 2x for 1 hour	1.25x base input, explicit mode only
Cache hit cost	0.1x base input	0.1x base input
Managed truncation	Context editing clears old tool results, client-side, beta	truncation: "auto" drops from the beginning
Managed compaction	Compaction, server-side, beta	compact_threshold, server-side
Provider-held history	No	Yes, with store, previous_response_id and Conversations

Every row is confirmed against the vendor docs read on the date below. Current prices, and whether the beta features have shipped, are left open in the tabs.

Two rows change a design. Provider-held history does not make the history free: previous input tokens are still billed on every turn. And an overflow either errors or silently drops the start of the conversation depending on one parameter, so set it deliberately rather than inheriting it.

Anthropic

What this maps to: three separate features. cache_control for prompt caching, context editing for clearing old tool results, and compaction for server-side summarisation.

Prompt caching

Question	Answer	Status
How is a breakpoint set?	A cache_control object of the form {"type": "ephemeral", "ttl": "5m"} on a content block or a tool definition. A top-level cache_control also exists and moves the breakpoint to the last cacheable block as the conversation grows	confirmed
How many breakpoints?	Up to four explicit ones, with a lookback of up to 20 blocks	confirmed
Minimum cacheable prompt?	Model-dependent. The table read today gives 512 tokens for Opus 5, 1,024 for Sonnet 5 and Opus 4.8, 2,048 for Opus 4.7, and 4,096 for Opus 4.6 and Haiku 4.5	confirmed
How long does a cache live?	5 minutes by default. A 1 hour ttl is available	confirmed
What does a write cost?	1.25x the base input price for a 5 minute entry, 2x for a 1 hour entry	confirmed
What does a hit cost?	0.1x the base input price for most models	confirmed
What invalidates it?	The prefix is a hierarchy of tools, then system, then messages. A change at one level invalidates that level and everything after it, so editing a tool definition invalidates the whole prefix. Toggling web search or citations invalidates from system onward	confirmed
Are the prices above current?	Rates move. Read the pricing page before quoting them	unconfirmed

Managed context management

Question	Answer	Status
Is there a managed way to trim tool results?	Yes. Context editing, in beta behind the context-management-2025-06-27 header, with a context_management.edits array. The clear_tool_uses_20250919 strategy triggers at 100,000 input tokens by default and keeps the last 3 tool uses	confirmed
Is there managed summarisation?	Yes. Compaction, a separate beta behind the compact-2026-01-12 header, edit type compact_20260112. It summarises the conversation at a threshold, emits a compaction block, and drops everything before that block on later requests	confirmed
What is the difference between the two?	The docs put it directly: context editing clears specific tool results on the client, compaction summarises the whole conversation on the server	confirmed
Can the saving be previewed?	Yes. The token counting endpoint supports previewing what context editing would clear	confirmed
Will either leave beta?	Not stated on the pages read today	unconfirmed

Their vocabulary

Standard term	Their term
Cache breakpoint	cache_control, type ephemeral
Truncating tool output	Context editing, clear_tool_uses_20250919
Compaction	Compaction, compact_20260112, producing a compaction block

Where to look

The usage object reports cache creation and cache read tokens separately, which is the only reliable way to tell whether a breakpoint is landing where you think. Responses also carry a context_management field describing what was cleared.

Last verified: 2026-09-09 against the prompt caching guide (https://platform.claude.com/docs/en/build-with-claude/prompt-caching), the context editing guide (https://platform.claude.com/docs/en/build-with-claude/context-editing) and the compaction guide (https://platform.claude.com/docs/en/build-with-claude/compaction).

OpenAI

What this maps to: prompt caching, which is on by default, plus three separate controls on the Responses API for conversation state, truncation and compaction.

Prompt caching

Question	Answer	Status
Does it need to be enabled?	No. The docs say prompt caching is enabled by default for supported models, with breakpoints placed automatically	confirmed
Can breakpoints be set by hand?	Yes on newer models, with prompt_cache_options.mode set to explicit and prompt_cache_breakpoint markers in content blocks	confirmed
Minimum cacheable prompt?	1,024 tokens	confirmed
What does a hit cost?	Cached input is billed at 0.1x the uncached rate. An explicit cache write is 1.25x	confirmed
How long does an entry live?	prompt_cache_options.ttl defaults to 30 minutes on newer models. Older models use prompt_cache_retention, either in_memory or 24h	confirmed
What invalidates it?	Any change to the rendered prefix: the model, tool names, descriptions, schemas or ordering, parallel_tool_calls, the output format, reasoning.effort, text.verbosity, or a compaction. Reuse requires the entire rendered prefix to match	confirmed
Can cache routing be influenced?	Yes. prompt_cache_key groups similar requests to improve hit rates, and replaces the older user field	confirmed

Conversation state and compaction

Question	Answer	Status
Does the platform hold conversation state?	Yes. store defaults to true and keeps response data for at least 30 days. previous_response_id chains turns. A Conversations object is durable and is not subject to the 30 day expiry	confirmed
Does server-held state make the history free?	No. With previous_response_id, all previous input tokens in the chain are billed as input tokens again	confirmed
Is there managed truncation?	Yes. truncation set to auto drops items from the beginning of the conversation to fit the window. The default is disabled, which returns a 400 instead	confirmed
Is there managed compaction?	Yes. context_management with a compact_threshold triggers server-side compaction and returns an encrypted compaction item. A standalone /responses/compact endpoint also exists	confirmed
Does compacting cost the cache?	Yes. Context compaction is listed among the things that change the rendered prefix	confirmed
Are the cache rates above current?	Rates move. Read the pricing page before quoting them	unconfirmed

Their vocabulary

Standard term	Their term
Cache breakpoint	prompt_cache_breakpoint, under prompt_cache_options
Truncation	truncation: "auto"
Compaction	Context management, compact_threshold, and a compaction item
Conversation history held by the provider	Conversation state, previous_response_id, the Conversations API

Where to look

usage.input_tokens_details.cached_tokens on a response is the number that tells you whether caching is working. The default truncation: "disabled" is the safer setting while developing, because it turns a context overflow into an error rather than into silent forgetting.

Last verified: 2026-09-09 against the prompt caching guide (https://developers.openai.com/api/docs/guides/prompt-caching), the conversation state guide (https://developers.openai.com/api/docs/guides/conversation-state), the context management guide (https://developers.openai.com/api/docs/guides/context-management) and the Responses create reference (https://developers.openai.com/api/reference/cli/resources/responses/methods/create).

---

Check your understanding

0 of 4 answered

1. What does truncation lose that compaction does not?

   Any record at all of the dropped turns. Compaction leaves a summary behind; truncation leaves nothing.The system prompt, which truncation removes first.The most recent turns, which truncation removes first.Nothing. The two are different names for the same operation.
2. An agent finishes a task, compacts, and then starts reading a file it already read and reasoned about. What happened?

   The summary did not record that conclusion, so from the agent's point of view the work was never done.Compaction cleared the model's weights, so it forgot how to read files.The file changed, so the earlier read was invalidated.The agent hit its output limit and restarted the turn.
3. An agent must keep a plan across a two-hour session that will compact several times. Where should the plan live?

   In a file on disk, with a short pointer to it in the context.In the first user message, so it is the oldest thing in the history.Repeated in full at the end of every reply the model gives.In the system prompt, rewritten each time the plan changes.
4. A team adds the current time to the top of its system prompt so the model knows the date. The bill goes up sharply. Why?

   A cache is matched on an exact prefix from position zero, so a value that changes every call invalidates everything after it.Timestamps are billed at a higher rate than ordinary text.The extra tokens for a timestamp are large enough to notice.Caching is disabled on any request that contains a date.