Prompt caching, compression, and compaction in coding agents

A coding agent can send the same project instructions many times while fixing one bug. After a tool result, the agent often makes another model request, and that request needs enough history to continue the work.

There are several ways to make this cheaper or faster. They act on different parts of the system, so it’s easy to misunderstand what an announcement promises.

Understanding the difference helps when a session suddenly consumes more usage than you expected.

Why a short instruction can use a lot of tokens

You type “fix the validation,” which takes very little space. The request sent to the model can also include project instructions, tool definitions, files the agent read, and earlier messages.

Then the agent runs a test. The next request needs the result, together with enough of the previous conversation to interpret it. A terminal command doesn’t itself require model inference, but deciding what command to run and what to do with its output usually does.

That is why a session with ten model requests can process much more input than the text you typed. The context window measures how much fits in one request. Total usage adds up the work across requests.

Caching can reduce the cost of repeatedly processing that history. It doesn’t remove the history from the context window.

Start with the repeated prefix

A prompt has a beginning: instructions, tool definitions, reference material, and the earlier conversation. A later request can start with that same content and append new messages.

OpenAI’s prompt caching documentation explains that a matching prefix can reuse saved model state. Keeping a session open doesn’t itself guarantee a hit. Content and relevant settings must match, and a suitable cache entry must still be available.

Imagine a coding task with stable project rules followed by a changing request:

Project rules: use the existing validation helper.
Reference: the form submits to /api/contacts.
Task: fix the missing email validation.

On the next request, preserve the earlier messages and append the test result. Rewriting the rules to include a timestamp at the beginning changes the prefix unnecessarily.

Matching is about the supplied content, not just its meaning. “Use the existing validation helper” and “Reuse our validation helper” may mean the same thing to you, but they are different input. A changed section near the beginning can prevent reuse of everything after it, while an unchanged earlier prefix can still be reusable.

The tiny example above only shows ordering. Providers also have minimum cacheable lengths and rules about eligible boundaries. Repeating a few sentences doesn’t guarantee that a request qualifies.

That doesn’t mean you should hide facts the agent needs. Put genuinely changing material where it belongs, and keep stable material stable.

A cache hit still produces a new answer

Prompt caching reuses earlier computation about the input. The model still processes new content and generates a response.

A completed answer is therefore different from a reusable prefix. Asking the same question twice can produce different answers even when both requests reuse cached input.

Anthropic’s caching guide documents automatic caching and explicit cache breakpoints. Its default lifetime is five minutes, with a one-hour option. A cache read refreshes its lifetime. Those API controls have their own requirements and pricing.

If you build an agent harness, choose settings from the documentation for that provider and model. If you use an existing coding app, inspect what the app exposes. An API setting doesn’t automatically become an environment variable supported by every CLI.

Cache reads and cache writes have different costs

A cache needs to be populated before later requests can reuse it. Some providers and models charge more for writing a prefix than for ordinary input, then charge less for reading it. A short task with little reuse can therefore benefit less than a long session.

Here’s a hypothetical example, using made-up prices rather than a particular model’s rate card. Suppose a reusable prefix contains 20,000 tokens. Ordinary input costs $1 per million tokens, cache writes cost $1.25, and reads cost $0.10.

That last row assumes every later request fully reuses the prefix. It excludes new messages, generated output, and any extra writes as the conversation grows. It shows why repeated reuse matters without pretending that the whole session costs $0.043.

Use the provider’s usage fields and actual model prices for a real calculation. In OpenAI’s Responses API, usage.input_tokens_details.cached_tokens reports cached input. Models with separately billed cache writes also expose cache_write_tokens and its lifetime breakdown. OpenAI’s calculation guidance explains how to avoid counting those writes twice.

Compression changes the journey to the server

A gzip-compressed request can be smaller on the wire while expanding to the same JSON at the destination, if the client and endpoint support that encoding.

Think of sending a large text file inside a ZIP archive. Extracting it gives you the original text. You saved transfer bytes; you didn’t shorten the document the recipient reads.

The same distinction applies to an agent request. Compression can help when network transfer is slow. It doesn’t establish a prompt-cache hit, and the compressed byte count doesn’t tell you how many input tokens the model processes.

When reading a performance claim, look for the measurement. Upload duration, request size, model input tokens, and time to the first response are different numbers. An improvement in one can leave the others unchanged.

Be careful with the word “compression” in product announcements. Some tools use it for shortening context or summarizing tool output. That changes the model’s input. Ask whether decompression reconstructs the original request exactly, or whether the tool has removed information.

Compaction changes what the model receives

A long conversation can accumulate old logs and failed attempts. Compaction replaces earlier work with a more compact representation so later requests carry less history.

That representation depends on the system. Some agents use a readable summary. OpenAI’s compaction API can return an encrypted, opaque item carrying prior state and reasoning, alongside retained context. You shouldn’t assume every compaction result is a paragraph you can read or edit.

That can change the prefix and reduce cache reuse immediately afterward. OpenAI’s cache guide explicitly describes that trade-off. Compare the total input cost, because fewer tokens can still cost less despite a lower hit rate.

Compaction can also omit something useful. The agent needs a way to recover important facts by reading project files.

For example, imagine a reset-token bug investigation with pages of failed test output. A useful handoff needs the actual requirement, the root cause, the files changed, and the current test result. It also needs a decision such as “keep the existing session policy.” A summary saying “working on authentication” loses too much.

Write durable project rules in AGENTS.md and task-specific decisions in a short project note. Keep the current code and tests available for the agent to inspect. That helps both after compaction and when you deliberately start a new conversation.

Don’t compact merely to make a meter look better. Use it when the conversation has become expensive or difficult to work with, and check that the next task still has the context it needs.

A lower cache hit rate can still mean a cheaper session

Suppose a request contains 100,000 input tokens and 90,000 come from cache. Its token cache-hit rate is 90%.

After compaction, a request might contain 15,000 tokens with no cache hit. The hit-rate meter looks worse, but the model receives far less context. Which request costs less depends on the rates for reads, ordinary input, and writes. There can also be a separate cost for generating the compacted representation.

Measure several subsequent turns. The shorter context may become reusable again, and the one-time compaction cost may pay off over those turns. A single request immediately after compaction gives an incomplete picture.

Look at usage before changing settings

Claude Code’s cost documentation describes a prompt-cache line in /usage, including hits, misses, and expected rebuilds. That line covers the main conversation; subagents have their own requests. The plan usage breakdown provides separate attribution.

For an API application, record input tokens and cached-input tokens alongside latency. Track cache writes separately when the provider reports them, and include output tokens and the number of requests. Compare similar tasks with the same model settings. A short run that only writes a cache is different from a long session that reuses it repeatedly.

Finally, keep API billing and subscription allowances separate. OpenAI documents that cached tokens still count toward API tokens-per-minute limits. A cheaper token rate doesn’t establish how a coding subscription calculates its weekly allowance.

What I would check when usage jumps

I would start with one task and look at what changed between requests. A big log read, a different model, and a newly launched subagent lead to different investigations.

A high cache-hit percentage can coexist with an expensive session. Reusing a large prefix hundreds of times still adds up, and every response has its own output cost.

I would also limit terminal output to what the next decision needs. Ask for the failing test and its error instead of dumping a whole build log into context. Keep the full log available when deeper inspection is necessary. Avoiding irrelevant input can help regardless of whether caching works.

Start a new conversation when the next task is unrelated. For an ongoing bug investigation, keep enough history to continue accurately. Saving a few tokens is a poor trade if the agent has to rediscover the requirement and repeat the work.

Want me to talk about your product? You can sponsor this site.