What 14 Days of AI Context Compaction Failures Taught Me

Posted August 30, 2026

In part one, I compared two ways to shrink the history of a long AI chat. One approach tried to preserve the model's existing context. The other started fresh, searched the full history, and built a summary from the evidence it found.

The search and summary approach produced better handoffs in that test, so I made it the production default in Friday on August 15, 2026.

Fourteen days later, the production data showed a problem: it failed about 30% of the time, and a failed attempt often cost more than a successful one.

The first benchmark gave me a useful answer about summary quality. It did not tell me enough about reliability, timing, or cost. I needed a broader test.

What is context compaction?

An AI chat can only send a limited amount of text to a model on each turn. The conversation, instructions, tool calls, and tool results all count toward that limit.

Context compaction replaces some of that older material with a smaller version. A good compacted history gives the model enough information to continue the work without sending the entire chat again.

This sounds like a summarization problem, but the production data showed that summarization was only one part of it.

What did the first production system do?

Friday waited until a chat was close to its context limit. It then started a separate compaction agent that could search the original history and write a new summary.

That agent had:

up to 50 tool calls
tools to search the chat and edit its draft
a structured format for claims and citations
a size target that the final result had to meet

These rules were meant to protect quality. In practice, they also gave the system many ways to keep working without finishing.

What happened in production?

From August 15 through August 29, Friday recorded about 136 accepted compactions, 58 terminal failures, and 132 attempts that were replaced by a later attempt. If I leave out the replaced attempts, about 30% ended in failure.

The failures were also expensive. On one provider, a failed attempt used an average of 6.36 million input tokens and reached a maximum of 11.7 million. On another provider, failures took an average of 18.5 minutes and reached a maximum of 77 minutes.

When I read the failed attempts, they fell into five groups:

Problem	What happened
The agent kept searching	One attempt used all 50 tool calls on search and never wrote a summary. Other attempts repeatedly read their own drafts.
Required content was already too large	Some messages had to remain in full for safety. Those messages alone were larger than the size target, so no summary could pass. One chat repeated the same failure four times because the system treated it as retryable.
Compaction started too late	Some chats grew beyond the compactor's own input limit. The model could not read the history it was supposed to shrink.
The provider was unavailable	Rate limits and outages caused several quick retries without enough time between them.
The output could not pass	Some summaries were larger than the model's output limit. Other models failed to produce the required claim format. In both cases, the entire result was discarded.

The problem was not simply that the summaries were bad. The process started late, allowed a long search loop, and rejected results that were useful but slightly larger than the target. Most of the safeguards added more work instead of giving the system a safe way to stop.

Why didn't the first benchmark catch this?

Part one's benchmark focused on the quality of the final handoff under controlled conditions. The search and summary approach did well at that job.

Production added several questions the benchmark did not cover:

Can the compactor read a very large chat?
How often does the full process finish?
What happens when the provider is down?
What should happen when a useful result misses the ideal size target?
How much does a failure cost?

The first result was not useless. It was incomplete. Summary quality mattered, but it was only one part of the system.

What was taking up all the space?

I exported eight real production chats. They included an example from each failure group, a successful compaction, and the largest recent chat, which had a 17MB transcript.

The existing live test used five messages totaling about 2,000 tokens. The production failures started above 460,000 tokens. That difference was large enough to hide the main problem.

Across the eight real chats, tool calls and tool results made up 70% to 99% of the bytes. The actual conversation between the person and the model was about 1%.

In other words, most of the context was not prose that needed a careful AI summary. It was old search results, command output, and other tool data that plain code could reduce first.

What do other AI tools do?

I reviewed six open-source tools: opencode, OpenAI's Codex CLI, gemini-cli, goose, aider, and cline. I also looked at the published behavior of Claude Code, Anthropic's context management API, and MemGPT/Letta.

Across the systems I checked, the common approach was much simpler:

None used a search loop with up to 50 model-driven tool calls. Most used one model call. The largest process I found used two.
They started compaction before the context was full. The thresholds ranged from about 50% to 90% of the context window.
They reduced old tool results with regular code before asking a model to summarize anything.
They accepted a result if it was smaller and could be sent to the model. Some also had a rule-based fallback that used no model call.

My production system had stricter citations, more model calls, and a harder size requirement than the other systems I reviewed. The question was whether that extra work produced enough value to justify its cost and failure rate.

What did I test next?

I built a new evaluation using the eight production chats. It used the same message formats, provider adapters, and size calculations as Friday. Only the compaction approach changed.

I compared five options:

Current production system: the existing search loop.
One summary call: ask the model for a structured Markdown summary, with no tools and no claim schema.
Rule-based cleanup plus one summary: shorten old tool calls and results first, then make one summary call.
Early rolling summaries: summarize older turns throughout the life of the chat instead of waiting for an emergency.
A more practical size rule: accept any result that is smaller and can be sent, even if it misses the ideal target.

The rule-based cleanup kept the tool name, shortened arguments, status, and a small result summary. It removed old reasoning text and replaced media with a short placeholder. Recent messages stayed untouched.

Which approach worked best?

The rule-based cleanup alone removed 69% to 95% of the tokens on every production example. It used no model calls. Six of the eight chats fell below the normal acceptance ceiling before a summary was created.

It also rescued the chat that had become too large for the old compactor to read. The old system could not start, but regular code reduced the chat in milliseconds.

A single summary call worked when the model could read the original history. It passed on three of eight chats and stopped immediately on the five that were too large. That is better than spending many calls on work that cannot finish, but it is not enough by itself.

Cleanup followed by one summary call passed on all eight chats:

Method	Accepted	Median time	Median input tokens	Model calls
Current search loop (2 chats)	2/2	460s	671k	13 to 17
One summary at the limit	3/8	146s	404k	1
Cleanup plus one summary	8/8	81s	165k	0 or 1
Early rolling summaries (3 chats)	3/3	about 40 to 300s per summary	40k to 417k over the life of the chat	2 to 15 over the life of the chat

The early approach also prevented the largest failure state from appearing. When I replayed the chat that had become too large, its working context never exceeded 185,644 tokens. The total cost of every summary across that chat was less than 1% of the cost of one production failure.

One test exposed another useful constraint. A single turn contained several megabytes of script output, so even an early summary could receive too much input at once. Running the rule-based cleanup before each rolling summary fixed the issue in all three rolling tests and reduced their cost by another seven times.

Did the old system do anything better?

Yes. In a direct comparison on two chats, the old search loop kept slightly more exact identifiers in the summary itself.

However, the cleaned tool records already preserved those identifiers next to the summary. A small fidelity test scored the old summary at 0.47 and the new summary at 0.40, while the rule-based tool summaries scored between 0.67 and 1.0.

The extra search loop was improving one document, but the surrounding context had already kept more of the information we cared about.

I also used a separate model to check whether each result kept enough information for the chat to continue safely. Nearly every accepted result scored 5 out of 5. One result scored 1 because the compaction boundary separated a tool call from its result. That is now a rule in the new design: keep each tool call and its result together.

What changes now?

The new design has four parts:

Clean up old tool data first. Older tool calls become a small record with the tool name, shortened arguments, status, and result summary. Recent messages remain intact. This step uses regular code and does not depend on an AI provider.
Make one structured summary. The model writes a Markdown snapshot with the goal, decisions, constraints, evidence, open errors, completed work, and next actions. It does not search or edit through a loop.
Accept useful reductions. The ideal size remains a target, but it is not the only passing result. If the new context is smaller and the model can read it, Friday can use it. If the summary call fails, the cleaned version remains available as a fallback.
Start earlier. Friday will begin rolling summaries around 150,000 projected tokens instead of waiting until the chat is almost full.

Each change addresses a production failure directly:

Old failure	New behavior
The agent kept searching	There is no search loop.
Required content was too large	Important errors move into the summary, while old tool data is reduced.
Compaction started too late	Rolling summaries start earlier, and cleanup can still reduce a large chat before a model reads it.
The provider was unavailable	Rule-based cleanup does not need a provider.
The output could not pass	There is no claim schema, and a useful smaller result is not discarded for missing the ideal target.

What did I learn?

The main lesson is that a good summary is not enough. The timing, acceptance rules, fallback, and cost of failure are also part of context compaction.

My updated checklist is:

Test the full process. Measure when it runs, whether it finishes, and what a failure costs.
Measure what fills the context. In Friday's production chats, old tool data was the main problem.
Use plain code for plain cleanup. A model does not need to shorten every old command result.
Treat the ideal size as a target. A smaller, usable result is still useful.
Keep a fallback that does not depend on a model provider.
Start before the chat becomes an emergency.

The runtime change is next. It will go through the normal review process, and the new production fixtures will stay in the evaluation suite. The old search loop will be removed rather than kept as a second path.

For anyone tracking model cost, the full evaluation used about 9.5 million input tokens on Qwen 3.8 Max through Alibaba's Token Plan. That is roughly the cost of two production failures from the old system.