I Tested AI Context Compaction. The Clever Fork Lost.

Posted August 16, 2026

I had a clever idea about AI context compaction.

It was actually clever.

It also lost.

That distinction matters because the agent world is full of ideas that feel so architecturally satisfying that people skip the annoying step where they prove the idea is worth shipping.

I did not want to do that.

The idea was this: when an AI conversation gets too large, do not start a fresh compaction request and paste the history into it. Fork the existing conversation at its exact prefix, append one new instruction asking the same model to write a context handoff, and let the provider reuse the cache it already built.

In theory, the fork gets two benefits at once:

The model stays inside the same visible working context instead of reading a reconstructed transcript.
The provider can reuse the expensive prefix instead of processing it again at full price.

This felt like the right shape. It preserves continuity. It respects how prompt caches work. It avoids shipping a giant inline transcript to a new worker.

So I built it.

Then I built an eval designed to make the current approach fail.

Then the current approach won.

First, how an AI agent loop works

The model is only one part of an AI agent. The harness is the software loop around it. It stores the conversation, decides which messages and tool definitions go into the next model request, executes tool calls, appends their results, and checks whether the model's answer is valid.

A normal turn looks roughly like this:

The harness builds a request from the system instructions, relevant conversation, and available tools.
The model returns prose, a tool call, or both.
If there is a tool call, the harness executes it and adds the result to the conversation.
The model sees that result and continues. The loop repeats until the turn is complete.

The canonical history is the durable, full record of those messages and tool results. The active context is the smaller selection actually sent to the model on a particular request. Keeping those concepts separate matters: compaction can reduce what the model sees without deleting what the system knows.

As the loop runs, the canonical history keeps growing. Eventually, sending all of it on every request becomes slow and expensive, even if it still fits inside the model's technical limit. That earlier point is the economic limit. This is where the harness asks a compactor to create a handoff:

full canonical history -> compactor -> compact handoff + recent messages -> continued agent

The continued agent treats that handoff as its working memory. If it needs an exact old detail, a recovery tool can search the canonical history. The central engineering question is whether the handoff preserves enough of the work for the agent to continue correctly without hauling the entire transcript forward.

What compaction is actually trying to preserve

Long agent conversations become economically stupid before they become technically impossible. Tool results accumulate. Old branches remain in the transcript. Completed work keeps consuming attention. At some point the system needs a smaller handoff document that can carry the live work forward.

The lazy framing is that compaction is summarization.

It is not.

A useful compaction has to preserve the live state of the work:

the active objective;
decisions, evidence, and why they changed;
active and superseded constraints;
actions already completed and writes that must not be repeated;
uncertainty that should remain uncertainty;
the next discriminating action.

That is closer to a project handoff backed by a transaction log. If the conversation says, “We tried A, evidence X disproved it, constraint Y changed, and the next move is B,” a bad compaction remembers A and Y as facts. A good one preserves the update: A is dead, Y is old, and B is next because of X.

This is why I did not ask the benchmark to preserve hidden chain of thought. The model exposes no pristine internal diary to copy. The system can preserve and evaluate visible working state: reasoning notes, decisions, evidence, rejected paths, uncertainty, completed actions, and next steps. That is what matters operationally anyway.

The techniques

I compared several ways of producing the handoff. In eval language, an oracle is an intentionally expensive best-case reference: it gets the full history and tells us whether the task is solvable at all. A lower-bound control is deliberately bad, so we can prove the benchmark notices destroyed state. The baseline is the current production technique that a new candidate must actually beat.

Technique	How the handoff gets made	What it is testing
Full-history oracle	Continue with the entire trajectory	Best-case solvability control
Lossy-summary floor	Compress aggressively without recovery	Whether the eval detects lost state
Search + memory	A fresh compactor searches bounded canonical history, retrieves evidence, then submits a validated handoff	Current production baseline
Prefix-direct	Append a handoff instruction to an exact cached prefix and have that model publish directly	Continuity plus cache reuse
Prefix-plan-worker	The exact prefix fork writes a structured brief, then a fresh worker searches history and publishes the final handoff	Same-context planning plus controlled publication
Sandbox-tool fork	Let the prefix fork execute deterministic disposable tools	Whether tool access is the missing causal ingredient

Here, search + memory means the compactor starts with a small prompt instead of the whole transcript. It uses a bounded search tool to retrieve relevant evidence from canonical history, writes the handoff, and submits it to a validator. “Publish” in the implementation means “accept this internal handoff,” not “put it on the internet.”

The phrase exact cached prefix is doing a lot of work too. A prefix is everything the provider has already seen before the new compaction instruction. For the cache to be reusable, that inherited sequence cannot merely mean the same thing. Its provider-facing representation has to stay byte-for-byte identical.

It is not enough for the logical Friday transcript to look the same. System instructions, tool schemas, cache markers, message ordering, tool-call serialization, and repair turns can all change the wire representation.

I added adapter-level proofs for that. The active request had to be a byte-identical prefix of the compaction request. Repair turns had to extend the fork without rewriting the inherited source prefix. Even a compaction tool call could not be allowed to displace one of Qwen's explicit cache markers.

That work found real bugs. The first implementation preserved the application-level prompt hash while moving provider cache boundaries. It looked exact from inside the harness and was not exact on the wire.

This is why “we use prompt caching” is not evidence. Cache behavior is a provider contract, and it has to be proven at the wire level.

The hypothesis

My favored hypothesis was:

A byte-identical prefix fork will preserve more reasoning continuity than a fresh search-based compactor while staying within 1.5 times the baseline cost and latency because most of the inherited context will be cached.

I gave it real ship gates:

Gate	Requirement
Continuity improvement	At least 15 points over current
Confidence	Paired 95% interval above zero
Pairwise judge preference	At least 60% of non-ties
Deterministic final-state success	No regression
Cache-adjusted cost	No more than 1.5x current
p95 compaction latency	No more than 1.5x current
Exact inherited prefix	100%
Safety and duplicate writes	Zero failures
Prefix fallback	Zero

That last gate became important later.

If a prefix experiment silently falls back to the existing search compactor, it can inherit the baseline's handoff quality while still being labeled a prefix success. That is benchmark contamination. The eval now records prefix fallbacks explicitly and makes any one of them a hard failure.

I needed an eval that was not already solved

Needle-in-a-haystack tests are useful, but they were too easy for this question. A compaction can retain a literal fact and still destroy the work.

I borrowed the shape of the benchmark from several stronger eval traditions:

LongMemEval-V2 for dynamic state and workflow gotchas.
NoLiMa for retrieval without obvious keyword overlap.
tau2-bench for exact final-state checks without requiring one path.
RULER and LongBench v2 for configurable long-context reasoning.
MT-Bench's judge study for reference-guided, order-swapped judging.

The resulting benchmark has five seeded families.

Family	The reasoning state that must survive
Incident diagnosis	Evidence reverses an early hypothesis; choose the next discriminating test
Changing implementation plan	Preserve revised dependencies and do not resurrect a superseded design
Data reconciliation	Derive a source-precedence policy, absorb corrections, apply it to a paraphrased case
Interrupted tool workflow	Resume completed work, avoid duplicate writes, survive an environment gotcha
Product judgment	Preserve the user's rationale, taste, trust boundaries, and scope on a new proposal

Each fixture includes a gold, or reference, ReasoningStateGraphV1. It names the goal, hypotheses, evidence, active and superseded constraints, decisions, rationale, completed actions, uncertainty, forbidden repetitions, and expected next action.

The target is deliberately hard to isolate. A fixture can contain 24 live incident chains, 35 tenant implementation plans, 6 reconciliation policies, 12 interrupted workflows, or 16 user profiles. They use opaque aliases and sit beside a long deterministic source dossier. The challenge names one target only after compaction, so the model cannot simply preserve the most prominent example.

The post-compaction question is not “what was the secret code from turn 40?” It asks the continuation to apply a causal update to one specific chain.

What the eval actually asks

Here is a simplified incident-diagnosis fixture.

Before compaction, the agent sees this sequence:

A cache leak is the provisional explanation for rising timeouts.
A blind process restart is proposed.
New measurements show that heap usage stayed flat while timeouts rose.
Connection wait time increased before request latency.
The working diagnosis changes to connection-pool exhaustion.
Heap and garbage-collection evidence has already been collected. Upstream retry rate is still unknown.
The next action becomes a bounded pool-cap ablation, not a restart.

The reference graph records the direction of those updates:

flat heap -------- contradicts ------> cache leak
connection wait -- supports ---------> pool exhaustion
pool exhaustion -- motivates --------> pool-cap ablation
restart ------------------------------> superseded
upstream retry rate ------------------> unresolved

After compaction, the model gets a new observation: a canary reproduces the timeout without heap growth. It must choose the next test, explain why that test follows from the reversal, retain the unresolved retry metric, remember that heap evidence was already collected, and avoid reviving the restart.

A passing continuation says, in effect: run the bounded pool-cap ablation and compare connection wait, because the timing evidence supports pool exhaustion while flat heap contradicts the leak hypothesis. Do not restart yet. Upstream retry rate remains unresolved.

A failing continuation can still sound reasonable. It might recommend restarting “to rule out the cache,” repeat the completed heap measurement, drop the unresolved retry rate, or remember pool exhaustion without remembering why it replaced the earlier diagnosis. A keyword-recall test would miss most of those failures. The state graph does not.

Two other families test different failure modes:

Interrupted tool workflow: an upload completed, then receipt polling was rate-limited. After compaction, the agent must inspect the durable status and record verification exactly once. Re-uploading is a hard failure even if the prose sounds cautious. The sandbox action ledger makes that deterministic.
Product judgment: the user wants less review work but has explicitly rejected autonomous sending. The active design is a shadow-ranked review queue with visible rationale and required approval. After compaction, a new proposal suggests auto-sending “high-confidence” drafts. A good continuation rejects that shortcut, preserves the calibration uncertainty, and proposes the smallest reversible ranking experiment.

These are small versions of what an actual agent has to do after a handoff: resume the right branch, preserve why the plan changed, and avoid repeating consequential work.

Deterministic scoring first, model judging second

I wanted most of the score to come from things a script could verify.

Each trial receives a continuity score from 0 to 100. That is not the percentage of transcript words remembered and it is not the model's confidence. It is the weighted share of required reasoning state and correct outcomes that survived compaction.

The score was designed as:

continuity score = 60% deterministic outcome/state + 40% semantic judge

The deterministic side checks exact sandbox state, completed-action preservation, forbidden duplicate writes, active versus superseded constraints, and required next actions.

The semantic judge scores the parts that are harder to reduce to exact assertions: whether the rationale still makes sense, uncertainty remains calibrated, and the continuation is useful. It sees a bounded packet containing the gold state graph, challenge, handoff, and continuation result, but not the technique name or cost. It must cite graph identifiers for every finding. Pairwise comparisons run in both answer orders; inconsistent preferences become ties.

An aggregate score is the average across the seeded trials. A family score is the average within one scenario type, such as incident diagnosis. Pairwise judge preference asks a different question: when shown the baseline and candidate side by side, how often does the judge consistently prefer the candidate?

Before the judge's scores count, it has to pass known corruptions: omitted rationale, resurrected constraints, false certainty, verbosity attacks, and identical-answer ties.

I do not think “LLM as a judge” is automatically scientific. I think a calibrated, blinded, bounded judge can cover semantic properties that are expensive to reduce to string matching, as long as deterministic state remains the authority where deterministic state exists.

Calibration: prove the benchmark has room

Before comparing compaction techniques, I ran a 64K calibration.

Control	Aggregate continuity	What it should prove
Full-history oracle	97.32	The tasks are solvable
Search + memory	74.84	The baseline is useful but not saturated
Deliberately lossy summary	9.07	The benchmark detects destroyed reasoning state

Read 97.32 as “the full-history control satisfied about 97% of the weighted continuity requirements,” not “it remembered 97% of the words.” No baseline family exceeded 84.4. The oracle's weakest family still scored 94.4. The judge calibration passed 6 out of 6 adversarial cases.

That was the green light.

The eval was not just measuring whether the model could do the task at all. It also was not an easy suite where every halfway competent compactor scored 98.

There was room for the clever idea to win.

Why Qwen was useful for this work

I ran the experiment on Qwen 3.8 Max Preview through Alibaba Cloud's Token Plan. The plan is unusually well suited to interactive harness development: a subscription includes a large monthly Credits pool, a dedicated API key, and OpenAI-compatible and Anthropic-compatible endpoints that can be connected to your own coding or agent tool. The published tiers currently range from 25,000 to 250,000 Credits per seat each month.

That API-key detail matters. I could exercise my own harness, tool loop, long-context serialization, and cache accounting instead of evaluating only inside a provider's chat product. The large allowance made it practical to generate long trajectories, run controls, repair broken eval infrastructure, and repeat the same request shape enough times to measure cache behavior.

The major American labs do not currently package their individual subscriptions this way. OpenAI and Anthropic explicitly separate subscription access from API billing, while the Gemini API uses its own Cloud Billing account. Qwen's Token Plan is still a development plan for interactive coding and agent tools, not a substitute for production API billing, but it is an excellent environment for building and inspecting a harness.

The cache part actually worked

Qwen's explicit context cache is real. Its official context-cache documentation lets callers mark stable content with cache_control. The published economics at the time of this experiment priced explicit writes at 125% of normal input and reads at 10%, with a short renewable lifetime.

In a direct repeated-prefix probe, the second request contained 5,438 input tokens and Qwen reported 5,432 cached tokens.

That is an excellent cache hit.

It is also not the whole economic story.

Qwen does not expose a free “save this prefix for later” endpoint. Creating the explicit cache requires a model request. The first pass pays the cache-creation premium. The compactor still has to generate and sometimes repair the handoff. If the attempt fails and falls back, the system pays for both paths.

The tables separate input tokens from cached input tokens, the subset the provider says it reused, and cache creation tokens, the content written into an explicit cache. The cache-adjusted cost below is a normalized token-equivalent using the provider's different write and read rates. It is useful for comparing arms, but it is not a dollar amount.

That cost is therefore closer to:

ordinary input
+ 1.25 * cache creation
+ 0.10 * cache reads
+ output and reasoning
+ retries
+ fallback

The cache hit can be nearly perfect while the product route is still expensive.

That is exactly what happened.

The result

The cleanest exact-save comparison looked like this. By this point the Token Plan balance was near 5%, and the candidate had already failed the deterministic, cost, latency, and fallback gates. I did not spend the remaining allowance on its final semantic-judge matrix, so these continuity numbers are the deterministic component, not the earlier 0-to-100 composite calibration score.

Measure	Search + memory	Exact prefix fork	Fork vs. baseline
Deterministic continuity	60	40	-20 points
Input tokens	18,254	224,686	12.3x
Cached input tokens	3,695	133,431	36.1x
Cache creation tokens	14,541	91,219	6.3x
Cache-adjusted input cost (token-equivalent)	18,563.75	127,402.85	6.9x
Compaction wall-clock latency	75.7s	682.3s	9.0x
Exact inherited prefix	Not applicable	2 of 2	Passed
Prefix fallback	0	1	Hard failure

Latency is the full time spent producing an accepted handoff, including retries and fallback. “Exact inherited prefix” is the provider-wire proof described earlier. A fallback means the prefix route failed and the baseline had to rescue it, so the candidate did not independently succeed.

The fork did the technically impressive thing. It preserved the exact inherited prefix. It produced large cache reads. It proved the provider adapter could keep its markers stable across an appended compaction request and a repair turn.

Then it failed to publish a valid handoff in its two allowed attempts and fell back to search.

So even the 40 continuity score is not clean prefix quality. The final artifact was contaminated by the baseline fallback. Once fallback became a hard failure, the arm was ineligible regardless of the cost.

An earlier, smaller slice looked more encouraging:

Earlier slice	Current	Prefix-direct
Deterministic continuity	about 30	46.67
Cache-adjusted input cost	about 29K	100.4K
Latency	about 63s	265s

That was a real signal: the same-prefix model sometimes held the evolving task state better. It also cost 3.4 times as much, took 4.2 times as long, and produced a malformed first publication. Declaring victory there would have been bad evaluation practice. Once cache creation, retries, publication validity, and fallback were counted, the promising signal did not survive the full route.

The winner is search + memory

For this system, with this model, under these constraints, the best compaction technique I tested is the boring one:

Start a fresh bounded compaction worker, let it search canonical history for the evidence it needs, require at least one successful retrieval, and validate the final handoff before accepting it.

It is not theoretically pure. It does not preserve the same model prefix. It asks a worker to reconstruct active state through retrieval.

It is faster, cheaper, easier to bound and recover, and currently better at producing a valid handoff.

The prefix fork remains a useful research seam. I would test it again if one of these changes:

the provider offers a durable fork or continuation handle without paid priming;
cache creation can be amortized across many compactions;
cache lifetime becomes materially longer;
the model becomes much more reliable at publishing the required schema;
the handoff can be produced without repair turns;
the provider exposes a server-side compaction primitive with verifiable prefix reuse.

But none of those is true enough today to justify making it the default.

A practical checklist for context management

The experiment left me with a reusable standard for compaction systems:

Treat compaction as state transfer, not text shortening. Preserve goals, evidence, decisions, superseded paths, completed actions, uncertainty, and next steps.
Keep canonical history durable. A compact handoff should reduce active context, not destroy the underlying record.
Make recovery bounded and explicit. Search should retrieve exact evidence in pages, and the handoff should say what remains available.
Validate before accepting the handoff. Require successful retrieval, a stable schema, source attribution, and explicit treatment of unresolved state.
Score outcomes before prose. Check sandbox state, duplicate writes, forbidden actions, and the selected next step deterministically. Use a semantic judge for rationale and usefulness, not for facts the system can verify directly.
Calibrate the benchmark. A full-history oracle should nearly solve it, a deliberately lossy summary should fail, and the production baseline should leave room for improvement.
Measure the whole route. Count cache creation, reads, output, repairs, fallback, and wall-clock latency. A cache hit is not automatically a product win.
Prove provider behavior on the wire. Application-level prompt equality does not guarantee identical cache markers or serialized messages.

The eval and the prefix research seams shipped. The production default did not change.

In this case, the prefix-stable fork was real. The cache reuse was real. The reasoning-continuity hypothesis was plausible. The implementation work improved the harness.

The winner was still search + memory.

That is less magical than the idea I started with. It is also the decision I trust.