Context-preparation timing benchmark

Measured timing change

In this self-benchmark, preparing repository summaries with a language model before the request reduced median context-preparation time by 61% (33.2 to 12.9 seconds) across three fixed questions on one 36-file, 9.7K-line repository. An immediate repeat prepared context in 0.81 milliseconds.

Recorded timings

Generated 2026-08-01 on an Apple M2 with Node 23.7.0. Each state contains one observation for each of three fixed questions.

Where the preparation work happens

The cold measurement starts from an empty isolated cache and pays for the whole-project summary pass inside the timed request. The prewarmed measurement starts from an empty cache too, but completes that pass before timing the request. This keeps the available repository context comparable while changing when its preparation is paid.

The recorded configuration used Haiku for basic summaries and Opus to combine those summaries into context and generate answers. Prewarming removes basic-summary work from the interactive path, but context folding and answer generation still require model calls. The immediate repeat reuses prepared context and records one model call for the answer.

The runner resets an isolated cache between independent cases and records hits, misses, model-call phases, and stage timings. This matters because an unrecorded warm cache could make an apparent latency improvement depend on earlier requests rather than the state being tested.

Per-question context-preparation times

These are individual observations, not repeated-trial averages. The question IDs match the raw result file so each timing can be traced to its selection, question, and telemetry.

Exact calculation

(33,223.190 − 12,865.299) / 33,223.190 = 61.276%

Advance preparation took a median 18,857.843 milliseconds and 59 model calls. This one-time cost is recorded separately from the request timings above; the reduction moves work ahead of the request rather than eliminating it.

Reproducibility materials

Reproduction materials include the dated result, all three fixed questions, code that resets caches between runs, timing instrumentation for each stage, measurement tests, and a runner using result-format version 2 (schema v2). The updated runner records token usage reported by the provider for each model and stage, and keeps request-time context work separate from one-time advance preparation.

With dependencies installed, a dry run checks all three benchmark targets and reports the current repository’s file, line, and byte counts without making model calls.

Study limitations

• One repository and three observations per state.
• Each median combines timings from three different fixed questions, rather than repeated timings of one request.
• The recorded source state was commit afc99fe plus uncommitted benchmark instrumentation.
• The reported 61% reduction covers pre-answer context preparation; answer generation is a separate stage.
• The original result format (schema v1) does not record token usage, so it cannot establish dollar cost.

Source records and benchmark code