Measure the Context Tax Before an AI Agent Starts Working

An AI coding agent can spend tokens before it has done any work that matters to you. It may load built-in instructions, user rules, repository guidance, tool definitions, prior messages, tool results, and cached prefixes before it reads the one-line task you care about. Then retries, verification turns, and subagents can multiply that starting load.

That is the context tax: model input and related execution overhead that exists because of the agent harness, configuration, and accumulated session state rather than because of the task itself.

The tax is not automatically waste. A safety rule can prevent a destructive command. A repository instruction can stop an invalid refactor. A tool schema can enable a successful test run. The practical question is not “Which agent has the smallest prompt?” It is:

How much context does this configuration process, what useful protection or capability does it buy, and how much does a verified completion cost?

This guide provides a vendor-neutral way to answer that question without inventing a benchmark. It does not publish measured winners. Instead, it gives you a protocol, a results table, a CSV schema, and formulas you can use with your own repository and accounts.

Product documentation and CLI behavior referenced below were checked on 2026-07-24. Limits, model defaults, output fields, and subscription policies can change, so preserve the documentation date and tool version with every run.

Why the context tax is easy to miss

The visible user prompt is only one part of a model request. Anthropic’s context-window documentation says that a request can include the system prompt, prior messages, tool results, images, documents, and tool definitions; generated output and extended thinking also consume context (official Anthropic context-window documentation, checked 2026-07-24). That list captures the main reason a short instruction can produce a surprisingly large usage record.

The same distinction matters in Codex. OpenAI documents a codex debug prompt-input command that renders the model-visible prompt input list as JSON for debugging instruction discovery, session context, and prompt construction (official OpenAI Codex command reference, checked 2026-07-24). That is a more useful starting point than counting the characters in the final prompt pasted into a terminal.

Users notice the problem in several forms:

• a fresh agent session feels “heavy” before the requested edit begins;
• a long conversation becomes more expensive or reaches compaction sooner;
• enabling many MCP or built-in tools appears to increase the initial payload;
• a retry repeats much of the same prefix;
• multiple subagents each start with their own instructions and tools;
• a usage limit arrives sooner than the visible prompt length would suggest.

One social post circulated a comparison of roughly 33,000 versus 7,000 tokens before useful work. That is a self-reported observation, not an OpenAI or Anthropic product specification (the original social post). It does not establish a universal baseline because the post cannot control your CLI version, model, enabled tools, instruction files, repository, session history, cache state, or accounting semantics. Treat it as a testable hypothesis: some configurations may carry materially different startup loads.

This distinction protects you from two bad conclusions. First, a large self-reported number does not prove that every installation of a product has that overhead. Second, a small number does not prove that the agent is efficient if it fails the task, omits verification, or silently delegates work to calls that were not counted.

Define the unit you are measuring

“Token use” is too vague for a defensible comparison. Record at least four units:

1. Run: one top-level invocation from process start or session turn to terminal result.
2. Model call: one request to one model, including calls made after tool results.
3. Agent branch: the root agent or one spawned worker, teammate, or subagent.
4. Verified completion: a run whose output passes the same predetermined checks.

The run is what the user experiences. The model call explains accumulation. The branch catches fan-out. The verified completion prevents a cheap failure from looking efficient.

Also separate three quantities:

• context occupancy: tokens that must fit within the active context window;
• processed usage: input and output reported for calls, including separately reported cache categories;
• billing or plan consumption: money, credits, or allowance charged under the current product and account.

Those quantities overlap but are not identical. Anthropic explicitly notes that cached prompt prefixes still occupy the context window even though caching changes what you pay for those tokens (official Anthropic context-window documentation, checked 2026-07-24). OpenAI’s API usage objects report input, output, total, cached-input, and reasoning-output fields, while Codex subscription usage can draw from an agentic usage and credit pool whose task consumption varies with task size, complexity, model, and execution location (official OpenAI Responses usage reference, official OpenAI Codex plan guidance, checked 2026-07-24).

Do not collapse these into one number called “tokens.” Keep the raw vendor fields and add normalized fields beside them.

Build a context inventory before benchmarking

Create an inventory for each configuration. Do not estimate token counts from file sizes when the product exposes model-visible input or usage. File bytes are still useful as a change detector, but different tokenizers and serialization formats can produce different token totals.

Fixed instructions

This layer includes vendor-provided system instructions, safety policies, global user instructions, and harness rules. Some content may be visible through a debug command or trace; some may remain managed by the product.

Record:

• CLI or app version;
• selected model and reasoning setting;
• account surface, such as subscription CLI or API;
• global instruction filenames and byte counts;
• whether the product exposes the complete model-visible prompt;
• a hash of each user-controlled instruction file.

Never assume that “no repository instructions” means “no instructions.” The harness still needs operational and safety behavior.

Repository files

Repository guidance may be discovered by filename and directory scope. It may include coding conventions, build commands, test rules, nested instructions, or generated memory. Record which files were loaded, their byte counts, their hashes, and their location relative to the task.

Separate always-loaded repository rules from task-relevant files opened during work. A 10,000-token style guide that prevents an invalid edit is overhead with value; a stale duplicate that repeats global rules is overhead without obvious value.

For the experiment, keep the repository commit or archive hash fixed. If one agent indexes the repository automatically while another reads files on demand, record that behavior rather than trying to force the implementations to look identical.

Tool schemas

Every exposed tool needs a name, description, and input schema. Tool definitions can be substantial even when no tool is called. Anthropic states that tool definitions are part of the context and documents that tools, system messages, text messages, tool use, and tool results can participate in prompt caching (official Anthropic context documentation, official Anthropic prompt-caching documentation, checked 2026-07-24).

Record:

• number of enabled tools;
• serialized schema bytes if export is possible;
• tool server names and versions;
• whether deferred or on-demand discovery is enabled;
• tools actually invoked;
• changes to the tool set between samples.

Do not compare “tool-enabled Agent A” against “tool-disabled Agent B” and call the result a vendor comparison. That is a configuration comparison.

Conversation history

Prior user and assistant messages, thinking artifacts where applicable, tool calls, and tool results can be carried forward. A long session therefore has a different starting point from a new session. Anthropic documents progressive accumulation of conversation turns and says Claude Code may auto-compact as a conversation nears the model’s maximum input size (official Anthropic context-window documentation, checked 2026-07-24). Its Claude Code cost guide separately warns that a context or auto-compact warning is not the same as a subscription usage limit (official Claude Code cost guide, checked 2026-07-24).

Measure fresh and continued sessions separately. For continued sessions, record the number of turns, transcript bytes, compaction events, and whether the continuation resumes a stored server-side state or replays a local transcript.

Cached input

Caching reduces repeated-prefix billing or latency under vendor-specific rules; it does not make the prefix disappear. Preserve raw cache fields.

Anthropic’s API examples distinguish input_tokens, cache_creation_input_tokens, and cache_read_input_tokens, and its documentation explains that modifications follow a tools-to-system-to-messages invalidation hierarchy (official Anthropic prompt-caching documentation, checked 2026-07-24). OpenAI’s Responses usage schema includes input_tokens_details.cached_tokens, while its aggregate usage documentation says input token totals include cached tokens (official OpenAI Responses usage reference, official OpenAI Usage API reference, checked 2026-07-24).

That difference alone makes naïve addition unsafe. For OpenAI API-shaped data, adding cached tokens to input tokens can double-count. For Anthropic-shaped data, total processed input may require combining uncached, cache-read, and cache-creation fields. Your adapter must follow the documented semantics of the exact output you captured.

Generated output and reasoning

Output is not startup context, but it belongs in total execution cost. Capture visible output, reasoning-token fields when reported, and every intermediate assistant message. Do not infer hidden reasoning length from the final prose.

OpenAI’s response usage structure exposes output-token details, including reasoning tokens where supported (official OpenAI Responses usage reference, checked 2026-07-24). Anthropic’s API response exposes usage, and its official SDK documentation shows reading the usage response property for input and output counts (official Anthropic TypeScript SDK documentation, checked 2026-07-24).

Retries

A retry is a new cost event even if the user sees one final answer. Count:

• automatic transport retries;
• model retries after invalid structured output;
• agent retries after a failed command;
• human-requested correction turns;
• full reruns after an incomplete or incorrect result.

Label the retry reason. A network retry is not the same failure mode as an agent inventing a passing test.

Subagent fan-out

Every branch must be included in the root run. Anthropic documents that agent teams spawn multiple Claude Code instances with their own context windows and that token usage scales with active teammates and their run duration (official Claude Code cost guide, checked 2026-07-24). The principle is vendor-neutral: parallel branches can reduce wall-clock time while increasing processed context.

Record each branch with parent_agent_id, start and end times, model, tool set, and usage. If the product exposes only a root total, mark branch-level usage unavailable rather than assigning the total evenly.

Use machine-readable output, but inspect what it contains

Machine-readable mode reduces transcription mistakes; it does not guarantee equivalent fields.

OpenAI documents codex exec as the stable non-interactive command. Its --json flag prints newline-delimited JSON events, and --output-last-message writes the final assistant message for downstream checks (official OpenAI Codex command reference, checked 2026-07-24). The same reference documents codex debug prompt-input for rendering the model-visible input list as JSON. Use both when available: event output for the run, prompt-input output for the starting context inventory.

Claude Code documents print mode with --output-format json or stream-json, plus --max-turns for limiting non-interactive agentic turns (official Claude Code CLI reference, checked 2026-07-24). Claude Code’s /usage screen reports session token statistics; Anthropic says the displayed dollar amount is a local estimate and directs users to the Console Usage page for authoritative API billing (official Claude Code cost guide, checked 2026-07-24).

Before collecting the real samples, run one disposable parser-validation task and answer these questions:

• Is usage reported per model call, per turn, per branch, or only per root run?
• Does input include cached input?
• Are cache reads and writes separate?
• Are reasoning tokens included in output?
• Is cost an authoritative billed amount, a local estimate, or absent?
• Are failed and retried calls present?
• Does the final event say success, or must success be established by external checks?

Store the raw JSON or JSONL unchanged. Parse into a normalized CSV in a separate step. That makes it possible to correct an adapter later without rerunning paid or rate-limited experiments.

A vendor-neutral experiment protocol

The protocol below compares configurations, not abstract model intelligence. Use the same small repository and the same task. Do not modify the task between vendors after seeing a failure.

1. Prepare one fixed repository

Choose a small repository that can be copied exactly for every sample and tested offline. Pin:

• the repository archive hash or commit identifier;
• dependency lockfiles;
• runtime and compiler versions;
• test command;
• formatting or type-check command;
• initial working-tree state;
• network policy;
• environment variables, excluding secrets from the recorded dataset.

The repository should contain enough structure to make agent instructions and file discovery realistic, but it should finish its deterministic checks quickly. Do not use a repository that requires changing external services.

2. Define one bounded task

Use a task with an objectively checkable result. For example:

Add support for rejecting an empty display name in the existing validation function. Preserve current behavior for non-empty names. Add or update focused tests. Do not change dependencies.

Write the task once and hash the exact UTF-8 prompt. Define completion before running any agent:

• the expected file scope;
• a required test command and exit code;
• a required new behavioral assertion;
• a prohibition on unrelated file changes;
• a maximum wall-clock time;
• a maximum number of top-level retries.

Do not grade style by intuition after seeing vendor names. If code review matters, create a blind rubric before the run.

3. Define configurations

Create configuration labels such as:

• minimal-tools-fresh;
• standard-tools-fresh;
• standard-tools-long-history;
• standard-tools-one-subagent;
• standard-tools-three-subagents.

For a cross-vendor comparison, match capabilities as closely as the products allow: same model class intent, same reasoning setting intent, same write permissions, same network policy, and the same validation commands. Preserve unavoidable differences as confounders.

Never disable safety instructions merely to reduce tokens. The goal is to measure deployable configurations.

4. Measure the pre-task input

For each configuration, capture a context manifest before the task runs:

configuration_id
vendor
product_surface
cli_version
model
reasoning_setting
fresh_or_continued
instruction_file_count
instruction_bytes
repository_rule_file_count
repository_rule_bytes
enabled_tool_count
tool_schema_bytes_if_available
history_turns
history_bytes_if_available
prompt_input_export_path
prompt_input_export_bytes

If the product exposes an exact model-visible prompt export, tokenize it only with a vendor-supported counter for the chosen model or use reported request usage. If it does not, report bytes and an “unavailable” token field. Do not substitute a third-party tokenizer and label the result exact.

To estimate fixed startup overhead by difference, add a control task that requires no repository work and produces a tiny structured acknowledgement. Run it under the same configuration. The control is not a perfect view of hidden system tokens: the harness may take different paths for a real coding task. Label the resulting median control_run_input, not “true system prompt size.”

5. Run cold samples

A cold sample starts with:

• a fresh process or product-defined fresh session;
• the same clean repository copy;
• no continuation history;
• no deliberate cache priming;
• the same enabled tools;
• the same task prompt.

Run at least five samples per configuration; seven is a practical odd-numbered default when allowance permits. Randomize configuration order so time-of-day or service-load changes do not always favor the same configuration. Record failures rather than replacing them silently.

“Cold” refers to your controlled session and cache procedure. You cannot prove the absence of every provider-side optimization. Use cache_state_observed=unknown unless the usage output establishes a miss.

6. Run warm samples

Warm samples repeat the identical task setup after a documented priming run, within the cache lifetime or continuation behavior supported by the product. Do not edit tool schemas, instructions, or the prompt between the priming run and measured warm runs.

Warm samples test economic and latency effects of repeated prefixes. They do not show a smaller context window footprint. Keep:

• priming run usage;
• warm run usage;
• elapsed time between runs;
• reported cache-read and cache-write fields;
• any cache invalidation event;
• whether the repository was reset.

7. Run long-history samples

Create a fixed synthetic conversation transcript that contains several prior turns and tool results but no answer to the benchmark task. Use the exact same transcript for every compatible configuration. Record transcript bytes and, where reported, input tokens.

Do not compare one vendor’s compacted summary with another vendor’s raw transcript without labeling the difference. Compaction is part of the product behavior, but it changes what the model receives.

8. Run fan-out samples

Only test fan-out if both the product and task permit a controlled branch count. Fix the maximum number of branches. Give each branch the same bounded role across configurations, such as one implementation branch and one review branch. Sum usage across the entire agent tree.

Record both elapsed time and aggregate processed tokens. A fan-out configuration may be faster but more expensive; those are separate results.

9. Verify every completion externally

The agent’s final claim is not the completion check. After each run:

1. capture the process exit code;
2. list changed files;
3. reject changes outside the allowlist;
4. run the fixed test command in a clean shell;
5. run the fixed type, lint, or formatting check if part of the rubric;
6. confirm the targeted assertion exists and fails against the unmodified baseline when feasible;
7. record verified_completion as 1 only if every mandatory check passes.

Preserve command stdout and stderr. A run that uses few tokens but leaves failing tests is not an efficient completion.

10. Summarize with medians and distributions

For each configuration and condition, report:

• sample count;
• verified completion count;
• median processed input;
• median cached input;
• median output;
• median elapsed seconds;
• median top-level retries;
• median aggregate branch count;
• median reported or normalized cost;
• minimum and maximum, or an interquartile range;
• cost per verified completion.

Medians reduce the influence of one unusually long run, but they do not replace raw samples. Publish the populated CSV alongside any future benchmark article.

Results table template

Leave cells blank until you have collected actual runs. A blank table is more honest than a plausible-looking fabricated benchmark.

Configuration	Condition	Samples	Verified completions	Median processed input	Median cached input	Median output	Median retries	Median branches	Median elapsed seconds	Total normalized cost	Cost per verified completion
Vendor A, minimal tools	Cold
Vendor A, standard tools	Cold
Vendor A, standard tools	Warm
Vendor A, long history	Continued
Vendor A, fixed fan-out	Cold
Vendor B, minimal tools	Cold
Vendor B, standard tools	Cold
Vendor B, standard tools	Warm
Vendor B, long history	Continued
Vendor B, fixed fan-out	Cold

CSV schema

Use one row per root run and retain a second call-level file if the product exposes individual calls. Empty fields mean unavailable, not zero.

experiment_id,run_id,configuration_id,vendor,product_surface,cli_version,model,reasoning_setting,repository_hash,task_prompt_hash,condition,started_at_utc,elapsed_seconds,exit_code,root_agent_id,total_agent_branches,total_model_calls,top_level_retries,automatic_retries,instruction_files,instruction_bytes,repository_rule_files,repository_rule_bytes,enabled_tools,tool_schema_bytes,history_turns,history_bytes,prompt_export_bytes,raw_input_tokens,raw_cache_read_tokens,raw_cache_write_tokens,raw_output_tokens,raw_reasoning_tokens,normalized_processed_input_tokens,normalized_output_tokens,reported_cost,cost_currency,cost_authority,plan_usage_delta,changed_files_count,disallowed_changes,test_command,test_exit_code,completion_checks_passed,verified_completion,raw_event_log_sha256,notes

Recommended enumerations:

• condition: cold, warm, continued, or fanout;
• cost_authority: vendor_bill, vendor_console, local_estimate, rate_card_calculation, or unavailable;
• verified_completion: 1 or 0;
• disallowed_changes: 1 or 0.

For call-level data, add:

experiment_id,run_id,call_id,parent_call_id,agent_id,parent_agent_id,sequence_index,model,started_at_utc,elapsed_seconds,raw_input_tokens,raw_cache_read_tokens,raw_cache_write_tokens,raw_output_tokens,raw_reasoning_tokens,reported_cost,stop_reason,error_type,retry_of_call_id,tool_calls_count

Keep hashes rather than raw secrets, authentication material, or private prompts in a public dataset. If repository content is private, publish the schema, generator, and verification commands without publishing the source.

Formulas that answer useful questions

The formulas below operate on normalized fields. Document each vendor adapter before combining results.

Normalized processed input

For an accounting shape where uncached input, cache reads, and cache writes are mutually exclusive:

processed_input
  = uncached_input
  + cache_read_input
  + cache_write_input

For an accounting shape where input_tokens already includes cached tokens:

processed_input = input_tokens

Never apply the first formula to the second shape. Preserve the raw fields so reviewers can audit the adapter.

Estimated fixed context tax

For a configuration (c), using the controlled no-work task:

estimated_fixed_context_tax(c)
  = median(control_run_processed_input for configuration c)

This estimates the configuration’s control-path load. It is not proof of the hidden system prompt’s exact size.

Task-attributable processed input

For run (r):

task_attributable_input(r)
  = max(0, processed_input(r) - estimated_fixed_context_tax(configuration(r)))

The floor at zero prevents noisy control estimates from producing a nonsensical negative value. Report the control distribution beside this derived metric.

Useful-work context ratio

Define task_relevant_input as task prompt tokens plus repository content and tool results required by the predetermined rubric. When exact component attribution is unavailable, use task_attributable_input as a clearly labeled proxy.

useful_work_context_ratio(r)
  = task_relevant_input(r) / processed_input(r)

A higher ratio is not automatically better. Removing a short safety rule could raise the ratio while making the system unacceptable. Use the ratio only among configurations that pass the same safety and completion requirements.

Retry amplification

retry_amplification(r)
  = total_processed_tokens_across_all_attempts(r)
    / processed_tokens_in_final_attempt(r)

A value of one means no extra processed tokens from prior attempts. Record automatic and human-triggered retries separately.

Fan-out amplification

fanout_amplification(r)
  = aggregate_processed_tokens_all_agents(r)
    / processed_tokens_root_agent_only(r)

This shows the token multiplier, not the latency benefit. Report elapsed time beside it.

Verified completion rate

verified_completion_rate
  = verified_completions / attempted_runs

Do not exclude timeouts, invalid outputs, or tool failures after the fact.

Cost per verified completion

cost_per_verified_completion
  = sum(normalized_cost for every attempted run)
    / count(verified_completion = 1)

If there are no verified completions, report the result as undefined, not zero.

For API billing, calculate normalized cost from the official rate card that applied on the run date, keeping cache reads, cache writes, uncached input, output, and tool charges separate. For subscription products, do not convert an allowance percentage into dollars unless the vendor supplies a defensible conversion. Report plan-usage delta and subscription fee separately.

Why vendor-reported usage may not be directly comparable

Two fields with the same name can describe different boundaries.

Different inclusion rules

One product may report a total that includes cached input; another may report cache reads and cache creation beside uncached input. One may expose reasoning output separately while another folds it into output. One CLI may report only the root session while an SDK trace exposes every call.

OpenAI’s documented API usage includes cached input within aggregate input totals, while Anthropic documents distinct cache-read and cache-creation fields (official OpenAI Usage API reference, official Anthropic prompt-caching documentation, checked 2026-07-24). This is a schema difference, not evidence that either product secretly uses fewer tokens.

Different product surfaces

An API call, a subscription CLI turn, a desktop-agent run, and a cloud task can use different harnesses and accounting. OpenAI says Codex task consumption varies with task complexity, model, and where the task runs, and that supported Codex surfaces can draw from the same agentic usage and credit pool on eligible plans (official OpenAI Codex plan guidance, checked 2026-07-24). Anthropic says Claude Code subscription users have usage included in their plan, so the CLI’s session dollar estimate is not their billing figure (official Claude Code cost guide, checked 2026-07-24).

Compare like with like: API to API when measuring token-priced economics, and subscription surface to subscription surface when measuring allowance consumption.

Different hidden or managed layers

Vendors may not expose every system instruction or preprocessing step. A prompt export can show model-visible items supported by that command, but it does not prove that two products expose the same boundary. Report visibility=partial when appropriate.

Different tokenizers and serialization

The same English text can tokenize differently across model families. Tool schemas may also be serialized with different wrappers. Cross-vendor token counts therefore measure each model’s processed units, not a universal amount of text.

Include bytes and file hashes as neutral reproducibility aids, but use vendor-reported tokens for within-vendor accounting.

Different caching behavior

Cache hits can depend on prefix stability, minimum cacheable length, lifetime, tool definitions, system messages, and request order. Anthropic documents that changing tool definitions invalidates the tool, system, and message caches in its hierarchy (official Anthropic prompt-caching documentation, checked 2026-07-24). A warm comparison is invalid if one configuration changed a tool description between samples.

Different context management

Products can compact, summarize, truncate, or resume context differently. A lower later-turn input could be the result of compaction, not lower startup overhead. Capture compaction events and evaluate whether the final task still passes.

Different plan limits

A context-window limit is not a subscription usage window, rate limit, or credit balance. Anthropic’s docs explicitly separate context or auto-compact warnings from session and weekly plan limits (official Claude Code cost guide, checked 2026-07-24). OpenAI likewise points users to the Codex usage page or limit banner for plan-specific options after reaching a limit (official OpenAI Codex plan guidance, checked 2026-07-24).

Record which ceiling was hit. Do not label every interruption “context exhaustion.”

Confounders to record

A credible report includes the conditions that could explain a difference:

• model identifier or alias changed during collection;
• reasoning or thinking settings were not equivalent;
• CLI, app, plugin, or MCP versions changed;
• one run had network access and another did not;
• repository state or dependency caches differed;
• background indexing occurred;
• tool schemas were loaded eagerly in one product and lazily in another;
• warm runs missed the cache;
• a session was compacted or summarized;
• a vendor performed an automatic retry that was not exposed;
• a subagent used a different model;
• service load affected latency;
• output verbosity differed;
• one run stopped at a usage limit;
• one product verified tests while another merely claimed success;
• rate-card prices or subscription allowances changed during collection;
• local usage estimates omitted other devices or surfaces;
• a task leaked into prior conversation history;
• the evaluator saw vendor identity and introduced grading bias.

If a confounder changes mid-experiment, start a new experiment ID. Do not merge the samples into one median.

Interpret the result without rewarding the wrong behavior

The smallest pre-task input is not the objective. The objective is the lowest context and cost that reliably produces the required, safe, verified outcome.

Use a decision ladder:

1. Reject configurations that violate safety or file-scope constraints.
2. Reject configurations that do not reach the required verified completion rate.
3. Among the remaining configurations, compare cost per verified completion.
4. Use elapsed time as a separate operational metric.
5. Inspect context-tax and useful-work ratios to find removable duplication.

Then optimize one layer at a time:

• delete duplicated instructions, not proven guardrails;
• move rarely needed reference material behind explicit discovery;
• expose only tools relevant to the task;
• shorten verbose schemas without weakening their contract;
• start a new session for unrelated work;
• keep stable prefixes stable enough to cache;
• bound retries;
• use subagents only when the completion or latency gain justifies fan-out.

Re-run the same samples after each change. If tokens fall but verified completion also falls, you did not improve efficiency; you moved cost into failure.

Limitations

This protocol cannot reveal content a vendor does not expose. Control-run subtraction estimates a configuration-specific baseline but cannot uniquely separate hidden system text, routing metadata, safety processing, or harness serialization.

Reported token counts can change with model and product updates even when your files do not. Subscription usage may be expressed as a limit bar or credit pool rather than an authoritative per-run currency amount. Local cost estimates may differ from billing records. Cached input can be cheaper while occupying the same context. Parallel work can reduce latency while increasing aggregate tokens. A tiny repository task cannot predict behavior on a million-line codebase.

Most importantly, this protocol does not turn one laboratory task into a universal ranking. Run a task suite that reflects your actual work: small edits, debugging, repository exploration, test repair, and review. Keep each task fixed, independently verifiable, and reported separately before aggregating.

The 33,000-versus-7,000 social comparison should remain what it was: a useful prompt to investigate, not a product fact. A defensible article begins only after raw event logs, configuration manifests, repeated samples, and completion evidence exist.

Frequently asked questions

Can I measure the hidden system prompt exactly?

Only if the product exposes the exact relevant model-visible input and a supported token count for it. Codex documents a prompt-input debug export, but cross-product visibility can still differ. Otherwise report visible components, raw usage, and a control-run estimate; do not label the estimate an exact hidden prompt count.

Should cached tokens count as context tax?

Yes for context occupancy and processed-prefix analysis. Treat them separately for cost because vendors can price and report cache categories differently. A cache discount does not remove the cached prefix from the context window.

Is a fresh session always cheaper?

Not necessarily. A fresh session removes conversation history but may miss a reusable prompt cache or repeat repository discovery. Measure both cold and warm conditions.

How many samples are enough?

Use at least five per condition and publish every run. Seven gives a stable middle observation without pretending to eliminate variance. Increase the sample count when results are close, failure rates are high, or service variance is large.

What if the CLI does not report per-call tokens?

Store the root-run output, mark per-call fields unavailable, and avoid claims about internal call distribution. You can still compare verified completion, elapsed time, root usage, changed files, retries visible in the trace, and plan-usage deltas.

Can subscription plans be compared by dividing the monthly fee by tokens?

Not responsibly unless the vendor supplies an authoritative token allowance and accounting conversion for that plan and surface. Report the subscription fee, observed usage-window delta, completed tasks, and limits encountered as separate fields.

The minimum publishable evidence

Before turning your populated results into a comparison article, require:

• an as-of date and exact tool versions;
• one immutable repository fixture;
• exact prompt and configuration hashes;
• raw JSON or JSONL logs;
• cold and warm samples;
• repeated runs with medians;
• branch-inclusive usage where available;
• deterministic completion checks;
• every failure retained;
• vendor-specific normalization notes;
• a populated CSV with no invented cells;
• official documentation linked next to accounting claims.

That evidence will not make every vendor number perfectly comparable. It will make the remaining differences visible, bounded, and honest. That is enough to replace “my prompt was only one line” with a measurement you can act on.