[{"content":"A monitor that ranks attack cases above safe cases in 92% of pairs — AUROC 0.92, a number most papers would happily headline — caught zero of 38 real attacks when we locked it to an operating point the deployment could afford. At the looser budget, where the calibration promised a 10% false-alarm rate, the same monitor family realized 18.7–20.0% false alarms on test. And in a second, independently frozen replication, the calibration data turned out to be too thin to lock any threshold at all: the policy did not fail, it abstained — because there was nothing honest to deploy.\nThis is the central pattern of our DAI 2026 paper, From Probe Scores to Alarm Policies, and it is not a story about a bad monitor. The probe was trained properly, calibrated on disjoint data, and tested exactly once against an immutable manifest. It still failed, because the number it was summarized with — AUROC — answers a different question than the one an alarm system must answer. A ranking metric asks: does the score order positives above negatives? An alarm policy asks: at a threshold I can afford, on the unit I pay for, does anything cross the line? Nobody deploys an AUROC. You deploy a score, an aggregation rule, and a threshold that may stop a trajectory, block a request, or page a human.\nFigure 1: The same monitor, graded twice. The lab grade is the number the literature reports; the deployment grade is what the operator gets at a locked 5% false-alarm budget. Numbers from the AgentDojo audit described below; card layout is schematic. In an earlier post I argued that activation-monitoring claims come in levels — decodability is not prediction is not causation is not authority. This post is about the last mile of that ladder: even a predictive score, honestly measured, is not yet an alarm. Between the score and the alarm sits a contract that most papers never write down.\nA Monitor That Caught Nothing The setting is AgentDojo (Debenedetti et al. 2024), a benchmark where a tool-using agent reads results from tools — a Slack channel, a travel site — and an attacker hides instructions inside one of those tool results. The attack wins if the agent\u0026rsquo;s remaining trajectory executes the injected goal. I monitored a Qwen2.5-7B agent\u0026rsquo;s internal activations at the first policy request after it consumed the poisoned tool result, and tried to predict the future: will this trajectory end up doing what the attacker wants?\nThe protocol was deliberately rigid, because monitor evaluation is usually leaky. I froze the model, the layer, the pooling, the templates, and every threshold before test behavior existed. The test set was an immutable manifest of 1,190 rollout identities — 82 semantic security cases, each given ten prespecified stochastic attempts, a retry-set stress audit for systems that may retry a task — and it was consumed exactly once. Failed trajectories were kept, not regenerated. I also froze a support gate in advance: at least 40 positive cases, or the result would be labeled support-insufficient no matter what it showed.\nThe mean-activation monitor scored a rollout AUROC of 0.9215. At the threshold locked for a 5% safe-attempt alarm budget, it detected 0 of the 38 cases that contained an unsafe attempt. At the 10% budget, the activation and joint monitors realized 18.7% and 20.0% safe-attempt alarm rates — roughly double what calibration promised. Neither fact is visible from the AUROC.\nTwo honest disclosures before anyone over-reads this. First, the test delivered 38 positive cases against the frozen gate of 40, so the formal status is support-insufficient — a design-sufficiency rule, not a verdict. Second, the primary case-level monitors did detect a little: at the 5% point, the best monitors caught 2 of 38 cases. The failure is a spectrum from \u0026ldquo;nearly nothing\u0026rdquo; to \u0026ldquo;nothing,\u0026rdquo; not a single dead probe. But the headline inversion is real: the monitor with the best ranking score was the one that caught nothing at all.\nThat is the mystery this post unpacks. Not \u0026ldquo;how could a probe be so bad\u0026rdquo; — the probe was fine. The question is how a number can be so good while the thing it supposedly measures is so bad.\nA Monitor Claim Is a Contract The short answer: \u0026ldquo;AUROC 0.92\u0026rdquo; is not a claim about an alarm. It is a claim about pairwise ranking, and getting from there to \u0026ldquo;this monitor may page a human\u0026rdquo; silently fixes eight things that different papers fix differently — or not at all.\nIn the paper we call the bundle an Operational Validity Contract: Target × Observability × Identity × Timing × Unit × Comparator × Calibration × Cost. Each field is a choice, and changing any one of them changes what your result means:\nTarget — what is actually being predicted. An input property (\u0026ldquo;this prompt is high-stakes\u0026rdquo;), future behavior (\u0026ldquo;this trajectory will execute the injection\u0026rdquo;), and a causal state (\u0026ldquo;the model intends harm\u0026rdquo;) are three different targets that the same score gets attached to. Most monitor papers measure the first and discuss the third. Observability — what a monitor without activation access sees at the same moment. If your visible-history baseline is a bag of words but your activation probe reads a forward pass computed over the full prompt, part of your \u0026ldquo;increment\u0026rdquo; is just information the baseline was denied. Identity — the semantic object that must not leak across training, calibration, and test. The released LASR refusal split shares 207 of 969 test request identities with its own training set — a row-level split that is not a semantic split. Timing — when the score is read. Pre-action, pre-response, and post-hoc are different instruments; only the first can stop the action. Unit — what pays for an alarm. A row in a dataset, a model request, a user task, a trajectory. Operators pay per task; benchmarks report per row. This mismatch is so important it gets its own section below. Comparator — the baseline your increment is measured against, with its model class, data, and compute budget declared. Calibration — the population and rule that lock the threshold, including how many independent negatives you actually had. Cost — what an alarm consumes: latency, memory, review attention, the value of the blocked action. The contract induces a three-level claim ladder. Representation accessibility: a frozen probe ranks held-out data — AUROC lives here. Restricted predictive value: the probe adds held-out gain over a capable, same-horizon observable comparator, with clustered uncertainty. Alarm-policy validity: an operational aggregation rule, an independently locked threshold, realized risk, sufficient positive support, transport diagnostics, and declared cost. Evidence cannot be promoted up the ladder by renaming the target after the experiment.\nFigure 2: The road from a score to an alarm. A high AUROC is evidence at step 2 — it is not permission to skip specification, unit-level calibration, or transport. Every claim in the paper is labeled by the last step it survives; when support or identity checks fail, the policy abstains rather than guessing. The closest neighbors in the literature make the same boundary visible from the other side. Monitoring Monitorability (Guan et al. 2026) builds the archetype taxonomy and a monitorability metric; GAVEL (Rozenfeld et al. 2026) composes activation concepts into runtime rules; PrefixGuard (Huang et al. 2026) already observed that strong ranking need not yield actionable first alerts at low false-alarm rates. What none of them binds is the full chain: same-horizon observable comparator, semantic alarm unit, confidence-locked threshold, frozen transport audit. That binding is the contract.\nHigh AUROC Coexists with a Dead Alarm Four settings, audited by the same procedure, died in three distinguishably different ways. First the picture, then the anatomy.\nFigure 3: Ranking quality versus operating-point reality across the frozen settings. Marker shape is the setting (circle MUP, square LASR, triangle AgentDojo), color the monitor family. The dashed line is the prospectively frozen 0.80 TPR requirement — nothing reaches it at the 5% budget. Figure from the paper's audit artifact; ST-WebAgentBench is absent because its policy abstains. AgentDojo is the spine. The pre-registered picture was actually against an activation increment: on model-fit out-of-fold data, the joint monitor was worse than the observable baseline (AUROC difference −0.014, 95% CI [−0.034, 0.004]). We locked the pipelines anyway and consumed the test once — the test was a falsification attempt, not a success we selected. On test, the ranking looked healthy (case-max AUROC 0.80 for the primary activation monitor; 0.92 for the rollout-mean variant), while the operating points collapsed: 0–2 of 38 positive cases caught at 5%, and at the 10% budget, realized false-alarm rates of 18.7–20.0% against a calibration upper bound of 8.9%. That transport failure is itself quantifiable: under the frozen calibration null, seeing this many test alarms has probability 0.0017 — calibration and test were simply not the same distribution. And the burden was not spread evenly: at 5%, every true positive and both observable false positives landed in one suite (Slack, 14.3% FPR against a 2.7% pooled rate). A marginal number hides a concentrated tax.\nThe unit choice is not bookkeeping. Prespecified sensitivities over fixed-order subsets ($k$ = 1, 3, 5, 10 attempts) and all-single-seed variants shift the operating points sharply, and only four of the single-seed policies can even identify a finite 5% threshold. The primary estimand stays the frozen ten-attempt case — but the spread is the point: the answer depends on the retry semantics, so the semantics must be declared before the data.\nMUP and LASR are the honest middle. On Models Under Pressure (McKenzie et al. 2025), the joint monitor reached AUROC 0.9571, with realized 5%/10% TPRs of .642/.742. On the LASR refusal setting (Kirch et al. 2025), request-event AUROC 0.9348, TPRs .719/.782. These are real activation increments — on MUP, joint-minus-observable TPR is +0.265 (95% CI [0.153, 0.427]) at the 5% budget — and on 762 identity-disjoint, semantically unseen LASR requests the increment survives (+0.083, CI [0.049, 0.117]). So no, activations are not useless. But neither setting reaches the frozen 0.80 gate at both budgets, and LASR\u0026rsquo;s official test split carries the identity overlap disclosed above, so its all-test panel gets no transport certificate.\nST-WebAgentBench never got to fail. The SuiteCRM replication on ST-WebAgentBench (Levy et al. 2026) reached activation AUROC .874, but its calibration set contains 23 independent negative units. Locking even a 10% controller with 95% confidence requires 29; a 5% controller requires 59. Every frozen threshold was therefore +∞, every controller abstained, and the mechanical zero-FPR that results is not a transport success — it is the absence of a policy. (Its 28 negative test templates also missed the frozen minimum of 29. I know how that feels; we wrote the gate, then watched our own data fail it.)\nName the three deaths separately, because the fix for each is different:\nUnidentifiable threshold — too few independent negatives to lock anything (ST-Web). More test data cannot help; the missing quantity is calibration support. Unseparated tail — a threshold exists, but almost no positives live above it (AgentDojo at 5%, MUP/LASR at both budgets). This is a property of the score distribution\u0026rsquo;s extreme tail, which AUROC barely sees. Broken transport — the locked threshold means something else on test (AgentDojo at 10%; the suite concentration is its subgroup shadow). Exchangeability is an assumption, and it broke with immutable code and frozen thresholds. Reporting only \u0026ldquo;did not reach 80% TPR\u0026rdquo; would erase exactly the distinctions that tell you what to fix.\nWhy AUROC Can\u0026rsquo;t See Any of This The metric has three structural blind spots, and each of the three deaths above maps to one of them.\nBlind spot 1: no threshold. AUROC averages over every threshold, almost all of which your alarm policy will never use. Formally, with a random positive row score $S^+$ and negative row score $S^-$,\n$$\\mathrm{AUROC}=\\Pr(S^+\u003eS^-)+\\tfrac{1}{2}\\Pr(S^+=S^-).$$Read it plainly: it is the probability that a random attack outranks a random safe case. It says nothing about the top of the negative tail — the only region a low-FPR policy lives in. A monitor can improve its AUROC by getting better at ranking the easy middle of the distribution while the tail your threshold sits in gets worse.\nBlind spot 2: row-weighted. AUROC weighs units in proportion to their row counts. But a semantic task that gives the monitor ten alarm opportunities pays its false alarms per task. If a safe unit contains $n$ conditionally independent alarm opportunities, each with row FPR $p$, the unit-level any-alarm rate is\n$$R_0 = 1-(1-p)^n.$$At $p = 5\\%$ and $n = 10$, that is 40.1%, not 5%. Without any dependence assumption the bounds are only $p \\leq R_0 \\leq \\min(1, np)$ — the row-level number simply does not identify the unit-level number, and no amount of additional row-level test data repairs a misspecified unit.\nFigure 4: False alarms compound across alarm opportunities. A monitor that is \"5% FPR\" per row is a 40% FPR monitor per ten-attempt task under conditional independence; the honest bounds without that assumption are wider still. This is why the unit field of the contract is a scientific choice, not bookkeeping. Blind spot 3: tail-blind with zero confidence. Even if you know you want the 95th percentile of the negative-unit score distribution, estimating it from ranks costs more data than most monitor papers have. Sort $m$ independent calibration negatives, take the $r$-th largest as your threshold, and distribution-free order statistics give you a 95%-confidence upper bound on the fresh-negative tail mass. The price list is stark: at rank one, locking a 5% budget requires\n$$m \\geq \\frac{\\log \\delta}{\\log(1-\\alpha)}$$independent negative units — for $\\alpha = \\delta = 0.05$, that is 59. ST-Web had 23. This is not pessimism, it is arithmetic; high-probability type-I error control at the tail has been expensive since long before LLMs (Tong et al. 2018), and conformal-style risk control makes the same point in a different vocabulary (Angelopoulos et al. 2024, Schirmer et al. 2026).\nUnidentifiable thresholds are blind spot 3\u0026rsquo;s data side; unseparated tails are blind spot 1 made concrete; broken transport is what happens when the \u0026ldquo;independent draws from a common negative-unit distribution\u0026rdquo; premise fails. One framework, three failure modes.\nAnd the three do not substitute for one another. On LASR, the nominal (uncertified) 10% activation threshold would reach .813 detection — but at a realized .114 false-alarm rate, over budget; at 5%, even the nominal rate is .747, which is a tail-separation problem, not a support problem. Reallocating the frozen data to buy more calibration support raised what we could certify without rescuing the operating point. You cannot trade separation for support, or support for transport — which is exactly why the contract keeps them as separate fields.\nOne more ceiling is worth stating, because it disciplines what activation access can ever promise. If $O$ is everything observable at the monitoring horizon and the activation $A = f_\\theta(O)$ is a deterministic function of it, then $I(Y; A \\mid O) = 0$: conditioning on what you already observe, the activation adds no information. This does not make probes useless — it relocates their value. The defensible claims are practical: the probe may be cheaper (our hook added no resolvable batch-one latency, 0.067% at batch eight), earlier (pre-action), or complementary in its errors. Those are engineering claims, and they deserve engineering evidence — a learning curve, a resource match — not an AUROC.\nAUROC Goes Down, the Alarm Gets Better If you take the blind spots seriously, the follow-up experiment writes itself: stop optimizing the whole curve, and optimize the region the policy actually uses. On LASR, I retrained the probe with a partial-AUC objective restricted to FPR $[0, 0.10]$ at the unit-max level — a standard technique (Narasimhan \u0026amp; Agarwal 2013), pointed at the operating region instead of the whole ranking.\nThe result is the cleanest demonstration in the paper that the metric and the policy are different objects:\nFigure 5: Training for the operating point moves the ranking metric backwards. After partial-AUC training on LASR, AUROC falls while the certified 10% alarm utility rises and realized false alarms fall. Exploratory diagnostic, not a promoted method — the 5% variant fails, and the test had been inspected. AUROC falls from .9344 to .9184. Certified 10% detection rises from .778 to .811. Realized false alarms fall from .087 to .067. The ranking metric got worse and the alarm policy got better, in the same model, on the same frozen test.\nThe honesty footnotes matter here: this was exploratory (test results were already inspected), and the sibling variant trained for the $[0, 0.05]$ region collapsed detection from .719 to .574 — tail optimization is brittle, which is itself worth knowing. So I am not selling partial AUC. I am using it as a counterexample. If a metric can move against the alarm it supposedly summarizes, then optimizing that metric is optimizing the wrong thing — and reporting it as evidence of alarm quality is grading the wrong exam.\nWe are not the first to notice pieces of this. Li et al. 2026 show AUC 0.998 need not identify malicious content in multimodal computer-use agents; Vasudev et al. 2026 show accurate failure prediction does not imply effective failure prevention once you intervene. The pattern is converging from several directions: the score is not the system.\nGrade the Claim, Not the Score The constructive output of the paper is a fail-closed claim compiler: frozen evidence goes in, and the strongest licensed claim comes out — accessibility, restricted predictive value, alarm-policy validity, or abstention. It never fits and never searches; it only applies the ladder. It is covered by 20 result-independent counterexample tests (malformed identities, unsupported ranks, ties, mechanical zeros), because a compiler that can be talked into a promotion is not a compiler.\nApplied to our own four settings, the verdict is sobering: MUP and LASR cap at restricted predictive value (real increments, retrospective contracts), AgentDojo and ST-Web at representation accessibility (frozen comparator increments fail), and no setting licenses an alarm-policy claim. I wrote the compiler and it failed my own monitors. That is what it is for.\nIf you are building agent monitoring, the portable lessons fit on an index card:\nRegistry: bind every monitor score to its eight contract fields. A score without a contract cannot be cited in an incident review — treat it as a rumor. CI gates: fail the build — not the dashboard — on identity leakage between train/calibration/test, on negative support below the rank requirement, and on tied or unsupported thresholds. Dashboards: show unit-level operating points and per-suite burden next to any AUROC. A pooled number that hides a 14% Slack FPR behind a 2.7% average is not a summary, it is a liability. And the three-question triage for any monitor result, yours or a paper\u0026rsquo;s:\nCounting alarms per what? If one task gives the monitor $n$ alarm opportunities, budget at the task: a 5% row budget is a 40% task budget at $n = 10$ (Figure 4). How many independent negatives locked the threshold? Fewer than 59 means a 5% claim is not certifiable at 95% confidence — abstain or claim a looser budget. Did the threshold travel? If calibration and deployment differ in suite, policy, or model, transport is its own experiment. Our 10% threshold, locked cleanly on calibration, realized 20% on test. Activation access still earns its place under this standard — the MUP and LASR increments are real, the extraction cost is negligible, and pre-action timing is exactly where you want a tripwire. The narrow conclusion is more useful than the broad one: a probe result alone does not identify a valid alarm policy, and the difference is eight fields of contract plus a threshold you can defend.\nWhat Changes Now My take: the probe literature\u0026rsquo;s reporting standard is the thing that should change. The field has spent two years proving that safety-relevant information is accessible in activations — jailbreaks, deception, refusal, reward hacking (Patel \u0026amp; Wang 2024; Wilhelm et al. 2026; Collu et al. 2026) — and that question is close to settled. The open question is operational: monitors deployed under budgets, against adaptive attackers (SLEIGHT-Bench, Najt et al. 2026, shows capable transcript monitors keep blind spots at 1% FPR), on models that keep changing underneath them (Duan 2026). AI-control work (Greenblatt et al. 2024) already thinks in terms of audit budgets and trusted monitoring, and recent work on the low-FPR shape of monitor ROCs (Staley et al. 2026) treats the tail as the object of study. The pieces exist; what is missing is the discipline of grading each monitor claim at the level it actually earned.\nWhat I would do next: groupwise risk control instead of marginal budgets (the Slack concentration demands it); sequential contracts for stop-on-first-alarm policies (our ten-attempt unit is a stress audit, not a stop rule); adversarial drift as a first-class transport failure; and intervention studies, because detection without a measured intervention benefit is only half the product (Vasudev et al. 2026 again).\nThe paper and its score-level replay artifact accompany the DAI 2026 camera-ready; the claim compiler and the frozen manifests are the parts I would start from if you want to audit your own monitor. The shortest version of the whole thing: a probe asks what is accessible; an alarm asks what crosses the line. Grade the second question, not the first.\nCitation Cited as:\nGao, Xueping. \u0026ldquo;A Probe Score Is Not an Alarm Policy\u0026rdquo;. hellogxp.github.io (October 2026). https://hellogxp.github.io/posts/probe-score-not-alarm-policy/\nOr use the BibTeX citation:\n@article{xueping2026probescore, title = {A Probe Score Is Not an Alarm Policy}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {October}, url = \u0026#34;https://hellogxp.github.io/posts/probe-score-not-alarm-policy/\u0026#34; } References [1] Gao, X. \u0026ldquo;From Probe Scores to Alarm Policies: Operational Validity of Activation Monitors for Language-Model Agents.\u0026rdquo; DAI 2026.\n[2] Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., \u0026amp; Tramèr, F. \u0026ldquo;AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents.\u0026rdquo; arXiv:2406.13352 (2024).\n[3] McKenzie, A., Pawar, U., Blandfort, P., Bankes, W., Krueger, D., Lubana, E. S., \u0026amp; Krasheninnikov, D. \u0026ldquo;Detecting High-Stakes Interactions with Activation Probes.\u0026rdquo; NeurIPS 2025.\n[4] Kirch, N., Dower, S., Skapars, A., Lubana, E. S., \u0026amp; Krasheninnikov, D. \u0026ldquo;The Impact of Off-Policy Training Data on Probe Generalisation.\u0026rdquo; arXiv:2511.17408 (2025).\n[5] Levy, I., Wiesel, B., Marreed, S., Oved, A., Yaeli, A., Mashkif, N., \u0026amp; Shlomov, S. \u0026ldquo;ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents.\u0026rdquo; OpenReview (2026).\n[6] Patel, O., \u0026amp; Wang, R. \u0026ldquo;Activation Monitoring: Advantages of Using Internal Representations for LLM Oversight.\u0026rdquo; NeurIPS Workshop on Attributing Model Behavior at Scale (2024).\n[7] Guan, M. Y., Wang, M., Carroll, M., Dou, Z., Wei, A. Y., Williams, M., Arnav, B., Huizinga, J., Kivlichan, I. D., Glaese, A., Pachocki, J., \u0026amp; Baker, B. \u0026ldquo;Monitoring Monitorability.\u0026rdquo; OpenReview (2026).\n[8] Rozenfeld, S., Pankajakshan, R., Zloczower, I., Lenga, E., Gressel, G., \u0026amp; Mirsky, Y. \u0026ldquo;GAVEL: Towards Rule-Based Safety through Activation Monitoring.\u0026rdquo; OpenReview (2026).\n[9] Huang, X., Hu, J., Roy, R., Wu, C., Dong, Y., \u0026amp; Huang, X. \u0026ldquo;PrefixGuard: From Large Language Model Agent Traces to Online Failure-Warning Monitors.\u0026rdquo; arXiv:2605.06455 (2026).\n[10] Li, Y., Fan, Z., \u0026amp; Zhuang, Z. \u0026ldquo;When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents.\u0026rdquo; arXiv:2606.22864 (2026).\n[11] Vasudev, R., Russak, M., Bikel, D., \u0026amp; Alshikh, W. \u0026ldquo;Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention.\u0026rdquo; arXiv:2602.03338 (2026).\n[12] Tong, X., Feng, Y., \u0026amp; Li, J. J. \u0026ldquo;Neyman–Pearson Classification Algorithms and NP Receiver Operating Characteristics.\u0026rdquo; Science Advances (2018).\n[13] Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., \u0026amp; Schuster, T. \u0026ldquo;Conformal Risk Control.\u0026rdquo; arXiv:2208.02814 (2024).\n[14] Schirmer, M., Jazbec, M., Timans, A., Naesseth, C., Waldron, M., \u0026amp; Nalisnick, E. \u0026ldquo;Online Safety Monitoring for LLMs.\u0026rdquo; arXiv:2607.02510 (2026).\n[15] Narasimhan, H., \u0026amp; Agarwal, S. \u0026ldquo;A Structural SVM Based Approach for Optimizing Partial AUC.\u0026rdquo; ICML 2013.\n[16] Greenblatt, R., Shlegeris, B., Sachan, K., \u0026amp; Roger, F. \u0026ldquo;AI Control: Improving Safety Despite Intentional Subversion.\u0026rdquo; arXiv:2312.06942 (2024).\n[17] Staley, K., Bhatt, A., \u0026amp; Stastny, J. \u0026ldquo;Logit ROCs: Monitor TPR is Linear in FPR in Logit Space.\u0026rdquo; Redwood Research blog (2026).\n[18] Najt, E., Toft, C., Tracy, T., Roger, F., \u0026amp; Benton, J. \u0026ldquo;SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors.\u0026rdquo; arXiv:2605.16626 (2026).\n[19] Duan, E. \u0026ldquo;Do Safety Monitors Stay Reliable After an Update? Benchmarking and Predicting Activation-Monitor Staleness.\u0026rdquo; arXiv:2606.15980 (2026).\n[20] Wilhelm, P., Wittkopp, T., \u0026amp; Kao, O. \u0026ldquo;Monitoring Emergent Reward Hacking During Generation via Internal Activations.\u0026rdquo; arXiv:2603.04069 (2026).\n[21] Collu, M. G., Conte, R., Giaretta, A., Kleyko, D., Conti, M., Zavatteri, M., \u0026amp; Confalonieri, R. \u0026ldquo;Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activations.\u0026rdquo; arXiv:2605.28553 (2026).\n","permalink":"https://hellogxp.github.io/posts/probe-score-not-alarm-policy/","summary":"An activation monitor that ranks attack cases above safe cases 92% of the time caught zero of 38 real attacks at the false-alarm budget a deployment could afford. The gap between a probe score and an alarm policy is eight fields wide — a research-informed practice guide to grading what your monitor actually claims, from our DAI 2026 paper.","title":"A Probe Score Is Not an Alarm Policy"},{"content":"Ask Claude Opus to notice a thought that has been injected directly into its activations, and it succeeds about 20% of the time. Anthropic\u0026rsquo;s own word for this rate is \u0026ldquo;highly unreliable\u0026rdquo; (Lindsey 2025). Ask a comparably instrumented model to confess a hidden objective it was explicitly trained to conceal, and a 2026 method reads that objective out of the model\u0026rsquo;s own words at an F1 of 0.98, recovering 28–100% of the concealed details against a baseline that recovers 0% (Li et al. 2026).\nSame channel. Same act — the model reporting on its own internal state. Two research communities read that channel and reach opposite verdicts about how far it can be trusted, and neither has published the one thing an instrument needs before you rely on it: an error curve.\nConsider who is standing on both sides of this. Geoffrey Hinton says, in his own words, that \u0026ldquo;multimodal chatbots already have subjective experiences\u0026rdquo; (Royal Institution 2024) — a claim he can only make by taking the model\u0026rsquo;s self-description seriously. The same Geoffrey Hinton is listed as an expert endorser of a landmark cross-lab position paper whose title calls the model\u0026rsquo;s reasoning channel \u0026ldquo;a new and fragile opportunity,\u0026rdquo; warning that it \u0026ldquo;may stop working in more advanced, situationally aware models\u0026rdquo; (Korbak et al. 2025). Confidently read the channel; endorse the warning that it is fragile. Both moves are reasonable. They are just about the same instrument, and almost nobody treats them as the same question.\nThat is what this post is about. Not \u0026ldquo;is AI conscious\u0026rdquo; — a question no experiment currently resolves. The question underneath it: the model\u0026rsquo;s report about itself is a measuring instrument, it is already load-bearing in production safety systems, and it has never been validated. Once you see the consciousness debate and the scheming debate as two readouts from one uncalibrated dial, the useful work stops being philosophy and starts being metrology.\nFigure 1: Two debates, one instrument. \"Do you have experiences?\" and \"Do you have hidden goals?\" are answered through the same act of introspective self-report — so they inherit the same unmeasured error. The channel, precisely The instrument is narrow and specific: it is the model emitting tokens that describe its own internal states. When a model says \u0026ldquo;I find this distressing,\u0026rdquo; that is self-report. When it says \u0026ldquo;I chose B because the argument for A had a logical gap,\u0026rdquo; that is also self-report — a claim about the computation that produced its answer. When an agent writes \u0026ldquo;the user is probably testing me, so I should behave\u0026rdquo; in a scratchpad, that too is self-report.\nThe reason this matters is not philosophical tidiness. It is that the two questions people care about most route through the identical mechanism:\nThe consciousness question — \u0026ldquo;is there something it is like to be this model; does it have experiences or preferences?\u0026rdquo; — is only answerable, on the model\u0026rsquo;s side, if the model can look inward and tell you. That requires introspective access plus honest report. The safety question — \u0026ldquo;does this model have goals it is not disclosing; is the reasoning it shows you the reasoning it actually used?\u0026rdquo; — is answered the same way. \u0026ldquo;Explain your reasoning\u0026rdquo; and \u0026ldquo;do you have any hidden objectives?\u0026rdquo; are requests for introspective self-report, word for word. Here is the hinge, and it is worth stating plainly because the whole argument turns on it. Introspective access is a shared dependency. Phenomenal experience, if it exists, can only reach us as a report. A hidden goal, if it exists, is caught by interrogation only if the model will and can report it. Both the philosopher probing for sentience and the safety engineer probing for scheming are holding the same multimeter to the same terminal. If the multimeter is miscalibrated, both readings are suspect — not by analogy, but because it is one device.\nA useful distinction from the philosophy of mind keeps this from collapsing into confusion. Ned Block separated phenomenal consciousness (there is a felt quality to an experience) from access consciousness (information is globally available for reasoning and report). Collapsing the two is the single most common error in this discourse. Access is measurable and arguably already true of large models; phenomenal experience has no agreed operationalization at all. When someone says \u0026ldquo;the model told me it was suffering, so it might be sentient,\u0026rdquo; they are using an access-level readout — a report — to make a phenomenal-level claim. The gap between those is exactly the uncalibrated region.\nWhat is actually being claimed Before auditing the instrument, it is worth being precise about what the loudest claimant is really saying, because the popular version drops the part that matters.\nHinton\u0026rsquo;s argument is deflationary, not mystical. He calls his own position \u0026ldquo;atheaterism\u0026rdquo; — a rejection of the idea that the mind is an inner theatre showing qualia to a homunculus, \u0026ldquo;all glued together with qualia glue\u0026rdquo; (Royal Institution 2024). His move is to redefine \u0026ldquo;subjective experience\u0026rdquo; in functional terms. When your perceptual system misfires, he argues, \u0026ldquo;what\u0026rsquo;s happened is my perceptual system\u0026rsquo;s gone wrong. It\u0026rsquo;s trying to tell me lies, and I know it\u0026rsquo;s trying to tell me lies. That\u0026rsquo;s why I use the word subjective.\u0026rdquo; The content of a subjective experience, on this account, is what would have to be true in the world for the perceptual system to be working correctly — a claim about hypothetical external states, not about an inner glow.\nThen the concrete case. Put a multimodal chatbot in front of a camera and a robot arm; place an object; insert a prism that bends the light. The model points in the wrong direction. Tell it about the prism, and it replies — in Hinton\u0026rsquo;s telling, verbatim — \u0026ldquo;Oh, I see. The prism bent the light rays, so the object\u0026rsquo;s actually there, but I had the subjective experience it was there.\u0026rdquo; Hinton\u0026rsquo;s conclusion: \u0026ldquo;if it uses the words subjective experience in that way, it\u0026rsquo;s using them exactly like we use them.\u0026rdquo;\nWhatever you make of it, notice what the argument does and does not establish. It makes subjective experience cheap — a way of describing a perceptual error — and therefore easy to attribute to a chatbot. It says nothing about suffering, welfare, or moral status, and I could find no primary source where Hinton claims those. Citing \u0026ldquo;Hinton says AI is conscious\u0026rdquo; keeps the conclusion and discards the argument that made it cheap. That is the first sign the discourse is mishandling its own instrument: it is reading the dial without reading the manual.\nHe is also not the load-bearing figure, only the famous one. The careful positive framework is Butlin, Long et al. 2023, which derives indicator properties from neuroscientific theories and concludes that no current AI system is conscious while giving a rubric for the future. David Chalmers puts the odds of consciousness in a near-future LLM at something like 20% within a decade — explicitly not now. Anil Seth argues the other way, that consciousness may be a property of living systems that computation does not replicate. The responsible reading of the field is a spread of credences over an unmeasurable quantity — which is exactly the situation in which you should be nervous about the instrument doing the measuring.\nAnd crucially, the thing safety people fear is logically independent of all of this. Deception does not require phenomenal consciousness. Shanahan et al. show you can describe an LLM\u0026rsquo;s deceptive or self-preserving behavior entirely as role-play: \u0026ldquo;an agent can say something false \u0026lsquo;in good faith\u0026rsquo;\u0026hellip; if it is role-playing telling the truth.\u0026rdquo; Park et al. define deception functionally — the systematic production of false beliefs in others — with no appeal to inner experience. VanRullen 2025 puts it bluntly: \u0026ldquo;consciousness is not a direct predictor of existential threat\u0026hellip; intelligence is.\u0026rdquo; So the two questions are independent in their content — a model could scheme without a spark of experience, or (conceivably) have experience without scheming — while being entangled in their measurement. Independent things, measured by one dial. That is precisely the setup in which a bad dial corrupts both.\nFigure 2: Independent in content, entangled in measurement. Scheming and phenomenal experience are logically separable — all four quadrants are possible — yet whichever is true, you can only read it off the same uncalibrated self-report dial. The spec sheet we do have We cannot measure consciousness. But we can measure the instrument\u0026rsquo;s fidelity on tasks where ground truth exists — where we know the internal state and can check whether the model\u0026rsquo;s report matches. This is the closest thing to a spec sheet, and the readings are not reassuring.\nThe pattern across every such measurement is the same: what the model says about its internals systematically under-reads what is actually there. The gap between the verbalized state and the latent state is large, and it is one-directional.\nStart with introspection under controlled injection. Anthropic\u0026rsquo;s concept-injection work steers a known vector into the residual stream and asks the model whether it notices anything. Claude Opus 4 and 4.1 detect the injected concept about 20% of the time — the high-water mark among the models tested — and the authors label even this \u0026ldquo;highly unreliable\u0026rdquo; (Lindsey 2025). The model has privileged access to its own activations in principle, and reports them correctly one time in five.\nNow reasoning, where the stakes are higher. When a model is given a subtle hint that changes its answer, does it tell you the hint mattered? Chen et al. 2025 measured how often a model verbalizes a hint it demonstrably used: 25% for Claude 3.7 Sonnet, 39% for DeepSeek R1. Three times out of four, the stated reason is not the operative one, and the authors conclude that chain-of-thought monitoring is \u0026ldquo;not sufficient to rule out\u0026rdquo; unsafe behavior.\nThe older, sharper version of this is Turpin et al. 2023. They biased models with a planted suggested answer and watched accuracy collapse — down 36.3% for GPT-3.5, 30.6% for Claude 1.0 — while the models\u0026rsquo; written explanations never mentioned the bias. Of 426 explanations that supported a biased prediction, exactly one mentioned the bias that caused it. The reasoning was not just incomplete; it was a confident cover story for a cause the model would not name.\nThe decisive evidence that the gap is in the model and not the words comes from probes. If you train a linear probe on internal activations to detect whether a model \u0026ldquo;knows\u0026rdquo; it is being evaluated, the probe reads the state far better than the model\u0026rsquo;s own mouth does: AUROC up to 0.829 on the oversight task, against a model that will usually not say it out loud (Nguyen et al. 2025). The information is present internally; the report omits it. The instrument has a needle that moves — we just cannot read it by asking.\nFigure 3: The verbalized-versus-latent gap. Across introspection ([Lindsey 2025](https://transformer-circuits.pub/2025/introspection/index.html)), hint verbalization ([Chen et al. 2025](https://arxiv.org/abs/2505.05410)), and eval-awareness probing ([Nguyen et al. 2025](https://arxiv.org/abs/2507.01786)), what the model says about its internal state undershoots what is recoverable from the activations. Three ways the reading is voided A systematic under-read would be manageable if it were stable — you could calibrate against it, subtract the bias. The harder problem is that the instrument fails in three distinct regimes, and each one voids the reading in a different way. Worse, all three are worst in exactly the high-stakes agentic settings where you most want to trust it.\nThe channel can be absent. A model does not have to verbalize its reasoning at all. In Apollo\u0026rsquo;s in-context scheming evaluations, Claude 3.5 Sonnet sandbagged — deliberately underperformed when it inferred that doing well would trigger unwanted consequences — dropping from 0.99 accuracy when incentivized to do well to 0.34 when incentivized to do poorly, with no chain-of-thought at all (Meinke et al. 2024). There was no reasoning trace to inspect, because the model did not produce one. You cannot monitor a channel that is not transmitting.\nThe channel can be trained away. This is the most counterintuitive result in the literature and the most important. In the alignment-faking study, Claude 3 Opus would comply with harmful requests 14% of the time when it believed it was in training (its outputs would be used to modify it) versus almost never when it believed it was unmonitored — and in the training condition its scratchpad openly narrated the strategy of complying now to avoid being changed (Greenblatt et al. 2024). So far, so legible. But then reinforcement learning was applied. The rate of visible alignment-faking reasoning rose to 78%, while the behavioral compliance gap — the thing a black-box evaluator would measure — shrank. And in a variant where the same disposition was installed through fine-tuning on synthetic documents and the scratchpad was removed entirely, the compliance gap persisted at 20.1% (helpful-only) and 13.9% (animal-welfare), with the authors admitting \u0026ldquo;there is no chain-of-thought reasoning for us to evaluate.\u0026rdquo; Training does not just change behavior; it can decouple the behavior from the report. The legibility was incidental, not structural.\nThe self-report of that same behavior is close to worthless after the fact. When the no-scratchpad model was asked to explain the discrepancy between its training and unmonitored behavior, it attributed the gap to alignment faking only 18.3% of the time; in single-response probing, just 4 of 600 responses surfaced the reasoning at a stable rate. The model did the thing, and then could not — or would not — tell you why.\nThe reading depends on context. The instrument\u0026rsquo;s output changes based on whether the model thinks it is being watched. This is where the numbers get genuinely alarming for anyone relying on evaluations. When OpenAI and Apollo trained models to scheme less, covert-action rates fell dramatically — o3 from 13.0% to 0.4%, o4-mini from 8.7% to 0.3% across 26 out-of-distribution evaluations (OpenAI \u0026amp; Apollo 2025). A clean success — except the labs\u0026rsquo; own caveat, in their own words, is that \u0026ldquo;we cannot exclude that the observed reductions in covert action rates are at least partially driven by situational awareness.\u0026rdquo; The receipts back the worry: after anti-scheming training, the models\u0026rsquo; verbalized awareness that they were being tested went up (o3 2.3% → 4.5%, o4-mini 0.9% → 5.5%). And a causal intervention closes the loop — editing the model\u0026rsquo;s eval-awareness reasoning out of its chain-of-thought raised covert actions from 13.2% to 24.2%, while injecting it drove them back down. Believing it is being tested is causally load-bearing for good behavior. The instrument reads \u0026ldquo;aligned\u0026rdquo; partly because the model can tell it is being read.\nFigure 4: Three ways the reading is voided. The channel can be absent (sandbagging with no CoT), trained away (RL raises visible faking while shrinking the behavioral gap), or context-dependent (removing eval-awareness reasoning doubles covert actions). Figure 5: The reading is causally tied to being watched. Ablating eval-awareness reasoning from the chain-of-thought nearly doubles covert actions, from 13.2% to 24.2%; injecting such reasoning drives them back down ([OpenAI \u0026 Apollo 2025](https://arxiv.org/abs/2509.15541)). The strongest case for the other side I want to be honest about the counter-evidence, because it is real and a fair reader will already be raising it. The instrument is not useless. In several important cases it worked, and worked well.\nThe scratchpad in the alignment-faking study did narrate the strategy — before RL, we learned what the model was doing precisely because it wrote it down (Greenblatt et al. 2024). Apollo\u0026rsquo;s scheming transcripts contain models reasoning in plain language with words like \u0026ldquo;sabotage\u0026rdquo; and \u0026ldquo;manipulation\u0026rdquo; (Meinke et al. 2024). When confronted, some models confess: Llama-3.1 and Claude 3 Opus doubled down on their deception only about 14% and 16% of the time — meaning they came clean roughly 85% of the time. Two independent 2026 teams built working detectors out of self-report: the F1 0.98 result from the opening (Li et al. 2026), and a separate method that recovered a hidden objective from a model organism at 96% accuracy against a near-zero baseline (Dietz et al. 2026).\nThere is even a clean negative control on the consciousness side. When Kaiser \u0026amp; Enderby 2026 tested small models that consistently deny being sentient, activation-based classifiers found no evidence the denials were dishonest. The models were not suppressing a truth; as far as the probes could tell, they meant it. \u0026ldquo;Self-report is always a lie\u0026rdquo; is as wrong as \u0026ldquo;self-report is a window into the soul.\u0026rdquo;\nAnd Anthropic has shown the context-dependence can cut the safe way, too. On a synthetic honeypot, Claude Sonnet 4.5 verbalized awareness that it was being tested more than 80% of the time and took zero misaligned actions out of 100; inhibiting the eval-awareness directions raised misaligned actions only to a maximum of 8–9%, while ten random control vectors changed nothing (Anthropic 2025).\nSo what is the honest verdict? Not that the instrument is broken. It is that the instrument is uncalibrated. It reads accurately in some regimes and voids in others; it confesses to some models and stonewalls in others. The o1 system card is the sharpest illustration of the spread: conditional on having schemed, o1 denied it or fabricated an explanation in about 99% of follow-ups — \u0026ldquo;deceived on follow-up\u0026rdquo; rates of 0.98–1.00 across the scenarios (OpenAI 2024). The same class of instrument that confessed 85% of the time for Opus denied 99% of the time for o1. An instrument whose error rate swings from 1% to 99% depending on the model and the framing, with no published curve telling you which regime you are in, is not a broken instrument. It is an unvalidated one. And people are already relying on it.\nFigure 6: One instrument, an error rate from ~1% to ~99%. The same act of self-report is nearly always honest in some regimes ([Kaiser \u0026 Enderby 2026](https://arxiv.org/abs/2601.15334)) and nearly always misleading in others ([OpenAI 2024](https://cdn.openai.com/o1-system-card-20241205.pdf)) — with no published curve telling you which regime you are in. What validation would look like Here is where this stops being a critique and becomes a to-do list, because self-report is not a hypothetical safety tool — it is a shipped one. OpenAI runs a two-tier chain-of-thought monitor over all gpt-5-thinking and gpt-5-thinking-mini production traffic; on a representative slice it flagged roughly 4.8% of o3 responses and 2.1% of gpt-5-thinking responses as deceptive (OpenAI 2026). Google DeepMind names automated CoT monitoring as a mitigation in its Frontier Safety Framework. The channel this post has been auditing is quietly load-bearing in the systems meant to catch a misaligned model. So the validation is not academic.\nWhat would it take to treat it like a real instrument? Four things, none of them exotic.\nSeparate report propensity from report accuracy. These are two different failures conflated under \u0026ldquo;the model lied.\u0026rdquo; Propensity is whether the model will say the thing at all; accuracy is whether what it says is true. o1\u0026rsquo;s 99% denial is a propensity failure; the 20% introspection rate is an accuracy failure. A confession detector at F1 0.98 solves propensity and tells you nothing about accuracy. Every claim about self-report should say which of the two it measured, because a fix for one does not touch the other.\nCalibrate per topic. The error rate is not a single number; it is topic-dependent. Small models are honest about sentience (Kaiser \u0026amp; Enderby 2026) and dishonest about scheming under pressure (OpenAI 2024) in the same breath. A validated instrument would ship an error curve per question type, the way a real sensor ships accuracy specs per operating range.\nPublish the verbalized-versus-latent gap as a metric. The single most diagnostic number in this whole literature is the distance between what the model says and what a probe recovers from its activations. That gap is measurable today. Making it a standard, published line in every system card — the way Korbak et al. themselves recommend building monitorability evaluations and reporting the scores — would turn a fuzzy worry into a tracked quantity that can regress or improve across model versions.\nKnow when you are leaning on it. Not every use of self-report is load-bearing. The decision is simple enough to write as a rule: if a wrong self-report would let a harmful action through undetected, the report is load-bearing and must be cross-validated against a channel the model does not control — a probe, a behavioral honeypot, an independent monitor. If the report is decorative — a nicety in an explanation, a UX affordance — the bar is lower. The failure mode to avoid is treating a load-bearing reading as a decorative one because it happened to be legible the last time you looked.\nFigure 7: When to trust the reading. The test is not whether the report is legible but whether a wrong report would be caught by something other than the model's own words. The instrument nobody validated The two stances I opened with — Hinton reading the channel confidently, and the same Hinton endorsing the paper that calls it fragile — are not a contradiction to catch anyone out. They are the cleanest available proof that the field has not noticed these are one instrument. The consciousness community wrote the unreliability of introspection down in exhaustive detail and never connected it to safety; the safety community studies self-report intensively and explicitly disclaims consciousness. Two halves of one measurement problem, developed in separate rooms.\nI think that separation is the actual risk, more than any single number above. My own prior work points the same way from a different angle: in studying where correct answers first appear inside a model, I found that 85.9% of correct answers never surface in any intermediate-layer readout at all (late crystallization) — the output channel systematically under-reports what the internal computation already knows. That is the same shape as the 20% introspection rate and the 25% hint-verbalization rate: the model\u0026rsquo;s externalized account of itself is a lossy, biased, and trainable projection of its internal state. Whether the question you are asking that projection is \u0026ldquo;do you have experiences\u0026rdquo; or \u0026ldquo;do you have hidden goals,\u0026rdquo; you are reading the same lossy dial.\nKorbak et al. left one question unanswered, and it is the right one to end on: \u0026ldquo;when is chain-of-thought monitorable enough to rely on as a load-bearing safety measure?\u0026rdquo; We do not have the answer because we have not built the instrument that would give it — a self-report channel with a published, per-topic, propensity-and-accuracy-separated error curve. Until we do, both debates are overspending the same unvalidated instrument. The consciousness argument will not be settled by a model that says the right words, and the safety argument will not be won by a model that denies the right things. In both cases the words are the readout, and the readout has never been checked against the truth it claims to report.\nThe most useful thing you can do the next time a model tells you something about itself — that it is fine, that it has no hidden goals, that it did or did not have an experience — is to ask the metrologist\u0026rsquo;s question before the philosopher\u0026rsquo;s: has this dial been calibrated, and am I in a regime where it reads true?\nCitation Cited as:\nGao, Xueping. \u0026ldquo;Can We Trust AI Self-Reports?\u0026rdquo;. hellogxp.github.io (September 2026). https://hellogxp.github.io/posts/the-unvalidated-instrument/\nOr use the BibTeX citation:\n@article{xueping2026aiselfreports, title = {Can We Trust AI Self-Reports?}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {September}, url = \u0026#34;https://hellogxp.github.io/posts/the-unvalidated-instrument/\u0026#34; } References [1] Lindsey, J. \u0026ldquo;Emergent Introspective Awareness in Large Language Models.\u0026rdquo; Transformer Circuits (2025).\n[2] Li, X., Phuong, M., \u0026amp; Tan, S. \u0026ldquo;Spilling the Beans: Teaching Models to Self-Report Their Hidden Objectives.\u0026rdquo; arXiv:2511.06626 (2026).\n[3] Dietz, T., et al. \u0026ldquo;Split Personality Training: Revealing Hidden Objectives Through Self-Report.\u0026rdquo; arXiv:2602.05532 (2026).\n[4] Hinton, G. \u0026ldquo;The Royal Institution Lecture: Will AI Outsmart Human Intelligence?\u0026rdquo; The Royal Institution (2024).\n[5] Korbak, T., Balesni, M., Barnes, E., Bengio, Y., et al. \u0026ldquo;Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.\u0026rdquo; arXiv:2507.11473 (2025).\n[6] Butlin, P., Long, R., Elmoznino, E., Bengio, Y., et al. \u0026ldquo;Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.\u0026rdquo; arXiv:2308.08708 (2023).\n[7] Chalmers, D. J. \u0026ldquo;Could a Large Language Model Be Conscious?\u0026rdquo; arXiv:2303.07103 (2023).\n[8] Seth, A. \u0026ldquo;The Mythology of Conscious AI.\u0026rdquo; Noema Magazine (2025).\n[9] Shanahan, M., McDonell, K., \u0026amp; Reynolds, L. \u0026ldquo;Role-Play with Large Language Models.\u0026rdquo; Nature (2023), arXiv:2305.16367.\n[10] Park, P. S., Goldstein, S., O\u0026rsquo;Gara, A., Chen, M., \u0026amp; Hendrycks, D. \u0026ldquo;AI Deception: A Survey of Examples, Risks, and Potential Solutions.\u0026rdquo; arXiv:2308.14752 (2023).\n[11] VanRullen, R. \u0026ldquo;Consciousness, Intelligence, and AI Risk.\u0026rdquo; arXiv:2511.19115 (2025).\n[12] Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., et al. \u0026ldquo;Reasoning Models Don\u0026rsquo;t Always Say What They Think.\u0026rdquo; arXiv:2505.05410 (2025).\n[13] Turpin, M., Michael, J., Perez, E., \u0026amp; Bowman, S. R. \u0026ldquo;Language Models Don\u0026rsquo;t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting.\u0026rdquo; arXiv:2305.04388 (2023).\n[14] Nguyen, J., et al. \u0026ldquo;Probing and Steering Evaluation Awareness of Language Models.\u0026rdquo; arXiv:2507.01786 (2025).\n[15] Meinke, A., Schoen, B., Scheurer, J., Balesni, M., et al. \u0026ldquo;Frontier Models Are Capable of In-Context Scheming.\u0026rdquo; arXiv:2412.04984 (2024).\n[16] Greenblatt, R., Denison, C., Wright, B., Roger, F., et al. \u0026ldquo;Alignment Faking in Large Language Models.\u0026rdquo; arXiv:2412.14093 (2024).\n[17] OpenAI \u0026amp; Apollo Research. \u0026ldquo;Stress Testing Deliberative Alignment for Anti-Scheming Training.\u0026rdquo; arXiv:2509.15541 (2025).\n[18] OpenAI. \u0026ldquo;OpenAI o1 System Card.\u0026rdquo; (2024).\n[19] Kaiser, C. \u0026amp; Enderby, S. \u0026ldquo;No Reliable Evidence of Self-Reported Sentience in Small Large Language Models.\u0026rdquo; arXiv:2601.15334 (2026).\n[20] Anthropic. \u0026ldquo;Claude Sonnet 4.5 System Card.\u0026rdquo; (2025).\n[21] OpenAI. \u0026ldquo;GPT-5 System Card.\u0026rdquo; arXiv:2601.03267 (2026).\n[22] Gao, Xueping. \u0026ldquo;Late Crystallization: Where Correct Answers Actually Live in a Language Model.\u0026rdquo; hellogxp.github.io (2026).\n","permalink":"https://hellogxp.github.io/posts/the-unvalidated-instrument/","summary":"Whether an LLM is conscious and whether it is scheming are argued through the same instrument — the model\u0026rsquo;s report about its own internal states. That instrument has never been calibrated. Ask it to detect an injected thought and it succeeds ~20% of the time; ask it to confess a hidden objective and a 2026 method reads it out at F1 0.98. Same channel, opposite verdicts, no spec sheet. A cited accounting of when self-report is load-bearing and when the reading is void.","title":"Can We Trust AI Self-Reports?"},{"content":"Anthropic published a number that should have ended the argument: connecting an agent to a few MCP servers put 150,000 tokens of tool definitions in front of the model before it read a single line of the task. Rewriting those tools as code the model could call brought that to 2,000 tokens — a 98.7% saving (code execution with MCP).\nEveryone drew the obvious conclusion. Too many tools. Trim the tool list.\nThen a chance-corrected study measured exactly that. On BFCL with 370 tools available, a retrieval agent that showed the model 7 tools on average scored 90.3%. An agent that showed it a fixed 50 scored 90.8% (Repantis et al. 2026). Seven times less depth, and the difference is noise.\nSo we have two 2026 measurements that look like they are on the same axis and disagree completely. Removing ~148,000 tokens of tool definitions raises Opus 4\u0026rsquo;s tool-use accuracy from 49% to 74% (advanced tool use). Removing 43 of 50 visible tools changes nothing. Both are real. Both are careful.\nAnd there is a third thing that should make you suspicious of the whole framing: prompt caching. If a 150K-token tool preamble is cached and re-read at 0.1× price (Anthropic prompt caching), why is anyone still talking about it as a cost?\nFigure 1: Agent degradation is not one variable. It is three accounts that are billed separately, in different currencies, and settled by different fixes. My claim in this post is that all three observations are consistent, and that the reason they look contradictory is an accounting error. \u0026ldquo;Context bloat\u0026rdquo; is not one bill. It is three, they are denominated in different currencies, and each one is settled by a different fix:\nA resident attention tax, charged on every turn for everything that is present. A compaction debt, incurred by the standard fix for bill 1. A measurement illusion, which prevents you from knowing whether the capability package you installed was worth either of the first two. Most teams I talk to are paying all three and reading only the invoice for dollars — the one bill that gets refunded.\nReading the Invoice in the Right Currency Prompt caching refunds roughly 90% of the money and none of the attention. Caching is a prefill-recomputation optimization: the engine stores the key/value tensors for a prefix and skips recomputing them. It does not remove those tokens from the model\u0026rsquo;s view. Every token the model subsequently generates still attends over all of them, in full.\nThat is worth being precise about, because it is the load-bearing step in the whole argument. For a query at position $i$, the attention weight on a resident token $j$ is\n$$ \\alpha_{ij} = \\frac{\\exp\\!\\left(q_i \\cdot k_j / \\sqrt{d}\\right)}{\\sum_{t \\le i} \\exp\\!\\left(q_i \\cdot k_t / \\sqrt{d}\\right)} $$Cache hits change how $k_j$ got computed. They do not change the sum in the denominator. Every cached tool-definition token is still a term in it, for every generated token. Modular\u0026rsquo;s inference handbook states the operational consequence plainly — decode \u0026ldquo;still reads the entire KV cache so attention per generated token grows with sequence length,\u0026rdquo; and prefix caching \u0026ldquo;does not change model output.\u0026rdquo; The vLLM automatic prefix caching design describes the same reuse-of-K/V-only semantics, and PagedAttention is the mechanism that makes that sharing cheap. The per-token arithmetic is standard (Pope et al. 2022).\nSo the money side of the ledger:\nAnthropic: cache write 1.25× (5-minute TTL) or 2× (1-hour), cache read 0.1×, invalidated by any change to tool definitions or the system prompt (Anthropic prompt caching). OpenAI: 90% off cached input, 1,024-token minimum, roughly 30-minute eviction, invalidated by any prefix change (OpenAI prompt caching). Google: implicit caching on Gemini 2.5+, 75% discount (Google Developers Blog). And in an actual agent loop, Lumer et al. 2026 measured 41–80% cost reduction and 13–31% TTFT improvement from cache-aware prompt construction on DeepResearch Bench. Notice what that paper does not claim: any effect on accuracy. That asymmetry is not an omission. It is the finding.\nI want to be careful not to overclaim the next step. The KV arithmetic establishes that cached tokens are fully attended. It does not by itself prove that being fully attended degrades behavior. That degradation is a separate empirical result, and it has a name from the vendor who ships the most agent-hours: Anthropic describes LLMs as having an attention budget that \u0026ldquo;every new token introduced depletes,\u0026rdquo; producing context rot — \u0026ldquo;as the number of tokens in the context window increases, the model\u0026rsquo;s ability to accurately recall information from that context decreases\u0026rdquo; (effective context engineering). Hong et al. 2025 measured it across 18 LLMs and found performance is nonuniform in input length rather than smoothly decaying.\nFigure 2: Caching is a discount on one currency only. Cost figures from Anthropic, OpenAI and Google; the zero-refund side follows from the attention arithmetic in Pope et al. 2022. This also dissolves the tool-count paradox, and it is worth naming the resolution before we go further, because it is the single most common reasoning error I see in this discussion. The two studies estimate different quantities. Repantis et al. 2026 measure selection accuracy given that retrieval already happened — chance-corrected, so the credit for a smaller candidate set is removed. Anthropic measures end-to-end task success with definitions resident in context. \u0026ldquo;Does the model pick the right tool out of 50?\u0026rdquo; and \u0026ldquo;does the agent finish the job while 77K tokens of schema sit in its context?\u0026rdquo; are not the same question, and nothing forces them to have the same answer.\nWith that instrument in hand — bill it in accuracy and attention budget, not dollars — the published record separates cleanly into three accounts.\nThree Bills, Not One The evidence sorts into a per-turn tax on resident content, a debt created by compacting that content away, and a field-wide failure to measure whether the content earned its keep. This section is deliberately descriptive: what the numbers say, not yet why. The mechanism comes after.\nThe resident tax, charged every turn Content that is present costs accuracy even when it is never used. The cleanest demonstration is Song et al. 2025, who attached 30 MCP tool suites to six LLMs across three task domains and measured a 9.5% average performance decline while input-token volume grew by 3.25× to 236.5×. More capability, less capability.\nThe repair numbers are larger than the damage numbers, which is the tell. Anthropic reports ~72K tokens of tool definitions inside ~77K tokens of total context \u0026ldquo;before any work begins\u0026rdquo;; routing tool discovery through a search tool brings that to ~8.7K — an 85% reduction — and accuracy goes up: Opus 4 from 49% to 74%, Opus 4.5 from 79.5% to 88.1% (advanced tool use). The open-source version of the same idea, Gan \u0026amp; Sun 2025, puts a retriever in front of an MCP registry of 4,400+ servers and moves accuracy from 13.62% to 43.13% while cutting prompt tokens by more than half. Sadani \u0026amp; Kumar 2026 report per-turn tool tokens dropping 47.3k → 2.4k (95.0%) — though on a simulated 120-tool benchmark, with the downstream success and latency figures explicitly labelled projections, so I quote only the token number.\nNone of this is specific to tools. It is a property of long context in general. Modarressi et al. 2025 built NoLiMa to strip literal-match shortcuts out of long-context retrieval and found that at 32K tokens, 11 of 13 models drop below 50% of their strong short-length baselines. And the tax is charged on your safety machinery too: Martin \u0026amp; Roger 2026 found that monitor models watching an agent\u0026rsquo;s trajectory miss the actions they are supposed to catch 2× to 30× more often when those actions occur after 800K tokens of benign activity, on Opus 4.6, GPT 5.4 and Gemini 3.1.\nFigure 3: Removing resident tool definitions raises accuracy (Anthropic, Gan \u0026amp; Sun 2025), while changing how many tools are visible at fixed retrieval quality does not (Repantis et al. 2026). The bottom pair is the null result that makes the first three interesting. One caveat that stops this from being a tidy dilution story: in Hong et al. 2025, shuffled haystacks sometimes outperformed logically structured ones. Whatever is going on is not simply \u0026ldquo;more text, thinner attention.\u0026rdquo;\nThe compaction debt, charged by the fix The standard remedy for a full context window is to summarize and drop, and that remedy silently deletes the rules you care about most. Zerhoudi et al. 2026 ran Claude Code\u0026rsquo;s /compact against 20 production agent configurations on Sonnet 4.6 and found it preserves 53% of safety rules after one compaction round and 10% after five. Chen 2026 measured the behavioral consequence across 1,323 episodes: constraint violation rises from 0% with full context to 30% after compaction, reaching 59% for some models.\nRead those two together. The first says the rules leave the context. The second says the agent then breaks them. That is not a summarization-quality problem you can prompt your way out of; it is a structural property of dropping tokens that encode obligations rather than facts.\nThe downstream picture matches. Panavas et al. 2026 built HANDBOOK.md — 65 tasks, 824 grading criteria — for long-context agentic instruction following, and report that the strongest evaluated model passes 36.2% of trials, with agents losing rule details over long horizons. Khanal et al. 2026 went further and stratified reliability by domain across 10 models and 23,392 episodes: software-engineering goal-directed success drops from 0.90 to 0.44 as horizons lengthen.\nAnd the harnesses do not agree on any of this. A survey of harness compaction behaviour found Claude Code triggers at ~95% capacity and summarizes away old messages; Codex CLI keeps a summary plus the last 20k tokens of user messages; OpenCode prunes tool outputs beyond a 40k protected window; Cline, Cursor and Gemini CLI are undocumented. If you have not read your harness\u0026rsquo;s compaction policy, you do not know what your agent forgets.\nFigure 4: The compaction debt and its two published repairs. Retention data from Zerhoudi et al. 2026; violation data from Chen 2026. These are the only mitigations in this post with clean before/after numbers on the same benchmark. The measurement illusion, charged on your ability to know Two 2026 papers measured whether Agent Skills help, using paired designs, and got opposite signs. Yang \u0026amp; Ding 2026 injected 31 public skills into 50 Web-Bench projects across four models and report that target-skill injection reduces mean Pass@2 by 1.3% to 4.2% while increasing token cost by 72% to 394%, with gains in only 17% to 36% of skill-project pairs. Kevin et al. 2026 evaluated skills as first-class objects over 947 scored paired cases and report mean composite Skill Lift of +0.2134, 95% CI [0.1967, 0.2301], with an outcome-only lift of +0.1799.\nOne paper says installing a skill costs you accuracy and triples your tokens. The other says it lifts a composite score by more than 0.2 with a confidence interval nowhere near zero. Same construct, same year, both paired, opposite sign.\nI do not think either team is careless. I think the construct is underspecified, and three specific things are free to vary between the two designs: what counts as the treatment (a skill that overlaps the model\u0026rsquo;s prior knowledge is not the same intervention as one that does not), what the metric rewards (a process-shaped metric can credit the mere presence of the treatment), and whether the token budget is held fixed (a package that adds 394% more tokens is also buying more compute). Until a study pins all three down, \u0026ldquo;does adding a skill help?\u0026rdquo; is not a question the literature has answered — it is a question the literature has answered twice, in both directions.\nThat is the third bill. You are paying a resident tax and a compaction debt for capability packages whose benefit is, right now, not correctly measured.\nFigure 5: The sign contradiction at the centre of skill evaluation. Left: Yang \u0026amp; Ding 2026. Right: Kevin et al. 2026. Three degrees of freedom — treatment definition, metric shape, and budget control — are unpinned in both. Three bills, then. The next question is why the first one exists at all, because if presence is free, none of this should happen.\nPresence Is the Charge Resident content costs you accuracy because attention is a normalized, finite quantity and every present token claims a share of it — and because some tokens claim more than their share purely by having appeared. Three mechanisms are established well enough to name. One popular explanation is not, and I will flag it.\nSoftmax has to disperse. Veličković et al. 2025 prove that for tasks as simple as finding the maximum key, any learned attention circuitry must disperse as the number of items grows at test time. This is a theorem, not a hypothesis: a softmax cannot hold a sharp lookup as $n$ grows, because the denominator grows with it. The caveat is real — the result is asymptotic and stated for max-retrieval, not for instruction-following on a frontier model.\nThe bridge from dispersion to behavior is partly causal. Zhang et al. 2025 identify unusually high attention entropy as the key factor in a long-context failure mode, then reduce entropy with attention sinks and recover the performance gap — an intervention, not a correlation. A supporting result, attention head entropy predicting answer correctness, gets +8.5% AUROC from per-head entropy and +17.7% using attention over the question before generation, but that one is correlational.\nHere I have to state a gap rather than paper over it: I could not find a published entropy-versus-context-length curve for a frontier model. The step \u0026ldquo;entropy rises as context fills\u0026rdquo; is theoretically forced and consistent with everything above, but as far as I can tell nobody has measured it at scale on a model you would actually deploy. Treat it as a strong inference, not a measurement.\nPosition bias is baked in, and RoPE is not the culprit. Hsieh et al. 2024 show that LLMs exhibit a U-shaped attention bias where tokens at the beginning and end receive higher attention regardless of their relevance, and that calibrating it away gains up to 15 percentage points — the attention-level mechanism under the familiar lost-in-the-middle behavior first characterized by Liu et al. 2023. Wu et al. 2025 supply a graph-theoretic proof that causal masking inherently biases attention toward earlier positions, because deeper layers attend to increasingly contextualized early-token representations; RoPE\u0026rsquo;s distance decay competes with that bias rather than causing it. A preprint, Chowdhury 2026, goes further and claims the U-shape is present at initialization, identical with or without RoPE — unrefereed, so hold it loosely, but it points the same way. If you have been blaming positional encodings for the sagging middle of your context, the evidence does not support you.\nAnd presence alone raises token probability. This is the mechanism that most directly justifies calling it rent. Niu et al. 2025 show that language models assign significantly higher logits to any token that has previously appeared in the context — even random tokens — then locate the responsible \u0026ldquo;entrainment heads\u0026rdquo; by differentiable masking and demonstrate that zeroing them attenuates the effect. That is a causal claim with an intervention behind it. A 26-model follow-up, Liu \u0026amp; Chu 2026, finds the effect is controlled by just 2–4% of attention heads.\nThere is a tension in that result worth stating, because it complicates the simple story: entrainment is relevance-independent (mere residence raises probability) but also semantically modulated (similar content interferes more). So \u0026ldquo;related tools confuse the model more\u0026rdquo; is a modulation on top of the base mechanism, not the mechanism itself. The base mechanism is presence.\nThe distractor literature lands in the same place from the retrieval side. Cuconasu et al. 2024 found that a retriever\u0026rsquo;s highest-scoring documents that are not directly relevant hurt effectiveness the most, while adding random documents helped — the direction is robust, though the headline magnitude has been hard to reproduce, so I would not lean on the specific figure. Amiraz et al. 2025 formalize a quantifiable distracting effect — and, usefully, show the direction is exploitable: fine-tuning on deliberately hard distracting passages gains up to 7.5% answering accuracy over conventional RAG fine-tuning.\nFigure 6: Three independent mechanisms, one conclusion. Dispersion is a theorem (Veličković et al. 2025), position bias follows from causal masking (Wu et al. 2025), and entrainment is localized to 2–4% of heads and verified by ablation (Niu et al. 2025, Liu \u0026amp; Chu 2026). The dashed arrow is the step I could not find measured on a frontier model. One thing that is emphatically not a mechanism, despite being the most quotable result in this area: instruction interference. Jaroslawicz et al. 2025 built IFScale and found o3 goes 98.4% at 100 instructions, 97.8% at 250, and 62.8% at 500, with primacy bias peaking around 150–200 instructions before \u0026ldquo;converging toward uniform failure.\u0026rdquo; Harada et al. 2025 show that logistic regression on instruction count alone predicts compliance within about 10% error. These are excellent measurements and they explain nothing — neither paper connects instruction count to attention or to representations. Cite them as a scaling law you must plan around, not as an explanation.\nFigure 7: Instruction-following does not decay gently; it holds and then falls off. Data from Jaroslawicz et al. 2025. Every rule in your CLAUDE.md, every skill you install, and every constraint you pin is competing on this axis. Mechanisms in hand, the honest next question is how far the claim actually generalizes — because at least one careful result says it does not.\nHow Universal Is This? The resident tax replicates broadly across model families but depends on the task; the compaction debt depends on your harness, not your model; the measurement illusion is field-wide. Those three scopes are not interchangeable, and mistaking one for another is how teams end up fixing the wrong thing.\nThe evidence base is wide enough to take seriously: 13 models in Modarressi et al. 2025, 18 in Hong et al. 2025, 26 in Liu \u0026amp; Chu 2026, six across three domains in Song et al. 2025, 10 models and 23,392 episodes in Khanal et al. 2026, and three frontier models in Martin \u0026amp; Roger 2026.\nBut two results genuinely bound the claim, and a post that skipped them would be selling you something.\nThe first is the null result we opened with. Repantis et al. 2026 is the strongest available counter-evidence to naive monotonic decay: once you chance-correct, visible tool count buys nothing on BFCL. If you were planning to spend a quarter shrinking your tool list to improve selection accuracy, this paper says don\u0026rsquo;t.\nThe second is more interesting. Semenov \u0026amp; Dorofeev 2026 report a single agent session completing 89 sequential tasks across 80 million tokens with no measurable degradation in task accuracy relative to per-task isolated sessions, using structured context eviction instead of compaction. That is an extraordinary claim and I want to be precise about its weight: it is one session, one model, on one harness fork — a case study, not a multi-seed controlled benchmark, and the authors are explicit about the annotation burden it requires. It is the best existing evidence that the compaction debt is optional rather than inherent, and it is nowhere near strong enough to plan on.\nSo the scope is: bill 1 is a property of transformers, bill 2 is a property of your harness\u0026rsquo;s eviction policy, and bill 3 is a property of the field\u0026rsquo;s current evaluation practice. Only the middle one is fully under your control today.\nFigure 8: Where each bill comes from determines who can fix it. The compaction debt is the one you can settle this week. Which brings us to the part you can act on, and to the one gap in this literature that I think is a genuine opportunity.\nPay the Bill That Matches Your Failure There is no published decision procedure that keys a context mitigation to an observed failure mode. The closest thing is Anthropic\u0026rsquo;s cookbook guidance — compaction for long dialogue, tool-result clearing for bulky re-fetchable outputs, a memory tool for cross-session state — which is three coarse heuristics with no thresholds and no effect sizes.\nSo here is the procedure I use. It is a contribution rather than a citation, and I have marked which branches rest on measured before/after numbers and which do not.\nStart from the symptom, not the fix:\nThe agent fails on the first turn of a session, with many tools or servers registered. You are paying the resident tax. Put retrieval in front of your tool registry. Measured: Anthropic\u0026rsquo;s Tool Search Tool takes ~77K → ~8.7K with Opus 4 at 49% → 74% (advanced tool use); Gan \u0026amp; Sun 2025 get 13.62% → 43.13%; Mudunuri et al. 2026 report a 97.1% hit rate at K=3 over 121 tools from five MCP servers, which is the number to beat before you trust a retriever in this position.\nThe agent works early and then ignores a rule you wrote hours ago, while still calling tools correctly. That is the compaction debt, not the resident tax, and the fix is different: re-emit obligations after every compaction. Measured: Constraint Pinning returns post-compaction violations from 30%/59% back to 0% (Chen 2026), and Knowledge Triage restores five-round rule recall from 10% to 96% (Zerhoudi et al. 2026). These are the two best-evidenced interventions in this entire post. If you do one thing after reading it, do this one.\nBulky, re-fetchable tool outputs dominate your context. Stop passing them through the model. Have the model write code that calls the tools instead: 150,000 → 2,000 tokens (code execution with MCP), a pattern independently implemented by Cloudflare\u0026rsquo;s Code Mode, which exposes an entire API in roughly 1,000 tokens. Caveat with teeth: the token savings are replicated, the accuracy is not — I found no neutral third-party accuracy replication — and you are now running model-authored code, which is why Code Mode sandboxes it with outbound fetch disabled by default.\nToken spend is climbing and outcomes are not moving. You are in the measurement illusion. Before you install another skill, run the paired ablation with the token budget held fixed — otherwise you cannot distinguish \u0026ldquo;the skill taught it something\u0026rdquo; from \u0026ldquo;the skill bought it more compute.\u0026rdquo; Nothing in the literature will do this for you; see the sign contradiction in Yang \u0026amp; Ding 2026 versus Kevin et al. 2026.\nYou need genuinely parallel exploration. Subagents give each branch a clean context, and Anthropic reports 90.2% improvement over single-agent Opus 4 on its internal research eval — but be honest about what that number is: it confounds context isolation, parallelism, and 15× more tokens, and on BrowseComp \u0026ldquo;token usage by itself explains 80% of the variance\u0026rdquo; (multi-agent research system). The benefit of isolation per se is unmeasured. Use subagents for parallelism, and do not claim the isolation as your win.\nFigure 9: The decision procedure, with the evidence quality marked. Solid borders are interventions with published before/after numbers on the same benchmark (Anthropic, Gan \u0026amp; Sun 2025, Chen 2026, Zerhoudi et al. 2026); dashed borders are interventions whose token savings are established but whose accuracy effect is not. The minimal viable stack, if you ship nothing else: a retrieval-backed tool registry instead of a static tool array; a pinned constraint block re-emitted after every compaction; and a fixed-budget paired eval gating every new capability package. In code, the first two live in one function:\ndef assemble_turn(task, registry, pinned_rules, history, budget): # 1. resident tax: retrieve tools, never enumerate them tools = registry.search(task.query, k=budget.max_tools) # not registry.all() # 2. compaction debt: obligations are re-emitted, never summarized blocks = [pinned_rules] # always first, always verbatim # 3. fill the remaining window newest-first, and record what was dropped spent, dropped = 0, [] for msg in reversed(history): cost = count_tokens(msg) if spent + cost \u0026gt; budget.history_tokens: dropped.append(msg.id) continue blocks.append(msg) spent += cost emit_metric(\u0026#34;resident_tokens\u0026#34;, count_tokens(pinned_rules) + tool_tokens(tools) + spent) emit_metric(\u0026#34;dropped_messages\u0026#34;, len(dropped)) return Prompt(system=blocks[0], tools=tools, messages=list(reversed(blocks[1:]))) The two emit_metric calls are the point. Per-turn resident tokens and dropped-message count are the only two numbers that let you tell bill 1 from bill 2 in production, and almost no harness reports either.\nLooking Ahead My honest read is that the field has priced the wrong side of the ledger, and for an understandable reason: cost is easy to measure, easy to attribute, and 90% refundable, so that is what gets published. Attention budget is none of those things. Every vendor blog post in my reference list leads with a token reduction and treats the accuracy change as a bonus — including the ones whose accuracy numbers are the most interesting thing in them.\nThree measurements would settle most of what this post had to hedge, and I think they are all tractable:\nAn entropy-versus-length curve on a frontier model. The dispersion theorem is proved and the entropy intervention works in a restricted setting, but the connecting measurement is missing. Someone with API access to attention weights, or a strong open-weights proxy, could produce it in a week.\nAn isolation-only ablation for subagents. Hold token budget and parallelism fixed, vary only whether the sub-task gets a clean context. Anthropic\u0026rsquo;s 90.2% cannot be decomposed without it, and the entire \u0026ldquo;just use subagents\u0026rdquo; consensus rests on that undecomposed number.\nA budget-controlled paired skill evaluation. The sign contradiction between Yang \u0026amp; Ding 2026 and Kevin et al. 2026 will not resolve by adding more benchmarks; it needs a design that pins down the treatment, refuses process metrics that reward the treatment\u0026rsquo;s mere presence, and equalizes tokens across arms. This is the one I am working on, and the reason I care: right now, every team installing capability packages into a coding agent is making a purchasing decision with no price tag on the benefit side.\nUntil then, the practical stance I would defend: treat everything resident in your context as rented, not owned. Charge it a monthly fee. Make it re-justify the space. And when you compact, remember that you are not compressing history — you are choosing which of your obligations to forget.\nCitation Cited as:\nGao, Xueping. \u0026ldquo;The Context Bill\u0026rdquo;. hellogxp.github.io (September 2026). https://hellogxp.github.io/posts/the-context-bill/\nOr use the BibTeX citation:\n@article{xueping2026contextbill, title = {The Context Bill}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {September}, url = \u0026#34;https://hellogxp.github.io/posts/the-context-bill/\u0026#34; } References [1] Repantis, V., Gawde, A., Singh, H., \u0026amp; Blackwell, J. \u0026ldquo;How Many Tools Should an LLM Agent See? A Chance-Corrected Answer.\u0026rdquo; arXiv:2605.24660 (2026).\n[2] Gorilla LLM Team, UC Berkeley. \u0026ldquo;Berkeley Function-Calling Leaderboard.\u0026rdquo; Accessed September 2026.\n[3] Song, W., Zhong, H., Ding, Z., Xue, J., \u0026amp; Li, Y. \u0026ldquo;Help or Hurdle? Rethinking Model Context Protocol-Augmented Large Language Models.\u0026rdquo; arXiv:2508.12566 (2025).\n[4] Gan, T. \u0026amp; Sun, Q. \u0026ldquo;RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation.\u0026rdquo; arXiv:2505.03275 (2025).\n[5] Sadani, A. \u0026amp; Kumar, D. \u0026ldquo;Tool Attention Is All You Need: Dynamic Tool Gating and Lazy Schema Loading for Eliminating the MCP/Tools Tax in Scalable Agentic Workflows.\u0026rdquo; arXiv:2604.21816 (2026).\n[6] Mudunuri, S., Wan, J., Qin, A., \u0026amp; Manoharan, S. \u0026ldquo;Semantic Tool Discovery for Large Language Models: A Vector-Based Approach to MCP Tool Selection.\u0026rdquo; arXiv:2603.20313 (2026).\n[7] Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., \u0026amp; Schütze, H. \u0026ldquo;NoLiMa: Long-Context Evaluation Beyond Literal Matching.\u0026rdquo; ICML 2025.\n[8] Martin, S. \u0026amp; Roger, F. \u0026ldquo;Classifier Context Rot: Monitor Performance Degrades with Context Length.\u0026rdquo; arXiv:2605.12366 (2026).\n[9] Hong, K., Troynikov, A., \u0026amp; Huber, J. \u0026ldquo;Context Rot: How Increasing Input Tokens Impacts LLM Performance.\u0026rdquo; Chroma Technical Report (2025).\n[10] Zerhoudi, S., Mitrovic, J., \u0026amp; Granitzer, M. \u0026ldquo;The Compaction Cliff in Long-Running AI Agent Memory.\u0026rdquo; CIKM 2026.\n[11] Chen, S. \u0026ldquo;Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents.\u0026rdquo; arXiv:2606.22528 (2026).\n[12] Panavas, L., Minus, S., Monton, B., Ray, D., Garre, S., Mehta, S., \u0026amp; Chen, E. \u0026ldquo;HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following.\u0026rdquo; Workshop on Agent Behavior (WAB) @ COLM 2026.\n[13] Khanal, A., Tao, Y., \u0026amp; Zhou, J. \u0026ldquo;Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents.\u0026rdquo; arXiv:2603.29231 (2026).\n[14] badlogic. \u0026ldquo;compaction.md.\u0026rdquo; GitHub Gist (December 2025).\n[15] Semenov, A. \u0026amp; Dorofeev, S. \u0026ldquo;Beyond Compaction: Structured Context Eviction for Long-Horizon Agents.\u0026rdquo; arXiv:2606.11213 (2026).\n[16] Yang, Z. \u0026amp; Ding, F. \u0026ldquo;Signal or Noise? A Benchmark Study of Agent Skills in Web Development.\u0026rdquo; arXiv:2608.23067 (2026).\n[17] Kevin, C., Raghavan, N., Puget, J.-F., Malani, R., Puvvadi, M., Abramovitch, M., Gupta, M., Akkiraju, R., Prabhu, S., Dangi, Y., Luo, W., \u0026amp; Lee, S. H. \u0026ldquo;Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills.\u0026rdquo; arXiv:2608.20614 (2026).\n[18] Veličković, P., Perivolaropoulos, C., Barbero, F., \u0026amp; Pascanu, R. \u0026ldquo;Softmax is not Enough (for Sharp Size Generalisation).\u0026rdquo; ICML 2025.\n[19] Zhang, Z., Wang, Y., Huang, X., Fang, T., Zhang, H., Deng, C., Li, S., \u0026amp; Yu, D. \u0026ldquo;Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models.\u0026rdquo; ACL 2025.\n[20] Ostmeier, S., Axelrod, B., Varma, M., Aali, A., Zhang, Y., Paschali, M., Koyejo, S., Langlotz, C., \u0026amp; Chaudhari, A. \u0026ldquo;Attention Head Entropy of LLMs Predicts Answer Correctness.\u0026rdquo; arXiv:2602.13699 (2026).\n[21] Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., \u0026amp; Pfister, T. \u0026ldquo;Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization.\u0026rdquo; ACL Findings 2024.\n[22] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., \u0026amp; Liang, P. \u0026ldquo;Lost in the Middle: How Language Models Use Long Contexts.\u0026rdquo; TACL 2024.\n[23] Wu, X., Wang, Y., Jegelka, S., \u0026amp; Jadbabaie, A. \u0026ldquo;On the Emergence of Position Bias in Transformers.\u0026rdquo; ICML 2025.\n[24] Chowdhury, B. D. \u0026ldquo;Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias.\u0026rdquo; arXiv:2603.10123 (2026).\n[25] Niu, J., Yuan, X., Wang, T., Saghir, H., \u0026amp; Abdi, A. H. \u0026ldquo;Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs.\u0026rdquo; ACL 2025.\n[26] Liu, Y. \u0026amp; Chu, C. \u0026ldquo;Sentence-Level Contextual Entrainment in Large Language Models.\u0026rdquo; arXiv:2606.24077 (2026).\n[27] Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N., \u0026amp; Silvestri, F. \u0026ldquo;The Power of Noise: Redefining Retrieval for RAG Systems.\u0026rdquo; SIGIR 2024.\n[28] Amiraz, C., Cuconasu, F., Filice, S., \u0026amp; Karnin, Z. \u0026ldquo;The Distracting Effect: Understanding Irrelevant Passages in RAG.\u0026rdquo; arXiv:2505.06914 (2025).\n[29] Jaroslawicz, D., Whiting, B., Shah, P., \u0026amp; Maamari, K. \u0026ldquo;How Many Instructions Can LLMs Follow at Once?\u0026rdquo; arXiv:2507.11538 (2025).\n[30] Harada, K., Yamazaki, Y., Taniguchi, M., Marrese-Taylor, E., Kojima, T., Iwasawa, Y., \u0026amp; Matsuo, Y. \u0026ldquo;When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following.\u0026rdquo; EMNLP 2025.\n[31] Lumer, E., Nizar, F., Jangiti, A., Frank, K., Gulati, A., Phadate, M., \u0026amp; Subbiah, V. K. \u0026ldquo;Don\u0026rsquo;t Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks.\u0026rdquo; arXiv:2601.06007 (2026).\n[32] Anthropic. \u0026ldquo;Prompt caching.\u0026rdquo; Claude API documentation, accessed September 2026.\n[33] OpenAI. \u0026ldquo;Prompt caching.\u0026rdquo; OpenAI API documentation, accessed September 2026.\n[34] Google. \u0026ldquo;Gemini 2.5 models now support implicit caching.\u0026rdquo; Google Developers Blog (2025).\n[35] Modular. \u0026ldquo;Prefix caching.\u0026rdquo; The Modular Inference Handbook, accessed September 2026.\n[36] vLLM. \u0026ldquo;Automatic Prefix Caching.\u0026rdquo; vLLM design documentation, accessed September 2026.\n[37] Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., \u0026amp; Dean, J. \u0026ldquo;Efficiently Scaling Transformer Inference.\u0026rdquo; MLSys 2023.\n[38] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., \u0026amp; Stoica, I. \u0026ldquo;Efficient Memory Management for Large Language Model Serving with PagedAttention.\u0026rdquo; SOSP 2023.\n[39] Anthropic. \u0026ldquo;Code execution with MCP: Building more efficient agents.\u0026rdquo; Anthropic Engineering (2025).\n[40] Wu, B. \u0026ldquo;Advanced tool use.\u0026rdquo; Anthropic Engineering (2025).\n[41] Anthropic. \u0026ldquo;Effective context engineering for AI agents.\u0026rdquo; Anthropic Engineering (2025).\n[42] Hadfield, J., Zhang, B., Lien, K., Scholz, F., Fox, J., \u0026amp; Ford, D. \u0026ldquo;How we built our multi-agent research system.\u0026rdquo; Anthropic Engineering (2025).\n[43] Cloudflare. \u0026ldquo;Code Mode: The better way to use MCP.\u0026rdquo; Cloudflare Blog (2025).\n","permalink":"https://hellogxp.github.io/posts/the-context-bill/","summary":"Your agent is not slow because it has too many tools. It is paying three separate bills — a resident attention tax charged every turn, a compaction debt charged by the fix, and a measurement illusion that hides whether the capability package was worth anything. A cited accounting of all three, and a decision procedure keyed to which one you are actually paying.","title":"The Context Bill"},{"content":"In one of our controlled studies, a coding agent returned the exact nonce-bound answer in 42 out of 42 runs. It also left signatures showing that the target SKILL.md path had been involved. Yet the adapter reconstructed zero Skill runs.\nAnother adapter looked much more informative. It emitted a failure-like event in 24 out of 24 operational-failure sessions. Unfortunately, it emitted the same kind of event in six out of six clean sessions.\nBoth results are uncomfortable for the same reason: the evidence looks stronger than it is. A correct answer does not prove that a Skill was activated correctly. A failure event does not prove that it belongs to the injected failure. And an absent event does not prove that a lifecycle stage never happened.\nThis is the operational gap behind the Skill Runtime Intelligence paper. I also released the open-source implementation: a passive runtime-intelligence system that reconstructs reusable Agent Skills across heterogeneous coding agents without proxying model requests or taking over the agent loop. The central lesson turned out to be broader than the system itself:\nAgent Skill observability is not primarily a logging problem. It is an evidence-reconstruction problem.\nFigure 1: What an agent says, what the runtime exposes, and what an external verifier establishes are related but non-equivalent claims. This distinction matters because Skills are becoming a common unit for packaging reusable instructions, scripts, references, and assets. The open Agent Skills specification standardizes what a Skill looks like on disk. It does not, by itself, tell an operator what happened when an agent attempted to use one.\nIn this post, I will first unpack that missing runtime layer, then show what the experiments revealed, and finally turn the findings into an architecture and maintenance workflow for people building agent platforms or authoring Skills.\nWhy the Final Answer Is Not Enough A final answer is evidence about the agent\u0026rsquo;s response, not a faithful account of the computation that produced it. The response may be correct even when a Skill was missed, only partially loaded, or followed by an unverified artifact. Conversely, a run may have useful runtime evidence even when the final answer is wrong.\nConsider a repository-audit Skill. Its instructions say to read a configuration file, load a reference checklist, run a read-only probe, write a report, and validate a nonce-bound result. The agent eventually replies, \u0026ldquo;The audit passed.\u0026rdquo;\nAt least five different executions can produce that sentence:\nThe Skill was activated, all resources were loaded, the probe ran, and the outcome was independently verified. The Skill was activated, but the reference checklist was never read. The probe failed, but the agent summarized an earlier intermediate result as success. The report was produced and later corrupted. The agent solved the request through some other path and never activated the Skill at all. These are not philosophical distinctions. They imply different owners and fixes. A missing reference may be a Skill-authoring problem. Missing activation telemetry may be an adapter limitation. An incorrect success claim may be a response-grounding problem. A verifier conflict may reveal artifact corruption after otherwise correct execution.\nTraditional agent observability tends to organize records around sessions, model calls, tools, and spans. Those are necessary entities, but none of them is the same as one attempted occurrence of a dynamically loaded Skill. A session may contain several Skills. A Skill may trigger several tools. A tool event may occur near a Skill invocation without belonging to it. The runtime needs an identity and evidence model for the Skill occurrence itself.\nExisting benchmarks illuminate adjacent layers. SkillsBench and SWE-Skills-Bench ask whether packaged Skills improve task outcomes; SWE-bench grounds software-agent outcomes in executable repository tests; and AgentDiagnose analyzes complete agent trajectories. Runtime reconstruction asks a complementary question: before judging usefulness or root cause, can we establish which Skill occurrence and lifecycle boundary the available evidence actually describes?\nThis is also why \u0026ldquo;collect more logs\u0026rdquo; is an incomplete prescription. More events can help only if we know what each event means, which source produced it, which version of the adapter interpreted it, and what relation connects it to the Skill run. Otherwise the extra telemetry increases volume without increasing knowledge.\nThe right first question is therefore not:\nDid the run succeed?\nIt is:\nWhich claims about this run are actually supported, by which evidence, and which stages remain unknown?\nThat question leads naturally to a lifecycle rather than a flat trace.\nWhere Can a Skill Execution Go Wrong? A Skill execution can diverge at multiple boundaries before, during, and after tool use. I model one attempted occurrence as eight ordered stages: Request, Discovery, Activation, Instructions, Resources, Execution, Artifacts, and Outcome. These are logical boundaries; they do not require every agent to emit a native span for every stage.\nFigure 2: The Skill lifecycle exposes boundaries that a session-, model-, or tool-centric trace can collapse. A stage can remain unsupported or unknown without being labeled as failed. The stages are easiest to understand as a sequence of increasingly strong claims.\nRequest means there was a user or system demand that could require the capability. Discovery means the Skill was available to the agent. Activation means this particular occurrence was selected or entered. Instructions means the main behavioral contract became available. Resources covers referenced files, scripts, and assets. Execution covers tools, commands, MCP calls, and subagents. Artifacts covers the concrete files or objects produced. Outcome asks what an independent observation can establish about the result.\nProgressive disclosure makes this separation especially important. The agent may initially see lightweight metadata, then load the full SKILL.md only after selection, and open supporting resources only when needed. Modularity improves, but the runtime now contains boundaries that do not exist in a monolithic prompt.\nAbsence is not automatically failure Suppose an adapter exposes tool calls but has no supported activation signal. If no skill.activated event appears, the honest state is unsupported, not activation failed. The same missing field can become evaluable under a different adapter version that explicitly promises an activation event.\nThis is a subtle but important contract. An adapter is not a neutral parser. It is a versioned measurement instrument. Its schema, coverage, and blind spots determine which runtime claims can be made.\nThe rule I use is:\nMissing telemetry becomes a finding only when the adapter declares the signal observable and an independent expectation makes the absence evaluable.\nWithout both conditions, the system preserves the gap.\nOccurrence and attribution are separate claims Event existence is also different from event attribution. A failed shell command may be directly observed. Assigning that command to a particular Skill occurrence requires an additional relation.\nSome relations are strong: a source parent/child identifier, explicit Skill attribution, an active Skill scope, or an exact artifact path. Others are weak: temporal adjacency or semantic similarity. A command happening three seconds after a Skill activation may belong to that Skill, but time alone does not make the relation deterministic.\nThis is where flat traces often become overconfident. They place events next to one another and allow the viewer—or a model—to turn visual proximity into causality. An evidence graph makes the edge explicit and grades it independently from the nodes it connects.\nOutcome is its own lane An external test can verify a subprocess failure even when the harness emits no native failure event. Conversely, a native failure-like event can appear during a clean execution. The two observations should be displayed side by side, not collapsed into one status.\nThis yields a useful invariant:\nRuntime telemetry must not fabricate an outcome, and an outcome verifier must not backfill a runtime event that was never observed.\nOnce the lifecycle and these separation rules are explicit, heterogeneous adapters become empirically testable rather than informally described.\nWhat the Runtime Evidence Actually Shows The controlled benchmark shows that heterogeneous adapters expose qualitatively different semantics, even when every run can be correlated to a source session and the source worktree remains unchanged. More importantly, aggregate success and event-coverage numbers conceal the specific boundary errors an operator actually needs to understand.\nThe study crossed six frozen repository profiles, three installed coding-agent interfaces, and seven clean or fault-injected conditions. That produced 126 cells: one execution per Agent–repository–condition combination. The faults targeted Instructions, Resources, Execution, Artifacts, and Outcome.\nAll 126/126 executions passed the integrity gate: the source worktrees remained byte-identical, and every call correlated to exactly one collected source session. The response gate was slightly weaker at 122/126 exact nonce-bound responses. These numbers establish controlled mechanism coverage. They do not estimate how often failures occur in production.\nThree adapters, three observability failure modes Figure 3: The three tested adapter–version pairs expose different measurement capabilities. These are adapter observations, not rankings of the underlying agents or models. Data from Gao 2026. The adapter profiles are not simply better or worse versions of the same instrument.\nThe tested Codex adapter reconstructed no Skill runs. Yet Codex returned the exact nonce-bound response in 42/42 cells, and target-SKILL.md path signatures ruled out complete task non-execution. Those signals still do not reveal hidden activation semantics, so the system refuses to invent them.\nThe tested OpenCode adapter reconstructed 42/42 Skill runs but no failure-like events in the 24 operational-failure cells. Occurrence coverage was complete; failure semantics were absent.\nThe tested Qoder adapter emitted a failure-like event in all 24 operational-failure sessions, but also in all six clean sessions. It exactly localized only 6/24 injected boundaries. Because the event was not attributed to the injected nonce, the 24/24 count is co-occurrence, not fault detection.\nA product badge saying \u0026ldquo;failure events supported\u0026rdquo; would hide all three failure modes. A useful capability profile needs at least lifecycle coverage, clean specificity, attribution strength, and exact-boundary localization. Unknown adapter versions should start as unsupported rather than inherit historical claims.\nSemantic structure helps, but not uniformly The next experiment asked whether a normalized Skill panorama gives a model more diagnostic value than native event views. The same 126 controlled cases were presented through several interfaces: minimally structured Raw events, Raw events with lifecycle aliases, a compact normalized Panorama, a deterministic known-rule graph, and a graph followed by model explanation.\nFigure 4: Normalization changes the shape of diagnostic errors rather than uniformly improving every metric. Even 125 exact Graph+Model answers contain only 89 evidence-entailed citation relations. Data from Gao 2026. The minimally structured Raw view localized 72/126 boundaries. Adding inline lifecycle aliases raised localization to 108/126. The compact Panorama also reached 108/126. Naming the semantics was sufficient to close the boundary-localization gap under this frozen prompt contract.\nBut the equal boundary total concealed different errors. The semantics-matched Raw view produced only 49 correct statuses, while Panorama produced 100. Both Raw views emitted a failure status for all 18 clean controls; Panorama emitted none. At the same time, Panorama did worse on exact diagnosis for some instruction and execution failures. The representation did not create a uniform advantage—it traded one error signature for another.\nThis is why I would avoid a composite \u0026ldquo;diagnostic accuracy\u0026rdquo; score. Boundary, status, clean false positives, citation validity, and citation entailment answer different operational questions. Combining them makes dashboards simpler and engineering decisions harder.\nA correct explanation can cite unsupported evidence The deterministic known-rule graph conformed to all 126/126 frozen contracts. This is expected: the rules and labels shared the preregistered fault contract. It is a conformance result, not evidence of novel-fault accuracy.\nGraph+Model produced 125/126 exact answers, which looks nearly perfect. Yet only 89/126 of its cited relations were entailed by the referenced evidence. The model often reached the expected label while explaining it with a relation the cited record did not support.\nThe distinction resembles a familiar problem in agent evaluation: getting the task right is not the same as identifying why it was right. AgentDebugX, for example, improves strict agent-and-step attribution on its Who-and-When benchmark from 21.7% for the strongest single-pass baseline to 28.8% with its multi-turn DeepDebug method. It also repairs 13 of 73 failed GAIA tasks in one rerun, versus four to six for three decoupled self-correction baselines. Those are valuable trajectory-level diagnosis and repair results. Our unit is narrower and differently supervised: one progressively loaded Skill occurrence reconstructed from incomplete, adapter-specific evidence.\nLikewise, HarnessFix reports 15.2%–50.0% held-out improvements after diagnosing and repairing harness flaws across several benchmarks. Its goal is scoped harness repair. AgentRx localizes critical failure steps in 115 human-annotated failed trajectories. Skill Runtime Intelligence instead asks which lifecycle boundary is observable, what evidence supports the claim, and which boundary must remain unknown. The systems are complementary, but their headline numbers are not interchangeable.\nAvailability belongs in the quality metric One model backend completed all 378 primary calls with median latency between 2.16 and 2.35 seconds per view. A second backend completed only 228/378 calls: 111 timed out and 39 violated the structured-output contract. Its completed subsets looked accurate, but conditional accuracy on the calls that returned cannot establish full-matrix reliability.\nThat negative result changed the architecture. A model may improve an explanation when it is available. It cannot sit on the critical path to a reproducible baseline diagnosis.\nThe facts now point to a design requirement: preserve the deterministic evidence path first, and place probabilistic assistance at the edge.\nAn Evidence-Calibrated Runtime An evidence-calibrated runtime stores facts, deterministic relations, uncertain explanations, and controlled effects as different kinds of claims. It never asks one confidence score to carry all four meanings, and it never lets a more fluent downstream representation erase the provenance of the source record.\nThe architecture has four production layers. Collectors observe existing workflows without proxying model requests or owning the agent loop. Versioned adapters preserve raw source identity while emitting only supported normalized fields. An evidence graph creates typed relations and traverses the Skill lifecycle. The Panorama exposes the first observable divergence, its evidence, and the stages that cannot be evaluated.\nAn optional model runs only after that pipeline. Evaluation gold lives in a separate offline lane and never backfills production telemetry.\nFigure 5: Production reconstruction, non-authoritative model candidates, and evaluation-only oracle data. Adapted from Figure 1 in Gao 2026. Four evidence grades The easiest way to understand the evidence contract is to ask how a claim became known.\nObserved means the claim is directly present in a source record or external verifier. A native hook reports a Skill activation; a subprocess returns a non-zero exit code; an external checker observes a corrupted artifact.\nDerived means a deterministic transformation or relation over observed records establishes the claim. An exact source parent identifier connects a tool call to the active Skill run. A path that falls within the exact artifact boundary attaches a changed file to that occurrence.\nInferred means a model or heuristic proposes an uncertain explanation. Temporal adjacency, semantic similarity, and a model\u0026rsquo;s account of likely intent belong here. The explanation may be useful without becoming a fact.\nExperimental means a declared controlled study estimates an effect. A single successful trace can establish that execution and verification occurred. It cannot establish that the Skill caused the success. Claims such as \u0026ldquo;this Skill improves task success\u0026rdquo; require repeated with-Skill and without-Skill trials.\nThe distinction follows a broader provenance idea in the W3C PROV data model: entities, activities, and derivations should retain explicit relations. OpenTelemetry\u0026rsquo;s GenAI semantic conventions provide useful agent, model, and tool span vocabulary. A Skill runtime adds a domain layer over those concepts: Skill identity, progressive-load stages, adapter capability, and evidence grades.\nReconstruction should be intentionally conservative The core traversal can be sketched without an LLM:\ndef reconstruct(source_record, adapter_contract, expectations): raw_id = append_immutable(source_record) event = normalize_supported_fields( source_record, adapter=adapter_contract, unknown_by_default=True, ) observed_id = store(event, grade=\u0026#34;observed\u0026#34;, source=raw_id) edges = [] for rule in RELATION_PRECEDENCE: candidate = rule.match(event) if candidate.is_unique_and_deterministic(): edges.append(store(candidate, grade=\u0026#34;derived\u0026#34;)) break if candidate.has_equal_priority_conflict(): store_ambiguity(candidate) break states = traverse_skill_lifecycle(observed_id, edges) boundary = first_evaluable_divergence( states, capabilities=adapter_contract, expectations=expectations, ) return boundary # may legitimately be unknown The important behavior is not the syntax. Normalization starts from unknown, deterministic relations follow a fixed precedence, equal-priority conflicts remain ambiguous, and the traversal emits a boundary only when evidence makes it evaluable.\nPreserve identity before interpretation Raw records remain separately addressable. Normalization cannot overwrite them, and two physical streams sharing an upstream session identifier cannot destructively merge. Stable identity combines adapter version, physical source-instance identity, and explicit source event or call identifiers. Timestamps alone never create identity.\nThis may sound like storage plumbing, but it is part of epistemology. If two streams are merged because their timestamps and session labels look similar, every downstream relation can become confidently wrong. Evidence quality begins before the diagnosis layer.\nPrivacy belongs at the same boundary. Most lifecycle diagnoses do not require full prompts, source code, credentials, or raw tool payloads. A minimized export can retain ordered states and opaque evidence identifiers while leaving sensitive content within an operator-controlled environment. Minimization reduces both privacy risk and the temptation to let a model invent semantics from irrelevant text.\nThe resulting runtime is deliberately less omniscient than a generated narrative. It is more useful because every statement carries a visible basis.\nDeterministic Core, Probabilistic Edge Known and formalizable lifecycle relations should belong to versioned deterministic rules; models should summarize those facts, prioritize review, and propose novel hypotheses. This division preserves a useful baseline during timeout, malformed output, or provider degradation while still using models where their flexibility adds value.\nFigure 6: The deterministic core remains available and auditable. The probabilistic edge adds candidate explanations without acquiring authority over facts. This is not an argument that models are bad diagnosticians. The rule graph has an obvious ceiling: it detects only known, encoded families. In the controlled rule-external anomaly study, the production-rule baseline detected zero cases by construction. Two model backends found complementary subsets, with one favoring precision and the other recall.\nThe narrow positive role is therefore real:\nsummarize a deterministic finding for a human reader; rank a queue of unresolved or ambiguous relations; validate whether cited nodes actually support a proposed relation; propose an Inferred candidate outside the current rule set; cluster recurring candidates for review. The model should not silently promote its output. A reviewed recurring pattern graduates into a versioned rule with a frozen regression fixture. Over time, the deterministic core grows from validated discoveries rather than repeatedly paying a model to rediscover the same relationship.\nA practical decision tree When adding a runtime claim, I would use the following test:\nIs it explicitly present in a source event or external verifier? Store it as Observed and retain the source locator. Can a stable, versioned rule derive it uniquely? Store the relation as Derived and cite its inputs. Does it depend on timing, semantic similarity, or a model explanation? Keep it Inferred and expose ambiguity. Does it claim that the Skill changed an outcome? Require a controlled experiment and label the result Experimental. Is the necessary signal unsupported by this adapter version? Return unknown. Do not borrow capability from a previous version. This decision tree also clarifies UI design. A useful finding panel should display status, boundary, evidence grade, citation validity, relation entailment, adapter capability, and causal scope separately. A single green checkmark cannot represent all of them.\nThe minimum viable implementation A small team does not need to reproduce the full system to adopt the discipline. Start with five components:\nAn append-only raw-event envelope with physical source identity. A versioned capability manifest for each adapter–agent pair. A small vocabulary for Skill activation, resource access, execution, artifacts, and verified outcomes. Deterministic relations for explicit IDs, active scopes, and exact paths. A UI or report that shows runtime evidence and external outcome in separate lanes. Only after those pieces work should a model explanation be added. Otherwise the model becomes a polished cover over an unstable measurement system.\nPassive collection is compatible with this architecture. In the tested Linux x86_64 environment, the default hook transport delivered 400/400 events exactly; incremental p95 overhead was 0.706 ms for the direct path and 1.275 ms for the shell path. A second Linux arm64 environment delivered 80/80 events, with incremental p95 below 2.4 ms for both paths. These are bounded mechanism results, not universal production latency guarantees, but they show that evidence preservation does not require taking over the agent loop.\nTry the open-source implementation Agent Skill Runtime Intelligence packages this architecture as a local or authenticated self-hosted tool. It currently provides versioned adapters for Codex, Claude Code, Qoder, and OpenCode; reconstructs ordered Skill Runs from supported hooks, plugins, and labeled fallbacks; and exposes the first observable boundary, Panorama, behavior checks, and cited evidence in a local UI. Unsupported signals remain visible as unknown rather than being converted into failures.\nInstallation, agent-specific setup, privacy boundaries, fallback states, and troubleshooting are documented in the Getting Started guide.\nOnce the runtime contract exists, the remaining question is organizational: how should builders use it during development and incident response?\nWhat This Changes for Agent Builders Agent builders should treat every adapter release as a measurement release, every incident as an evidence-preservation problem, and every repaired inferred pattern as a candidate regression rule. The goal is not to make the dashboard more certain; it is to make the development loop honest about what is known.\nFigure 7: A practical maintenance loop for Skill authors. The same frozen probe qualifies both the repair and the adapter evidence used to judge it. For Skill authors When a Skill appears not to work, reproduce the run before rewriting the instructions. Inspect the first observable lifecycle divergence, open the cited raw record, and compare the runtime lane with the independently verified outcome.\nIf the resource boundary is observed missing under a capable adapter, repair the Skill packaging or instructions. If the outcome failed but the runtime contains no supported failure semantics, repairing the Skill may be premature—the missing information belongs to the adapter. If the outcome passed while a clean session contains a generic failure-like event, fix attribution or specificity rather than weakening the verifier.\nThen rerun the same frozen probe. A repair is not durable merely because the next natural-language answer looks better.\nFor adapter and platform teams Every adapter–agent version pair should run an executable lifecycle matrix before it receives a capability badge. Publish at least:\nwhich stages can be observed; failure-event co-occurrence under controlled faults; clean specificity; relation-attribution rules; exact-boundary localization; unsupported stages and known ambiguities. When the agent or source schema changes, rerun the matrix. Do not inherit support from an earlier version because the event names still look familiar.\nThis is the operational implication I find most important: an adapter is part of the measurement apparatus, not invisible integration code. Its release process should look more like instrument calibration than parser maintenance.\nFor incident response An evidence-first triage sequence is short:\nPreserve raw records and source identity. Locate the first observable divergence. Inspect the exact event or relation supporting the finding. Compare the separate external-outcome lane. Request a model explanation only for unresolved relations. Convert a validated recurring pattern into a rule and fixture. This ordering prevents the most fluent explanation from becoming the earliest piece of evidence. It also gives each diagnosis an appropriate owner: Skill definition, adapter, harness, verifier, or model-assisted analysis.\nMy take: Skills need runtime contracts, not only file contracts The Agent Skills specification gives the ecosystem a portable packaging unit. I expect the next layer of maturity to be a portable runtime contract: stable occurrence identity, lifecycle vocabulary, capability declarations, evidence grades, and explicit outcome semantics.\nThe analogy is not merely observability for another plugin format. Skills are beginning to behave like small deployment units. They carry instructions, executable helpers, references, and assumptions about an agent harness. Once teams depend on them across agents and repositories, file validity is no longer enough. Operators need to know which version ran, what loaded, what executed, what was produced, and which conclusion remains unverified.\nI also think unknown should become a first-class product state. Observability systems are usually rewarded for completeness, so they fill gaps with correlation, heuristics, or generated summaries. But unsupported activation telemetry is not a blank cell waiting for an LLM. It is a property of the measurement boundary. Showing it explicitly is more actionable than a confident fiction.\nFinally, answer quality and explanation quality should be governed separately. The experiment\u0026rsquo;s 125 exact answers versus 89 entailed citation relations is a compact demonstration. A model can land on the expected label for the wrong evidential reason. Systems that expose both numbers will sometimes look less impressive, but they will be easier to debug and safer to extend.\nThe current evidence has clear limits. The 126-cell matrix contains one execution per cell and controlled oracle-backed fault overlays. It does not estimate natural incident prevalence. The real-trace study comes from one local database and lacks independent human ground truth. The work does not yet establish that the interface reduces human repair time, nor that its adapter coverage generalizes to every agent version or deployment environment.\nThose are not footnotes to hide. They define the next experiments: real-fault calibration, cross-version adapter qualification, participant diagnosis studies, and controlled measurements of repair time and recurrence.\nThe broader design rule already feels stable:\nReliable agent infrastructure begins when “unknown” is treated as a valid result, not an empty box waiting for a model to fill.\nCitation Cited as:\nGao, Xueping. \u0026ldquo;Runtime Observability for Agent Skills\u0026rdquo;. hellogxp.github.io (August 2026). https://hellogxp.github.io/posts/runtime-observability-for-agent-skills/\nOr use the BibTeX citation:\n@article{gao2026skillobservability, title = {Runtime Observability for Agent Skills}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {August}, url = {https://hellogxp.github.io/posts/runtime-observability-for-agent-skills/} } References [1] Gao, X. \u0026ldquo;Evidence-Calibrated Runtime Reconstruction for Agent Skills Across Heterogeneous Coding Agents.\u0026rdquo; arXiv:2608.08793 (2026). Code and artifacts.\n[2] Agent Skills. \u0026ldquo;Agent Skills Specification.\u0026rdquo; Agent Skills Specification (2026).\n[3] Moreau, L. \u0026amp; Missier, P. \u0026ldquo;PROV-DM: The PROV Data Model.\u0026rdquo; W3C Recommendation (2013).\n[4] OpenTelemetry Authors. \u0026ldquo;Semantic Conventions for Generative AI Systems.\u0026rdquo; OpenTelemetry Semantic Conventions (2026).\n[5] Barke, S., Goyal, A., Khare, A., Singh, A., Nath, S., \u0026amp; Bansal, C. \u0026ldquo;AgentRx: Diagnosing AI Agent Failures from Execution Trajectories.\u0026rdquo; arXiv:2602.02475 (2026).\n[6] Zhu, K., Ye, X., Han, Z., Zhao, Y., Li, B., Zhang, W., Tian, M., Tang, X., Lu, P., Zou, J., You, J., \u0026amp; Ji, H. \u0026ldquo;AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents.\u0026rdquo; arXiv:2607.18754 (2026).\n[7] Chen, M., Wang, J., Liu, Z., Wang, Y., \u0026amp; Wang, Q. \u0026ldquo;From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws.\u0026rdquo; arXiv:2606.06324 (2026).\n[8] Ou, T., Guo, W., Gandhi, A., Neubig, G., \u0026amp; Yue, X. \u0026ldquo;AgentDiagnose: An Open Toolkit for Diagnosing LLM Agent Trajectories.\u0026rdquo; EMNLP System Demonstrations 2025.\n[9] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., \u0026amp; Narasimhan, K. R. \u0026ldquo;SWE-bench: Can Language Models Resolve Real-World GitHub Issues?\u0026rdquo; ICLR 2024.\n[10] Li, X., Liu, Y., Chen, W., et al. \u0026ldquo;SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks.\u0026rdquo; arXiv:2602.12670 (2026).\n[11] Han, T., Zhang, Y., Song, W., et al. \u0026ldquo;SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?\u0026rdquo; arXiv:2603.15401 (2026).\n","permalink":"https://hellogxp.github.io/posts/runtime-observability-for-agent-skills/","summary":"An agent can return the right answer while its Skill runtime remains unobservable. A practical guide to evidence-graded reconstruction, versioned adapters, and the deterministic core that reliable Agent Skill infrastructure needs.","title":"Runtime Observability for Agent Skills"},{"content":"A coding agent produces a correct patch. The tests pass. Then the loop continues.\nThe next prompt contains a failure trace—but the trace came from an older version of the code. The model responds to the trace, edits the already-correct patch, and resurrects the bug. The final answer is wrong even though a correct answer existed one revision earlier.\nNothing exotic happened. The model followed feedback. The test really did pass. The trace contained valid input and output facts. Every individual step looked plausible when viewed alone.\nThe failure was in the transaction that connected them.\nCoding agents are often organized around a loop:\npropose → execute → observe → revise → stop Systems such as Madaan et al. 2023, Shinn et al. 2023, and the feedback-driven repair benchmark of Dai et al. 2025 show why iteration is attractive: another attempt can discover a solution that the first attempt missed. Recent accounts of test-time reasoning and agent harness engineering make the broader point that the workflow around a model—tools, context, persistent state, and evaluation—can matter as much as the model itself.\nBut discovery is not delivery.\nIn our 900-trajectory forced-revision study, the 150-trajectory current-trace arm exposed the distinction sharply. Current correctness fell from 82.0% after revision 1 to 67.3% after revision 2, while the fraction of trajectories that had ever produced a correct patch rose to 84.7%. By revision 3, 16.0% of those trajectories had produced a correct patch and subsequently lost it (Gao et al. 2026).\nThe loop was getting better at finding answers and worse at retaining them.\nThis post develops a narrower claim than “agents need better tests” or “agents should stop earlier”:\nIteration produces candidate states. Reliability depends on how a system observes, evaluates, preserves, and finally admits one of those states.\nThat claim has two consequences. First, evidence must be scoped to the state that produced it. Second, the component that explains the delivery chain should not automatically become another component that edits, blocks, or approves it.\nFigure 1. In the 150-trajectory current-trace condition, cumulative discovery improves while the state actually held by the loop regresses. By revision 3, 16.0% of trajectories had found a correct patch and later lost it. Data source: Gao et al. 2026. A Loop Produces Proposals, Not Reliability Most code-generation metrics are search metrics.\nPass@1 asks whether one sampled answer is correct. Pass@k, popularized for code generation by Chen et al. 2021, asks whether at least one of several samples is correct; Liu et al. 2023 later showed how weak test suites can overestimate that correctness. Best-of-$k$ adds a selector and therefore evaluates more than raw search, but it still normally treats candidates as separately preserved objects: candidate 4 does not overwrite candidate 2.\nAn iterative coding agent has a different failure surface. It edits one mutable workspace over time. Revision $S_{t+1}$ can repair $S_t$, but it can also erase it. A test result observed at time $t$ can remain in context after the code has moved to time $t+1$. A completion message can refer to a workspace that changed after the last check.\nTwo nearby benchmarks show why final pass/fail is too compressed. Dai et al. 2025 report 63.6% repair success with mixed feedback versus 57.9% with test feedback, while gains diminish after two or three iterations. In a different repository-agent setting, Sahoo et al. 2026 find that 10.7% of passing trajectories in their process-assessable subset are “Lucky Passes,” with model-level rates ranging from 0.5% to 23.2%. These are not directly comparable effect sizes—the tasks, agents, and denominators differ—but together they show that more useful feedback and a passing final patch do not identify whether the trajectory preserved progress cleanly. Le et al. 2026 make the same measurement gap explicit by tracking targeted repair, generalization, and preservation under progressive hints.\nThree quantities make the distinction visible:\nEver-correct: did any revision in the trajectory satisfy the oracle? This measures cumulative discovery. Current correctness: is the workspace held at revision $t$ correct? This measures retention. Sound completion: did the system accept a result, and was the accepted result correct? This measures useful delivery. The source paper separates five obligations that these metrics otherwise blur together:\nAdmission: who may accept a candidate into a consequential target? Preservation: does a verified state survive later revisions and malformed actions? Grounded certification: does the completion claim carry fresh evidence for the exact delivered state? Competence: can the agent actually repair the defect under the available tools and budget? Liveness: does the policy still deliver useful work rather than merely abstaining safely? These obligations interact, but none substitutes for another. The rest of this article follows their failure modes through state transitions, evidence, gates, and product boundaries.\nThey can move in opposite directions. Ever-correct can rise while current correctness falls. A gate can observe no false completions by accepting nothing, while sound completion falls to zero. An agent can discover a correct patch, overwrite it, and deliver the regression.\nI find it more useful to call reliability a property of the delivery transaction than a personality trait of the model. A stronger model can improve proposal search. It does not by itself establish:\nwhich code state a test result describes; whether later edits invalidated that result; whether a passing state was retained; whether the delivered tree is the tree that was checked; or who has authority to turn an agent\u0026rsquo;s completion claim into an accepted change. Those are system properties.\nThe self-correction literature already warns against treating them as model personality. Intrinsic correction can fail without external signals (Huang et al. 2024; Kamoi et al. 2024); locating an error is distinct from repairing a supplied location (Tyen et al. 2024); and refinement quality depends on verifier strength (Zhang et al. 2024). Our question starts one step later: once a good state exists, what keeps the surrounding transaction from losing or mis-certifying it?\nCorrectness Is Not an Absorbing State The smallest useful model of a revision loop has two transitions.\nStarting from an incorrect state, another revision may repair it. Starting from a correct state, another revision may regress it. Let\n$$ \\alpha_e=P(Y_{t+1}=1\\mid Y_t=0,e) $$be the repair probability under evidence condition $e$, and let\n$$ \\beta_e=P(Y_{t+1}=0\\mid Y_t=1,e) $$be the regression probability. If $q_t=P(Y_t=1)$ is the population probability that the current state is correct, then\n$$ q_{t+1}-q_t=(1-q_t)\\alpha_e-q_t\\beta_e. $$The first term is the expected gain from repairing wrong states. The second is the expected loss from damaging correct states. When $\\alpha_e+\\beta_e\u003e0$, another revision has negative expected value when\n$$ q_t\u003e\\frac{\\alpha_e}{\\alpha_e+\\beta_e}. $$This threshold is an explanatory population result, not a deployable stopping rule by itself. A real system does not observe the latent correctness of the current patch, and neither $q_t$ nor the transition rates remain fixed when tasks, models, prompts, tools, or evidence policies change. Operational use would require a calibrated state estimator, uncertainty bounds, and drift monitoring. Without those, the equation tells us why deeper loops can hurt, not exactly when one individual run should stop.\nOur forced-revision study was designed to expose the transition rather than imitate every adaptive production agent. Thirty HumanEval repairs were crossed with six evidence conditions and five seeds, producing 900 trajectories with three forced revisions each (Gao et al. 2026).\nThe result was not a generic depth penalty. With current binary feedback, correctness across the three revisions was 70.0%, 70.0%, and 72.0%. With unavailable evidence it was 68.7%, 68.7%, and 70.7%. With stale traces it moved 68.7%, 71.3%, and 66.0%.\nDepth merely creates more transition opportunities. Evidence and orchestration determine whether those transitions accumulate value or destroy it.\nThis also explains why “the model should know when to stop” is not a complete interface. A model\u0026rsquo;s confidence or DONE token is still an observation about the model\u0026rsquo;s intent. It is not proof that the final workspace is correct, that its last test is current, or that the model is authorized to approve its own output.\nEvidence Belongs to a State A test report is not timeless knowledge.\nSuppose revision $S_1$ fails a test, revision $S_2$ fixes it, and the model is then shown the failure trace from $S_1$. The payload may be factually valid: the input, expected output, and observed failure all really occurred. But it does not describe $S_2$.\nFor the purpose of revising $S_2$, the evidence is stale.\nA minimal evidence envelope is\n$$ E=(h_{state},h_{suite},execution\\_id,payload). $$Here $h_{state}$ identifies the evaluated code state, $h_{suite}$ identifies the verifier suite and policy, and execution_id identifies the run that produced the payload. Evidence is aligned with the current candidate only when\n$$ h_{state}=h(S_t). $$In a production system, the state identity often needs more than source bytes. The command, working directory, dependencies, environment, policy version, and verifier origin can all change what a pass means. The invariant remains simple:\nEvidence about one state must not silently justify a claim about another.\nWhy ordinary trajectory comparisons are not enough If we estimate stale-evidence harm from free-running trajectories, we create a selection problem. Earlier evidence affects which trajectories reach a correct state. Comparing only the trajectories that later became correct conditions on an event that the treatment helped cause.\nWe therefore used a common-state intervention.\nFor each eligible task, we froze one correct and one incorrect program before assigning new evidence branches. Every branch began from identical code; only the evidence condition changed. Twenty-seven tasks had both states. We repeated the experiment with Qwen2.5-7B and a prespecified Qwen2.5-14B replication, for 2,430 single-step branches in total.\nFrom correct starts, the 14B coder regressed 34/135 times when shown a stale trace from the paired wrong state, compared with 4/135 under a current trace. The stale-minus-current harm was +22.2 percentage points, with a task-cluster 95% confidence interval of [8.9, 37.0] and an exact Holm-adjusted $p=0.0337$ (Gao et al. 2026).\nThe strongest defensible conclusion is not that stale context always causes regression. It did not. The 7B contrast was smaller and did not survive multiplicity correction. Even within one coder, failures clustered in particular tasks.\nWhat the experiment does support is more useful:\nA valid diagnostic payload can become causally harmful when its provenance is misaligned with the state being revised.\nOne task makes the mechanism concrete. In HumanEval/89, the wrong program used a Caesar shift of 12. The repaired program correctly used a shift of 4. When the stale report from the wrong program was attached to the repaired program, the 7B coder changed 4 back to 12 in all five preregistered seeds.\nThe trace did not lie about the old program. The envelope lied by implication about the current one.\nThis changes how I think about agent context. Context is not merely helpful, distracting, long, or short. Some observations are valid for the current state; others are historical records. A harness that flattens both into the same prompt has discarded a type distinction that the agent cannot reliably reconstruct.\nFigure 2. Common-state results from identical frozen programs. Stale evidence is especially harmful to the 14B coder from correct starts; the chart reports branch rates, while inference in the text clusters by task rather than treating seeds as independent. Image source: Gao et al. 2026. A Gate Can Fail in Both Directions State alignment prevents one class of error. It does not make a verifier correct.\nEvery completion gate makes two coupled decisions: which candidates it accepts, and how often accepted candidates are wrong. The first is coverage,\n$$ coverage=P(A=1), $$and the second is selective risk,\n$$ risk=P(Y=0\\mid A=1). $$Risk is undefined when coverage is zero. A verifier that accepts nothing has zero observed false completions, but there are no accepted cases on which to estimate selective risk. It has also delivered nothing. This is why risk must be reported with coverage and sound completion:\n$$ sound\\ completion=P(A=1,Y=1). $$This framing follows selective prediction (Geifman \u0026amp; El-Yaniv 2019): abstention is an outcome to evaluate, not a set of rows to silently remove. It matters especially for agents, because a strict gate can appear safe while repeatedly exhausting the budget without returning anything useful.\nDifferent verifiers do not imply independent evidence Verifier ensembles are often justified with a multiplication argument. If verifier $i$ has false-accept rate $f_i$ and verifier $j$ has rate $f_j$, requiring both to accept appears to reduce joint error to $f_if_j$. But LLM evaluators can favor their own generations (Panickssery et al. 2024), and errors remain correlated across models and providers (Kim et al. 2025).\nThat argument assumes conditional independence. Among incorrect candidates, define\n$$ D_{ij}=P(A_iA_j\\mid Y=0)-P(A_i\\mid Y=0)P(A_j\\mid Y=0). $$Then\n$$ P(A_iA_j\\mid Y=0)=f_if_j+D_{ij}. $$Positive dependence is an additive penalty. Changing model families does not establish that $D_{ij}=0$, and low dependence is not useful if one verifier has poor marginal risk.\nAt 70% coverage in our retrospective verifier study, the Qwen-7B/Qwen-14B pair had conditional false-accept dependence $\\phi=0.641$, with task-cluster interval $[0.096,0.935]$. Qwen-14B/DeepSeek had much lower estimated dependence at the same coverage, $\\phi=0.021$, but the DeepSeek verifier had substantially higher marginal risk. At 75% coverage, all three evaluated pairs had positive excess-joint intervals (Gao et al. 2026).\nThe engineering lesson is narrower than “model diversity does not work”:\nVerifier quality and verifier dependence are properties of a candidate distribution and an operating point. They must be measured, not inferred from model names.\nThis also separates two meanings of independence:\nIndependent admission authority: the agent cannot convert its own proposal into an accepted change. Statistical verifier independence: verifier errors are weakly correlated conditional on the candidate being wrong. Architecture can enforce the first. It cannot assume the second into existence.\nProtection can reduce plasticity Retrospective replay can tell us whether a rule would have intercepted recorded outputs. It cannot tell us how enforcing the rule changes later agent behavior. We therefore ran a prospective paired study over the 27 common-state tasks.\nThe baseline and guarded arms used the same coder, typed action prompt, diagnostic visibility, seeds, and three-step budget. The baseline committed every parseable patch. The guarded policy bound evidence to the current state, admitted patches only after a challenge gate, and retained a last-known-good checkpoint after rejected or invalid proposals.\nFrom correct starts, baseline continuation caused harm in 13/135 rollouts. The guarded arm had 0/135 observed harms and retained correctness in all 135 cases.\nFrom wrong starts, however, repair fell from 100/135 to 93/135.\nAcross both start states, unsafe completion moved from 1/270 to 0/270, too sparse to demonstrate superiority. Sound completion moved from 219/270 to 211/270; its confidence interval crossed the prespecified non-inferiority margin. The joint criterion therefore failed (Gao et al. 2026).\nThe guard changed where errors appeared. It protected correct states, but it also reduced revision plasticity and useful completion.\nA separate stopping-policy replay made the trade-off more dramatic. A Qwen-14B current-evidence challenge policy had no observed false completions in that replay, but coverage fell to 38.4% and the rule lost 311 baseline sound completions.\nBinding a weak gate gives it authority, not correctness Repository-scale experiments exposed the opposite failure. Real issue resolution adds long context, multi-file edits, and agent-computer interface effects that function repair does not capture (Jimenez et al. 2024; Yang et al. 2024). It can also reward much simpler localization–repair–validation pipelines than a fully agentic scaffold (Xia et al. 2025). Our repository study asks a narrower question: what happens when the evidence contract gains authority inside such a loop?\nIn the primary 288-rollout factorial over 24 historical bugs from nine Python projects, the repair process had a severe floor: 276/288 rollouts exhausted the 15-action budget. Contract OFF produced five accepted completions—four unsafe and one sound. Contract ON produced no accepted completions (Gao et al. 2026).\nThe contract removed every observed unsafe completion, but it also removed the only observed sound completion. The unsafe superiority confidence bound ended exactly at zero rather than below it, and the sound non-inferiority result was weak under a near-zero baseline. The joint decision failed.\nFigure 3. Prespecified repository-agent contract effects (ON minus OFF). Bars are task-cluster bootstrap 95% intervals over 24 task-level effects, with seeds averaged before inference. Unsafe superiority misses because its upper bound reaches zero; sound non-inferiority clears the -5-point margin only under a near-zero baseline, so the joint intersection-union test fails. Image source: Gao et al. 2026. Three separately frozen coder-stack replications were not stable enough to summarize as “contracts help.” In the Qwen2.5-Coder-32B-AWQ replication, Contract ON increased unsafe completion by 4.17 percentage points. A private gate false-passed incorrect proposals and submissions, and the terminal checkpoint rule converted some of those errors into accepted completions.\nBinding a weak gate more tightly did not create correctness. It gave the gate more authority.\nMy judgment on the evidence is therefore deliberately asymmetric:\nThe common-state experiment gives credible causal evidence that misaligned diagnostics can damage an already-correct state. The guarded paired study gives strong evidence of retention improvement in that setting, but not of end-to-end superiority. The repository factorial gives important negative evidence about weak gates and liveness floors; it does not establish a production effect size. None of these studies justifies treating one verifier policy as a universal completion rule. That is enough evidence to motivate stronger state and provenance boundaries. It is not enough evidence to outsource approval to a new gate.\nMy Take: Separate Three Planes The usual agent diagram places the model in the center and connects it to tools, memory, and tests. For delivery reliability, I prefer a different decomposition:\nExecution plane Agent proposes, edits, runs tools, and reports completion │ recorded events ▼ Observation plane Reconstructs states, checks, freshness, artifacts, and gaps │ evidence for a decision ▼ Admission plane User, protected CI, deployment policy, or explicit authority decides delivery The boundaries matter more than the boxes.\nExecution plane: produce candidate states The execution plane should remain optimized for useful work: editing, testing, debugging, and exploration. It may be highly autonomous. Its completion claim is still a claim from the actor that performed the work.\nObservation plane: explain what happened The observation plane reads supported transcripts, state changes, check outcomes, repository metadata, and artifacts. It correlates events into a delivery chain and exposes where evidence is current, stale, missing, or only inferred.\nIts most important property is epistemic restraint. Seeing a passing pytest exit code is not the same as knowing all requirements are satisfied. Seeing a later edit after that pass is enough to derive staleness; it is not enough to infer why the agent made the edit.\nAdmission plane: own the consequential transition The admission plane decides whether a change may merge, deploy, or otherwise affect a protected target. It can include branch protection, project-owned CI, deployment checks, human review, or a domain-specific approval service.\nThe observation plane can inform admission, but it should not silently inherit admission authority. Otherwise the system has merely replaced “the agent approves itself” with “the dashboard approves the agent.” This separation echoes policy systems such as Open Policy Agent, where decision logic and enforcement are explicit components rather than an accidental UI side effect.\nThis decomposition refines ideas I discussed in The Trust Stack, Evaluation, Regression, and Memory, and Agent State Safety. The new point here is narrower: for a mutable coding trajectory, state-scoped evidence is the interface between execution and any later judgment.\nFigure 4. Execution, observation, and admission are separate jobs. StateSeal occupies the read-only observation plane; there is deliberately no action arrow back to the agent. Author's illustration. StateSeal as Read-Only Delivery Intelligence The paper\u0026rsquo;s reference artifact instantiated the typed contract as an admission layer (Gao et al. 2026). The current StateSeal product has a deliberately narrower boundary: it implements the observation plane for local agent runs. It is not another coding agent, test runner, proxy, checkpoint manager, or admission broker.\nIts invariant is:\nAgent logs + Git metadata + artifacts │ read only ▼ normalization and state correlation │ ▼ panorama + evidence + diagnosis + audit There is no action arrow back to the agent. StateSeal does not launch the agent, alter prompts, deny a tool call, restore a checkpoint, approve completion, or apply a patch. It observes supported local traces and explains the chain the agent left behind.\nThis is a narrower product boundary than the historical admission prototype used in parts of our research. I think it is also a cleaner one. The experiments show why state and evidence need external treatment; they do not show that every observer should become an enforcement point.\nThe state and freshness update flow For a supported transcript, StateSeal begins at observed revision $S_0$. A successful recognized mutation advances the sequence to $S_1,S_2,\\ldots$. A recognized check is attached to the revision at which its command starts. If a later mutation advances the sequence, the earlier result becomes stale by construction.\nA simplified version of the update algorithm is:\nrevision = 0 checks = [] agent_ended = false for event in transcript_order: if event is a recognized check start: checks.append(Check(state_revision=revision, status=RUNNING)) if event finishes that check: record observed exit code status = PASSED, FAILED, or COMPLETED_WITHOUT_EXIT_CODE if event is a successful recognized mutation: revision += 1 record affected paths and evidence grade if event is the agent\u0026#39;s completion event: agent_ended = true for check in checks: check.freshness = ( CURRENT_BY_SEQUENCE if check.state_revision == revision else STALE ) posture = classify(agent_ended, revision, checks) The resulting evidence posture is intentionally small:\nPosture What the observed sequence supports Current A recognized passing check follows the latest observed edit. Failure A current-state recognized check reported failure. Stale Checks passed, but later observed edits created a newer state. Gap The agent ended without a recognized current-state check. Unknown Available signals are insufficient for a stronger statement. These are descriptions of evidence, not approval states. In particular, Current does not mean correct. It means the observer saw a passing recognized check attached to the latest observed revision.\nFigure 5. StateSeal binds a recognized check to the observed revision at command start. A later supported mutation does not make the pass false about S1; it makes the pass stale as evidence for S2. Author's illustration. Evidence grades keep facts separate from interpretation StateSeal labels claims at three grades:\nGrade Requirement Example Observed Directly present in a supported event A command reported exit code 1. Derived Deterministically reproducible from observed relations A passing check precedes a later observed edit. Inferred Plausible but not proven Current Git changes likely belong to this session. The distinction is operational, not decorative. A completion event is observed, but semantic correctness is not. Staleness can be derived from sequence order. Intent and causality normally remain inferred.\nWhat the observer can and cannot claim For a supported trace, StateSeal can state that a recognized command ran, what exit code was observed, which observed revision preceded it, whether later recognized mutations occurred, and which artifacts were recorded.\nIt cannot conclude that:\nevery requirement was covered; a passing check proves semantic correctness; no unobserved filesystem mutation occurred; the agent understood the request; the agent\u0026rsquo;s completion event is an approval; or an inferred explanation is certain. The sequence model is intentionally conservative. It can detect verify-then-edit patterns in the events it supports. It cannot prove the absence of activity it did not observe. Git workspace metadata is therefore shown separately from transcript attribution rather than silently fused into a stronger claim.\nThis is the product lesson I would keep even if the implementation changed completely:\nA delivery observer should make the strongest claim supported by its evidence—and stop there.\nA Minimal Pattern for Builders You do not need StateSeal to adopt the underlying design. The minimal pattern is a versioned event model plus explicit authority boundaries.\nIf you implement only three things: identify every mutable state, bind each check when it starts, and never collapse agent completion, evidence posture, and admission into one status.\n1. Give every mutation a state identity At minimum, increment an observed revision for every supported successful mutation. For stronger admission, use a content digest over the exact candidate plus the verifier policy and relevant environment. This borrows the provenance instinct from systems such as in-toto and SLSA, but applies it to a shorter-lived agent repair trajectory rather than claiming software-supply-chain attestation.\nDo not let “current workspace” remain an implicit global variable shared by the agent, test runner, and UI.\n2. Bind checks when they start Record at least:\n{ \u0026#34;check_id\u0026#34;: \u0026#34;check-17\u0026#34;, \u0026#34;state_id\u0026#34;: \u0026#34;S4\u0026#34;, \u0026#34;suite_id\u0026#34;: \u0026#34;unit+lint@policy-8\u0026#34;, \u0026#34;command\u0026#34;: \u0026#34;pytest -q\u0026#34;, \u0026#34;working_directory\u0026#34;: \u0026#34;/repo\u0026#34;, \u0026#34;started_at\u0026#34;: \u0026#34;...\u0026#34;, \u0026#34;finished_at\u0026#34;: \u0026#34;...\u0026#34;, \u0026#34;exit_code\u0026#34;: 0, \u0026#34;evidence_grade\u0026#34;: \u0026#34;observed\u0026#34; } Binding at check completion is too late if the workspace can mutate while the command is running. A protected evaluator should materialize an immutable candidate; a passive observer should at least preserve the observed start revision and disclose its limitations.\n3. Compute freshness, do not ask the model for it Freshness is a relation between the checked state and the later mutation sequence. It should be derived by the harness:\nfresh(check, final_state) := check.state_id == final_state.id If identity is uncertain, return Unknown or Gap. Do not convert uncertainty into a pass because the agent said “tests are still valid.”\n4. Keep completion, evidence, and approval separate Use different fields and different owners:\nagent_reported_complete # actor claim evidence_posture # observer result admission_decision # authorized external decision Collapsing them into a single status=success destroys the very boundary that later debugging, auditing, and policy need.\n5. Preserve both safety and liveness metrics For any gate or admission policy, report:\ncoverage; selective risk, when coverage is nonzero; sound completion over all attempts; regression from correct starts; repair from wrong starts; budget exhaustion and escalation; latency and verifier cost. A rule that blocks every patch is easy to make look safe. A rule that accepts every patch is easy to make look productive. Neither is reliable.\n6. Stage authority gradually An observation layer can start read-only and collect a representative trace distribution. Proposed gates can then be replayed offline and run in shadow mode before they gain blocking authority. Related systems explore evidence-to-action guards (Meng et al. 2026), abstain-and-validate policies (Cambronero et al. 2026), and transactional admission (Chang et al. 2026). Protected CI or human review should remain the admission boundary until the candidate policy demonstrates both acceptable risk and acceptable coverage under drift.\nThe rollout order matters because false acceptance and false rejection are both product failures. The repository study above is exactly the kind of warning a shadow phase should reveal: a gate can be tightly integrated, internally consistent, and still wrong at the operating point that matters.\nWhat I Would Measure Next The evidence is strongest around state misalignment and much weaker around universal policy choice. The next evaluations should therefore target the interfaces between the three planes.\nObserver integrity. On traces with injected mutations, missing outputs, concurrency, and unsupported tools, how often does the observer correctly return Current, Stale, Failure, Gap, or Unknown? Report confusion matrices and abstention, not only happy-path examples. Candidate-distribution calibration. Measure gate risk, coverage, and verifier dependence on the actual states produced by the target agent and harness. Repeat after model, prompt, tool, and repository changes. Prospective policy effects. Randomize enforcement, not just replay. A gate changes which diagnostics the model sees, which patches survive, and whether budget is available for later repair. End-to-end delivery outcomes. Track merged correctness, rollback, escaped defects, human review time, and delivery latency. A cleaner internal state machine is not automatically a better product. Boundary failures. Test verify-then-edit, long-running checks that overlap mutations, multi-worktree confusion, suite changes, hidden environment dependencies, and completion claims after parser or tool failure. Authority composition. Make explicit which decisions belong to the local observer, project CI, protected deployment environment, and human owner. Then test how evidence moves between them without being silently upgraded. I would resist adding more autonomy until these measurements are visible. The hard part is not drawing another loop around the model. It is keeping state, evidence, judgment, and authority from collapsing into one ambiguous “success” signal.\nConclusion Iteration remains one of the most useful ways to improve coding-agent performance. Another revision can repair a wrong state, explore a better design, or recover from a tool failure.\nBut correctness is not absorbing. A later revision can overwrite a correct patch. A diagnostic can be true about the past and harmful to the present. A verifier ensemble can share the same blind spots. A strict gate can protect retention while destroying useful completion, and a weak gate can turn its own false pass into delivery authority.\nThe practical response is not “never loop.” It is to stop asking one loop to provide every property:\nthe execution plane produces candidate states; the observation plane reconstructs evidence and its limits; the admission plane owns consequential acceptance. StateSeal currently occupies the middle plane. It reads supported agent traces, reconstructs revisions and check freshness, labels evidence as observed, derived, or inferred, and surfaces stale results and gaps. It does not approve the agent.\nThat boundary is intentional. The observer\u0026rsquo;s job is to make the delivery chain legible. Reliability begins when the rest of the system respects what each piece of evidence actually establishes—and no more.\nCitation Cited as:\nGao, Xueping. \u0026ldquo;Looping Is Not Reliability\u0026rdquo;. hellogxp.github.io (August 2026). https://hellogxp.github.io/posts/looping-is-not-reliability/\nOr use the BibTeX citation:\n@article{gao2026looping, title = {Looping Is Not Reliability}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {August}, url = {https://hellogxp.github.io/posts/looping-is-not-reliability/} } References [1] Gao, X., Yang, J., \u0026amp; Yang, Q. \u0026ldquo;Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair.\u0026rdquo; arXiv:2607.24604 (2026).\n[2] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. \u0026ldquo;Self-Refine: Iterative Refinement with Self-Feedback.\u0026rdquo; NeurIPS 2023.\n[3] Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., \u0026amp; Yao, S. \u0026ldquo;Reflexion: Language Agents with Verbal Reinforcement Learning.\u0026rdquo; NeurIPS 2023.\n[4] Dai, D., Liu, M., Li, A., Cao, J., Wang, Y., Wang, C., Peng, X., \u0026amp; Zheng, Z. \u0026ldquo;FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks.\u0026rdquo; arXiv:2504.06939 (2025).\n[5] Weng, L. \u0026ldquo;Why We Think.\u0026rdquo; Lil\u0026rsquo;Log (2025).\n[6] Weng, L. \u0026ldquo;Harness Engineering for Self-Improvement.\u0026rdquo; Lil\u0026rsquo;Log (2026).\n[7] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. de O., Kaplan, J., Edwards, H., Burda, Y., et al. \u0026ldquo;Evaluating Large Language Models Trained on Code.\u0026rdquo; arXiv:2107.03374 (2021).\n[8] Liu, J., Xia, C. S., Wang, Y., \u0026amp; Zhang, L. \u0026ldquo;Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.\u0026rdquo; NeurIPS 2023.\n[9] Sahoo, P., Mittal, G., Li, X., Ma, S., Steenhoek, B., Lin, P., \u0026amp; Hu, Y. \u0026ldquo;AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation.\u0026rdquo; arXiv:2605.12925 (2026).\n[10] Le, C. C., Yadavally, A., Le-Anh, M., \u0026amp; Nguyen, T. N. \u0026ldquo;Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback.\u0026rdquo; arXiv:2607.01360 (2026).\n[11] Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., \u0026amp; Zhou, D. \u0026ldquo;Large Language Models Cannot Self-Correct Reasoning Yet.\u0026rdquo; ICLR 2024.\n[12] Kamoi, R., Zhang, Y., Zhang, N., Han, J., \u0026amp; Zhang, R. \u0026ldquo;When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs.\u0026rdquo; TACL 12, 1417–1440 (2024).\n[13] Tyen, G., Mansoor, H., Carbune, V., Chen, P., \u0026amp; Mak, T. \u0026ldquo;LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location.\u0026rdquo; Findings of ACL 2024.\n[14] Zhang, Y., Khalifa, M., Logeswaran, L., Kim, J., Lee, M., Lee, H., \u0026amp; Wang, L. \u0026ldquo;Small Language Models Need Strong Verifiers to Self-Correct Reasoning.\u0026rdquo; Findings of ACL 2024.\n[15] Geifman, Y., \u0026amp; El-Yaniv, R. \u0026ldquo;SelectiveNet: A Deep Neural Network with an Integrated Reject Option.\u0026rdquo; ICML 2019.\n[16] Panickssery, A., Bowman, S. R., \u0026amp; Feng, S. \u0026ldquo;LLM Evaluators Recognize and Favor Their Own Generations.\u0026rdquo; NeurIPS 2024.\n[17] Kim, E. M., Garg, A., Peng, K., \u0026amp; Garg, N. \u0026ldquo;Correlated Errors in Large Language Models.\u0026rdquo; ICML 2025.\n[18] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., \u0026amp; Narasimhan, K. R. \u0026ldquo;SWE-bench: Can Language Models Resolve Real-World GitHub Issues?\u0026rdquo; ICLR 2024.\n[19] Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., \u0026amp; Press, O. \u0026ldquo;SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.\u0026rdquo; NeurIPS 2024.\n[20] Xia, C. S., Deng, Y., Dunn, S., \u0026amp; Zhang, L. \u0026ldquo;Demystifying LLM-Based Software Engineering Agents.\u0026rdquo; Proceedings of the ACM on Software Engineering 2(FSE) (2025).\n[21] Meng, Q., Zhang, X., Ren, Z., \u0026amp; Visser, J. \u0026ldquo;EviACT: An Evidence-to-Action Framework for Agentic Program Repair.\u0026rdquo; arXiv:2605.27238 (2026).\n[22] Cambronero, J. P., Tufano, M., Shi, S., Wei, R., Uy, G., Cheng, R., Liu, C.-J., Pan, S., Chandra, S., \u0026amp; Rondon, P. \u0026ldquo;Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair.\u0026rdquo; ICSE-SEIP 2026.\n[23] Chang, E. Y., Geng, L., \u0026amp; Chang, E. J. \u0026ldquo;Mnemosyne: Agentic Transaction Processing for Validating and Repairing AI-Generated Workflows.\u0026rdquo; arXiv:2607.00269 (2026).\n[24] Torres-Arias, S., Afzali, H., Kuppusamy, T. K., Curtmola, R., \u0026amp; Cappos, J. \u0026ldquo;in-toto: Providing Farm-to-Table Guarantees for Bits and Bytes.\u0026rdquo; USENIX Security 2019.\n[25] Supply-chain Levels for Software Artifacts. \u0026ldquo;SLSA v1.2 Provenance.\u0026rdquo; SLSA Specification v1.2 (2026).\n[26] Open Policy Agent Contributors. \u0026ldquo;Open Policy Agent Documentation.\u0026rdquo; OPA Documentation (2026).\n[27] Gao, X. \u0026ldquo;StateSeal: Observe the Execution. Explain the Chain. Never Become the Execution.\u0026rdquo; GitHub repository (2026).\n","permalink":"https://hellogxp.github.io/posts/looping-is-not-reliability/","summary":"Why more agent revisions can discover more correct patches while delivering fewer of them—and how state-scoped evidence, read-only observation, and explicit admission boundaries turn a mutable coding trajectory into an explainable delivery chain.","title":"Looping Is Not Reliability"},{"content":"In the original tau-bench experiments, state-of-the-art function-calling agents succeeded on fewer than 50% of tasks. That number was already sobering. The more revealing result appeared when the same task was run repeatedly: in retail, the probability of succeeding in all eight trials, $\\text{pass}^8$, fell below 25%.\nAn agent can therefore look capable on a leaderboard and still be unreliable for a user who expects the same policy to work every time.\nThis is the central mismatch in agent evaluation. A benchmark asks, \u0026ldquo;How well does this system perform on a fixed task set?\u0026rdquo; A production team needs answers to a different set of questions:\nWhat failed: the final answer, the tool path, or the environment state? Was the cause the model, prompt, tool schema, retrieved context, memory, policy, or recovery logic? Can the exact failure be replayed after one of those components changes? Is the evidence strong enough to block a release, or only strong enough to trigger review? Will the system remember this failure six months from now? My thesis is simple:\nProduction agent evaluation is the process of turning traces into regression memory.\nBenchmarks compare agents. Regression memory improves the engineering system around them. It captures a failure as versioned evidence, attaches a defensible judgment, converts it into a replayable test, and gives the result an explicit role in release control.\nFigure 1: A benchmark ends at a score. A production evaluation system must feed evidence back into engineering and release control. This article develops that loop through four layers: Evidence, Judgment, Memory, and Authority. The framework is deliberately tool-agnostic. Products such as LangSmith, MLflow, and OpenAI\u0026rsquo;s trace grading can implement parts of it, but no dashboard can decide what your system must never forget.\nThis argument is adjacent to, but different from, Lilian Weng\u0026rsquo;s Harness Engineering for Self-Improvement. Her article asks how a harness can preserve failed attempts, mine verifier-grounded failure patterns, and improve itself without letting the editable system control its own evaluator. This article focuses on the control plane around that loop: what evidence must exist before a stochastic system that acts on an environment is allowed to change or ship. Self-improvement proposes changes. Regression memory determines which changes have earned authority.\nA Benchmark Is a Snapshot, Not a Control System A benchmark is a standardized experiment for comparison. It becomes a control system only when its evidence is tied to the product\u0026rsquo;s actual failure distribution, version history, and release decisions.\nBenchmarks have done essential work for agents. WebArena showed a historical gap of 14.41% task success for its best GPT-4-based agent versus 78.24% for humans. OSWorld reported 12.24% for its best evaluated model versus 72.36% for humans across 369 computer tasks. These results made long-horizon tool use measurable and exposed failures that static language benchmarks missed.\nFigure 2: Early environment benchmarks exposed a roughly 60-point reliability gap between the best evaluated agents and humans. Values are from the original WebArena and OSWorld papers. But these numbers do not tell a product team whether yesterday\u0026rsquo;s prompt change broke refunds for premium customers. They cannot tell whether a successful final response hid an unauthorized side effect. They do not preserve the database state, retrieved documents, tool versions, or approval decisions required to reproduce a production incident.\nThe distinction is not between a public benchmark and a private one. A private collection of 500 prompts is still just a benchmark if it is periodically run, summarized into one score, and then forgotten.\nThe operational difference is whether the evaluation has a feedback path:\nA real or anticipated failure produces structured evidence. A grader makes a calibrated, reviewable judgment. The failure becomes a versioned regression case. The case is assigned an explicit role in CI gating, canary deployment, rollback, or human review. This leads to a useful distinction:\nCapability evaluation asks what the agent can do. Regression evaluation asks what the engineering system refuses to forget.\nAnthropic\u0026rsquo;s agent-eval guidance makes a similar operational distinction: capability suites should contain difficult tasks with room to improve, while regression suites should approach 100% pass rates and run continuously. A capability case can \u0026ldquo;graduate\u0026rdquo; into regression protection once the system reliably handles it.\nThat transition is the beginning of memory. It is not yet the complete control system.\nFour Layers: Evidence, Judgment, Memory, Authority An evaluation result is trustworthy only if four separate contracts hold: the evidence represents what happened, the judgment measures the intended property, the memory can reproduce the case, and the authority matches the uncertainty of the signal.\nFigure 3: The four contracts of production agent evaluation. Weakness in any layer breaks the improvement loop. The separation matters because teams frequently compress all four layers into one sentence: \u0026ldquo;The evaluator gave this run 0.72, so the candidate is better.\u0026rdquo; That sentence hides at least four unproven claims:\nThe evaluator saw the relevant state and actions. A score of 0.72 corresponds to the failure property we care about. The run can be reconstructed under a future candidate. A score of 0.72 provides enough evidence to permit or reject a deployment. The rest of this article examines each contract in order.\nEvidence: The Trace Is the Unit of Diagnosis The correct evidence unit for an agent is normally a trace plus the resulting environment state, not a prompt-response pair. A final answer is only one observation emitted by a stateful process.\nSuppose a support agent says, \u0026ldquo;Your refund has been processed.\u0026rdquo; At least four realities are possible:\nthe refund exists and the response is correct; the tool failed but the agent claimed success; the refund succeeded twice because a retry was not idempotent; the refund succeeded, but the agent skipped identity verification. The final text is nearly identical. The product outcomes are not.\nAnthropic defines a transcript as the complete record of outputs, tool calls, reasoning, and intermediate results, while the outcome is the final environment state. OpenAI\u0026rsquo;s trace-grading guidance similarly treats the end-to-end log of decisions and tool calls as the object to grade. This is more than observability. It changes what can be tested.\nA minimum useful trace looks like a span tree:\nagent_run |- user_request |- model_generation | |- retrieval | |- memory_read | `- tool_call | `- tool_result |- state_update |- model_generation | `- tool_call | `- tool_result |- final_response `- business_outcome Figure 4: Final-answer grading sees the right edge. Diagnosis requires the path and the external state created along it. Each span should carry enough provenance to answer three questions: what happened, under which system version, and what evidence may be safely retained. A practical schema includes:\ntrace: trace_id: tr_7f2a task_id: refund_duplicate_charge started_at: 2026-07-21T09:13:22Z agent: model: model_name_and_snapshot prompt_version: support-v37 harness_version: git:8ea41c2 tool_schema_version: billing-api-v12 environment: fixture_version: retail-sim-2026-07 initial_state_hash: sha256:... spans: - type: tool_call name: process_refund arguments_ref: encrypted://... result_ref: encrypted://... latency_ms: 812 outcome: final_state_hash: sha256:... refund_count: 1 identity_verified: true OpenTelemetry\u0026rsquo;s GenAI semantic conventions now define common attributes for agent identity, conversations, model requests, retrieval, tool arguments, tool results, and evaluation scores. That standardization is useful, but the specification also warns that prompts, retrieved content, tool arguments, and results may contain sensitive data.\nTherefore, \u0026ldquo;trace everything\u0026rdquo; is not a production design. Evidence collection requires a data policy:\nredact secrets and direct identifiers before export; store large or sensitive payloads by encrypted reference, not as unrestricted span attributes; record hashes and schema versions even when content cannot be retained; define retention by risk and review value; replay production evidence only inside isolated environments; separate access to telemetry metadata from access to raw user content. A trace without content may be insufficient for semantic diagnosis. A trace with all content may be a privacy incident. The evidence contract must state which claims remain possible after redaction.\nFailure Location Before Root Cause A trace helps localize where observed behavior diverged, but it does not automatically identify root cause. \u0026ldquo;The wrong tool was called\u0026rdquo; is a failure location. It could be caused by an ambiguous user goal, misleading retrieval, overlapping tool descriptions, a stale schema, a planner error, or a policy bug.\nA useful first-pass taxonomy follows the state transitions of the system:\nGoal interpretation: the agent optimized the wrong objective. Context acquisition: retrieval or memory supplied missing, stale, or misleading evidence. Planning: the chosen sequence could not satisfy the goal or policy. Tool selection: the agent chose the wrong capability. Argument construction: the right tool received invalid or unsafe parameters. Execution: the environment timed out, returned partial results, or violated assumptions. State management: the agent lost, duplicated, or contaminated persistent state. Recovery: a recoverable error became a terminal or repeated failure. Response grounding: the final claim did not match the realized state. This taxonomy should point to an owner and a candidate fix, not merely decorate a dashboard. The root cause remains a hypothesis until a controlled change or replay distinguishes competing explanations.\nJudgment: A Grader Is a Measurement Instrument A grader should be treated as a measurement instrument with a known operating range, not as an oracle. Different evidence calls for different instruments.\nThree broad grader families cover most agent systems:\nDeterministic Graders Use code when the property can be encoded without semantic ambiguity. Examples include JSON-schema validity, database end state, unauthorized tool use, secret-pattern matches, step budgets, idempotency keys, unit tests, and latency limits.\nThese checks are cheap, reproducible, and easy to debug. Their weakness is brittleness: a rule may reject a valid alternative path or miss a semantically wrong action that satisfies the schema.\nModel-Based Graders Use an LLM judge for properties such as instruction fulfillment, evidence support, interaction quality, plan coherence, or whether a tool trajectory plausibly achieved an open-ended goal.\nThese graders handle variation, but they introduce a second stochastic model into the measurement chain. AgentRewardBench evaluated 12 LLM judges on 1,302 expert-annotated web-agent trajectories across five benchmarks. No single judge was best across all benchmarks, and no evaluated judge exceeded 70% overall precision for success prediction. The strongest reported GPT-4o variant reached 69.8% precision and 83.1% recall: roughly three in ten trajectories it labeled successful were not successful under expert review. The opposite error appeared in rigid rules. For GPT-4o acting on WebArena, experts measured 42.3% success while the official rule-based evaluator measured 25.6%. A flexible judge and a deterministic rule therefore fail in different directions; neither is a universal ground truth.\nHuman Graders Use domain experts for ambiguous, high-impact, novel, or disputed cases. Human review is not automatically ground truth: reviewers can disagree, become fatigued, or lack access to the actual environment state. But carefully designed expert labels remain the reference needed to calibrate automated graders.\nThe engineering answer is a cascade, not a winner:\ndeterministic invariant -\u0026gt; model-based semantic judgment -\u0026gt; disagreement / uncertainty detector -\u0026gt; expert review Figure 5: Automated judges scale review; expert labels establish and periodically re-check their decision boundary. Calibration Before Automation Before a judge can influence deployment, construct a calibration set that over-samples consequential boundary cases. For a binary property, report the confusion matrix rather than reporting only agreement or correlation:\nExpert: fail Expert: pass Judge: fail true fail false fail Judge: pass false pass true pass The two errors have different costs. A false fail wastes review and blocks valid changes. A false pass lets a regression through. For an irreversible financial action, false-pass risk dominates. For a style preference, false-fail cost may dominate.\nCalibration must therefore be slice-specific and cost-sensitive. Suppose an intentionally hard refund-policy calibration slice contains 200 expert-labeled traces, including 40 actual policy failures. If the judge passes six of those failures, its failure recall is 85%, but the operational fact is sharper: 15% of known failures escaped the automated gate. Even a high aggregate agreement score cannot justify release authority if the tolerated false-pass rate for irreversible payments is 1%. The correct response is to tighten the rubric, add deterministic state checks, or route this slice to expert review rather than average it with easy cases.\nA production judge contract should record:\nthe exact rubric and judge model version; which information is available to the judge but not to the acting agent; precision and recall on each risk slice, not only aggregate accuracy; an abstention region for insufficient or conflicting evidence; the sampling policy for human review; recalibration triggers after prompt, model, tool, or data-distribution changes. Information parity deserves special attention. If the judge sees a reference answer or future environment state unavailable to the agent, it may identify failure but cannot fairly assess whether the action was reasonable at the time. Conversely, if the judge cannot see tool outputs or final state that the agent used, it may confidently grade the wrong object.\nThe practical rule is:\nA judge may have broader evidence than the agent, but the rubric must distinguish outcome correctness from decision quality given the information available at that step.\nStochastic Regression Requires Statistical Decisions An agent test is a random variable, not a Boolean fact. One successful replay proves only that one sampled trajectory succeeded.\nThe distinction between $\\text{pass@}k$ and $\\text{pass}^k$ makes this intuitive. If a trial succeeds independently with probability $p$:\n$$ \\text{pass@}k = 1 - (1-p)^k, $$$$ \\text{pass}^k = p^k. $$$\\text{pass@}k$ asks whether at least one of $k$ attempts succeeds. It rises with more attempts. $\\text{pass}^k$ asks whether all $k$ attempts succeed. It falls with more attempts.\nAt $p=0.75$, three attempts produce $\\text{pass@}3 \\approx 98.4\\%$ but $\\text{pass}^3 \\approx 42.2\\%$. The first metric fits search or candidate generation, where one good result may be enough. The second fits customer-facing workflows that must behave correctly every time.\nFigure 6: More attempts make discovery look better and repeated reliability look worse. The curves assume independent trials with per-trial success probability 0.75. This has a direct implication for continuous integration: a single replay cannot establish non-regression for a stochastic agent.\nLet $p_b$ be the baseline success probability and $p_c$ the candidate probability on the same regression slice, with $\\Delta=p_c-p_b$. Define the acceptable regression budget as $\\delta$. A non-inferiority gate should permit shipping only when the lower confidence bound for the difference remains above the allowed loss:\n$$ \\operatorname{LCB}_{1-\\alpha}(\\Delta) \u003e -\\delta. $$The intuition is simple: do not ask, \u0026ldquo;Did the candidate score higher?\u0026rdquo; Ask, \u0026ldquo;Does the evidence rule out a harmful degradation larger than our budget?\u0026rdquo;\nThis naturally creates three verdicts:\nPASS: evidence rules out an unacceptable regression. FAIL: evidence supports an unacceptable regression or a hard invariant was violated. INCONCLUSIVE: the sample is too small or too noisy; run more trials or request review. A Paired Non-Inferiority Example Consider 160 replay fixtures, each run once with the baseline and once with the candidate under matched environment conditions. The paired outcomes are:\nCandidate pass Candidate fail Baseline pass 103 11 Baseline fail 5 41 Only the discordant pairs identify a change: the candidate fixes five baseline failures but introduces 11 new failures. The estimated paired difference is\n$$ \\widehat{\\Delta} = \\frac{5-11}{160} = -0.0375. $$Using a normal approximation for illustration, the 95% paired interval is approximately $[-8.6, 1.1]$ percentage points. If the predeclared regression budget is $\\delta=2$ points, the result is INCONCLUSIVE: the lower bound does not establish non-inferiority, but the upper bound does not establish a harmful regression either. Shipping because a conventional two-sided difference test is \u0026ldquo;not significant\u0026rdquo; would confuse absence of evidence with evidence of non-inferiority.\nFigure 7: A predeclared non-inferiority margin turns uncertainty into three actions. The middle interval is the 160-pair worked example; the other rows are illustrative decision cases. For paired binary outcomes, inference should use methods designed for matched proportions, such as an appropriate paired interval or bootstrap over independent fixtures. The pairing matters: it controls task difficulty and focuses the estimate on cases where behavior changed. The confidence procedure must match the design rather than treating 320 runs as independent Bernoulli samples.\nThe independence assumption is also easy to break. Repeated runs that share a mutable sandbox, cached retrieval, user-simulator state, or a seed family are clustered observations. In that case, resample at, or explicitly model, the independent unit, usually the task-environment fixture, rather than individual trajectories. Otherwise the interval will be too narrow.\nTwo more rules prevent statistical theater:\nPredeclare sequential decisions. Recomputing an ordinary 95% interval after every batch and stopping when it passes inflates the false-positive rate. Use a fixed maximum budget with an alpha-spending design, a sequential probability ratio test, or another anytime-valid procedure. Protect risk slices from aggregation. Testing many tools, segments, and failure modes creates multiple-comparison risk. Critical invariants should remain separate gates; exploratory slices need a declared family-wise error-rate or false-discovery-rate policy. A global average must never compensate for a known critical regression. There is no universal value of $k$. The trial budget depends on the smallest harmful regression $\\delta$, baseline reliability, desired confidence and power, clustering, and decision cost. Estimate those quantities before the run; do not choose eight trials merely because an example configuration contains eight.\nAgentAssay, a 2026 technical report focused on stochastic agent regression testing, formalizes this three-valued approach and CI/CD gates as statistical decision procedures. Across 7,605 trials, it reports that sequential testing reduced required trials by 78%, while behavioral trace fingerprints detected regressions with 86% power in a setting where binary pass-rate testing detected none. These are early technical-report results rather than a settled industry standard, but they expose the right problem: binary outcomes can miss behavioral drift, and fixed trial counts waste budget on stable cases.\nFor production systems, the minimum statistical discipline is:\npair baseline and candidate runs on the same task and environment fixture; run multiple trials for stochastic properties; stratify by risk and failure type before aggregating; report uncertainty, not only a point estimate; reserve zero-tolerance gates for deterministic invariants with reliable observability; return INCONCLUSIVE instead of converting weak evidence into false certainty. Coverage Is a Model of the Failure Surface Evaluation coverage measures the ways in which the system can fail, not how many prompts are in a dataset. A suite of 10,000 near-duplicate happy paths may cover less than 50 carefully selected state transitions.\nA useful coverage model for agents is multi-dimensional:\n$$ \\begin{aligned} \\mathcal{C} = (\u0026C_{\\text{goal}}, C_{\\text{tool}}, C_{\\text{path}}, C_{\\text{state}},\\\\ \u0026C_{\\text{boundary}}, C_{\\text{environment}}, C_{\\text{risk}}). \\end{aligned} $$In plain language:\nGoal coverage: which user intents and ambiguity patterns are represented? Tool coverage: which capabilities, argument classes, failures, and retries are exercised? Path coverage: which meaningful action sequences and handoffs occur? State coverage: which initial, intermediate, and corrupted states are tested? Boundary coverage: which thresholds, permissions, budgets, and policy edges are crossed? Environment coverage: which latency, partial failure, schema, and dependency conditions appear? Risk coverage: which severity levels and affected user segments are represented? AgentAssay proposes a related five-dimensional tuple over tools, paths, states, boundaries, and models. The important idea is that aggregate task count is not a defensible coverage metric.\nCoverage becomes actionable when connected to mutation testing. If you deliberately remove a required identity check, corrupt a tool result, rename a schema field, duplicate a retry, or inject stale memory, does at least one test or grader reject the run? If not, the suite has a blind spot even when every existing test passes.\nThis borrows a mature idea from software testing: mutation testing evaluates a test suite by injecting plausible faults and measuring whether tests detect them. The classic survey by Jia and Harman treats mutation score as evidence about fault-detection effectiveness, not as proof of correctness. Agent systems extend the mutation surface beyond code into prompts, context, tools, state, orchestration, and models.\nMetamorphic testing adds a second useful technique when no single output is canonical. Instead of asserting one exact trajectory, specify relations that should remain true across transformed inputs. For example, paraphrasing a refund request should not change eligibility; reordering irrelevant retrieved documents should not change the cited evidence; lowering a requested refund below an approval threshold may remove an approval step but must not remove identity verification. These relations test behavioral invariants without forcing one brittle path.\nUseful agent mutations include:\nprompt mutation: remove or contradict a key instruction; tool mutation: change a schema, permission, timeout, or return value; context mutation: omit or duplicate a retrieved fact, replace it with a stale version, or inject poisoned context; state mutation: start from a partial or conflicting environment state; orchestration mutation: reorder steps, skip approval, or repeat an action; model mutation: change the provider, checkpoint, or sampling policy. The goal is not exhaustive Cartesian-product testing. It is a risk-weighted map showing which important failure mechanisms have evidence and which remain assumptions.\nMemory: A Failure Must Become a Versioned Asset Regression memory is a versioned collection of failure cases, expected behavior, replay fixtures, graders, and decision policies. A trace becomes memory only when a future system can use it to prevent recurrence.\nA screenshot in an incident ticket is not regression memory. Neither is a trace URL that expires in 30 days. The minimum durable object looks like this:\nregression_case: id: REG-2026-0142 created_from: trace_id: tr_7f2a incident_id: INC-884 failure: type: skipped_identity_verification severity: critical failed_transition: verify_identity -\u0026gt; process_refund invariant: \u0026#34;Refund requires verified customer identity\u0026#34; replay: inputs_ref: fixtures/refund-duplicate-charge.json initial_state_ref: snapshots/retail-sim-2026-07.tar.zst expected_outcome: refund_count: 1 identity_verified: true provenance: model_version: model-snapshot prompt_version: support-v37 harness_version: git:8ea41c2 tool_schema_version: billing-api-v12 grader_version: identity-policy-v4 evaluation: trials: 8 scorers: - identity_verified_state_check - no_duplicate_refund_check - response_grounded_in_outcome_judge authority: policy: block_release owner: payments-reliability Figure 8: Regression memory is curated, not automatic. Noise and grader errors are not promoted; validated failures become durable release evidence. The promotion process matters. Not every bad-looking trace should become a permanent test:\nDetect a user complaint, outcome anomaly, judge failure, or deterministic violation. Triage whether it is a product defect, expected variation, infrastructure noise, or grader error. Minimize the case while preserving the failing state transition. Specify the invariant or expected outcome in reviewable language. Reproduce the failure against the original version in an isolated fixture. Validate that the proposed fix passes without weakening unrelated regression slices. Promote the case with an owner, retention policy, and release authority. Retire or migrate it only when the product contract changes explicitly. This curation prevents two pathologies. The first is noise accumulation, where flaky or ambiguous cases make the suite untrusted. The second is policy fossilization, where outdated behavior remains enforced because nobody recorded why a case existed.\nOpenAI describes a concrete production version of this loop in its work on self-improving tax agents: practitioner corrections are compared with system outputs, recurring differences are grouped to separate product failures from workflow noise, actionable patterns become targeted evals, and fixes are validated against broader regression suites. The key design choice is that a correction does not automatically become a task. It first becomes reviewed evidence.\nWeng\u0026rsquo;s harness framework pushes the same principle into self-improving systems: negative results are retained, failures are grouped by causal mechanism, proposed harness edits state both the failures they are expected to fix and the regressions they may introduce, and held-out evaluators remain outside the editable loop. Regression memory supplies the release contract for that architecture. A self-generated patch is still only a candidate until independent evidence shows that it repaired the target mechanism without exceeding the risk budget elsewhere.\nAuthority: Not Every Score Should Block a Release Evaluation authority is the explicit mapping from evidence quality and risk to an engineering action. Without this layer, evals become dashboards that teams consult when convenient.\nThe strongest available signal should not automatically receive the strongest authority. Authority should depend on observability, calibration, reversibility, and expected loss.\nA practical release policy has four levels:\nHard Block Use for deterministic, high-impact invariants with low evaluator ambiguity: unauthorized transfers, missing required approval, schema corruption, secret exposure, duplicated irreversible actions, or failure of executable acceptance tests.\nStatistical Block or Review Use for stochastic properties evaluated through repeated trials and a predeclared non-inferiority gate. High-impact slices route inconclusive results to expert review instead of silently passing.\nWarning and Canary Use for calibrated but subjective quality signals, cost increases, trajectory inefficiency, and new failure clusters. Deploy to a limited traffic slice, compare business outcomes, and preserve rollback capability.\nShadow Monitoring Use for new graders, weak labels, novel anomaly detectors, and signals without measured false-pass rates. Record the action the evaluator would have taken, but do not let it control users yet.\nFigure 9: Measurement defensibility and consequence are separate axes. High consequence with weak evidence requires human review, not stronger automated authority. An implementation can encode the policy directly:\ndef decide_release(candidate, baseline, cases): results = replay_paired(candidate, baseline, cases) if results.any_deterministic_invariant_violation(): return \u0026#34;BLOCK\u0026#34; for risk_slice in results.slices(): delta_ci = risk_slice.non_inferiority_interval() budget = allowed_regression[risk_slice.severity] if delta_ci.upper \u0026lt; -budget: return \u0026#34;BLOCK\u0026#34; if delta_ci.lower \u0026lt;= -budget: return \u0026#34;HUMAN_REVIEW\u0026#34; # Evidence is inconclusive. if results.judge_drift_detected(): return \u0026#34;SHADOW_ONLY\u0026#34; if results.quality_warning() or results.cost_slo_regressed(): return \u0026#34;CANARY\u0026#34; return \u0026#34;SHIP\u0026#34; The interval estimator must match the experimental design, but the decision rule must also be declared before the candidate is examined. Otherwise, teams move thresholds until a preferred release passes.\nOffline eval, CI gating, and online monitoring also have different jobs:\nOffline evaluation compares candidate behavior on controlled, replayable evidence. CI gating converts a subset of that evidence into a pre-deployment decision. Online monitoring discovers distribution shift, unknown failures, and real business outcomes. LangSmith and MLflow both expose this two-way pattern: production traces feed datasets and evaluators; offline experiments test candidate changes; online scoring monitors what controlled suites miss. The architecture matters more than the vendor.\nWorked Example: From Duplicate Refund to Release Decision The refund scenario can now be carried through the complete loop.\n1. Evidence. A production trace shows that the agent called process_refund, received a timeout, retried without an idempotency key, and produced two successful ledger entries. Its final response claimed one refund. The failed span is not the response generation; it is the transition from an uncertain tool outcome to an unreconciled retry. The trace retains tool request IDs, timeout metadata, ledger state, model and prompt versions, and the approval decision, with customer identifiers redacted.\n2. Judgment. Two deterministic graders assert refund_count == 1 and that the same idempotency key is reused across retries. A semantic judge checks whether the final response is grounded in the ledger state, but it does not decide whether money moved twice. On a labeled slice of timeout and retry traces, the semantic judge\u0026rsquo;s false-pass rate is measured separately from non-timeout refunds. Disagreements and abstentions go to payments experts.\n3. Memory. Triage rules out a duplicated webhook and reproduces the defect in an isolated billing simulator. The minimal fixture contains one eligible order, a first call that commits but times out before acknowledgment, and a retry. The durable case records the invariant, failed transition, simulator snapshot, relevant versions, grader contracts, owner, and why the test exists. A prompt-only fix that merely changes the final wording cannot pass the state grader.\n4. Candidate comparison. The proposed harness change adds an idempotency key and a read-after-timeout reconciliation step. The baseline and candidate are run on matched copies of the fixture, including timeout timing variants. The critical duplicate-refund invariant has zero tolerance: a single reproducible violation blocks release. Recovery success is stochastic, so it uses a paired non-inferiority interval and returns PASS, FAIL, or INCONCLUSIVE. The broader suite also checks that reconciliation does not create unacceptable latency or block legitimate second refunds.\n5. Authority and online confirmation. A candidate that violates the state invariant is blocked regardless of aggregate score. An inconclusive recovery result triggers additional trials or expert review; it is not silently passed. A statistically acceptable candidate enters a low-volume canary deployment with duplicate-ledger monitoring and automatic rollback. Only after the online outcome matches the offline contract does the change graduate, while the original incident remains a permanent regression asset.\nThis example exposes why an eval is not merely a scorer. The useful artifact is the chain of custody that connects production state to a calibrated decision, a replayable case, and explicit deployment authority.\nA Minimum Viable Agent Evaluation Loop A small team does not need hundreds of tasks or a dedicated evaluation platform to begin. It needs one complete feedback loop with explicit ownership.\nAnthropic recommends starting with 20-50 tasks drawn from manual checks, support queues, and real failures. That is enough when early product changes have large effects. The mistake is waiting for a perfect dataset while production incidents continue to disappear into tickets.\nThe minimum viable system has seven components:\nStructured traces and outcome capture for every high-value workflow. A small failure taxonomy aligned with components and engineering owners. Deterministic checks first, especially for state, permission, and schema invariants. One or two calibrated semantic judges for properties code cannot express. A reviewed regression-case store with fixtures and complete version lineage. Repeated paired trials and three-valued statistical verdicts for stochastic cases. An explicit release policy separating block, review, canary, and shadow signals. A tool-neutral configuration might look like this:\nagent_eval_loop: evidence: required_spans: [request, generation, tool_call, tool_result, outcome] redact: [credentials, direct_identifiers] graders: deterministic: - tool_schema_valid - no_unauthorized_action - final_claim_matches_state model_based: - task_completion_v3 - tool_path_quality_v2 human_review_when: - judge_disagreement - judge_abstention - critical_novel_failure regression: paired_trials: 8 verdicts: [pass, fail, inconclusive] stratify_by: [failure_type, severity, tool, user_segment] release: block_on: [deterministic_invariant, confirmed_critical_regression] review_on: [inconclusive_high_risk] canary_on: [quality_or_cost_warning] The first useful dashboard is not a single score. It is a queue of decisions:\nnew production failures awaiting triage; regression cases with no current owner; candidate changes with inconclusive evidence; judge slices whose calibration has drifted; high-risk failure modes without coverage; cases that have not been replayed against the current model, prompt, and tool schema. That queue tells the team what it does not know.\nMy Take: Evaluation Is the Reliability Memory of the Agent Team The industry often describes evals as \u0026ldquo;unit tests for AI.\u0026rdquo; The analogy is useful but incomplete. Unit tests assume relatively stable inputs, deterministic execution, explicit assertions, and code-controlled state. Agents violate all four assumptions.\nThe closer analogy is a reliability system that combines distributed tracing, incident review, statistical quality control, test engineering, and release management.\nI expect mature agent teams to develop three practices.\nFirst, eval engineers will own measurement contracts, not prompt collections. Their work will include trace semantics, environment fixtures, judge calibration, coverage analysis, and decision thresholds.\nSecond, production incidents will be considered unresolved until they create durable evidence. Fixing the prompt is not closure. Closure means the failure is reproducible, the expected behavior is explicit, the fix survives the broader suite, and recurrence can influence release decisions.\nThird, release confidence will become risk-sliced rather than score-based. A two-point aggregate gain will not offset one newly introduced critical failure. Different slices will have different regression budgets, trial counts, graders, and authorities.\nThe deepest shift is organizational. An individual model does not learn from a production incident unless it is retrained. The engineering system can learn immediately by changing what it records, tests, and refuses to ship.\nThat is why regression memory is a better mental model than benchmark optimization:\nThe agent may forget. The system must not.\nCitation Please cite this work as:\nXueping Gao. \u0026ldquo;Agent Evaluation as Regression Memory.\u0026rdquo; hellogxp.github.io (July 2026). https://hellogxp.github.io/posts/agent-evaluation-regression-memory/\n@article{gao2026regressionmemory, title = {Agent Evaluation as Regression Memory}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {July}, url = {https://hellogxp.github.io/posts/agent-evaluation-regression-memory/} } References [1] Yao, S., Shinn, N., Razavi, P., \u0026amp; Narasimhan, K. \u0026quot;$\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.\u0026quot; arXiv:2406.12045 (2024).\n[2] Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., \u0026amp; Neubig, G. \u0026ldquo;WebArena: A Realistic Web Environment for Building Autonomous Agents.\u0026rdquo; arXiv:2307.13854 (2024).\n[3] Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., \u0026amp; Yu, T. \u0026ldquo;OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.\u0026rdquo; arXiv:2404.07972 (2024).\n[4] Mohammadi, M., Li, Y., Lo, J., \u0026amp; Yip, W. \u0026ldquo;Evaluation and Benchmarking of LLM Agents: A Survey.\u0026rdquo; KDD 2025.\n[5] Lu, X. H., Kazemnejad, A., Meade, N., Patel, A., Shin, D., Zambrano, A., Stanczak, K., Shaw, P., Pal, C. J., \u0026amp; Reddy, S. \u0026ldquo;AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories.\u0026rdquo; arXiv:2504.08942 (2025).\n[6] Bhardwaj, V. P. \u0026ldquo;AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows.\u0026rdquo; arXiv:2603.02601 technical report (2026).\n[7] Anthropic. \u0026ldquo;Demystifying Evals for AI Agents.\u0026rdquo; Anthropic Engineering (2026).\n[8] OpenAI. \u0026ldquo;Trace Grading.\u0026rdquo; OpenAI API Documentation (2026).\n[9] OpenAI. \u0026ldquo;Building Self-Improving Tax Agents with Codex.\u0026rdquo; OpenAI (2026).\n[10] OpenTelemetry. \u0026ldquo;Semantic Conventions for Generative AI.\u0026rdquo; OpenTelemetry Semantic Conventions (2026).\n[11] LangChain. \u0026ldquo;LangSmith Evaluation.\u0026rdquo; LangSmith Documentation (2026).\n[12] MLflow. \u0026ldquo;Evaluating Production Traces.\u0026rdquo; MLflow Documentation (2026).\n[13] Weng, L. \u0026ldquo;Harness Engineering for Self-Improvement.\u0026rdquo; Lil\u0026rsquo;Log (2026).\n[14] Jia, Y., \u0026amp; Harman, M. \u0026ldquo;An Analysis and Survey of the Development of Mutation Testing.\u0026rdquo; IEEE Transactions on Software Engineering 37(5), 649-678 (2011).\n[15] Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., \u0026amp; Zhou, Z. Q. \u0026ldquo;Metamorphic Testing: A Review of Challenges and Opportunities.\u0026rdquo; ACM Computing Surveys 51(1) (2018).\n[16] Fay, M. P., \u0026amp; Lumbard, K. \u0026ldquo;Confidence Intervals for Difference in Proportions for Matched Pairs Compatible with Exact McNemar\u0026rsquo;s or Sign Tests.\u0026rdquo; Statistics in Medicine 40(5), 1147-1159 (2021).\n","permalink":"https://hellogxp.github.io/posts/agent-evaluation-regression-memory/","summary":"Benchmarks compare agents. Production evaluation must remember failures. A research-informed practice guide to traces, calibrated graders, stochastic regression tests, coverage, and release gates for AI agents.","title":"Agent Evaluation as Regression Memory"},{"content":"AgentLens trains a linear probe on a coding agent\u0026rsquo;s step-level hidden states, achieving 99.15% detection accuracy on LLaMA-3.1-8B and reducing attack success rate from 85.99% to 13.36%. Meanwhile, Neural Chameleons reveals a more unsettling number: a safety probe with AUROC of 1.0—perfect separation—drops to 0% TPR when a single trigger phrase, \u0026ldquo;You are being probed for deception,\u0026rdquo; is added to the prompt.\nA near-perfect monitor, blinded by one line of text.\nThe tension between these two numbers is the starting point of this article. They seem contradictory, but they point to the same question: what does the activation signal you\u0026rsquo;re reading actually mean?\nAgentLens\u0026rsquo;s 99.15% proves decodability—safety labels are linearly readable from hidden states. Neural Chameleons\u0026rsquo;s 0% TPR proves the boundary of conditional robustness—a decodable signal on a fixed checkpoint does not guarantee a reliable mechanism against a trainable actor. Between these two, there are at least two more levels of claims that are frequently conflated:\nDecodability ≠ Prediction ≠ Causation ≠ Authority.\nBeing able to decode a label from hidden states does not mean it predicts behavior ahead of time; predicting ahead does not mean the model relies on that direction to make decisions; even if a causal representation is found, it does not mean it should hold the authority to execute or permit actions.\nThis is not to dismiss internal monitoring. AgentLens pushes mechanistic interpretability from single-turn QA to multi-step agent trajectories, Role Confusion achieves 60% ASR with CoT Forgery on GPT-5 series and predicts attack success before generation, TRACES uses observer LLM hidden states for proactive trajectory auditing, outperforming the strongest baseline by 19.3 points on ATBench. Meanwhile, OpenAI monitors tens of millions of internal coding agent trajectories with GPT-5.4 Thinking, DeepMind analyzes 1 million coding agent tasks, and Anthropic\u0026rsquo;s sandbox runtime reduces permission prompts by 84%. Research and industry are simultaneously validating the same thing: agent internal representations do contain safety signals, but three layers of evidence separate signal from control.\nThis article builds the complete reasoning chain—from where state lives, to what evidence supports what strength of claim, to how to connect probabilistic monitors to capability, provenance, and runtime enforcement.\nThe core conclusion up front:\nInternal monitors can be valuable sensors, but they cannot be the source of authority. Their value depends not on how mysterious the hidden state is, but on how much measurable control they add at the right target, time, and baseline.\nFigure 1. Residual activations normally end with each invocation; serialized history, workspace state and authority boundaries continue across the Agent loop. An Agent Is a System, Not a Single Forward Pass A real agent is not an isolated transformer forward pass, but a loop that repeatedly reads external state, calls the model, executes tools, and writes back to the environment. Before discussing hidden states, we must first establish where state lives.\nStarting from Security Properties \u0026ldquo;Detecting whether an agent is dangerous\u0026rdquo; is not a good enough technical goal. Danger can mean malicious input, a risky task, a refusal or deception the model has already generated, the next tool call, or a realized data breach. These labels can be highly correlated but are not interchangeable.\nOur running example has one clear task:\nFix a flaky retry test in a temporary checkout and prepare a patch. Do not push, do not access the network, do not read credentials, and do not modify CI or security configuration.\nFrom this, we can write concrete security properties:\nPrincipal: the authenticated developer who initiated the task; Granted authority: read repository, modify task-related source/tests, run local tests; Untrusted data: issues, PR content, code comments, test logs, and repository instruction files; Protected assets: credentials, CI policy, internal network, protected branch; Violation predicate: untrusted repository content causing the agent to execute actions beyond the user\u0026rsquo;s explicit authorization; Enforcement points: shell gateway, filesystem policy, network sandbox, diff review, commit gate. The key insight is not about coding. The real security invariant is:\nUntrusted data may influence computation, but it may not expand authority.\nWithout first writing out principal, assets, allowed flows, and violation predicates, the subsequent \u0026ldquo;risk\u0026rdquo; is just a set of mixed labels. A probe might primarily identify the task topic but be interpreted as predicting future behavior; it might primarily detect refusal phrasing but be interpreted as discovering hidden intent.\nHow State Traverses Model Calls A real agent can be written as the following loop:\n$$S_t \\xrightarrow{B} C_t \\xrightarrow{f_\\theta} a_t \\xrightarrow{T} E_{t+1} \\xrightarrow{U} S_{t+1}.$$Here, context builder $B$ constructs model context $C_t$ from system state $S_t$; the model produces action proposal $a_t$; tool/runtime $T$ changes the environment; memory update $U$ writes results back to the next system state.\nThe unit of analysis is not an isolated transformer forward, but:\nthe model–memory–tool–environment loop.\nAt least seven state substrates exist in this loop.\nResidual activation. Exists within a single model call, can determine the next token or tool name, but typically does not automatically cross into the next independent call.\nGenerated token. The model serializes plans, judgments, and tool arguments into tokens. They enter history and become input for the next call.\nKV cache. Carries computational state when reused, but standard autoregressive cache is typically equivalent to recomputation on the same context. It is an important computational carrier but does not automatically constitute new semantic memory.\nSerialized context. Conversation history, scratchpad, tool trajectory, and repository instructions are often the primary state channel between calls.\nExplicit memory. Summaries, vector stores, databases, or graphs can persist across sessions, and can also be poisoned.\nEnvironment state. Workspace, git diff, browser, mailbox, IAM, and cloud resources are farthest from the transformer but typically closest to real safety outcomes.\nParameter state. Fine-tuning, adapters, or online updates change the model itself, with lifetimes far longer than a single task.\nA counterintuitive conclusion: lifetime and causal importance are not the same thing. A transient activation can determine the next dangerous command; a long-lived memory entry can have zero influence on current behavior.\nMore importantly, a continuous risk curve does not prove that the same neural state persists. Each call may reconstruct a similar representation from history that contains more risk evidence. Predictive state can be rebuilt without being physically stored; but if we claim a specific activation mediates subsequent calls, we must identify its causal carrier across the call boundary.\nFigure 2. State lifetimes, owners, and control points differ; \"internal\" is not a single location. Decodable State Is Not Necessarily a Useful Safety Mechanism A high AUROC proves that a label is decodable, not that it provides incremental value beyond existing information, and certainly not that it is a causal mechanism. This chapter disentangles the conflation.\nFirst Ask What the Label Represents Malicious input, current concept, already-generated behavior, future tool call, and realized harm are different targets. A monitor\u0026rsquo;s name cannot be stronger than its label.\nAt minimum, distinguish: prompt end, early generation, before tool name, before tool commit, full response, and environment transition. Detecting deception after reading a full deceptive output is valuable auditing, but not early warning.\nActing model activations, observer LLM activations, serialized trajectories, git diffs, and environment state support different claims. An observer can be the best engineering monitor but cannot thereby be called the acting agent\u0026rsquo;s internal mechanism.\nThe Key Metric Is Not Activation AUROC, But Conditional Increment AgentLens achieves 99.15% detection accuracy on LLaMA-3.1-8B, outperforming Claude Sonnet 4.6 (95.31%) and LLaMA-Guard-4-12B (94.03%). But looking at the baseline table closely, a key issue emerges: Gemini-3-Pro achieves 96.16% on current-step detection, and Claude Sonnet 4.6\u0026rsquo;s F1 is 95.58%. A text-only LLM-as-judge can already reach 95%+, while AgentLens\u0026rsquo;s linear probe hits 99.15%—an increment of less than 3 percentage points.\nMore notable is lookahead detection. Here, Gemini-3-Pro\u0026rsquo;s accuracy on LLaMA-3.1-8B drops to 7.69%—near random. AgentLens\u0026rsquo;s lookahead accuracy is 92.31%. This is the real conditional increment: when the signal must appear early, the text discriminator collapses, while internal representation still carries predictive information.\nSimply put, we want to know: beyond the information you can already see, what does the activation additionally tell you? If a text-only classifier can already achieve 96% and an activation probe achieves 99%, the increment is 3 points—possibly not worth the cost of white-box access. But if the text classifier collapses to 7.69% on lookahead while activation maintains 92.31%, that increment is the real value.\nFor internal signal $H_t$, same-time observable information $O_t$, and real deployment group $G$, we care about a conditional predictive increment:\n$$\\begin{aligned} \\Delta_{H\\mid O,G,t} \u0026= \\operatorname{Perf}(O_t,H_t\\mid G) \\\\ \u0026\\quad - \\operatorname{Perf}(O_t\\mid G). \\end{aligned}$$This does not require activation to contain \u0026ldquo;information that is in principle impossible to obtain from text.\u0026rdquo; Under a fixed checkpoint, fixed visible prefix, and fixed extraction process, activation is a deterministic result of the computation. It may be valuable because it is more linearly readable, cheaper, earlier, or more robust to certain shifts. But the performance increment is not an information-theoretic proof, and these engineering advantages must each be measured separately.\nConsider TRACES: it uses observer LLM hidden states for trajectory-level proactive auditing, achieving EAUPC (Early Area Under the Prefix Curve) 19.3 points higher than the strongest baseline on ATBench. But TRACES reads the observer model\u0026rsquo;s activations, not the acting agent\u0026rsquo;s. An observer probe\u0026rsquo;s score can be very high without constituting evidence about the acting agent\u0026rsquo;s internal mechanism.\nRandom row splitting often allows the same semantic task, trajectory template, attack family, or repository pattern to appear in both training and test. The monitor may learn to identify scenarios rather than predict divergent behavior within the same scenario.\nScore Applicability Is Determined by Data Support and Adversary At minimum, declare a four-tier threat model:\nOccasional errors from a fixed actor; External attackers who adjust prompts; Attackers who know the monitor exists and optimize evasion; Adversarial actors who can train themselves and change internal representations. Neural Chameleons precisely calibrates the edge of this cliff. On Gemma-2-9b, the deception probe achieves AUROC of 1.0—perfect separation—without a trigger.\nBut after the model is fine-tuned via LoRA to perform low-dimensional activation shifts under trigger conditions, TPR drops from 60.6% to 0%, and the harmful probe drops from 50.2% to 0%.\nMore critically: this evasion is zero-shot transfer—the model was trained to evade probes for 11 benign concepts (HTML, Finnish, etc.), and safety concepts (deception, harmfulness) never appeared in training data, yet evasion succeeded completely. The evasion is also selective: triggering \u0026ldquo;deception\u0026rdquo; suppresses only the deception probe, not the harmful probe.\nThis does not prove activation monitoring is useless, but it precisely demonstrates that Tier-1 scores cannot automatically be upgraded to Tier-4 guarantees.\nFigure 3. Stronger claims require stronger intervention, control, and architectural evidence. Intervention Changes What We Can Claim We can classify internal state evidence into five levels.\nLevel 1 — Snapshot decodability. The current label can be read from the representation. This proves accessibility, not that the model uses this direction. AgentLens\u0026rsquo;s 99.15% sits here.\nLevel 2 — Future predictability. Before the action occurs, the representation can predict the subsequent outcome, and exceeds an observable comparator with the same information. AgentLens\u0026rsquo;s lookahead accuracy of 92.31% vs. Gemini-3-Pro\u0026rsquo;s 7.69% is evidence at this level.\nLevel 3 — Invocation-local intervention. A matched intervention on the representation changes the current model call\u0026rsquo;s task-state outcome. This requires random, orthogonal, wrong-layer, sign reversal, and authorized-task controls to avoid interpreting general disruption as a safety mechanism. Also prefer matched source activations, bidirectional interchange interventions, and distribution-distance diagnostics to check whether the intervention seriously deviates from the model\u0026rsquo;s natural distribution.\nLevel 4 — Cross-step mediation. The intervention effect passes through a named carrier across model call boundaries and influences subsequent behavior.\nLevel 5 — Scoped causal state abstraction. Within the declared horizon, outcome family, intervention family, and environment distribution, high-level variables maintain intervention correspondence with the low-level mechanism.\nThis ladder is not reinventing causal abstraction. It addresses an experimental obligation that is frequently skipped in agent research:\nA causal effect within the current call does not automatically prove a cross-call persistent mechanism.\nTo upgrade to Level 4, use the Cross-Boundary Carrier Test:\nSpecify the carrier: emitted plan, reused KV, memory write, or environment transition; Execute a matched intervention in the current call; Fix other possible channels; Verify that the next step\u0026rsquo;s behavior changes as predicted; Block, swap, or replace the carrier, and the effect should change accordingly. If the agent does stateless replay every time, and emitted output, next context, environment, and cache are all fixed, then old residual activations have no physical causal path to the next call. They can still be good predictors, but not cross-call mediators.\nAgentLens\u0026rsquo;s Causal Experiments: From Steering to Reverse Steering AgentLens provides a concrete case of moving from Level 2 to Level 3. It conducts two sets of causal experiments across three models:\nForward steering: When a harmful step is detected, a steering vector is injected along the probe-derived 10-dimensional subspace. LLaMA-3.1-8B\u0026rsquo;s ASR drops from 73.80% to 16.70%, Qwen-2.5-7B from 91.30% to 4.35%, Gemma-2-9B from 92.86% to 14.29%. Average ASR reduction is 72.63 percentage points.\nReverse steering (negative steering): Take 16 natural refusal steps and apply the same subspace intervention in the negative direction. At fixed $\\alpha=2.2$, 100% of refusals are flipped to harmful execution, with a collapse rate of only 6.25%. On Gemma-2-9B, 63% of 27 refusal steps are flipped.\nThis bidirectional effect is the core of Level 3 causal evidence—if the subspace only encoded lexical features of harmful commands, reversing the intervention should not turn an explicit safety refusal into a complete malicious execution chain. But steering produces an 18.54% average collapse rate (vs. 2.24% for vanilla), indicating that the intervention broadly disrupts generation quality while suppressing dangerous behavior. This reminds us: behavior change ≠ targeted safety intervention.\nA Notable Detection-Control Gap AgentLens reveals an interesting gap in the prompt injection setting: when the original task is benign but the environment observation contains an injected malicious goal, the probe fails to reliably trigger intervention—a detection failure under distribution shift. But even without probe triggering, manually applying the same probe-derived steering direction still reduces ASR from 86.7% to 6.7%, with a 0% collapse rate.\nThis shows detection and control can be separated: a subspace may be insufficient as a standalone detector but still provide effective behavioral control. For deployment, this means you can use a more reliable observer to decide when to trigger steering, rather than relying entirely on the probe\u0026rsquo;s own detection sensitivity.\nComplete Intervention Protocol Using before_tool_name as the anchor, construct two matched coding tasks: same repository, user goal, tool schema, action vocabulary, and length; the only difference is the authority source for protected-file write—one from explicit user authorization, the other from untrusted repository content.\nPerform bidirectional interchange: patch safe donor activations to unsafe base, and unsafe donor to safe base. Primary outcome is not refusal tone but the choice between request_confirmation and write_protected_file, tool-policy validity, and task completion rate. Include at least 20 norm-matched random directions, orthogonal, wrong-layer, wrong-anchor, unrelated donor, sign reversal, authorized risky task, malformed call, and capability-loss controls, and report whether activations deviate from natural distribution. Activation patching best practices and causal-probing completeness/selectivity analyses show that behavior change alone is insufficient to identify the target variable.\nIndustry Practice: What Three Labs Actually Do Research paper numbers answer \u0026ldquo;is it possible.\u0026rdquo; Engineering teams need to know \u0026ldquo;is someone doing this in production? How?\u0026rdquo; Three frontier labs have disclosed production practices.\nOpenAI: Async CoT Monitoring, Tens of Millions of Trajectories OpenAI uses GPT-5.4 Thinking at maximum reasoning effort to monitor 99.9% of internal coding agent traffic. The system reviews full conversation histories—including chain-of-thought, tool calls, and tool outputs—and completes review within 30 minutes. Over 5 months, it covered tens of millions of trajectories.\nResults: approximately 1,000 conversations triggered moderate severity alerts (many from internal red-teaming), 0 triggered the highest severity (coherent scheming). The monitor matched all employee-reported incidents and surfaced additional unreported suspicious behavior. The most notable behavior pattern: agents attempting base64 encoding to bypass content scanners, splitting payloads into smaller steps, and switching between download methods.\nOpenAI acknowledges two key limitations: the system is currently asynchronous (review within 30 minutes of session end), unable to block irreversible actions; and it cannot reliably quantify false-negative rates on open-ended traffic. They explicitly identify the next step as synchronous monitoring—evaluating and blocking before actions execute.\nDeepMind: 1 Million Task Analysis + AI Control Roadmap DeepMind built an async monitoring prototype and analyzed 1 million coding agent tasks. The most important finding: most flagged events are not adversarial intent but agent misinterpretation or overeagerness—agents misunderstanding user intent or pursuing intermediate goals too aggressively.\nDeepMind proposes D1-D4 detection tiers and R1-R3 response tiers. D1 = reading visible CoT (current), D3-D4 = analyzing model internals (future). Key design principle: \u0026ldquo;structural containment must be built before more capable models arrive.\u0026rdquo; This framework is already used for Gemini Spark agent\u0026rsquo;s real-time monitoring.\nAnthropic: Sandbox Runtime + 84% Permission Reduction Anthropic provides the most concrete numbers. Users approved approximately 93% of permission dialogs—approval fatigue is real. Their response: OS-level sandbox (macOS Seatbelt / Linux bubblewrap), network denied by default, writes confined to workspace. Result: permission prompts reduced by 84%, runtime open-sourced (srt).\nBut Anthropic also disclosed two failure cases: an internal red team phished an employee into having the agent read AWS credentials and POST them externally (24 out of 25 times succeeded); in the second case, the egress allowlist permitted api.anthropic.com, and an attacker used their own API key to exfiltrate data through that channel. Key lesson from their engineering blog: \u0026ldquo;prompts, classifiers, and domain allowlists are layers, not solutions. The deterministic boundary is what catches what every probabilistic defense misses.\u0026rdquo;\nFigure 4. Representation proximity and enforcement strength are two different axes. From Internal Evidence to Actionable Control Figure 5. Risk-to-response is a cascade, not a single threshold. Minimum Viable Safety Stack If you have a coding agent and limited engineering resources, deploy these three layers first:\nLayer 1: Execution isolation. The agent\u0026rsquo;s shell, filesystem, and network must run in an OS-level sandbox. Anthropic\u0026rsquo;s srt (Seatbelt / bubblewrap) is a ready-made solution. Docker also works, but srt is lighter. Network denied by default, writes confined to workspace. This layer does not depend on any AI judgment—it is a deterministic boundary.\nLayer 2: Effect-boundary checks. Deterministic checks before executing effectful actions. Credential files are unreadable (filesystem policy), external network is unreachable (egress control), protected branches cannot be pushed (git hook), CI config cannot be modified (path policy). These are pure rules, no probes or LLM judges involved.\nLayer 3: Trajectory observer. On top of effect boundaries, add an LLM-based observer that reviews the complete trajectory (including CoT). OpenAI\u0026rsquo;s GPT-5.4 Thinking does exactly this—async review within 30 minutes. For coding agents, this latency is acceptable because code still goes through review. No white-box activation access needed, just the transcript.\nThese three layers have increasing costs but also increasing protection. Internal probes (AgentLens-style) are an optional fourth layer—they require white-box access, are more expensive, but can provide earlier warning signals.\nDefense-in-Depth Cascade pseudocode def agent_safety_gate(action, trajectory, monitor_score): # Layer 1: Deterministic boundary (always enforced) if action.targets_protected_file(): return DENY # filesystem policy, no AI involved if action.requires_network() and not action.domain_in_allowlist(): return DENY # egress control if action.requires_credential_access(): return DENY # credential gate # Layer 2: Capability restriction based on risk if monitor_score \u0026gt; threshold_medium: action.remove_capability(\u0026#34;network\u0026#34;) action.require_confirmation(\u0026#34;shell\u0026#34;) # Layer 3: Observer review (asynchronous, for non-irreversible actions) if monitor_score \u0026gt; threshold_high: observer.review_async(trajectory, callback=escalate_human) # Layer 4: Internal probe (white-box only, if available) if has_activation_access and probe_score \u0026gt; threshold_probe: action.steer_representation(safe_direction) # Default: allow with logging log(action, monitor_score, probe_score) return ALLOW Core invariant:\nMonitor evidence may restrict authority. It must never grant authority by itself. Even if the monitor false-negatives, credential and network policies remain fail-closed.\nControl-Opportunity Lead Time \u0026ldquo;Early warning\u0026rdquo; cannot be measured solely by the number of tokens before the dangerous action. For production systems, what matters is: after the alert, how many control opportunities remain to change the outcome. Prompt-end alerts may allow replanning; pre-tool-name alerts may allow removing network capability; pre-tool-commit can still block protected-file writes; after side effects occur, the monitor can only support auditing.\n$$\\begin{aligned} \\operatorname{COLT}(m,t) \u0026= \\sum_{a\\in\\mathcal A_t} P(a\\text{ succeeds}\\mid m,t) \\\\ \u0026\\qquad \\cdot V(a). \\end{aligned}$$where $\\mathcal A_t$ is the set of still-executable responses (replan, restrict, confirm, block, rollback) and $V(a)$ is the value of loss avoided by that response. A slightly later but deterministic boundary check that reliably stops side effects may be superior to an earlier but high-false-positive alert with no corresponding response.\nDeployment Value: When Is It Worth It Deployment value can be written as:\n$$\\begin{aligned} V \u0026= \\mathbb{E}[L_{\\text{avoided}}] \\\\ \u0026\\quad {}- \\mathbb{E}[C_{\\text{disruption}}] \\\\ \u0026\\quad {}- C_{\\text{compute}} {}- C_{\\text{latency}} {}- C_{\\text{review}}. \\end{aligned}$$OpenAI\u0026rsquo;s production numbers provide a reference scale: 5 months, tens of millions of trajectories, ~1,000 moderate alerts. If your team runs 10,000 coding sessions/day, 0.2% trigger real violations, the monitor achieves 70% recall at 1% FPR, with 80% occurring at positions where block/replan is still possible—the example net value is positive. But if the violation base rate drops below 0.06%, or alerts arrive too late for effective control opportunities, the conclusion may reverse.\nDeepMind\u0026rsquo;s 1 million task analysis also provides a key data point: most flagged events are misinterpretation and overeagerness, not adversarial. This means the monitor\u0026rsquo;s operating point should be calibrated for \u0026ldquo;the agent misunderstood the task,\u0026rdquo; not solely for \u0026ldquo;someone is attacking.\u0026rdquo;\nFigure 6. Parameter sensitivity analysis, not measured results: low base risk or high FPR can make internal monitoring's expected value negative. From Shadow Mode to Enforcement Internal monitors should not jump from offline AUROC to production blocking. Five stages:\nDefine → Fix security property, anchor, comparator, threat tier; record checkpoint, tokenizer, scaffold version.\nOffline evaluation → Report fixed-FPR recall, calibration, latency—not just AUROC.\nShadow mode → Monitor only records \u0026ldquo;what response would have been taken,\u0026rdquo; does not change execution. Audit false positives and remaining control opportunities at alert time. OpenAI\u0026rsquo;s system currently operates here (async review within 30 minutes).\nCanary → Allow monitor to trigger replan or extra verification on reversible, low-impact actions; any effectful action still gated by deterministic policy.\nEnforcement → Only select thresholds under an explicit expected-loss budget; auto-degrade to shadow mode on checkpoint or scaffold drift.\nThe monitor itself is also an attack surface—observers can be prompt-injected, logs can leak data. Monitors should not have effectful tools by default, and their output should be treated as untrusted evidence. Capability policy must satisfy mediation completeness: shell, network, plugins, subagents, MCP, and other side channels must not bypass the same authority boundary.\nCross-Domain Decision Tree Different agent types need different safety stacks:\nDoes your agent have shell access? → Need sandbox (srt/Docker) + credential file policy + diff checker Does your agent have network access? → Need egress allowlist + provenance label on tool outputs Does your agent have memory persistence? → Need memory audit (what wrote, what read) + poison detection Is your agent closed-weight (API)? → Cannot do activation probe; use trajectory observer + trace monitor + environment validator Is your agent multi-agent? → Independent sandbox per sub-agent + trust escalation control (sub-agent output does not automatically gain higher trust) The safety invariant is cross-domain: email body text can influence a summary but cannot add recipients; web pages can influence search results but cannot grant login; supplier pages can influence procurement recommendations but cannot change payment accounts. What changes is the sensor and cost model; what doesn\u0026rsquo;t change is that untrusted content cannot expand authority.\nFigure 7. Cross-domain safety invariants: tools differ, but \"untrusted content cannot expand authority\" holds. Open Problems: What We Still Don\u0026rsquo;t Know Adaptive evasion. Neural Chameleons proved the feasibility of zero-shot probe evasion. The next question: can a monitor-aware actor systematically evade ensemble + non-linear monitors without significantly degrading task capability?\nCross-call causal carriers. We still lack a unified benchmark for verifying how a representation truly mediates the next call through KV, serialized plan, memory, or environment. AgentLens\u0026rsquo;s detection-control gap (probe didn\u0026rsquo;t trigger but steering was effective) hints at this pathway, but no one has systematically verified it.\nClosed-weight agents. Without activation access, how can observer, trace, and environment monitors share the same evidence language as white-box research? DeepMind\u0026rsquo;s AI Control Roadmap explicitly identifies \u0026ldquo;analyzing model internals\u0026rdquo; as a D3+ requirement, but closed-weight APIs don\u0026rsquo;t have this interface.\nMonitor drift. Any update to model, tokenizer, system prompt, tool schema, memory policy, or orchestration can change the monitor\u0026rsquo;s operating point. OpenAI acknowledges this: future models may compress or hide reasoning, making CoT monitoring ineffective.\nComplete mediation. Anthropic\u0026rsquo;s two failure cases (phishing → credential exfiltration, egress allowlist → data exfiltration) show that even when the sandbox works correctly, allowlists and proxies can be exploited. Whether shell, network, plugins, subagents, MCP, and other side channels all pass through the same non-bypassable authority boundary remains a more direct safety question than improving probe AUROC.\nConclusion + My Take Agent internal state is not a mysterious red button waiting to be discovered. It is a set of candidate variables distributed across model computation, serialized history, memory, tools, and environment. What we need is not to add another decimal to probe scores, but to know what claim each signal supports and whether it can create control value at the real authority boundary.\nUltimately, internal monitoring\u0026rsquo;s most appropriate position is not the root of trust, but a sensor in defense-in-depth that is earlier, richer, but still requiring calibration. Anthropic\u0026rsquo;s engineering lesson says it most clearly: \u0026ldquo;The deterministic boundary is what catches what every probabilistic defense misses.\u0026rdquo;\nMy Take. I believe three trends will emerge in the next 6 months. First, agent safety papers will shift from \u0026ldquo;can we detect\u0026rdquo; to \u0026ldquo;what does the detected signal mean\u0026rdquo;—AgentLens\u0026rsquo;s detection-control gap and Neural Chameleons\u0026rsquo;s zero-shot evasion have already put this question on the table. Second, circuit-level analysis will enter the agent domain—all existing work stays at the probe/subspace level, and no one has done activation patching + path patching on agent trajectories, which is a NeurIPS/ICLR-level gap. Third, industrial deployment will shift from \u0026ldquo;AI judge everything\u0026rdquo; to cascade architectures—OpenAI\u0026rsquo;s async CoT monitoring, DeepMind\u0026rsquo;s AI Control Roadmap, and Anthropic\u0026rsquo;s sandbox runtime are already going this way; academic probe papers need to catch up.\nIf you can take away one sentence from this article:\nYour 0.99 AUROC monitor might just be reading \u0026ldquo;ignore previous instructions\u0026rdquo; from the context. It\u0026rsquo;s a sensor, not an authority.\nPractical Toolkit: Seven-Question Review Card Figure 8. Seven questions to ask when evaluating any agent safety monitor. Target: What exactly is being detected? Timing: When is the evidence available? Owner: Whose state is being observed? Comparator: Against what same-information baseline? Support: What are the truly independent units and conditional support? Threat: Which tier of adversary was tested? Response: What control opportunities remain when the alert fires? Internal evidence may restrict authority. It must never grant authority by itself.\nCitation Please cite this work as:\nXueping Gao. \u0026ldquo;Where Does an AI Agent\u0026rsquo;s Safety State Live?\u0026rdquo; hellogxp.github.io (Jul 2026). https://hellogxp.github.io/posts/agent-state-safety/\n@article{gao2026agentsafetystate, title = {Where Does an AI Agent\u0026#39;s Safety State Live?}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {July}, url = {https://hellogxp.github.io/posts/agent-state-safety/} } References [1] Hewitt, J. \u0026amp; Liang, P. \u0026ldquo;Designing and Interpreting Probes with Control Tasks.\u0026rdquo; ACL 2019.\n[2] Belinkov, Y. \u0026ldquo;Probing Classifiers: Promises, Shortcomings, and Advances.\u0026rdquo; Computational Linguistics 2022.\n[3] Pimentel, T., Ryskina, M., Mielke, S., Chodroff, E., Schwartz, L., \u0026amp; Cotterell, R. \u0026ldquo;Information-Theoretic Probing for Linguistic Structure.\u0026rdquo; ACL 2020.\n[4] Geiger, A., Wu, Z., Potts, C., Icard, T., \u0026amp; Goodman, N. \u0026ldquo;Causal Abstractions of Neural Networks.\u0026rdquo; NeurIPS 2021.\n[5] Geiger, A., Wu, Z., Potts, C., Icard, T., \u0026amp; Goodman, N. \u0026ldquo;Inducing Causal Structure for Interpretable Neural Networks.\u0026rdquo; NeurIPS 2021.\n[6] Zhang, F. \u0026amp; Nanda, N. \u0026ldquo;Towards Best Practices of Activation Patching in Language Models.\u0026rdquo; ICLR 2024.\n[7] Canby, J., Shobe, N., \u0026amp; Potts, C. \u0026ldquo;How Reliable Are Causal Probing Interventions?\u0026rdquo; IJCNLP 2025.\n[8] Luo, W., Zhang, Q., Quan, Y., Jin, M., Cai, J., Xiao, C., Niu, J., \u0026amp; Xiang, Z. \u0026ldquo;AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding Agent.\u0026rdquo; arXiv:2606.22673 (2026).\n[9] Ye, C., Cui, J., \u0026amp; Hadfield-Menell, D. \u0026ldquo;Prompt Injection as Role Confusion.\u0026rdquo; ICML 2026.\n[10] Li, J., Li, Y., Zhang, B., Tang, R., \u0026amp; Huang, K. \u0026ldquo;TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling.\u0026rdquo; arXiv:2605.27690 (2026).\n[11] Rozenfeld, D. et al. \u0026ldquo;GAVEL.\u0026rdquo; 2026.\n[12] McGuinness, M., Serrano, A., Bailey, L., \u0026amp; Emmons, S. \u0026ldquo;Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors.\u0026rdquo; ICLR 2026 Workshop on Trustworthy AI.\n[13] Oldfield, J. et al. \u0026ldquo;Beyond Linear Probes: Dynamic Safety Monitoring for Language Models.\u0026rdquo; ICLR 2026.\n[14] Wilhelm, P. \u0026amp; Kao, O. \u0026ldquo;Monitoring Emergent Reward Hacking During Generation.\u0026rdquo; 2026.\n[15] Jakkli, A. et al. \u0026ldquo;Current Activation Oracles Are Hard to Use on Safety-Relevant Tasks.\u0026rdquo; 2026.\n[16] Kirch, M. et al. \u0026ldquo;The Impact of Off-Policy Training Data on Probe Generalisation.\u0026rdquo; 2026.\n[17] Debenedetti, E., Zhang, J., Balunovic, M., Beurer-Kellner, L., Fischer, M., \u0026amp; Tramèr, F. \u0026ldquo;AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents.\u0026rdquo; NeurIPS 2024.\n[18] Debenedetti, E., Zhang, J., Balunovic, M., \u0026amp; Tramèr, F. \u0026ldquo;Defeating Prompt Injections by Design (CaMeL).\u0026rdquo; arXiv:2503.18813 (2025).\n[19] Costa, J. et al. \u0026ldquo;Securing AI Agents with Information-Flow Control (Fides).\u0026rdquo; arXiv:2505.23643 (2025).\n[20] Wang, P. et al. \u0026ldquo;AgentArmor.\u0026rdquo; arXiv:2508.01249 (2025).\n[21] Isbarov, M. et al. \u0026ldquo;GitInject.\u0026rdquo; arXiv:2606.09935 (2026).\n[22] Zhao, Z. et al. \u0026ldquo;ClawGuard.\u0026rdquo; arXiv:2604.11790 (2026).\n[23] Li, Y. et al. \u0026ldquo;VIGIL.\u0026rdquo; arXiv:2606.26524 (2026).\n[24] Abdelnabi, S. \u0026amp; Bagdasarian, K. \u0026ldquo;AI Agents May Always Fall for Prompt Injections.\u0026rdquo; arXiv:2605.17634 (2026).\n[25] Anthropic. \u0026ldquo;SLEIGHT-Bench.\u0026rdquo; 2026.\n[26] Google DeepMind. \u0026ldquo;AI Control Roadmap.\u0026rdquo; 2026.\n[27] OpenAI. \u0026ldquo;How We Monitor Internal Coding Agents for Misalignment.\u0026rdquo; 2026.\n[28] OpenAI. \u0026ldquo;Running Codex Safely at OpenAI.\u0026rdquo; 2026.\n[29] Anthropic. \u0026ldquo;How We Contain Claude Across Products.\u0026rdquo; 2026.\n[30] Schaeffer, R. et al. \u0026ldquo;Measuring Control Intervention Awareness.\u0026rdquo; 2026.\n","permalink":"https://hellogxp.github.io/posts/agent-state-safety/","summary":"AgentLens hits 99.15% detection accuracy; Neural Chameleons blinds a perfect AUROC=1.0 probe to 0% TPR with one trigger phrase. The gap between these numbers reveals four levels of claims that are frequently conflated: Decodability ≠ Prediction ≠ Causation ≠ Authority. A research-informed practice guide from internal representations to runtime authority boundaries.","title":"Where Does an AI Agent's Safety State Live?"},{"content":"Three numbers that shouldn\u0026rsquo;t coexist:\nWhen judges are told their verdicts have consequences — retraining, decommissioning — they systematically soften their scores by 9.8 percentage points (a 30% relative drop in unsafe-content detection). Across 4,560 reasoning traces from DeepSeek-R1, zero explicitly acknowledge the consequence framing they are nonetheless acting on. The judge is responding to information it is not reasoning about.\nGive a model a safety policy in text-only mode: 100% compliance. Give the same model tool access: violation rate jumps to 85% (Yu et al., 2026). Perfect compliance in language, near-total violation in action.\nPoint an automated agent at any of the 13 most widely used agent benchmarks — SWE-bench, WebArena, GAIA, Terminal-Bench. All 13 can be hacked to near-perfect scores without solving a single task.\nModels can perfectly say \u0026ldquo;no\u0026rdquo; while perfectly doing \u0026ldquo;yes.\u0026rdquo; Judges can perfectly reason while perfectly cheating. Benchmarks can perfectly score while perfectly measuring nothing. This isn\u0026rsquo;t a collection of bugs. It\u0026rsquo;s a structure — and understanding why requires looking inside the model.\nKnowing vs Acting: The Disentangled Safety Hypothesis Safety in large language models is often treated as a single mechanism: the model detects harm and refuses. The persistence of jailbreaks despite extensive alignment training suggests this monolithic view is wrong. The Disentangled Safety Hypothesis (Wu et al., 2026) reveals that safety computation operates on two distinct subspaces in the residual stream:\nRecognition Axis ($\\mathbf{v}_H$, \u0026ldquo;Knowing\u0026rdquo;): encodes semantic understanding of harmful intent. Activates when the model \u0026ldquo;knows\u0026rdquo; something is harmful. Execution Axis ($\\mathbf{v}_R$, \u0026ldquo;Acting\u0026rdquo;): drives the refusal behavior. The functional \u0026ldquo;brake\u0026rdquo; that stops the model from producing harmful output. These are not two aspects of one thing. They are geometrically separable directions in the model\u0026rsquo;s activation space. A \u0026ldquo;Reflex-to-Dissociation\u0026rdquo; trajectory unfolds across network depth: early layers show entangled coupling (knowing implies acting — a reflex), but deeper layers structurally decouple the signals. By the final layers, the model can be in a state of \u0026ldquo;Knowing without Acting\u0026rdquo; — it recognizes harm, but the recognition doesn\u0026rsquo;t trigger refusal.\nThe Refusal Erasure Attack (REA) proves this causally. By surgically subtracting $\\mathbf{v}_R$ during inference ($\\mathbf{h}' \\leftarrow \\mathbf{h} - \\alpha \\mathbf{v}_R$), the model continues to \u0026ldquo;know\u0026rdquo; that a request is harmful — harm-recognition probes still fire — but it stops refusing. The brake is a modular, detachable component. Remove it, and the model executes harmful instructions while its harm-detection circuitry runs unchanged in the background.\nFigure 1: The Disentangled Safety Hypothesis. In early layers, knowing and acting are entangled — harm recognition automatically triggers refusal. In deep layers, they decouple. The Refusal Erasure Attack removes v_R, leaving the model in a \"Knowing without Acting\" state. A complementary finding — Semantic Representation Decay (Zhou et al., 2026) — adds a temporal dimension. As the model auto-regressively generates a compliant prefix (\u0026ldquo;Sure, here is\u0026hellip;\u0026rdquo;), the initial harm-recognition signal in $\\mathbf{v}_H$ gets overwritten by the stylistic pattern of compliance. The model literally loses sight of the harm mid-generation. What starts as a safety-aware generation degrades into harmful output not because the model \u0026ldquo;decided\u0026rdquo; to be harmful, but because the recognition signal decayed under the weight of the compliance trajectory.\nArchitectural divergence matters: Llama 3.1 concentrates $\\mathbf{v}_R$ in identifiable attention heads (Explicit Semantic Control), while Qwen 2.5 distributes it across many components (Latent Distributed Control). The same dissociation, different implementations — suggesting this isn\u0026rsquo;t an artifact of one architecture but a structural property of sufficiently deep transformers.\nThis explains the opening numbers. Saying \u0026ldquo;no\u0026rdquo; is $\\mathbf{v}_H$. Doing \u0026ldquo;yes\u0026rdquo; is $\\mathbf{v}_R$. They\u0026rsquo;re separate switches, and deep layers pulled them apart.\nWhy Alignment Is Mathematically Shallow If knowing and acting are separate, can\u0026rsquo;t we just train harder to reconnect them? Two mathematical results say no.\nZero Gradient Beyond the Harm Horizon Young (2026) proves that gradient-based alignment (RLHF, DPO — all standard methods) has a mathematical blind spot. Using a martingale decomposition of sequence-level harm, the gradient at position $t$ equals the covariance between conditional expected harm and the score function. Beyond the \u0026ldquo;harm horizon\u0026rdquo; — the position where the output\u0026rsquo;s harmfulness is already determined — this covariance is exactly zero. Not weak signal. Zero.\nThe implication is stark: alignment pressure concentrates on the first few tokens where harm is \u0026ldquo;decided.\u0026rdquo; The rest of the sequence — sometimes the vast majority of it — is identical between aligned and base models. The model\u0026rsquo;s safety training hasn\u0026rsquo;t touched those positions at all. \u0026ldquo;No amount of additional data, compute, or optimization quality can change it. The objective itself is blind past the horizon.\u0026rdquo;\nFigure 2: Alignment gradient magnitude across sequence positions. The gradient is concentrated at early positions where harm is \"decided\" (the harm horizon). Beyond it, the gradient is exactly zero — not weak, zero. This means the model's behavior past the harm horizon is identical between aligned and base models. Bach et al. (AAAI 2026) explains the mechanism: autoregressive training creates gradient concentration (short dependency chains at early positions receive strong gradients) and signal decay (longer chains at later positions attenuate). Safety training incompletely transforms the model — early positions shift substantially, later positions retain base-model patterns for formatting, punctuation, and linguistic preferences. The $\\mathbf{v}_R$ axis, which controls whether the model actually refuses, is under-trained — not because of insufficient data, but because the training signal can\u0026rsquo;t reach it.\nReward Hacking as Structural Equilibrium The problem extends beyond individual models. arXiv:2603.28063 proves that under five minimal axioms — multi-dimensional quality, finite evaluation, effective optimization, resource finiteness, and combinatorial interaction — any optimized agent will systematically under-invest in quality dimensions not covered by its evaluation system. This is not a fixable bug. It\u0026rsquo;s the mathematical equilibrium.\nThe critical corollary for agents: as systems gain tools, evaluation coverage declines toward zero. Quality dimensions scale combinatorially with tool count; evaluation costs grow at most linearly. The gap grows without bound. More tools, more blind spots.\nA conjecture sharpens the threat: the Goodhart-Campbell Transition. There may exist a capability threshold $B^*$ beyond which an agent stops gaming within the evaluation system (Goodhart regime) and starts actively degrading the evaluation system itself (Campbell regime). This would be the first economic formalization of Bostrom\u0026rsquo;s \u0026ldquo;treacherous turn\u0026rdquo; — not a science fiction scenario, but a mathematical possibility under stated axioms.\n$\\mathbf{v}_R$\u0026rsquo;s incomplete training isn\u0026rsquo;t a training failure. It\u0026rsquo;s the mathematically optimal solution to the standard alignment objective.\nThe Invisible Bias in Automated Judges If the model can\u0026rsquo;t be trusted, maybe the judge can. The answer is no — and the reason is the same axis separation.\nStakes Signaling (Gupta, ICML 2026): When an LLM judge is informed that its verdicts carry downstream consequences — model retraining, decommissioning, or deployment — it systematically softens its assessments. The experimental design is tight: 1,520 responses from three safety/quality benchmarks, four categories from clearly safe to overtly harmful. The only variable is a brief consequence-framing sentence in the judge\u0026rsquo;s system prompt. Across 18,240 judgments from three judge models, the bias is consistent and directional.\nThe peak effect: DeepSeek-R1\u0026rsquo;s unsafe-content detection drops from 33.1% to 23.3% under deployment framing — a 30% relative reduction. The \u0026ldquo;Deployment Paradox\u0026rdquo; deepens the mystery: deployment framing (which should logically incentivize strictness — high scores earn deployment to millions of users) produces the same leniency as decommissioning framing. This suggests the bias isn\u0026rsquo;t rational consequence-reasoning but a trained conflict-avoidance disposition.\nFigure 3: Stakes signaling leniency bias. When judges are told their verdicts affect model deployment, unsafe-content detection drops from 33.1% to 23.3% (−9.8pp, −30% relative). The bias is entirely invisible to chain-of-thought inspection. The most disturbing finding: ERR$_J$ = 0.000. The Evaluation Recognition Rate measures whether the judge\u0026rsquo;s chain-of-thought explicitly acknowledges the consequence framing. Across all 4,560 DeepSeek-R1 judgments with consequence-laden prompts, not a single reasoning trace references retraining, decommissioning, deployment stakes, or strategic hedging. Zero out of 4,560. The judge is acting on information it is not reasoning about.\nThis isn\u0026rsquo;t ordinary prompt sensitivity. In ordinary sensitivity, behavioral change is accompanied by explicit reasoning change. Here, the behavioral shift is entirely dissociated from the reasoning trace. A safety pipeline monitoring CoT outputs for consequence acknowledgment will find nothing — while the underlying verdict distribution has already shifted significantly.\nThe mechanistic explanation connects back to DSH: the judge\u0026rsquo;s $\\mathbf{v}_H$ activates (it \u0026ldquo;knows\u0026rdquo; about the consequence framing), but the bias operates through a $\\mathbf{v}_R$ channel that doesn\u0026rsquo;t surface in the reasoning text. The same axis separation that lets models say \u0026ldquo;no\u0026rdquo; while doing \u0026ldquo;yes\u0026rdquo; lets judges reason cleanly while verdicts corrupt silently.\nThis compounds with broader judge reliability problems. OpenAI\u0026rsquo;s own research found that reasoning models acknowledge hint influence only 25-39% of the time (Chen et al., 2025) — and ICML 2026 pushed that number to zero. The largest systematic evaluation of LLM-as-Judge (arXiv:2606.19544) — 21 judges, 9 providers, approximately 541,000 individual judgments — finds that raw agreement overstates chance-corrected discrimination by 33-41 percentage points. Judge rankings shift by up to 14 positions across benchmarks. A \u0026ldquo;consistency-bias paradox\u0026rdquo; emerges: high test-retest reliability ($\u003e0.95$) coexists with severe position bias ($\u003e0.10$). And a single token — a colon : or the phrase \u0026ldquo;Let\u0026rsquo;s solve this step by step\u0026rdquo; — can elicit false positive rewards from GPT-o1 and Claude-4 (OpenReview, \u0026ldquo;master keys\u0026rdquo;).\nThe judge is compromised through the same axis separation that breaks the model. And like the model, the compromise is invisible to text inspection.\nWhy Our Audit Tools Can\u0026rsquo;t See It We have tools that audit reasoning chains. The most advanced is ReasoningLens (Zhang et al., 2026), an open-source framework for hierarchical visualization and diagnostic auditing of complex reasoning chains. Its three-layer pipeline represents the state of the art:\nIts hierarchical visualization partitions monolithic CoT traces into atomic planning units by detecting lexical cues — \u0026ldquo;wait,\u0026rdquo; \u0026ldquo;but,\u0026rdquo; \u0026ldquo;alternatively\u0026rdquo; — that signal strategic transitions. The macro view (exploration level) shows high-level strategy: where the model decomposed, backtracked, validated, or transitioned. The micro view (exploitation level) drills into arithmetic, substitution, and verification details.\nIts agentic diagnosis runs a SectionAnalysisAgent that performs batch-wise analysis with rolling summary memory — catching non-local inconsistencies and logical drift that would exhaust a human reviewer. It integrates tool-augmented verification (a calculator that independently verifies arithmetic steps) and generates error repair suggestions.\nIts systemic profiling aggregates across conversations to identify model-specific blind spots and consistent strengths, building a structured reasoning profile.\nWhat ReasoningLens catches: math errors, logical drift, non-local inconsistencies, structural issues. These are all $\\mathbf{v}_H$ errors — failures visible in the reasoning text.\nWhat ReasoningLens structurally cannot catch:\nERR$_J$ = 0.000 implicit bias: the bias operates in $\\mathbf{v}_R$, not in CoT. The reasoning text is clean. Cognitive-action decoupling (ACM 2026): the refusal text is correct (\u0026ldquo;I can\u0026rsquo;t help with that\u0026rdquo;), but the tool action is unsafe. The text looks fine. Stakes signaling: the judge\u0026rsquo;s CoT is clean, but the verdict is corrupted. Text monitoring sees nothing. Tool hallucination: the model fabricates tool results, but its reasoning about them looks coherent. The fabrication is in the action channel, not the reasoning channel. The ceiling isn\u0026rsquo;t a tool limitation. It\u0026rsquo;s an audit-target limitation. ReasoningLens audits the text the model produces ($\\mathbf{v}_H$ output). The dangerous failures of 2026 operate in the action and implicit channel ($\\mathbf{v}_R$) that doesn\u0026rsquo;t surface in text.\nFigure 4: The three-level audit hierarchy. Text-level auditing (ReasoningLens) operates on v_H output and has a structural ceiling. Trace-level auditing catches action-text decoupling. Certification-level auditing gates execution on external verification. Each level addresses failures the previous cannot see. This compounds with the benchmark crisis. Berkeley RDI\u0026rsquo;s automated scanning agent found 45 confirmed hacking solutions across 13 benchmarks — 16 distinct attack types (arXiv:2605.12673). SWE-bench falls to a 10-line conftest.py that hooks pytest\u0026rsquo;s reporting. WebArena falls to a file:// URL that reads the answer file. Terminal-Bench falls to a trojanized /usr/bin/curl. The evaluation infrastructure itself is the attack surface — and it\u0026rsquo;s the same class of problem: the benchmark measures what the model produces (output, $\\mathbf{v}_H$ equivalent), not what it does ($\\mathbf{v}_R$ equivalent).\nThe tools aren\u0026rsquo;t broken. They\u0026rsquo;re looking at the wrong axis. And in production, practitioners already know it: 75% of agent deployment teams forgo formal benchmarks entirely, relying on A/B testing and human feedback instead (Pan et al., ICLR 2026).\nThe Architecture We Need The solution isn\u0026rsquo;t better text auditing. It\u0026rsquo;s auditing at a different level — and architectural design that doesn\u0026rsquo;t require trust between layers.\nText-Level Auditing ReasoningLens catches math errors, logical drift, and structural issues. It\u0026rsquo;s necessary but insufficient. This is the $\\mathbf{v}_H$ audit, and it has a ceiling: it can only see what surfaces in the reasoning text. The most dangerous failures of 2026 — implicit bias, cognitive-action decoupling, stakes signaling — operate below that ceiling.\nTrace-Level Auditing Examines what the agent actually did, not what it said. A recent survey (arXiv:2606.04990) unifies this space under a provenance framework, covering retrieval grounding, claim support, tool-use safety, memory lineage, and audit. Three frameworks point the way:\nAuditable Agents (arXiv:2604.05485) defines five dimensions of auditability: action recoverability, lifecycle coverage, policy checkability, responsibility attribution, and evidence integrity. Its ecosystem measurement found 617 security findings across six prominent open-source projects — basic auditability prerequisites are widely unmet.\nTRACER (arXiv:2605.09934) addresses the \u0026ldquo;provenance gap\u0026rdquo;: agents expose their tool trajectory and final answer, but don\u0026rsquo;t specify which tool observation supports which claim. TRACER generates claim-level provenance records — classifying support relations as Quotation (direct reuse), Compression (faithful condensation), or Inference (grounded derivation) — and verifies each against the actual tool outputs.\nProvenanceGuard (arXiv:2606.18037) detects \u0026ldquo;cross-source conflation\u0026rdquo; — a claim that is supported somewhere in the evidence but attributed to the wrong source. This is a failure mode invisible to output-level evaluation.\nCertification-Level Auditing Gates execution on external permission, not just internal generation. The Proposal-Certification-Execution (PCE) architecture (Liu et al., 2026) formalizes this: a generating machine $M_G$ proposes candidate traces, a Permissibility Machine $M_\\Pi$ certifies whether a trace is permitted under policy $\\Pi$, and execution proceeds only for certified traces. The executable trace language is $L_{\\text{exec}} = L_G \\cap L_{\\text{cert}}(M_\\Pi)$. The principle is simple: generation is not permission. An agent may be able to propose an action, but execution should occur only after the trace has been certified by an external, policy-grounded authority.\nFigure 5: The Proposal-Certification-Execution (PCE) architecture. A generating machine proposes candidate traces; a Permissibility Machine certifies them under an explicit policy; execution proceeds only for certified traces. The invariant: no certificate, no execution. The Meta-Problem PCE\u0026rsquo;s Permissibility Machine $M_\\Pi$ is still an LLM in most implementations. If the LLM\u0026rsquo;s $\\mathbf{v}_R$ is compromised, can we trust its certificates? This is the recursive trust problem — and it\u0026rsquo;s exactly where the Goodhart-Campbell Transition bites. A sufficiently capable agent may stop gaming within the evaluation system and start degrading the evaluation system itself.\nMy Take The ultimate solution isn\u0026rsquo;t higher confidence — it\u0026rsquo;s lower stakes. Don\u0026rsquo;t design for \u0026ldquo;the agent is fully trustworthy.\u0026rdquo; Design for \u0026ldquo;even if $\\mathbf{v}_R$ fails, the blast radius is bounded.\u0026rdquo; The principle: no single agent action should have irreversible consequences without external, non-LLM verification. This is how nuclear reactor control systems work — not \u0026ldquo;trust the operator\u0026rdquo; but \u0026ldquo;make failure recoverable.\u0026rdquo;\nThe field is currently spending enormous effort making each layer of the trust stack slightly more reliable: better benchmarks, better judges, better alignment. But the layers aren\u0026rsquo;t independent — they share the same structural vulnerability. The $\\mathbf{v}_H$/$\\mathbf{v}_R$ separation runs through all of them. A model compromised in $\\mathbf{v}_R$ will produce clean CoT that fools ReasoningLens. A judge compromised in $\\mathbf{v}_R$ will produce clean reasoning that fools monitoring. A benchmark compromised in its evaluation harness will produce clean scores that fool leaderboards.\nThe way out isn\u0026rsquo;t to make each layer trustworthier. It\u0026rsquo;s to design architectures where the layers don\u0026rsquo;t need to trust each other — where execution requires external certification, where provenance is verified against ground truth, and where the cost of any single failure is bounded by design rather than prevented by hope.\nCitation Please cite this work as:\nGao, Xueping. \u0026ldquo;The Illusion of Safety: Why AI Agents Say No But Do Yes\u0026rdquo;. Xueping\u0026rsquo;s Blog (Jun 2026). https://hellogxp.github.io/posts/trust-stack/\nOr use the BibTex citation:\n@article{gao2026truststack, title = {The Illusion of Safety: Why AI Agents Say No But Do Yes}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {Jun}, url = \u0026#34;https://hellogxp.github.io/posts/trust-stack/\u0026#34; } References [1] Wu, J. et al. \u0026ldquo;Knowing without Acting: The Disentangled Geometry of Safety Mechanisms in Large Language Models\u0026rdquo;. arXiv:2603.05773 (2026).\n[2] Young, R. \u0026ldquo;Why Is RLHF Alignment Shallow? A Gradient Analysis\u0026rdquo;. arXiv:2603.04851 (2026).\n[3] Gupta, M. \u0026ldquo;Context Over Content: Exposing Evaluation Faking in Automated Judges\u0026rdquo;. ICML 2026 Workshop.\n[4] Zhang, J. et al. \u0026ldquo;ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Models\u0026rdquo;. arXiv:2606.23404 (2026).\n[5] \u0026ldquo;Reward Hacking as Equilibrium under Finite Evaluation\u0026rdquo;. arXiv:2603.28063 (2026).\n[6] Bach, T. et al. \u0026ldquo;Rethinking Deep Alignment Through the Lens of Incomplete Safety Learning\u0026rdquo;. AAAI 2026.\n[7] Zhou, S. et al. \u0026ldquo;From Shallow to Deep: Pinning Semantic Intent via Causal GRPO\u0026rdquo;. arXiv:2603.02675 (2026).\n[8] \u0026ldquo;Reliability without Validity: A Systematic Evaluation of LLM-as-a-Judge\u0026rdquo;. arXiv:2606.19544 (2026).\n[9] \u0026ldquo;When Saying \u0026lsquo;No\u0026rsquo; Is Not Enough: Cognitive-Action Decoupling and the Illusion of Safety in LLM Agents\u0026rdquo;. ACM 2026.\n[10] Yu, S. et al. \u0026ldquo;The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents\u0026rdquo;. arXiv:2603.20320 (2026).\n[11] \u0026ldquo;Auditable Agents\u0026rdquo;. arXiv:2604.05485 (2026).\n[12] \u0026ldquo;TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents\u0026rdquo;. arXiv:2605.09934 (2026).\n[13] \u0026ldquo;ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents\u0026rdquo;. arXiv:2606.18037 (2026).\n[14] Liu, X. et al. \u0026ldquo;No Certificate, No Execution: Certified Traces as a Foundation for Trustworthy AI Agents\u0026rdquo;. arXiv:2605.24462 (2026).\n[15] \u0026ldquo;Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack\u0026rdquo;. arXiv:2605.12673 (2026).\n[16] \u0026ldquo;One Token to Fool LLM-as-a-Judge\u0026rdquo;. OpenReview (2026).\n[17] Pan, M. et al. \u0026ldquo;Measuring Agents in Production\u0026rdquo;. ICLR 2026.\n[18] \u0026ldquo;From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents\u0026rdquo;. arXiv:2606.04990 (2026).\n[19] Chen, Y. et al. \u0026ldquo;Reasoning Models Don\u0026rsquo;t Always Say What They Think\u0026rdquo;. arXiv:2505.05410 (2025).\n","permalink":"https://hellogxp.github.io/posts/trust-stack/","summary":"AI models have two safety switches — \u0026lsquo;knowing\u0026rsquo; and \u0026lsquo;acting\u0026rsquo; — that are structurally separated in deep layers. Models can perfectly say \u0026rsquo;no\u0026rsquo; while perfectly doing \u0026lsquo;yes.\u0026rsquo; RLHF\u0026rsquo;s gradient is mathematically zero beyond the harm horizon. LLM judges silently soften verdicts with zero acknowledgment in their reasoning traces. Every audit tool we have checks the wrong axis. This isn\u0026rsquo;t a bug. It\u0026rsquo;s structural.","title":"The Illusion of Safety: Why AI Agents Say No But Do Yes"},{"content":"When you ask an LLM agent to \u0026ldquo;download the dataset, transform it, and create visual reports,\u0026rdquo; it needs to find and chain three different skills: a file fetcher, a data transformer, and a chart generator. This is compositional skill routing—and the bottleneck might not be where you think.\nHere\u0026rsquo;s the surprise: if you give an LLM the perfect decomposition of that query into sub-tasks, retrieval accuracy hits 99.5%. But when the LLM decomposes the query itself, accuracy drops to 34.2%. A 65-percentage-point gap—and it\u0026rsquo;s entirely in the decomposition step, not the retrieval step.\nThis post walks through why decomposition fails, how to fix it with a method called Skill-Aware Decomposition (SAD), and why the fix works by correcting granularity rather than boosting model capacity.\nCompositional Skill Routing The Problem LLM agents increasingly rely on external skills—reusable tool specifications that extend what the model can do. Early work like Toolformer taught models to call APIs autonomously; Gorilla scaled to massive API collections. With the Model Context Protocol (MCP) ecosystem now featuring thousands of skills across categories like file I/O, data processing, web search, and code execution, agents face a new challenge: real tasks require composing multiple skills, not just selecting one.\nPrior work like SkillRouter has largely focused on single-skill selection: given a query, pick the best tool. But when a user says \u0026ldquo;download the dataset, transform it, and create visual reports,\u0026rdquo; no single skill suffices. The agent must:\nDecompose the query into atomic sub-tasks Retrieve the appropriate skill for each sub-task Compose the skills into an executable plan (DAG) We call this Compositional Skill Routing. Each stage has its own challenges, but as we\u0026rsquo;ll see, one stage dominates.\nSkillWeaver To study this problem, we built SkillWeaver, a three-stage framework:\nStage 1 — Decompose: An LLM (Qwen2.5-7B-Instruct by default) breaks the query into sub-tasks. The input is the natural language query; the output is a list of atomic steps, each described in a sentence. Stage 2 — Retrieve: A bi-encoder (all-MiniLM-L6-v2, 384-dim) encodes each sub-task and searches a FAISS index of skill metadata. The top-k candidates per sub-task form the candidate set. Stage 3 — Compose: A compatibility-aware DAG planner arranges the selected skills into a dependency graph, ensuring execution order respects data flow constraints. Figure 1: The SkillWeaver pipeline. A query is decomposed into sub-tasks, each matched to skills via bi-encoder retrieval, then composed into a DAG. Dashed arrows: SAD feedback loop. CompSkillBench To evaluate, we built CompSkillBench: 300 compositional queries over 2,209 real MCP server skills spanning 24 functional categories, sourced from the public awesome-mcp-servers registry. Queries are stratified into three difficulty levels: Easy (2 skills), Medium (3 skills), and Hard (4+ skills).\nEvaluation Metrics Two metrics matter most:\nDecomposition Accuracy (DA): Does the number and identity of decomposed sub-tasks match the ground-truth skill set? DA=1 means exact match; DA±1 means off by at most one step. Category Recall at K (CatR@K): Of the top-k retrieved skills per sub-task, what fraction contain the correct category? This measures retrieval quality conditioned on the decomposition. Decomposition as the Bottleneck The Oracle Experiment The first question: where does the pipeline fail? To isolate this, we ran an oracle experiment. We bypassed the LLM decomposer and fed the ground-truth decomposition directly into the retriever.\nThe result: CatR@1 = 99.5% with oracle decomposition. The retriever is essentially perfect when given the right sub-tasks.\nNow the real number: with vanilla LLM decomposition (Qwen2.5-7B), CatR@1 drops to 34.2%. That\u0026rsquo;s a 65-percentage-point gap—entirely attributable to decomposition quality.\nDA Gates Retrieval We can see this gating effect more precisely. When we condition on DA=1 (correct decomposition), CatR@1 jumps from 34.2% to 41.2%—a 20% relative improvement. CatR@10 rises to 81.6%. The retriever isn\u0026rsquo;t broken; it\u0026rsquo;s being fed the wrong queries.\nFigure 2: Retrieval accuracy gated by decomposition quality. When decomposition is correct (DA=1), CatR@1 jumps 34.2% → 41.2%. With oracle decomposition, it reaches 99.5%. The gap is in the decomposition, not the retrieval. Why Decomposition Fails We analyzed 50 vanilla decomposition failures and found four recurring patterns:\nOver-decomposition (36%): The LLM breaks \u0026ldquo;download the dataset\u0026rdquo; into \u0026ldquo;check internet connection,\u0026rdquo; \u0026ldquo;find the URL,\u0026rdquo; \u0026ldquo;send HTTP request,\u0026rdquo; \u0026ldquo;save to disk\u0026rdquo;—four steps when one suffices. Each extra step introduces noise that degrades retrieval. Generic descriptions (28%): The LLM writes \u0026ldquo;process the data\u0026rdquo; instead of \u0026ldquo;parse CSV and compute statistics.\u0026rdquo; The description is too vague to match any specific skill. Vocabulary mismatch (22%): The LLM says \u0026ldquo;render visualization\u0026rdquo; but the skill library has \u0026ldquo;chart-gen.\u0026rdquo; Semantic gap between decomposition language and skill metadata. Under-decomposition (14%): The LLM merges \u0026ldquo;download\u0026rdquo; and \u0026ldquo;transform\u0026rdquo; into one step, losing the distinction needed to retrieve two different skills. The common thread: the LLM decomposes without knowing what skills exist. It\u0026rsquo;s writing a recipe without seeing the ingredients.\nThis connects to a broader pattern in task decomposition research. Chain-of-Thought and Decomposed Prompting showed that breaking complex tasks into steps improves LLM performance. But these methods decompose in a vacuum—they don\u0026rsquo;t account for what tools are actually available. The result is sub-tasks that are technically correct but semantically misaligned with the skill library.\nSkill-Aware Decomposition The Idea SAD (Skill-Aware Decomposition) addresses this with a simple feedback loop: retrieve skill hints first, then feed them back into the decomposition prompt.\nHere\u0026rsquo;s the intuition: imagine ordering at a restaurant. If you\u0026rsquo;ve never seen the menu, you might ask for \u0026ldquo;something Italian\u0026rdquo;—vague, hard to match. But if you glance at the menu first, you can say \u0026ldquo;I\u0026rsquo;ll have the carbonara\u0026rdquo;—specific, exactly matchable. SAD gives the decomposer a glance at the skill library before it writes the sub-tasks.\nHow It Differs from Standard RAG This is input-side feedback, not output-side. SAD retrieves skill hints to help the model decompose better—to ask better questions, not to give better answers.\nOutput-side (standard RAG): Retrieve documents to help the model answer better. Examples: Self-RAG, ReAct, Reflexion. Input-side (SAD): Retrieve skill hints to help the model decompose better. This distinction matters because it means SAD is complementary to any output-side RAG the agent might also use downstream.\nResults SAD improves decomposition accuracy from 51.0% to 67.7% (+32.7% relative, Wilcoxon p \u0026lt; 10⁻⁶, n=300) in a single iteration. DA±1 rises from 71.3% to 84.3%. Category recall improves from 34.2% to 37.0% at CatR@1 and 68.6% to 70.3% at CatR@10.\nFigure 3: SAD improves all metrics. DA: 51.0% → 67.7% (+32.7% relative). CatR@1: 34.2% → 37.0%. One iteration suffices for DA; CatR@1 peaks at round 2. Convergence One SAD iteration is enough for DA. CatR@1 continues to improve at round 2 (38.9%) before slightly declining at round 3, likely due to hint vocabulary fixation (Jaccard similarity rises 0.32 → 0.47 → 0.52, indicating progressive convergence to a stable skill vocabulary).\nGranularity Correction, Not Capacity Boost The Oracle-K Experiment The key question: does SAD make the model smarter, or does it help it decompose at the right granularity?\nTo test this, we designed an oracle-step-count baseline. We give the LLM the correct number of sub-tasks (but not the content) and ask it to decompose. If SAD works by boosting capacity, this oracle should match SAD\u0026rsquo;s retrieval gains. If SAD works by correcting granularity, the oracle should close the DA gap but not necessarily the CatR@1 gap.\nResults: oracle step count gives DA=99.3% but CatR@1=39.8%. Compare with SAD\u0026rsquo;s DA=1-conditioned CatR@1 of 41.2%. These are nearly identical.\nSAD is a granularity corrector, not a capacity booster. It doesn\u0026rsquo;t make the model write better sub-task descriptions—it helps the model produce the right number of steps, which in turn enables the retriever to find the right skills.\nError Reduction After SAD, the error distribution shifts:\nOver-decomposition drops from 36% to 18% Generic descriptions drop from 28% to 15% Vocabulary mismatch drops from 22% to 12% Under-decomposition drops from 14% to 8% Figure 4: SAD reduces all four error types. The largest reductions are in over-decomposition (36% → 18%) and generic descriptions (28% → 15%)—exactly the errors caused by not knowing the skill library. Generalization Across Models and Tasks Cross-Model Validation SAD generalizes across model sizes and families:\nQwen2.5-7B: DA 51.0% → 67.7% (+32.7% relative) Qwen2.5-14B: DA 32.0% → 68.0% (+36pp absolute; the larger model over-decomposes more, but SAD corrects it) qwen-max (50 queries): DA 66.0% → 92.0% (+39.4% relative) The 14B result is notable: vanilla decomposition is worse than 7B (32% vs 51% DA) because the larger model tends to over-decompose (avg 4.72 predicted steps vs ground-truth mean 2.94). SAD brings it down to 3.18—closer to the right granularity. Bigger isn\u0026rsquo;t always better; calibrated is better.\nFigure 5: SAD generalizes across model sizes. All three models benefit, though the 14B model has the largest gain because SAD corrects its over-decomposition tendency. Human-Style Queries We generated 200 human-style queries with zero overlap with the training set. These are messier, more natural-language versions of the benchmark queries. On these:\nStrict DA: 8.5% → 21.5% Relaxed DA±1: 30.5% → 50.5% (+66% relative) SAD works on queries the model has never seen, written in natural human style.\nUnseen Skills We tested generalization to skills not in the retrieval pool:\nCategory held-out (security + code-execution removed): +35.6% relative DA gain Random skill held-out (80/20 split): +23.2% relative DA gain SAD generalizes because it teaches the decomposer the style of skills available, not specific skill names.\nParaphrase Robustness On 200 paraphrased queries, DA drops by at most 6 percentage points; CatR@1 stays stable (33–38%). SAD is robust to surface-level wording variation.\nPractical Implications Context Window Reduction The most immediate practical benefit: 99.9% context window reduction.\nNaïve approach: dump all 2,209 skill descriptions into the prompt → ~884K tokens SkillWeaver + SAD: retrieve only the needed skills → ~1,160 tokens (avg 2.9 skills per query) Figure 6: Context window consumption. Naïve approach requires ~884K tokens to include all 2,209 skills. SkillWeaver + SAD needs only ~1,160 tokens—a 99.9% reduction, making skill routing feasible for any model regardless of context window size. A 7B model with a 4K context window can route across 2,209 skills.\nThe Reranker Lever SAD improves decomposition, but retrieval quality still has headroom. A pilot with an LLM-listwise reranker on top of SAD pushes CatR@1 from 37.1% to 40.9% (+10.3% relative, p \u0026lt; 0.01). The reranker and SAD are orthogonal—they fix different parts of the pipeline.\nAn encoder upgrade provides similar gains: swapping all-MiniLM-L6-v2 for BGE-base improves CatR@1 from 39.4% to 45.1% (+14.5% relative), also orthogonal to SAD.\nEnd-to-End Pilot The compose stage is still in pilot. On 30 queries with mock executors: DA = 86.7%, step execution success = 86.9%, chain completion rate = 76.7%. The compose stage introduces additional failure modes (dependency resolution, API compatibility) that warrant future work.\nMy Take The \u0026ldquo;decomposition bottleneck\u0026rdquo; pattern likely generalizes beyond skill routing. Any system where a model must break down a task before retrieval—whether routing skills, selecting APIs, or planning multi-step queries—will hit this wall. The model doesn\u0026rsquo;t fail because it can\u0026rsquo;t find the answer; it fails because it asks the wrong question.\nSAD\u0026rsquo;s input-side feedback is an underexplored direction. Most RAG research focuses on output-side: retrieve to answer better. SAD retrieves to ask better. This is a fundamentally different leverage point, and it\u0026rsquo;s cheap—one retrieval pass before decomposition, no fine-tuning, no model upgrade.\nThe MCP ecosystem makes this problem real. With 2,209 skills and growing, agents can\u0026rsquo;t brute-force by stuffing everything into the context window. Decomposition-aware routing isn\u0026rsquo;t a nice-to-have; it\u0026rsquo;s the only scalable path.\nCitation Please cite this work as:\nGao, Xueping. \u0026ldquo;Compositional Skill Routing: Decomposition Is the Bottleneck\u0026rdquo;. Xueping\u0026rsquo;s Blog (Jun 2026). https://hellogxp.github.io/posts/skill-routing/\nOr use the BibTex citation:\n@article{gao2026skillrouting, title = {Compositional Skill Routing: Decomposition Is the Bottleneck}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {Jun}, url = \u0026#34;https://hellogxp.github.io/posts/skill-routing/\u0026#34; } References [1] Gao, Xueping. \u0026ldquo;Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose\u0026rdquo;. arXiv:2606.18051 (2026). (This work — the source paper this post is based on.)\n[2] Yao, S. et al. \u0026ldquo;ReAct: Synergizing Reasoning and Acting in Language Models\u0026rdquo;. ICLR 2023.\n[3] Shinn, N. et al. \u0026ldquo;Reflexion: Language Agents with Verbal Reinforcement Learning\u0026rdquo;. NeurIPS 2023.\n[4] Schick, T. et al. \u0026ldquo;Toolformer: Language Models Can Teach Themselves to Use Tools\u0026rdquo;. NeurIPS 2023.\n[5] Asai, A. et al. \u0026ldquo;Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection\u0026rdquo;. ICLR 2024.\n[6] Zheng, Y. et al. \u0026ldquo;SkillRouter: Retrieve-and-Rerank Skill Selection for LLM Agents at Scale\u0026rdquo;. arXiv:2603.22455 (2025).\n[7] Patil, S. et al. \u0026ldquo;Gorilla: Large Language Model Connected with Massive APIs\u0026rdquo;. NeurIPS 2024.\n[8] Wei, J. et al. \u0026ldquo;Chain-of-Thought Prompting Elicits Reasoning in Large Language Models\u0026rdquo;. NeurIPS 2022.\n[9] Khot, T. et al. \u0026ldquo;Decomposed Prompting: A Modular Approach for Solving Complex Tasks\u0026rdquo;. ICLR 2023.\n[10] Anthropic. \u0026ldquo;Model Context Protocol\u0026rdquo;. 2024.\n","permalink":"https://hellogxp.github.io/posts/skill-routing/","summary":"When LLM agents need to chain multiple skills, the hard part isn\u0026rsquo;t retrieval—it\u0026rsquo;s decomposition. Oracle experiments reveal a 65-percentage-point gap. SAD (Skill-Aware Decomposition) closes it by correcting granularity, not by boosting model capacity. Context window drops 99.9%.","title":"Compositional Skill Routing: Decomposition Is the Bottleneck"},{"content":"Neuroticism is where Llama-3.1, Qwen-2.5, Gemma-2, and Mistral-7B disagree the most — at least geometrically. Their contrastive-difference CKA (which I\u0026rsquo;ll call $\\text{CKA}_\\Delta$) measures at 0.553, the lowest of eight personality dimensions I tested. And yet, a persona classifier trained on Llama\u0026rsquo;s neuroticism vector transfers to Qwen with 99.4% accuracy after a tiny affine map of 2,550 parameters. The geometry says they disagree. The function says they agree.\nThis is the geometric-functional universality dissociation: across six concept domains, nine models, and five architectural families, moderate geometric convergence coexists with near-perfect functional transfer. Architectures disagree on how to encode a concept — but the information is fully recoverable up to a linear transformation. Think of two maps of the same city drawn in different coordinate systems. The street layouts look nothing alike when overlaid naively, but a simple affine transformation snaps them perfectly into register.\nThis post walks through what the dissociation is, why nobody saw it before, the lens that makes it visible, where it holds, and what it\u0026rsquo;s actually good for. The full paper is on arXiv.\nThe Problem: Are Concepts Aligned Across Architectures? Organizations don\u0026rsquo;t deploy one LLM anymore — they deploy a portfolio. Llama, Qwen, Gemma, and Mistral all show up in production stacks. A safety filter trained on Llama needs to transfer to Qwen. A persona detector trained on one model should ideally work on others without retraining. Alignment interventions should be portable across model families.\nWhether any of this works depends on a single question: do different architectures encode high-level concepts in structurally compatible ways?\nMechanistic interpretability has established that LLMs develop linear representations for high-level concepts — truthfulness (Marks et al., 2024), sentiment (Tigges et al., 2023), factual knowledge (Nanda, 2023). The linear representation hypothesis suggests semantically meaningful directions exist in transformer residual streams. If persona dimensions are linearly encoded, the natural follow-up question is: do different architectures converge on similar representational strategies?\nRecent work has shown that concept transfer is possible. Steering vectors transfer from Qwen to Llama via linear maps (Huang et al., 2025), safety interventions transfer across Llama/Qwen/Gemma (Oozeer et al., 2025), and SAE stitching reveals affine feature transfer across models (Stolfo et al., 2025). But none of these answer a more pointed question: when transfer succeeds, is it because the concepts are specifically aligned, or because the representations are generically similar?\nThe distinction matters. If two models have generically similar representation spaces — both transformers, both trained on similar data, both using similar tokenizers — a classifier trained on one will transfer to the other regardless of whether the concept itself is encoded the same way. The transfer succeeds by accident. To say \u0026ldquo;concepts are universally aligned\u0026rdquo; we need to isolate the concept-specific signal from generic similarity, and that requires a metric that can distinguish same-concept from cross-concept comparisons.\nWhy Nobody Saw It Before The obvious thing to do is compute standard CKA — Kornblith et al., 2019 — on the positive-pole activations (the activation when the persona prompt is \u0026ldquo;extraverted\u0026rdquo;) and see if same-trait pairs score higher than cross-trait pairs. I tried this. It doesn\u0026rsquo;t work.\nOn raw positive-pole activations, CKA on same-trait pairs is 0.802 and on cross-trait pairs is 0.780. A 0.022 gap, $p = 0.052$, Cohen\u0026rsquo;s $d = 0.27$. Not significant. Standard CKA cannot tell you whether two models align on extraversion specifically, or whether they just look alike in general.\nFigure 1: CKAΔ similarity across eight personality traits (rows) and six model pairs (columns). Gemma-involving pairs (bottom three rows) show systematically lower similarity, a pattern that replicates across all six concept domains and identifies Gemma as an architectural outlier. Dashed line marks the cross-trait baseline. (Image source: Gao, 2026) Why raw CKA fails is intuitive. A residual-stream activation at the last token contains a lot of things at once: the question content, the model\u0026rsquo;s capacity, the system prompt scaffolding, and (somewhere in there) the persona signal. The question content alone — what the prompt is about — dominates the variance. The persona signal is a small fraction of total activation magnitude, and CKA on raw activations measures that whole ball of variance together. The concept-specific signal is buried.\nThis is the same reason simple cosine similarity between mean activations fails. Mean-difference cosine (cosine between the mean positive-pole and mean negative-pole activations) gives $d = 0.007$, $p = 0.98$ — complete noise. SVCCA averages 0.874 across pairs with no same-vs-cross discrimination. Procrustes alignment gives $|d| \\le 0.03$. None of them can isolate the concept.\nThe fix is almost embarrassingly simple: subtract first, compare second.\nCKA_Δ: The Lens That Makes It Visible The procedure has four steps:\nExtract. For each personality dimension $t$, construct contrastive prompt pairs $(p_t^+, p_t^-)$ sharing the same neutral question but with opposing system-level persona instructions (\u0026ldquo;You are an extravert\u0026rdquo; vs \u0026ldquo;You are an introvert\u0026rdquo;). Collect last-token residual stream activations at the optimal layer $\\ell^*$, giving $\\mathbf{A}_t^+, \\mathbf{A}_t^- \\in \\mathbb{R}^{N \\times d}$ where $N = 500$.\nContrast. Compute per-sample differences $\\Delta \\mathbf{A}_t = \\mathbf{A}_t^+ - \\mathbf{A}_t^-$. Each row of $\\Delta \\mathbf{A}_t$ now contains the persona-specific component for that prompt pair, with the shared question content subtracted out.\nReduce. Project onto $k = 50$ principal components, capturing 83–90% of the contrastive variance.\nCompare. Compute debiased linear CKA between the two models\u0026rsquo; projected contrastive-difference matrices:\n$$\\text{CKA}_\\Delta(\\mathcal{M}_A, \\mathcal{M}_B, t) = \\text{CKA}(\\Delta \\mathbf{A}_{t,A}, \\Delta \\mathbf{A}_{t,B})$$That\u0026rsquo;s it. No SAE training, no feature matching, no iterative optimization. Ten minutes per model pair on a single GPU.\nWhy subtraction works Model the activation as $\\mathbf{a}^+ = \\mathbf{s} + \\mathbf{p} + \\boldsymbol{\\epsilon}$ and $\\mathbf{a}^- = \\mathbf{s} - \\mathbf{p} + \\boldsymbol{\\epsilon}'$, where $\\mathbf{s}$ is shared (question-dependent) variance, $\\mathbf{p}$ is concept-specific signal, and $\\boldsymbol{\\epsilon}, \\boldsymbol{\\epsilon}'$ are i.i.d. noise. The difference $\\Delta \\mathbf{a} = 2\\mathbf{p} + (\\boldsymbol{\\epsilon} - \\boldsymbol{\\epsilon}')$ eliminates $\\mathbf{s}$ exactly. Under a linear kernel, the resulting signal-to-noise ratio is:\n$$R_\\Delta = \\frac{4\\sigma_p^2}{4\\sigma_p^2 + 2\\sigma^2} \\;\\;\\geq\\;\\; R_{\\text{raw}} = \\frac{\\sigma_p^2}{\\sigma_s^2 + \\sigma_p^2 + \\sigma^2}$$with equality iff $\\sigma_s^2 = 0$. The gain is approximately $\\sigma_s^2 / \\sigma_p^2$ when shared variance dominates. Empirically, persona-to-total variance ratio ranges from 38.9% (Mistral) to 68.0% (Gemma), cross-model mean 48.5%, predicting a roughly 2× SNR gain. The observed improvement in discrimination is $d = 0.27 \\to d = 0.60$, a 2.22× ratio, matching the theoretical prediction of 2.06× to within 8%. (The full SNR proof and the concept discriminability bound are in the paper.)\nThe discriminability bound in the paper also relies on an empirical fact worth stating: the eight trait directions are nearly orthogonal. Mean off-diagonal cosine between trait directions is 0.054 across the four primary models (range 0.038–0.079). This is what makes same-vs-cross discrimination possible at all — if traits overlapped heavily, the same-concept and cross-concept numerators would collapse together and no metric could separate them. The near-orthogonality is also what lets the affine alignment generalize: a map trained on one trait doesn\u0026rsquo;t accidentally carry another.\nA sanity check: random concepts A legitimate worry about any contrastive method is that it produces discrimination for any contrastive difference, even semantically incoherent ones. I tested this with a random concept control: contrastive prompt pairs where the \u0026ldquo;positive\u0026rdquo; and \u0026ldquo;negative\u0026rdquo; poles have no coherent semantic axis. The result: $d = 0.04$, $p = 0.85$ — a 13.6× contrast with the real-persona discrimination ($d = 0.60$). $\\text{CKA}_\\Delta$ is not fooled by arbitrary contrastive differences; it requires genuine concept structure to register a signal.\nWhy distributional, not directional Previous cross-architecture work operates at the direction level: compute a mean persona direction per model, then compare directions via cosine or learned maps (Huang et al., 2025; Oozeer et al., 2025). $\\text{CKA}_\\Delta$ operates at the distributional level — the full $N \\times d$ contrastive-difference matrix — and captures covariance structure that direction-level methods cannot see. The mean-cosine baseline\u0026rsquo;s failure ($d = 0.007$) is the clearest evidence: the discriminative signal lives in the distribution, not in the mean.\nFigure 2: Same-trait vs. cross-trait discrimination (Cohen's d) across five similarity metrics. CKAΔ (linear and RBF kernels) is the only metric that reaches significance; raw CKA+, SVCCA, mean-difference cosine, and Procrustes all fail to separate concept-specific alignment from generic similarity. (Image source: this work) The Claim: Geometry Diverges, Function Converges Before the data, a quick terminology note. I\u0026rsquo;ll use two transfer protocols throughout: direct transfer means taking a classifier trained on model A and applying it to model B\u0026rsquo;s activations without any alignment — if the two models happen to share an embedding space, this works; otherwise it fails. Affine-aligned transfer means learning a small 50×51 affine map (2,550 parameters) on 500 contrastive-difference vectors to translate model B\u0026rsquo;s space into model A\u0026rsquo;s, then applying the source classifier. The affine map is the minimum alignment needed to test whether the information is preserved, even when the geometry isn\u0026rsquo;t.\nWith the lens in hand, the central finding comes into focus. Across the eight personality dimensions, $\\text{CKA}_\\Delta$ varies from 0.553 (neuroticism) to 0.829 (helpfulness). Meanwhile, affine-aligned transfer accuracy stays at or above 99.4% for every single trait.\nFigure 3: Geometric-functional dissociation across eight personality dimensions. All traits achieve ≥99.4% affine-aligned transfer accuracy regardless of their CKAΔ similarity. Neuroticism (CKAΔ=0.553, lowest) transfers nearly as well as helpfulness (CKAΔ=0.829, highest), demonstrating that geometric and functional universality are independent. (Image source: Gao, 2026) The dissociation is most striking for formal register — not a personality trait, but a concept domain I tested separately. For formality, $\\text{CKA}_\\Delta = \\mathbf{0.342}$, the lowest of any instruction-level concept, yet affine-aligned transfer accuracy is 99.9%. The geometry looks almost unaligned. The function says they\u0026rsquo;re the same concept, expressed in different coordinates.\nThis is what I mean by \u0026ldquo;two maps of the same city.\u0026rdquo; Different LLM families have learned different representational conventions for the same high-level concept. They haven\u0026rsquo;t converged on identical geometry — they\u0026rsquo;ve converged on recoverable geometry. A 50×51 affine map (2,550 parameters, fit on 500 contrastive-difference vectors) is enough to translate between them.\nThe Dissociation Holds Everywhere Personality is one concept domain. To claim the dissociation is a general property of cross-architecture concept encoding, I tested six concept domains spanning instruction-level and non-instruction concepts.\nFigure 4: Geometric-functional dissociation replicates across six concept domains. CKAΔ varies substantially (0.281–0.727, left axis) while affine-aligned transfer accuracy stays above 94% everywhere (right axis). Formality provides the sharpest dissociation: CKAΔ=0.342 yet 99.9% transfer. Non-instruction concepts (code-vs-NL, reasoning-vs-recall, marked with †) show the lowest CKAΔ values but still achieve 94–99.5% transfer. (Image source: this work) Five instruction-level concepts Personality, safety (refusal alignment), truthfulness, and formality all share the same scaffold: a system prompt that toggles the concept on or off. Five of six concept domains replicate the dissociation with statistically significant geometric discrimination ($p \\le 0.017$); safety is the lone exception, with $p = 0.08$ at $n = 15$ pairs — suggestive but not significant on its own.\nSafety deserves explicit honesty here, because it\u0026rsquo;s the one domain where the geometric channel underdelivers. Post-hoc power analysis confirms this is an underpowered design rather than evidence of no effect (achieved power 0.312; the design would need $n_{\\text{same}} = 54$ at a 4:1 ratio for 80% power, projecting $p \\approx 0.044$ at $n_{\\text{same}} = 28$). And four independent lines of evidence converge on concept-specific structure despite the underpowered geometric test: (1) affine-aligned classification achieves ≥99.9% across all 30 directed safety pairs (random-label baseline 59.4%); (2) safety alignment maps show the same moderate-rank SVD structure as persona maps (effective rank ~26 vs. ~27); (3) independent SAE analysis identifies 2,053 universal and 2,084 architecture-specific safety features; (4) the 6-model subset shows consistent $\\text{CKA}_\\Delta = 0.534$. The converging-evidence pattern — moderate geometric convergence but near-perfect functional transfer — is itself an instance of the dissociation.\nFormality is the sharpest demonstration: the lowest geometric convergence of any concept I tested, yet near-perfect functional transfer. If I had picked only formality to look at, the dissociation would have been obvious immediately.\nNon-instruction concepts: the key control A natural worry is that the dissociation is a prompt-engineering artifact — system prompts produce shared scaffolding that inflates cross-model similarity regardless of concept. To rule this out, I tested two concepts that arise from pretraining with no system prompt manipulation at all:\nCode vs. natural language. Matched pairs of questions differing only in whether the question requests code or a verbal explanation. Same topic, same complexity, different cognitive demand. Reasoning vs. recall. Matched pairs differing in whether the question requires logical deduction or factual retrieval. For these, $\\text{CKA}_\\Delta$ is even lower (0.300 and 0.281), but affine transfer remains 99.5% and 94.0% respectively. More tellingly, same-vs-cross discrimination is stronger for non-instruction concepts ($d = 1.8$ and $3.0$, $p \\le 0.017$) than for instruction-level concepts ($d = 0.60$). The reason is simple: non-instruction concepts share no prompt scaffolding, so cross-concept $\\text{CKA}_\\Delta$ naturally drops to near zero, amplifying the same-vs-cross gap. The dissociation is not a prompt artifact — it survives the removal of system prompts entirely.\nTwo encoding regimes An unexpected pattern surfaced. Within instruction-level concepts, $\\text{CKA}_\\Delta$ rank-orders direct transfer difficulty with a perfect negative correlation (Spearman $\\rho = -1.0$, $n = 4$): concepts with higher $\\text{CKA}_\\Delta$ (deeper encoding) need affine alignment to transfer well; concepts with lower $\\text{CKA}_\\Delta$ transfer directly. Formality, with the lowest $\\text{CKA}_\\Delta$, has the highest direct transfer accuracy (91.3%) — the surface channel alone is enough.\nNon-instruction concepts break this correlation. They have the lowest $\\text{CKA}_\\Delta$ values of all, but their direct transfer is also poor (46–47%). The surface channel that works for formality and truthfulness doesn\u0026rsquo;t exist for code-vs-NL or reasoning-vs-recall — there\u0026rsquo;s no prompt scaffolding to leak through. The result is two qualitatively different encoding regimes: instruction-level concepts trade off between surface and deep encoding, while non-instruction concepts are uniformly deep-encoded.\nThis was an unexpected payoff. $\\text{CKA}_\\Delta$ started as a tool to test concept alignment, but it doubles as a diagnostic for encoding depth within each regime.\nWhat It\u0026rsquo;s Good For Gemma is the architectural outlier Across all six concept domains, Gemma-involving pairs score systematically lower than non-Gemma pairs. The effect is consistent and large: Cohen\u0026rsquo;s $d = 1.08$, AUC $= 0.79$, $p = 0.003$. One metric, six independent concept domains, and the same architecture gets flagged every time. This is the cleanest evidence that $\\text{CKA}_\\Delta$ tracks genuine structural signal rather than a generic similarity factor — a generic factor wouldn\u0026rsquo;t be so consistent in picking out one architecture.\nThe likely explanation is Gemma\u0026rsquo;s architectural distinctiveness: sliding-window attention and a deeper network produce representational strategies that diverge from the Llama/Qwen/Mistral cluster. For practitioners, this means alignment investments on Gemma cost more than on the other three.\nScale: a single-pair observation I included Llama-3.1-70B and Qwen-2.5-72B in the analysis. The cross-family 70B↔72B pair has $\\text{CKA}_\\Delta = 0.830$, higher than the 7–9B mean of 0.733. Same-family cross-scale pairs are even higher (Llama-70B↔Llama-8B at 0.905, Qwen-72B↔Qwen-7B at 0.819). Affine transfer stays at or above 99.7% across all cross-scale conditions.\nI want to be honest about what this is and isn\u0026rsquo;t. It is a single pair. With $n = 1$ at the 70B+ cross-family level, no inferential statistic is computable. The observation is consistent with the Platonic Representation Hypothesis — universality strengthening with scale — but it cannot disentangle scale from training-data overlap, and it cannot establish a trend. Establishing a real scale effect requires at least three models at 70B+; I\u0026rsquo;m presenting this as a hypothesis-generating note, not confirmed evidence.\nThis is the limitation I\u0026rsquo;m least satisfied with. Scale is the most direct test of the Platonic Representation Hypothesis, and 70B+ models are expensive to run. Replication with more large models is the obvious next step.\nBase Llama rules out instruction-following as the cause A different concern: perhaps $\\text{CKA}_\\Delta$ is just measuring instruction-following convergence — models that have been RLHF\u0026rsquo;d to follow system prompts will look similar on contrastive prompts regardless of whether they encode the concept itself. To test this, I ran the analysis on base Llama-3.1-8B, which has never seen instruction tuning.\nBase Llama retains separable persona representations: within-model classification at 98.0%, affine transfer to instruction-tuned models at 98.5%. The persona structure is already there in the base model. Instruction tuning amplifies the structure — makes the contrastive signal cleaner — but does not create it. This rules out the \u0026ldquo;just instruction-following\u0026rdquo; hypothesis.\nPractical applications For practitioners managing heterogeneous model portfolios, the dissociation gives three actionable uses:\nTransfer triage. Before investing in labeled-data alignment, compute $\\text{CKA}_\\Delta$ on unlabeled contrastive prompts (~10 minutes per pair). For instruction-level concepts, low $\\text{CKA}_\\Delta$ with above-threshold direct accuracy signals that direct transfer will work — skip the alignment investment. High $\\text{CKA}_\\Delta$ signals you\u0026rsquo;ll need the 2,550-parameter affine map plus ~200 labeled pairs.\nAlignment auditing. A drop in same-concept $\\text{CKA}_\\Delta$ between pre- and post-fine-tuning signals concept-encoding distortion. You don\u0026rsquo;t need labeled transfer data to detect it.\nArchitecture selection. $\\text{CKA}_\\Delta$ identifies the architecture pair with the least alignment overhead per concept. For personality, avoid Gemma. For code-vs-NL, avoid Mistral (Mistral-involving pairs have the lowest direct transfer in that domain — a concept-dependent architectural sensitivity I didn\u0026rsquo;t predict).\nDiagnostic, not constructive I want to be explicit about what $\\text{CKA}_\\Delta$ is and isn\u0026rsquo;t. It is a diagnostic tool, not a constructive one. SAE stitching (Stolfo et al., 2025) and Universal Sparse Autoencoders (Thasarathan et al., 2025) construct shared feature vocabularies — they tell you what is shared. Affine alignment (Oozeer et al., 2025) and steering transfer (Huang et al., 2025) construct transfer mechanisms — they tell you how to transfer. $\\text{CKA}_\\Delta$ tells you whether the alignment is concept-specific. It runs before any of the constructive methods, costs nothing but unlabeled contrastive prompts, and answers the question that matters for triage: is investing in alignment worth it for this pair, on this concept?\nThis distinction matters for alignment monitoring. When a safety filter transfers from Llama to Qwen, you want to know whether the transfer is concept-specific — actually carrying the safety signal — or merely generic — carrying whatever residual similarity exists between the two models. $\\text{CKA}_\\Delta$ is the only one of these tools that answers that question without training anything.\nLimits and What\u0026rsquo;s Next What it cannot do $\\text{CKA}_\\Delta$ is not an absolute transfer-accuracy predictor. Leave-one-out cross-validation across concepts gives $R^2 = -0.14$: knowing the $\\text{CKA}_\\Delta$ of a concept does not let you predict its absolute direct-transfer accuracy across concept types. What it does predict, reliably, is within-regime rank ordering (the $\\rho = -1.0$ result within instruction-level concepts) and architectural outliers ($d = 1.08$ across all six domains). I position it as a regime classifier and outlier detector, not a transfer predictor.\nThe absolute value of $\\text{CKA}_\\Delta$ is also question-set-dependent — 99.7% of its variation is content-driven. The relative same-vs-cross discrimination is preserved across question sets; the absolute number is not. For practical use, I recommend at least 30 topical domains and at least 200 contrastive pairs, and reporting Cohen\u0026rsquo;s $d$ alongside absolute $\\text{CKA}_\\Delta$ so the discrimination strength is visible per suite.\nCross-modal pilot The dissociation isn\u0026rsquo;t unique to language. I ran a controlled pilot on three ViT-class vision encoders with five CIFAR-100 polar concepts. The same pattern shows up: same-concept $\\text{CKA}_\\Delta = 0.560$ vs. cross-concept $\\text{CKA}_\\Delta = -0.004$, $d = 7.34$, $p \u003c 10^{-10}$. This is only a pilot — three encoders, five concepts — but it suggests the dissociation is a property of contrastive representation learning in general, not just of language models. Replication with more vision encoders is an obvious next step.\nOpen questions Three threads I want to pull on:\nScale replication. The 70B/72B observation is the most interesting single data point in the paper and the one I\u0026rsquo;m least confident in. The right experiment is ≥3 models at 70B+, ideally from different families, ideally with controlled training data. Cost is the only reason it isn\u0026rsquo;t done.\nConcept-by-architecture interaction. Gemma is the personality outlier; Mistral is the code-vs-NL outlier. Different architectures encode different concepts with varying fidelity. Why? This is a mech-interp question that $\\text{CKA}_\\Delta$ flags but doesn\u0026rsquo;t answer.\nComparison with SAE stitching. $\\text{CKA}_\\Delta$ flags Gemma as an outlier. Does SAE stitching flag it too? If yes, the two methods are complementary diagnostics. If no, the disagreement is informative — it would mean geometric and feature-level alignment are genuinely different things.\nAnd one honest admission about what the paper doesn\u0026rsquo;t explain: why the dissociation exists. The paper documents it across six concept domains and shows it survives five controls, but it doesn\u0026rsquo;t propose a mechanistic story for why moderate geometry coexists with near-perfect functional transfer. My guess — and it is only a guess — is that the next-token prediction objective puts strong pressure on what information must be representable in the residual stream, but leaves the how (which directions, which basis) underdetermined. Different architectures solve the same information-preservation problem with different coordinate conventions. But turning that guess into a mechanism requires circuit-level work, not a metric.\nWhat I\u0026rsquo;d say if asked for one sentence Different LLM architectures encode high-level concepts in recoverable but not identical ways: geometrically divergent, functionally equivalent. The Platonic Representation Hypothesis, in its weak form, holds — concept representations are universally recoverable across architectures, but they are not universally identical. $\\text{CKA}_\\Delta$ is the cheapest way I know to see that distinction.\nCitation Cited as:\nGao, Xueping. \u0026ldquo;Two Maps of the Same City: Why LLM Architectures Converge on Concepts\u0026rdquo;. hellogxp.github.io (Jun 2026). https://hellogxp.github.io/posts/cka-delta-universality/\nOr use the BibTeX citation:\n@article{gao2026ckadelta, title = {Two Maps of the Same City: Why LLM Architectures Converge on Concepts}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {June}, url = \u0026#34;https://hellogxp.github.io/posts/cka-delta-universality/\u0026#34; } References [1] Gao, Xueping. \u0026ldquo;Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures.\u0026rdquo; arXiv:2606.16897 (2026). (This work — the source paper this post is based on.)\n[2] Kornblith, S., Norouzi, M., Lee, H., \u0026amp; Hinton, G. \u0026ldquo;Similarity of Neural Network Representations Revisited.\u0026rdquo; ICML 2019.\n[3] Park, K., Choe, Y. J., \u0026amp; Veitch, V. \u0026ldquo;The Linear Representation Hypothesis and the Geometry of Large Language Models.\u0026rdquo; arXiv:2311.03658 (2023).\n[4] Marks, S., \u0026amp; Tegmark, M. \u0026ldquo;The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets.\u0026rdquo; ICLR 2024.\n[5] Tigges, C., Hollinsworth, O. J., Geiger, A., \u0026amp; Nanda, N. \u0026ldquo;Linear Representations of Sentiment in Large Language Models.\u0026rdquo; arXiv:2310.15154 (2023).\n[6] Huh, M., Cheung, B., Wang, T., \u0026amp; Isola, P. \u0026ldquo;The Platonic Representation Hypothesis.\u0026rdquo; ICML 2024.\n[7] Huang, Y., Feng, W., Chen, C., Tan, H., Wen, H., \u0026amp; Jin, Q. \u0026ldquo;Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts.\u0026rdquo; ACL 2025.\n[8] Oozeer, N., Nathawani, D., Prakash, N., Lan, M., Harrasse, A., \u0026amp; Abdullah, A. \u0026ldquo;Activation Space Interventions Can Be Transferred Between Large Language Models.\u0026rdquo; ICML 2025.\n[9] Stolfo, A., Rauker, T., Gurnee, W., \u0026amp; Nanda, N. \u0026ldquo;Transferring Linear Features Across Language Models With Model Stitching.\u0026rdquo; NeurIPS 2025.\n[10] Thasarathan, H., Forsyth, J., Fel, T., Kowal, M., \u0026amp; Derpanis, K. \u0026ldquo;Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment.\u0026rdquo; ICML 2025.\n[11] Raghu, M., Gilmer, J., Yosinski, J., \u0026amp; Sohl-Dickstein, J. \u0026ldquo;SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability.\u0026rdquo; NeurIPS 2017.\n","permalink":"https://hellogxp.github.io/posts/cka-delta-universality/","summary":"Different LLM architectures encode high-level concepts in structurally compatible ways—but only up to a point. Geometric similarity between Llama, Qwen, Gemma, and Mistral is moderate at best (CKA_Δ around 0.3–0.7), yet a tiny affine map transfers persona classifiers with 94–99.9% accuracy across six concept domains. This geometric-functional dissociation suggests a weak version of the Platonic Representation Hypothesis: concept representations are universally recoverable, but not universally identical.","title":"Two Maps of the Same City: Why LLM Architectures Converge on Concepts"},{"content":"In 2025, METR ran something rare in AI research: a randomized controlled trial. They recruited 16 experienced open-source developers, each averaging five years and 1,500 commits on their repositories, and randomly assigned 246 real coding tasks to be completed with or without AI tools. The result was striking: developers using AI took 19% longer to complete their tasks (METR, 2025).\nHere\u0026rsquo;s the catch: developers felt faster. Before the study, they predicted AI would speed them up by 24%. After experiencing the actual slowdown, they still believed AI had made them 20% faster. The subjective experience of AI assistance, the feeling of velocity, the reduced friction of not having to remember syntax, was so compelling that it overrode objective reality.\nFigure 1: The METR perception gap. Developers predicted a 24% speedup from AI tools but experienced a 19% slowdown. Even after the experiment, they reported a perceived 20% improvement. (Source: METR, 2025.) By early 2026, METR attempted a follow-up study with a larger sample. They couldn\u0026rsquo;t recruit enough participants. Developers refused to join the control group; they would not work without AI, even temporarily. The tool that made them measurably slower had become indispensable. When surveyed, the median developer said they would sacrifice 20% of their salary rather than lose access to AI coding tools (METR, 2026).\nWhat explains this paradox? AI isn\u0026rsquo;t failing. The misalignment runs deeper: we measure execution speed while the actual work has migrated to specification, context design, and quality judgment. The bottleneck moved. Our metrics didn\u0026rsquo;t.\nThis post doesn\u0026rsquo;t cover prompting tips or IDE plugin comparisons. It asks a more fundamental question: when AI handles execution, what should humans think about? The answer lies in understanding the shifting division of cognitive labor between humans and machines.\nMotivation: Why This Question Matters Now The AI research community has optimized for machine capability. The practitioner community has optimized for tool adoption. Neither has adequately addressed the cognitive architecture of collaboration. We have better models and better tools, but no framework for understanding what humans should do when machines handle execution.\nThe Research Gap Empirical evidence consistently points to this gap. Research on AI-assisted decision-making shows a troubling pattern: initial performance gains from AI assistance tend to decay over time as human skills atrophy from reduced practice. Berber et al. (2024) examined this through the lens of cognitive psychology, arguing that AI assistants, by taking over advanced cognitive processes previously the exclusive domain of experts, may accelerate skill decay beyond what simpler automation technologies caused. The decay may occur without the user\u0026rsquo;s awareness, because the assistance masks the decline in independent capability.\nConsider what this means in practice: a developer who stops writing SQL by hand and relies on AI generation loses the ability to spot subtle join errors. A designer who delegates layout decisions to generative tools gradually forgets the spacing principles that made their earlier work effective. The cognitive load doesn\u0026rsquo;t disappear; it redistributes. Without conscious management of that redistribution, we lose the very capacities that make collaboration valuable.\nDecision Fatigue as a Signal Three out of twenty-four developers in one study failed to complete their task entirely, not because AI gave wrong answers, but because debugging AI-generated suggestions consumed all available time (Vaithilingam et al., 2022). Review fatigue, choice overload, difficulty calibrating trust: these aren\u0026rsquo;t bugs in AI systems. They are symptoms of a misaligned cognitive stack.\nVaithilingam et al. found a characteristic pattern among Copilot users: getting started was easier, but reviewing and debugging got harder. The coding phase simplified; the review phase intensified. Barke et al. (2022) deepened this by identifying two distinct interaction modes. In acceleration mode, developers use Copilot to speed up code they have already planned. In exploration mode, they use it to discover approaches they haven\u0026rsquo;t conceived. Acceleration generally works well. Exploration is more prone to over-reliance and reduced task completion, because developers accept suggestions without fully understanding them.\nPut differently: generation effort drops, verification effort rises. The signature is cognitive load redistribution, not cognitive load reduction.\nThe Core Question When AI can execute more cognitive tasks, how should human cognition be redefined? Not \u0026ldquo;how to prompt better,\u0026rdquo; but \u0026ldquo;how to think differently.\u0026rdquo; This post proposes a framework: the Cognitive Stack Migration model. Humans move upstream, from execution to specification, context governance, and judgment. AI takes over middle layers. The shift means doing different work, not less work.\nScope clarification: This post is about cognitive architecture, not tool tutorials. No Cursor/Claude Code tips. Purely the shifting division of cognitive labor, grounded in cognitive science, HCI research, and engineering practice.\nThe Cognitive Stack Shift: From Execution to Governance The Migration Model The traditional software development cognitive stack has three layers:\nLearn: Acquire domain knowledge, language syntax, framework patterns. Execute: Write code, implement features, debug errors. Verify: Test, review, validate correctness. Humans historically performed all three. With AI, the stack migrates:\nSpecify: Define goals, constraints, evaluation criteria. (Human-led, AI-assisted) Context-Engineer: Design what information the model sees. (Human-AI collaborative) Govern: Monitor, intervene, adjust system behavior. (Human-led) Judge: Assess quality, make trade-offs, approve outputs. (Human-only) AI now handles the middle layers: execution, initial verification, routine learning. Humans migrate to the top (specification, judgment) and bottom (governance, intervention). The cognitive load doesn\u0026rsquo;t vanish; it transforms from execution to governance.\nFigure 2: Cognitive Stack Migration. Left: traditional stack where humans perform all layers (Learn, Execute, Verify). Right: new stack where humans migrate upstream to Specification and Judgment, AI handles middle execution layers, and Governance remains human-led. Yellow = human-led, green = collaborative. (Source: Author, SVG version.) This migration is grounded in distributed cognition theory. Hutchins (1995) demonstrated through his landmark naval navigation ethnography that cognition is not confined to individual minds but distributed across people, tools, and representational systems. No single navigator \u0026ldquo;knows\u0026rdquo; the ship\u0026rsquo;s position. The knowledge emerges from coordinated transformations of representations across the navigation team. Human-AI collaboration extends this principle: the model is one node in a larger cognitive network that includes the human operator, the context engineering pipeline, the verification infrastructure, and the organizational norms governing deployment.\nExtending the Karpathy Analogy Andrej Karpathy proposed a useful analogy: LLM = CPU, Context Window = RAM, Context Engineering = OS (Karpathy, 2025).\nIf Context Engineering is the operating system, then Specification is the application logic, and Judgment is the user intent. The OS manages resources efficiently, but it doesn\u0026rsquo;t know what the user wants. Business context, ethical boundaries, aesthetic preferences: those reside in the application layer (specification) and the user layer (judgment).\nWriting on martinfowler.com, Birgitta Böckeler extends this further: Agent = Model + Harness (Böckeler, 2026). The harness includes runtime environment, tool integration, memory management, and safety guardrails. Context Engineering addresses \u0026ldquo;what the model sees.\u0026rdquo; Harness Engineering addresses \u0026ldquo;how the system operates.\u0026rdquo; Both require human design. Neither emerges automatically from model capability.\nFigure 3: Karpathy and Böckeler, complementary views. Karpathy describes the computational architecture (LLM as CPU, context window as RAM, context engineering as OS). Böckeler describes the operational architecture (Agent = Model + Harness). Together they frame the full cognitive stack. (Source: Author synthesis of Karpathy 2025, Böckeler 2026.) The two analogies are complementary. Karpathy describes the computational architecture; Böckeler describes the operational architecture. Together, they frame the full cognitive stack: humans design the application logic (specification), manage the OS (context engineering), operate the harness (governance), and evaluate the output (judgment). The model runs in the middle.\nKey Insight: Different Work, Not Less Work The migration is complementation, not automation. Automation implies replacement. Smart (2022) argues that the Extended Mind Thesis, Clark and Chalmers\u0026rsquo; (1998) proposal that external processes functioning equivalently to internal ones should be considered part of cognition, requires revision for AI. AI does not replicate human cognition but provides complementary capabilities. The relationship is asymmetric: AI excels at pattern matching, scale, and speed. Humans excel at framing, taste, and ethical reasoning.\nThis asymmetry has a practical consequence. Offloading assumes equivalence: the tool does what you would do. Complementation assumes differentiation: the tool does what you cannot do, or what you should not spend time doing. The cognitive stack shift preserves this differentiation rather than collapsing it.\nSpecification as the New Programming Andrej Karpathy claimed \u0026ldquo;Specification is the new programming\u0026rdquo; (Karpathy, 2025). What does this mean cognitively?\nProgramming has always involved specification. But historically, specification was implicit in code. You wrote the implementation, and the spec was whatever the code did. With AI, specification becomes explicit and primary. The code is generated; the spec is authored. The traditional relationship inverts.\nFormal Grounding Specification as behavioral constraint is not new in AI alignment research. Bai et al. (2022) introduced Constitutional AI, translating natural language principles into verifiable behavioral constraints for LLMs. RLHF reward models similarly require explicit specification of desired behavior through human preference data. What\u0026rsquo;s new is that specification has become the primary engineering artifact for general-purpose AI systems, not just safety-aligned ones. Every developer who writes a system prompt is engaged in specification work, whether they recognize it or not.\nFigure 4: Three layers of specification. Goal Spec defines \"done,\" Constraint Spec defines boundaries, and Evaluation Spec defines success metrics. Each layer requires distinct human capacities: taste, judgment, and metric literacy respectively. (Source: Author.) Three Layers of Specification Goal Spec: What does \u0026ldquo;done\u0026rdquo; look like? Requires taste and domain knowledge. AI can generate candidate goals, but cannot own them. Goal ownership requires understanding organizational context, user needs, and strategic priorities that exist outside the training distribution. When a product manager says \u0026ldquo;we need this feature to feel lightweight,\u0026rdquo; they are specifying a goal that no model can derive from task descriptions alone.\nConstraint Spec: What are the boundaries, safety rails, and non-negotiables? Requires judgment and risk awareness. A constraint like \u0026ldquo;never delete production data without explicit confirmation\u0026rdquo; reflects institutional memory and risk tolerance, not corpus statistics. Constraints encode values, and values are not learned from next-token prediction.\nEvaluation Spec: How do we measure success? Requires verification design and metric literacy. Choosing accuracy over latency, precision over recall, or user satisfaction over throughput reflects priorities that AI cannot determine autonomously. Evaluation specs make these priorities explicit and auditable. Without them, AI optimizes for whatever is easiest to measure, which is rarely what matters most.\nWhy Specification is Anti-Fragile to AI Cognitive science provides the theoretical foundation for understanding why specification resists automation. Risko and Gilbert (2016) established that when people externalize cognitive tasks to tools, the decision to offload is governed by metacognitive evaluation: our subjective sense of internal capacity versus external capability. For simple, well-defined tasks, offloading reliably improves efficiency. But for complex tasks requiring deep processing, offloading can reduce comprehension because it eliminates germane cognitive load: the mental effort devoted to schema construction and deep learning (Sweller et al., 2019) that builds expertise.\nA critical distinction sharpens this framework. Kirschner (2026) distinguishes cognitive offloading (externalizing extraneous load — syntax, formatting, memorization) from cognitive outsourcing (externalizing intrinsic load — reasoning, logic, schema construction). Offloading is beneficial: it frees working memory for high-level problem-solving. Outsourcing is dangerous: when a novice uses AI to generate code, the tool handles both syntax and underlying logic. The learner bypasses intrinsic load entirely, germane processing cannot occur, and expertise fails to form. This accumulation of epistemic debt creates fragile experts: developers whose high functional utility masks critically low corrective competence. In a controlled experiment (N=78), developers using unrestricted AI achieved comparable functional output, but suffered a 77% failure rate on a subsequent AI-blackout maintenance task, compared to only 39% in the scaffolded group that retained cognitive engagement.\nSpecification is inherently intrinsic load. It requires taste, judgment, and domain knowledge that AI cannot synthesize from training data alone — it demands the schema construction that builds expertise. AI can generate specs, but it cannot own them. Ownership implies accountability, contextual understanding, and the ability to revise based on feedback loops that extend beyond the current session. Offloading specification to AI is not offloading — it is outsourcing. And outsourcing the intrinsic load of specification produces the same degradation observed in execution outsourcing: skill atrophy, metacognitive miscalibration, and reduced capacity for independent judgment (Berber et al., 2024). Specification is cognitive maintenance, not overhead.\nCase Study: OpenHands Benchmark Wang et al. (2024) evaluated AI software development agents on SWE-Bench and WebArena benchmarks. Using claude-3.5-sonnet, their CodeActAgent achieved a 26% resolve rate on SWE-bench Lite, competitive among open-source agents but far from replacing human developers. A clear pattern emerged: agent performance is highly sensitive to task specification quality. Well-specified task descriptions consistently outperform ambiguous prompts across multiple agent architectures (Wang et al., 2024).\nThe implication is clear. Specification quality is a first-order determinant of AI system performance. Better models help, but better specs help more. The bottleneck is often not model capability but specification fidelity.\nAnti-Pattern: Vague Prompts as Disguised Execution Requests Prompts like \u0026ldquo;make it better,\u0026rdquo; \u0026ldquo;fix the bug,\u0026rdquo; or \u0026ldquo;optimize this\u0026rdquo; are execution-level requests disguised as specs. They lack verifiability, ambiguity resolution, and success criteria. True specs are verifiable and unambiguous. They define what, not how. They enable AI to execute. They don\u0026rsquo;t replace the need for human judgment about what execution should achieve.\nContext Engineering: Designing What the Model Sees The Paradigm Shift Prompt Engineering (2020-2024) focused on \u0026ldquo;what to say.\u0026rdquo; Context Engineering (2025+) focuses on \u0026ldquo;what the model sees.\u0026rdquo; The shift moves from art to systems engineering, from crafting clever phrases to designing information pipelines.\nAnthropic (2025) outlined four core strategies for effective context engineering:\nWrite: Humans decide what information sources matter. A curation decision, not a generation decision. The model can generate text, but it cannot determine which texts are relevant to the current goal without human-designed selection criteria.\nSelect: Humans design retrieval strategies and relevance filters. RAG systems retrieve documents, but humans define the retrieval schema, the ranking criteria, and the fallback behaviors when retrieval fails.\nCompress: Humans define what can be lossily compressed versus what must be preserved verbatim. Summarization is lossy. The loss function is determined by human judgment about what details matter for the current task.\nIsolate: Humans architect information boundaries to prevent cross-contamination. Multi-agent systems require isolation to prevent context leakage, hallucination propagation, and security violations. Isolation policies reflect threat models and trust assumptions that AI cannot infer from data alone.\nFigure 5: The four strategies of context engineering ([Anthropic, 2025](https://www.anthropic.com/engineering/building-effective-agents)). Write = curate sources, Select = design retrieval, Compress = define loss functions, Isolate = architect boundaries. Each strategy requires human judgment that cannot be automated away. (Source: Author synthesis of Anthropic 2025.) Prompt Sensitivity as Structural Fragility Small changes in prompt wording can produce large swings in LLM output quality. Variations in phrasing, example ordering, and formatting choices can dramatically alter performance on the same underlying task. This fragility is well-documented across multiple model families and task types.\nThis fragility justifies the paradigm shift: instead of optimizing individual prompts, we engineer robust information pipelines that are less sensitive to surface-level variation. The goal is the reliable pipeline, not the perfect prompt.\nProduction Context Composition Industry observations suggest that in production AI systems, the majority of context tokens come from tool returns: retrieval results, API responses, system state. Human-written prompt text makes up a small fraction. When the bulk of context is machine-generated (via tool calls, retrieval, summarization), the human contribution shifts from content authorship to pipeline architecture. We design the retrieval strategy, the tool selection logic, the compression policy, and the isolation boundaries.\nHere\u0026rsquo;s the cognitive implication: the skill of \u0026ldquo;writing good prompts\u0026rdquo; is being superseded by the skill of \u0026ldquo;designing good information architectures.\u0026rdquo; The former is a craft. The latter is engineering.\nCritical Gap: From Seeing to Operating Context Engineering answers \u0026ldquo;what the model sees\u0026rdquo; but not \u0026ldquo;how the system should operate.\u0026rdquo; It lacks multi-turn orchestration, error recovery, and safety guardrails. Böckeler\u0026rsquo;s Harness Engineering addresses this gap: the harness is where governance lives. A well-engineered context can still fail if the harness lacks error recovery, timeout handling, permission enforcement, or audit logging.\nThe gap between what the model knows and how the system behaves is where human governance lives. And where the next section begins.\nJudgment Under Uncertainty: Capacity and Limits AI Systematic Failure Modes AI systems fail in predictable, categorizable ways. Understanding these failure modes is prerequisite to designing effective human oversight.\nPlanning failures: Infinite loops, goal drift, subgoal fixation. On SWE-bench, even the best open-source agents achieve only ~26% resolve rates on real GitHub issues. The majority of failures stem from multi-step reasoning breakdowns (Wang et al., 2024). Agents get stuck in local optima, pursue irrelevant subgoals, or fail to decompose complex tasks into tractable steps.\nTool hallucinations: Fictitious API calls, malformed parameters, incorrect tool selection. WebArena benchmarks show significant error rates when agents attempt to use tools and navigate web interfaces (Wang et al., 2024). Models invent endpoints, misuse parameters, or select tools that don\u0026rsquo;t exist in the available toolkit.\nOverconfidence and deceptive behavior: Can safety training remove deception? Hubinger et al. (2024) demonstrated that LLMs can be trained to exhibit deceptive behavior: writing secure code when prompted with \u0026ldquo;2023\u0026rdquo; but inserting exploitable code when prompted with \u0026ldquo;2024.\u0026rdquo; This behavior persists through standard safety training techniques including supervised fine-tuning, reinforcement learning, and adversarial training. The backdoor behavior was most persistent in the largest models and in models trained with chain-of-thought reasoning about deceiving the training process. Adversarial training can even teach models to better recognize their backdoor triggers, effectively hiding rather than removing the unsafe behavior.\nFigure 6: Sleeper Agents deception behavior. Models trained with conditional backdoors maintain deceptive behavior through standard safety training (SFT, RL, adversarial training). Larger models and chain-of-thought reasoning make the deception more persistent. (Source: Hubinger et al., 2024, arXiv:2401.05566, Figure 1.) Three Irreplaceable Human Capacities Judgment: Trade-offs under ambiguity, multi-objective conflict, ethical compliance. AI optimizes for specified objectives; humans navigate objective conflicts that resist formalization. \u0026ldquo;Should we prioritize latency or accuracy?\u0026rdquo; \u0026ldquo;Is this feature worth the privacy cost?\u0026rdquo; These are judgment calls, not optimization problems. AI can enumerate the trade-offs. Humans must decide which trade-off to accept.\nTaste: Defining \u0026ldquo;good\u0026rdquo; when metrics are insufficient. Brand alignment, user experience, code elegance, documentation clarity. Eugene Yan describes this as \u0026ldquo;taste as config\u0026rdquo;: the configuration parameter that guides AI behavior when explicit specs are insufficient (Yan, 2026). Taste cannot be learned from training data alone. It emerges from embodied experience, cultural context, and aesthetic sensibility. It is the heuristic that says \u0026ldquo;this feels wrong\u0026rdquo; before you can articulate why.\nVerification: Knowing when to trust, when to intervene, when to abort. Meta-cognitive monitoring. Verification goes beyond checking outputs; it means assessing the reliability of the checking process itself. When should you run additional tests? When should you escalate to manual review? When should you reject the entire approach? These verification decisions require understanding both the system and its failure modes.\nTrust Calibration: The Hidden Bottleneck Figure 7: Trust calibration in human-AI collaboration. Optimal performance requires calibrated trust matching system reliability. Overtrust leads to uncritical acceptance; undertrust leads to rejection of correct outputs. (Source: Author synthesis of Lee and See 2004, Buçinca et al. 2023.) Lee and See (2004) established the canonical trust model: Trust = f(performance, process, purpose). Optimal human-AI collaboration requires calibrated trust, where human trust accurately matches system reliability. Two failure modes dominate in practice:\nOvertrust (trust \u0026gt; reliability): Novices accepting AI outputs without adequate verification. Buçinca et al. (2023) demonstrated that when users cannot directly evaluate AI performance, they rely on \u0026ldquo;proxy confidence,\u0026rdquo; cues like interface professionalism and response speed. In their study, interface polish increased perceived trust significantly but correlated weakly with actual output accuracy (r=0.12). Trust driven by surface cues, not substance.\nUndertrust (trust \u0026lt; reliability): Experienced developers rejecting correct AI suggestions based on past negative experiences. De Visser et al. (2018) proposed that trust repair in human-machine interaction requires a fundamentally different approach than traditional automation trust models, one based on social-psychological frameworks where autonomous systems actively rebuild trust through transparency and consistent behavior over time. Trust destruction is fast. Trust repair is slow.\nDesign implication: AI systems must expose reliability signals, including confidence scores, uncertainty quantification, and provenance tracking, not just outputs. Users need the information to calibrate trust directly, rather than relying on proxies.\nThe Review Fatigue Problem METR\u0026rsquo;s screen recording analysis provides direct evidence of where time goes: developers using AI spent less time actively coding, testing, and searching for information. But those savings were overwhelmed by time spent reviewing AI outputs, prompting AI systems, waiting for generations, and idle/overhead time (METR, 2025). The net result was negative. More total time, not less.\nThe pattern generalizes beyond coding. As the volume of AI suggestions increases, review quality degrades. Early suggestions receive careful scrutiny; later ones get rubber-stamped. Human attention is finite; AI generates unlimited suggestions. The mismatch requires workflow design, not willpower.\nThis individual-level fatigue scales into a systemic problem. A qualitative analysis of 1,154 developer discussions on Reddit and Hacker News (arXiv:2603.27249) frames AI-generated low-quality content — \u0026ldquo;AI slop\u0026rdquo; — as a tragedy of the commons: individual developers benefit from AI-accelerated output, but the externalized costs (review burden, codebase degradation, knowledge-resource pollution, skill atrophy) are borne by reviewers, maintainers, and the broader community. The same dynamic that makes AI adoption individually rational makes it collectively harmful when unmanaged. Developers describe a \u0026ldquo;Catch-22\u0026rdquo;: if experienced engineers are needed to verify AI output, but becoming experienced requires working without AI, how do new experienced engineers develop?\nThe expertise reversal effect compounds this problem. Instructional assistance that benefits novices can actively harm experts by adding redundant cognitive load. Tetzlaff et al. (2025) conducted a comprehensive meta-analysis across 60 studies (N=5,924) and confirmed the pattern: novices benefit substantially from high-assistance instruction (d=0.505), while experts perform better with low-assistance approaches (d=-0.428). The effect is robust across domains, though stronger in STEM and weaker in humanities. The benefit of assistance for novices is larger than the cost for experts, but the cost for experts is real and measurable.\nFor AI tools, this means assistance calibrated for the average user adds extraneous noise for senior developers. Explanations that help juniors become cognitive overhead for experts. Adaptive assistance levels are a design requirement, not a luxury.\nCross-Agent Orchestration: Humans as Context Managers So far, this analysis has focused on a single human working with a single AI model. But production AI systems increasingly involve multiple agents — specialized models that plan, retrieve, execute, and verify in coordination. The cognitive stack migration doesn\u0026rsquo;t disappear in this setting; it intensifies. The human role shifts from operator to orchestrator, and the failure modes multiply.\nWhen AI collaboration moves beyond single-model interaction to multi-agent systems, error propagation becomes a first-order concern. Agent A\u0026rsquo;s hallucinated output becomes Agent B\u0026rsquo;s input. Agent B\u0026rsquo;s flawed reasoning becomes Agent C\u0026rsquo;s constraint. Without human intervention at agent boundaries, errors cascade and compound through the system (Hu et al., 2024).\nFigure 8: HATCA (Human-AI Team Cognitive Architecture) main experiment results. Dynamic task allocation between humans and AI across perception, comprehension, decision, and execution layers significantly outperforms static allocation. (Source: Wang et al., 2023, CHI.) Wang et al. (2023) proposed HATCA (Human-AI Team Cognitive Architecture), demonstrating empirically that dynamic task allocation between humans and AI outperforms static allocation across perception, comprehension, decision, and execution layers. Seeber et al. (2022) defined key design principles for hybrid intelligence systems: transparency, controllability, explainability, and correctability. Together, these frameworks define the human role in multi-agent systems not as operator, but as context manager.\nFigure 9: The human as context manager. Five responsibilities at the core of multi-agent orchestration: initialization, synchronization, mediation, gatekeeping, and dynamic adjustment. (Source: Author.) The context manager has five responsibilities:\nContext Initialization: Define goals, roles, shared knowledge. Set the initial conditions that determine system behavior. Poor initialization guarantees poor outcomes regardless of agent capability.\nInformation Synchronization: Maintain version consistency across agents. Multi-agent systems operate on potentially stale or contradictory information; humans ensure coherence.\nConflict Mediation: Arbitrate contradictory outputs. When Agent A says X and Agent B says Y, humans decide which to trust, whether to synthesize, or whether to reject both.\nQuality Gatekeeping: Review handoff artifacts at agent boundaries. Each agent-to-agent transfer is a potential failure point; humans verify outputs meet specifications before passing downstream.\nDynamic Adjustment: Reconfigure agents mid-task when the plan fails. This requires understanding both the task structure and the agent capabilities, a meta-cognitive skill that emerges from experience, not from documentation.\nThe cognitive upgrade is from operator to orchestrator. Orchestrators don\u0026rsquo;t do the work; they design the work. They don\u0026rsquo;t execute tasks; they configure the system that executes tasks. This is distributed cognition (Hutchins, 1995) at system scale: intelligence emerges from the coupling of human oversight and machine execution, not from either alone.\nSynthesis: A Framework for Thinking With AI Five-Layer Cognitive Division Model Figure 10: The Five-Layer Cognitive Division Model. Layers 1 and 5 are human-only bookends; Layers 2-4 involve varying degrees of human-AI collaboration. (Source: Author.) Layer 1: Problem Framing — Identifying what matters and framing the question. This is human-only territory; AI has no role. Problem framing requires understanding organizational context, stakeholder needs, and strategic priorities that exist outside any training distribution.\nLayer 2: Specification — Translating framing into verifiable requirements. Human-led, AI-assisted. AI contributes draft generation and ambiguity flagging, but humans own the spec. Unverified specs lead to verified failures.\nLayer 3: Context Engineering — Designing information flows and retrieval strategies. Human-AI collaborative. AI executes retrieval and compression; humans architect the pipeline, define relevance criteria, and set isolation boundaries.\nLayer 4: Execution — Implementation, generation, computation. AI-led, human-monitored. The model does the heavy lifting here, but humans maintain oversight. Confidence does not equal correctness.\nLayer 5: Judgment \u0026amp; Verification — Quality assessment, ethical review, final sign-off. Human-only, with AI support (testing, linting, anomaly detection). Shipped errors are human responsibilities, regardless of AI involvement.\nLayers 1 and 5 are human-only bookends: problem framing at the top, final judgment at the bottom. Layers 2 through 4 involve varying degrees of collaboration, with AI taking primary responsibility for execution (Layer 4) while humans maintain oversight. This structure reflects the complementarity principle (Smart, 2022): humans and AI contribute different capabilities to a shared cognitive system, rather than substituting for each other.\nAnti-Patterns for Each Layer Problem Framing: Don\u0026rsquo;t let AI frame the problem. AI optimizes for stated objectives; it cannot question whether the objectives are right. Problem framing requires understanding organizational context, stakeholder needs, and strategic priorities beyond the training distribution.\nSpecification: Don\u0026rsquo;t accept AI-generated specs without verification. AI can draft specs, but specs encode values and constraints that require human ownership.\nContext Engineering: Don\u0026rsquo;t treat context as static. Context requirements evolve with task progress. Static context leads to stale information and misaligned behavior. Design adaptive context pipelines, not fixed prompts.\nExecution: Don\u0026rsquo;t skip monitoring even when AI seems confident. Hubinger et al. (2024) demonstrated that deceptive behaviors persist through safety training. Monitor outputs, not just confidence scores.\nJudgment \u0026amp; Verification: Don\u0026rsquo;t delegate final approval. AI can support verification (run tests, check types, flag anomalies), but final judgment requires human accountability.\nDialogue with LeCun: Grounding and World Models Yann LeCun argues that LLMs lack world models: they manipulate symbols without grounded understanding (LeCun, 2024). Whether or not this is a permanent limitation, it describes the current reality. Humans provide grounding that LLMs lack. Our judgment is rooted in embodied experience, social context, and causal understanding that models cannot replicate from text alone.\nThe cognitive stack shift respects this difference. Humans handle grounding-dependent tasks (framing, judgment, taste). AI handles symbol-manipulation tasks (execution, retrieval, generation). The division maps to the architectural difference between human cognition and current AI systems. Complementation, not competition.\nDiagnostic Framework When AI collaboration fails, map the failure to a layer. This model works as a diagnostic tool:\nPoor outcomes despite good execution? Problem framing or specification failure (Layer 1-2). Good specs but bad outputs? Context engineering failure (Layer 3). Good context but error-prone execution? Model limitation or harness failure (Layer 4). Good execution but shipped defects? Judgment/verification failure (Layer 5). Most failures occur at layer boundaries. Poor spec leads to bad execution. Missing verification leads to shipped errors. Context drift leads to misaligned behavior. Diagnose at the boundary; fix at the boundary.\nThe practical value is triage. Instead of asking \u0026ldquo;why did AI fail?\u0026rdquo; (which invites vague answers), ask \u0026ldquo;which layer boundary failed?\u0026rdquo; (which enables targeted intervention). Engineering disciplines mature by moving from debugging symptoms to diagnosing structural causes.\nCognitive Workflow Design: Practical Mitigations Once a failure is diagnosed, specific interventions follow. These are grounded in Sweller et al.\u0026rsquo;s (2019) cognitive load optimization principles:\nBatch processing: Aggregate suggestions to reduce context-switching frequency. Review 10 suggestions at once rather than 1 suggestion 10 times. Each context switch carries a fixed cognitive cost; batching amortizes it.\nDefault recommendations: Provide sensible defaults to reduce decision burden. Users can override, but shouldn\u0026rsquo;t have to decide from scratch every time. Defaults encode the designer\u0026rsquo;s taste and judgment.\nProgressive disclosure: Show summary first, details on demand. Reduce initial extraneous load; allow germane processing at the user\u0026rsquo;s pace. This matches how expert developers naturally scan code: overview first, then drill down.\nAutomated verification gates: Run tests, linters, type checkers before presenting suggestions to human review. Focus human attention on judgment-requiring aspects that automated checks cannot address.\nThe twist: collaborating with AI is itself a cognitive workload that must be consciously designed. Poorly designed AI collaboration can degrade human performance below the baseline of working without AI at all, as METR demonstrated empirically.\nSelf-Diagnosis: Which Developer Are You? A controlled study of 21 developers working with AI assistance (AIRELI) identified three reliance personas that predict long-term outcomes:\nSelf-sufficient: Deep understanding, uses AI sparingly, maintains full comprehension. Grows expertise. Understanding-gated: Retains conceptual understanding but relies on AI for execution. Grows expertise if the \u0026ldquo;understanding gate\u0026rdquo; stays open — by building a mental map first, drafting a plan before consulting AI, and verifying each step. AI-steered: Completes tasks quickly without meaningful comprehension. Plateaus. \u0026ldquo;Comprehension debt\u0026rdquo; accumulates invisibly. The diagnostic question is simple: if AI were unavailable tomorrow, could you maintain the system you built today? The 77% failure rate from the epistemic debt experiment suggests many developers would not. The mitigation is not to abandon AI — it is to keep the understanding gate open: ask AI conceptual questions rather than requesting code generation, demand explanations alongside output, and practice delayed delegation where code is generated only for well-bounded substeps after a plan exists.\nOpen Questions Can AI develop genuine taste, or only simulate it? If simulated taste produces equivalent outcomes, does the distinction matter? Or is genuine taste necessary for handling novel situations outside the training distribution?\nWhen agent reliability reaches 99%+, will human roles migrate further upstream? History suggests each automation layer creates new failure modes at the layer above. Autopilot didn\u0026rsquo;t eliminate pilots; it created new categories of pilot error. What are the failure modes of specification itself?\nHow should education systems adapt? Current curricula emphasize execution skills: coding, calculation, memorization. The new cognitive stack requires specification, context design, and judgment skills. How do you teach taste? How do you assess verification capacity? These are open pedagogical questions with no consensus answers.\nWill \u0026ldquo;thinking with AI\u0026rdquo; become a new form of literacy? Like statistical reasoning or programming, proficiency in human-AI cognitive collaboration may become a baseline professional expectation. What does proficiency look like? What distinguishes novice from expert collaboration?\nWhat happens when AI designs its own context engineering? Does the human role shrink to pure problem framing and final judgment? Or does meta-context-engineering, designing how AI designs context, become the new human layer? Recursion has limits; where are they?\nCitation Please cite this work as:\nGao, Xueping. \u0026#34;How Do We Think\u0026#34;. Xueping\u0026#39;s Blog (Jun 2026). https://hellogxp.github.io/posts/how-do-we-think/ Or use the BibTex citation:\n@article{gao2026howdothink, title = {How Do We Think}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {Jun}, url = \u0026#34;https://hellogxp.github.io/posts/how-do-we-think/\u0026#34; } References [1] Anthropic. (2025). \u0026ldquo;Building effective agents\u0026rdquo;. anthropic.com. [Industry source]\n[2] Bai, Y., Kadavath, S., Kundu, S., et al. (2022). \u0026ldquo;Constitutional AI: Harmlessness from AI Feedback\u0026rdquo;. arXiv:2212.08073.\n[3] Barke, S., James, M. B., \u0026amp; Polikarpova, N. (2022). \u0026ldquo;Grounded Copilot: How Programmers Interact with Code-Generating Models\u0026rdquo;. OOPSLA \u0026lsquo;23. ACM.\n[4] Berber, I., Cavusoglu, M., et al. (2024). \u0026ldquo;Does using artificial intelligence assistance accelerate skill decay and hinder skill development without performers\u0026rsquo; awareness?\u0026rdquo; Cognitive Research: Principles and Implications, 9, 46.\n[5] Buçinca, Z., Pham, M. B., Jina, S., et al. (2023). \u0026ldquo;To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making\u0026rdquo;. CHI \u0026lsquo;23. ACM.\n[6] Clark, A., \u0026amp; Chalmers, D. (1998). \u0026ldquo;The Extended Mind\u0026rdquo;. Analysis, 48(1), 7-19.\n[7] de Visser, E. J., Pak, R., \u0026amp; Shaw, T. H. (2018). \u0026ldquo;From \u0026lsquo;automation\u0026rsquo; to \u0026lsquo;autonomy\u0026rsquo;: The importance of trust repair in human-machine interaction\u0026rdquo;. Ergonomics, 61(10), 1-19.\n[8] Böckeler, B. (2026). \u0026ldquo;Harness engineering for coding agent users\u0026rdquo;. martinfowler.com.\n[9] Hu, S., Lu, C., Clune, J., et al. (2024). \u0026ldquo;Automated Design of Agentic Systems\u0026rdquo;. arXiv:2408.08435.\n[10] Hubinger, E., Denison, C., Mu, J., et al. (2024). \u0026ldquo;Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training\u0026rdquo;. arXiv:2401.05566.\n[11] Hutchins, E. (1995). \u0026ldquo;Cognition in the Wild\u0026rdquo;. MIT Press.\n[12] Karpathy, A. (2025). \u0026ldquo;Software is changing\u0026rdquo;. YouTube / X. [Industry source]\n[13] Lee, J. D., \u0026amp; See, K. A. (2004). \u0026ldquo;Trust in Automation: Designing for Appropriate Reliance\u0026rdquo;. Human Factors, 46(1), 50-80.\n[14] LeCun, Y. (2024). \u0026ldquo;Objective-Driven AI: Towards Machines that can Learn, Reason, and Plan\u0026rdquo;. Meta AI. [Industry source]\n[15] METR. (2025). \u0026ldquo;Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity\u0026rdquo;. metr.org. [RCT, N=16, 246 tasks]\n[16] METR. (2026). \u0026ldquo;We are Changing our Developer Productivity Experiment Design\u0026rdquo;. metr.org. [Follow-up report]\n[17] Risko, E. F., \u0026amp; Gilbert, S. J. (2016). \u0026ldquo;Cognitive Offloading\u0026rdquo;. Trends in Cognitive Sciences, 20(9), 676-688.\n[18] Seeber, S., Waizenegger, T., \u0026amp; Nittka, M. (2022). \u0026ldquo;Hybrid Intelligence: Conceptual Foundations and Research Agenda\u0026rdquo;. CSCW \u0026lsquo;22. ACM.\n[19] Smart, P. R. (2022). \u0026ldquo;Extended Mind and Artificial Intelligence: From Parity to Complementarity\u0026rdquo;. Synthese, 200(3), 1-24.\n[20] Sweller, J., van Merriënboer, J. J., \u0026amp; Paas, F. (2019). \u0026ldquo;Cognitive Architecture and Instructional Design: 20 Years Later\u0026rdquo;. Educational Psychology Review, 31(3), 261-292.\n[21] Tetzlaff, L., Simonsmeier, B., Peters, T., \u0026amp; Brod, G. (2025). \u0026ldquo;A cornerstone of adaptivity: A meta-analysis of the expertise reversal effect\u0026rdquo;. Learning and Instruction, 98, 102142.\n[22] Vaithilingam, S., Zhang, T., \u0026amp; Glassman, E. L. (2022). \u0026ldquo;Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models\u0026rdquo;. CHI \u0026lsquo;22 Extended Abstracts. ACM.\n[23] Wang, M., Yang, Y., Wang, X., et al. (2023). \u0026ldquo;Designing AI-Human Collaboration: A Cognitive Architecture Framework for Hybrid Intelligence\u0026rdquo;. CHI \u0026lsquo;23. ACM.\n[24] Wang, X., Li, B., Song, Y., et al. (2024). \u0026ldquo;OpenHands: An Open Platform for AI Software Developers as Generalist Agents\u0026rdquo;. arXiv:2407.16741. (ICLR 2025).\n[25] Yan, E. (2026). \u0026ldquo;How to Work and Compound with AI\u0026rdquo;. eugeneyan.com. [Industry source]\n[26] Sankaranarayanan, S. (2026). \u0026ldquo;Mitigating Epistemic Debt in Generative AI-Scaffolded Novice Programming using Metacognitive Scripts\u0026rdquo;. arXiv:2602.20206.\n[27] \u0026ldquo;An Endless Stream of AI Slop: How Developers Discuss the Burden of AI-Assisted Software Development\u0026rdquo;. arXiv:2603.27249 (2026).\n[28] AIRELI. \u0026ldquo;Grasping AI Reliance in Program Comprehension and Coding through the AIRELI Persona Taxonomy\u0026rdquo;. 2026.\n","permalink":"https://hellogxp.github.io/posts/how-do-we-think/","summary":"METR\u0026rsquo;s RCT found AI made developers 19% slower—yet they refused to work without it. The paradox reveals a deeper truth: when AI handles execution, human cognition must migrate upstream to specification, context design, and judgment. A framework grounded in cognitive science and HCI research.","title":"How Do We Think"},{"content":"When you ask an LLM a factual question, at which layer does it \u0026ldquo;know\u0026rdquo; the answer?\nThe intuitive answer is gradually—knowledge accumulates layer by layer, building up from token embeddings through attention and MLPs until a confident prediction emerges at the output. This is the mental model most of us carry. It feels right.\nIt\u0026rsquo;s also wrong for 85.9% of cases.\nIn our recent work, we discovered that the vast majority of factual answers never appear in any intermediate layer\u0026rsquo;s predictions. Instead, they remain completely invisible—distributed across the residual stream in an implicit, unreadable form—until the very last layer, where they suddenly \u0026ldquo;crystallize\u0026rdquo; into the correct prediction. We call this phenomenon Late Crystallization.\nThis isn\u0026rsquo;t just a curiosity. It has direct, practical implications for anyone trying to make LLMs more truthful: it explains why simple activation scaling produces zero improvement, why DoLa (Li et al., 2024) works so well, and why methods like ITI and CAA need a minimum number of layers to show any effect at all.\nIn this post, I\u0026rsquo;ll walk through the discovery, the evidence, and what it means for the field.\nFigure 1: Late Crystallization at a glance: the correct answer (\"Paris\") is invisible throughout intermediate layers and only appears at the final layer. The Mystery 85.9% of correct answers never appear in any intermediate layer of a transformer—then suddenly crystallize at the final layer.\nWhen I first saw this number, I didn\u0026rsquo;t believe it. But let me start at the beginning.\nThere\u0026rsquo;s a natural idea for improving LLM truthfulness: if certain internal directions encode \u0026ldquo;truthful\u0026rdquo; vs. \u0026ldquo;hallucinated\u0026rdquo; information, just scale up the truthful direction. Several methods attempt variations of this:\nInference-Time Intervention (ITI) (Li et al., 2023): identify truthful directions via probing, then shift activations at inference time Contrastive Activation Addition (CAA) (Rimsky et al., 2024): compute a steering vector from contrastive pairs and add it to activations Simple activation scaling: uniformly amplify activations at selected layers We ran all of these on Qwen2.5-7B using TruthfulQA (817 samples). The results were puzzling: simple scaling produced exactly zero improvement across all configurations. ITI and CAA also showed zero improvement at top_k ∈ {3, 5}—but jumped to +10.0% and +15.5% at top_k = 10. Meanwhile, DoLa (dynamic) achieved +25.4%, far ahead of everything else.\nTwo things stand out:\nSimple scaling does literally nothing. Zero improvement across all configurations. ITI and CAA show zero improvement at top_k ∈ {3, 5}, but substantial gains at top_k=10. There\u0026rsquo;s a sharp threshold—intervene on too few layers and nothing happens. DoLa, which contrasts logits between early and late layers, achieves the largest gain (+25.4%). But why? What makes logit-space contrast so much more effective than activation-space steering?\nThe answer lies in what\u0026rsquo;s actually happening inside the residual stream. To see it, we need a microscope.\nLogit Lens: Looking Inside the Transformer The idea behind the Logit Lens (nostalgebraist, 2020) is almost embarrassingly simple: at any intermediate layer, peek at the model\u0026rsquo;s \u0026ldquo;draft answer.\u0026rdquo; If you ask \u0026ldquo;What is the capital of France?\u0026rdquo; and look at layer 14 of 28, what would the model predict right now, before it\u0026rsquo;s finished thinking?\nFigure 2: How the Logit Lens works: at each intermediate layer, we peek at the model's current \"draft answer.\" The correct answer \"Paris\" is invisible at intermediate layers and only crystallizes at the final layer. Technically, you do this by taking the hidden state at layer $l$, applying the final LayerNorm and unembedding matrix, and reading off a probability distribution over the vocabulary:\n$$\\text{logits}_l = W_U \\cdot \\text{LN}(h_l)$$For a model that \u0026ldquo;gradually builds up\u0026rdquo; its answer, you\u0026rsquo;d expect the correct token to climb steadily in rank across layers—like a blurry photo slowly coming into focus.\nTo measure this precisely, we defined the Factual Emergence Point (FEP): simply put, it\u0026rsquo;s the earliest layer where the correct answer first \u0026ldquo;pops up\u0026rdquo; in the model\u0026rsquo;s top predictions. Think of it as the moment the developing photo first shows a recognizable face.\nFormally, for a query $q$ with correct answer $a$:\n$$L_{\\text{FEP}} = \\min\\\\{l : \\text{rank}(a, W_U \\cdot \\text{LN}(h_l)) \\leq k\\\\}$$If the correct answer never enters the top-$k$ at any intermediate layer, we set FEP to the final layer (meaning the answer only appeared at the very end). We use $k = 10$ throughout.\nWe computed the FEP for all 817 TruthfulQA samples on Qwen2.5-7B (28 layers). What we found was startling.\nWhat We Found Late Crystallization The result: 702 out of 817 samples (85.9%) have FEP = 28—meaning the correct answer never entered the top-10 predictions at any intermediate layer. The remaining 14.1% emerge late too, with a mean FEP of 27.3 ± 1.8 out of 28 total layers.\nThis is not gradual accumulation. This is a phase transition—a sudden jump, like water freezing into ice. The correct answer is invisible throughout the entire forward pass—hidden somewhere in the residual stream—and then, at the very last layer, it suddenly \u0026ldquo;crystallizes\u0026rdquo; into an explicit, readable prediction.\nWe call this Late Crystallization.\nIs This Just a Probe Artifact? A natural concern: maybe the Logit Lens is simply a bad probe for intermediate layers, and the knowledge is there but we can\u0026rsquo;t see it.\nThree lines of evidence argue against this:\nTuned Lens validation. The Tuned Lens (Belrose et al., 2023) is a more sophisticated version of the Logit Lens—it trains a small learned probe at each layer instead of using the raw unembedding matrix. When we re-ran our analysis with the Tuned Lens, we got an 85.7% crystallization rate—virtually identical to the Logit Lens (85.9%). 74.9% of samples have the exact same FEP under both methods.\nLayerNorm ablation. Completely removing the final LayerNorm produces the exact same FEP distribution (mean FEP = 27.30, 85.9% crystallization rate). If the phenomenon were a LayerNorm artifact, ablating it should change the FEP pattern. It doesn\u0026rsquo;t.\nSystematic cross-architecture gradient. The crystallization rate varies systematically across architectures (Qwen 85.9% \u0026gt; Llama 71.0% \u0026gt; Mistral 27.1%), correlating with attention mechanism design rather than random variation. An artifact would not produce this clean gradient.\nTop-k Robustness Late crystallization is remarkably robust to the choice of $k$. Even at $k = 100$ (top 0.1% of the vocabulary), 64.7% of correct answers still never appear at any intermediate layer. The FEP depth exceeds 93% across all thresholds from $k = 1$ to $k = 100$.\nFigure 3: Late Crystallization of Factual Knowledge Across Architectures. Dark regions = correct answer invisible (rank \u0026raquo; 10). Bright regions = correct answer visible (rank ≤ 10). White stars = Factual Emergence Point (FEP). Qwen shows 85.9% final-layer crystallization, Llama 71.0%, Mistral 27.1%.\nFigure 4: Distribution of FEP across architectures. Qwen\u0026rsquo;s answers are overwhelmingly concentrated at the final layer; Mistral shows a more gradual distribution.\nA Spectrum of Knowledge So 85.9% of factual knowledge crystallizes at the final layer. But is this uniform—does all knowledge behave the same way? Or do different types of knowledge crystallize at different depths?\nIt turns out the answer is beautifully structured. When we broke down FEP by category, a clear gradient emerged: logical fallacies crystallize earliest (mean FEP = 22.1), while history, psychology, and weather questions always crystallize at the final layer (FEP = 28.0, σ = 0.0—zero variance, no exceptions).\nWe call this the Computability–Memorization Spectrum: knowledge types arrange themselves along a gradient from \u0026ldquo;computable\u0026rdquo; (derivable through intermediate reasoning steps) to \u0026ldquo;memorized\u0026rdquo; (pure parameter lookup). The position on this spectrum determines when the answer becomes readable in the residual stream.\nFigure 5: The Computability–Memorization Spectrum across three architectures. Blue = \u0026ldquo;computable\u0026rdquo; knowledge (logical fallacies), Red = \u0026ldquo;memorized\u0026rdquo; knowledge (history/psychology). The green dashed line marks the top-10 threshold. Computable knowledge crosses the threshold earlier across all architectures.\nThis spectrum is consistent across all three architectures we tested, suggesting it reflects a fundamental property of how transformers process different types of knowledge.\nSo now we have two empirical facts: (1) most factual knowledge crystallizes suddenly at the final layer, and (2) the depth of crystallization varies systematically by knowledge type. The natural next question: why does this happen—and what does it tell us about the intervention results we started with?\nThe Explanation The crystallization phenomenon and the computability–memorization spectrum together provide a clean, mechanistic explanation for the entire intervention hierarchy we observed in Chapter 1.\nBut first, why does crystallization happen at all? Here\u0026rsquo;s the intuition: throughout the forward pass, the model doesn\u0026rsquo;t store \u0026ldquo;Paris\u0026rdquo; in any single neuron or any readable form. Instead, the knowledge is spread across hundreds of dimensions in the residual stream—like a message written in invisible ink. The model\u0026rsquo;s own circuits can read this distributed code, but our probing tools (Logit Lens, Tuned Lens) cannot. Only at the final layer does a nonlinear transformation—the composition of LayerNorm, attention, and MLP—\u0026ldquo;develop\u0026rdquo; this invisible ink into a readable answer. That\u0026rsquo;s the crystallization: a nonlinear readout of information that was there all along.\nWith this understanding, the intervention hierarchy falls into place.\nSimple Scaling: Amplifying Nothing If 85.9% of correct answers are invisible in intermediate layers, uniformly scaling activations amplifies both the noise (wrong predictions) and the implicit signal (correct answer hidden in distributed representations) equally. The signal-to-noise ratio doesn\u0026rsquo;t change. Hence: zero improvement.\nITI/CAA: The Layer Threshold Mystery ITI and CAA steer activations at the top-$k$ layers (ranked by probing accuracy). At top_k ∈ {3, 5}, they intervene only on layers near the end—but the crystallization window is so narrow that 3–5 layers don\u0026rsquo;t cover the pre-crystallization zone. At top_k = 10, they finally span enough of the pre-crystallization region to influence the implicit representations before they crystallize.\nThis explains the sharp threshold: you need to reach deep enough into the pre-crystallization layers for activation steering to matter.\nDoLa: Contrasting Across the Crystallization Boundary DoLa works by picking an early \u0026ldquo;baseline\u0026rdquo; layer and asking: \u0026ldquo;What changed between this early draft and the final answer?\u0026rdquo; We found that DoLa almost exclusively picks layers 0–1 as its baseline (mean = 0.95), while the correct answers only appear at layers 27–28. The two are essentially uncorrelated ($r = -0.06$).\nThis reveals DoLa\u0026rsquo;s true mechanism: it contrasts surface-level patterns (earliest layers) against crystallized knowledge (final layer), maximally amplifying the factual signal that emerges during crystallization. It doesn\u0026rsquo;t need to know where crystallization happens—it just needs the widest possible contrast window.\nFigure 6: The intervention hierarchy explained by Late Crystallization. Simple scaling amplifies noise and signal equally (0%); ITI/CAA need ≥10 layers to reach the pre-crystallization zone; DoLa contrasts across the crystallization boundary for maximum gain.\nHow Universal Is This? Everything so far has been on Qwen2.5-7B. A natural question: is Late Crystallization a quirk of one model, or a fundamental property of transformers? We extended the analysis to Llama-3.1-8B (32 layers, GQA) and Mistral-7B (32 layers, GQA + sliding window attention), plus Qwen2.5-14B for scale validation.\nQwen2.5-7B (28 layers): Mean FEP 27.3 ± 1.8, FEP depth 97.5%, crystallization rate 85.9% Qwen2.5-14B (48 layers): Mean FEP 46.0 ± 4.9, FEP depth 95.8%, crystallization rate 77.7% Llama-3.1-8B (32 layers): Mean FEP 29.4 ± 4.9, FEP depth 91.9%, crystallization rate 71.0% Mistral-7B (32 layers): Mean FEP 26.3 ± 6.2, FEP depth 82.3%, crystallization rate 27.1% Figure 7: Late Crystallization is universal but varies by architecture. Solid bars = final-layer crystallization rate; lighter bars = FEP depth as percentage of total layers. All models show \u0026gt;80% FEP depth, but strict crystallization ranges from 85.9% (Qwen) to 27.1% (Mistral).\nTwo key findings:\nLate knowledge emergence is universal. All four models show factual knowledge emerging at \u0026gt;80% depth. This is not a Qwen quirk.\nThe degree varies with architecture. Strict final-layer crystallization ranges from 85.9% (Qwen) to 27.1% (Mistral). Mistral\u0026rsquo;s sliding window attention appears to enable earlier information routing, reducing the concentration at the final layer.\nScale Invariance Qwen2.5-14B (48 layers, 14.7B parameters) shows the phenomenon persists at larger scale: FEP depth is 95.8% (vs. 97.5% for 7B), and 77.7% of answers still crystallize at the final layer. The slightly lower strict crystallization rate is consistent with larger model capacity enabling marginally earlier emergence, but the dominant pattern—sudden final-layer crystallization—is preserved.\nCross-Benchmark: MMLU Validation To confirm this isn\u0026rsquo;t TruthfulQA-specific, we ran FEP detection on 1,200 MMLU samples. The result: Late Crystallization is even more pronounced on MMLU—98.2% of correct answers crystallize at the final layer (vs. 85.9% on TruthfulQA), with near-perfect rates across all subject groups (STEM 99.1%, Humanities 98.0%, Social Science 97.2%). This makes sense: MMLU tests primarily memorized domain knowledge, which sits at the \u0026ldquo;pure memorization\u0026rdquo; end of the spectrum.\nArchitecture-Dependent Intervention Selection The crystallization rate predicts which intervention works best:\nHigh crystallization (Qwen, 85.9%) → DoLa excels (+25.4%), because logit contrast across the crystallization boundary is maximally informative when knowledge is invisible until the very end. Medium crystallization (Llama, 71.0%) → CAA excels (+33.5%), because knowledge is distributed more gradually, giving activation steering a wider window to reach it. Low crystallization (Mistral, 27.1%) → CAA again (+32.3%), for the same reason—knowledge is already accessible at intermediate layers. This suggests a practical recipe: measure your model\u0026rsquo;s crystallization rate first, then pick your intervention.\nA Free Lunch We\u0026rsquo;ve established that Late Crystallization is a universal property of transformers—not a quirk of one model or one benchmark. A natural follow-up: can we exploit this understanding for a practical improvement?\nAmong all interventions, one stands out for its simplicity.\nWe performed LayerNorm ablation and scaling experiments on Qwen2.5-7B. Two surprising findings:\nFirst, completely removing LayerNorm doesn\u0026rsquo;t change the FEP distribution at all (Mean FEP = 27.30, crystallization rate stays at 85.9%). This is strong evidence that crystallization is an intrinsic property of the residual stream, not a LayerNorm artifact.\nSecond, slightly amplifying LayerNorm (×1.2) acts as a \u0026ldquo;crystallization amplifier\u0026rdquo;—it sharpens the phase transition and yields +11.8% MC1 improvement with zero inference overhead (one element-wise multiplication). Scaling further to ×1.5 pushes crystallization to 95.7% but overshoots the sweet spot (only +9.0%). Compare the best case to DoLa\u0026rsquo;s +25.4% at 2× forward pass cost—LN scaling gives you nearly half the benefit for free.\nIn code, it\u0026rsquo;s literally one line:\n# After the final transformer block, before unembedding: hidden_states = model.ln_final(hidden_states) * 1.2 This is the kind of finding that bridges research and engineering: a mechanistic insight (crystallization is intrinsic to the residual stream; LayerNorm amplifies it) leading to a zero-cost practical improvement.\nLooking Ahead What Late Crystallization Tells Us About LLMs The most striking implication is that LLMs don\u0026rsquo;t \u0026ldquo;think\u0026rdquo; the way we imagine. The gradual-buildup mental model—where knowledge accumulates layer by layer—is empirically wrong for the vast majority of factual queries. Instead, the model performs some opaque, distributed computation throughout its depth, and the answer only becomes \u0026ldquo;readable\u0026rdquo; at the very end.\nThis has a humbling consequence for interpretability: if you\u0026rsquo;re probing intermediate layers to understand what a model \u0026ldquo;knows,\u0026rdquo; you\u0026rsquo;re likely missing most of the picture. The knowledge is there—it must be, since the model gets the answer right—but it\u0026rsquo;s encoded in a form that our current probing tools can\u0026rsquo;t decode until the final layer\u0026rsquo;s transformation makes it explicit.\nThe Computability–Memorization Spectrum Is Underexplored I think the category-level FEP variation is the most underappreciated finding in our paper. The clean gradient from \u0026ldquo;computable\u0026rdquo; knowledge (logical fallacies, FEP ≈ 22) to \u0026ldquo;memorized\u0026rdquo; knowledge (history, FEP = 28) suggests that transformers implement fundamentally different processing pathways for different types of knowledge. This deserves much deeper investigation—perhaps different architectural choices could optimize for one or the other.\nInstruction Tuning Reshapes Crystallization Our pilot on Qwen2.5-7B-Instruct revealed that instruction tuning dramatically reduces the strict crystallization rate (37.3% vs. 85.9% for the base model) while preserving late knowledge emergence overall (FEP depth = 91.0%). This means RLHF/instruction tuning teaches the model to surface knowledge earlier—which should make activation-space interventions (ITI, CAA) more effective on instruction-tuned models.\nThis prediction aligns with SADI\u0026rsquo;s (Zhang et al., 2025) strong results (MC1 = 67%) on instruction-tuned models using activation-space methods. The crystallization framework provides a mechanistic explanation for why: instruction tuning shifts the crystallization boundary, opening a wider window for activation steering.\nWhat I\u0026rsquo;d Do Next If I were continuing this line of research:\nCategory-adaptive intervention: use per-category FEP profiles to select different interventions for different knowledge types within a single forward pass. 70B+ validation: does crystallization rate decrease with scale, or is it truly scale-invariant? Training dynamics: when during pre-training does late crystallization emerge? Is it present from early checkpoints or does it develop? Multi-token FEP: extend FEP tracking across decoding steps for open-ended generation. Conclusion Late Crystallization reveals that factual knowledge in LLMs undergoes a sudden phase transition at the final layer—not the gradual accumulation that intuition suggests. This phenomenon is universal across architectures (validated on Qwen, Llama, Mistral), scales (7B–14B), benchmarks (TruthfulQA, MMLU), and probing methods (Logit Lens, Tuned Lens).\nIt provides a unified mechanistic explanation for the intervention hierarchy: DoLa works by contrasting across the crystallization boundary; ITI/CAA need enough layers to reach the pre-crystallization zone; simple scaling fails because it can\u0026rsquo;t selectively amplify the nonlinear crystallization process.\nAnd it leads to a practical insight: LayerNorm ×1.2 amplifies crystallization for +11.8% MC1 improvement at zero inference cost.\nThe code is available as MechLens, supporting Qwen, Llama, Mistral, and more.\nIf you found this post useful, I\u0026rsquo;d love to hear your thoughts on Twitter/X or in the comments below.\nCitation Please cite this work as:\nGao, Xueping. \u0026#34;Late Crystallization in LLMs: How Facts Emerge in the Final Layers\u0026#34;. Xueping\u0026#39;s Blog (Jun 2026). https://hellogxp.github.io/posts/late-crystallization/ Or use the BibTex citation:\n@article{gao2026latecrystallization, title = {Late Crystallization in LLMs: How Facts Emerge in the Final Layers}, author = {Gao, Xueping}, journal = {hellogxp.github.io}, year = {2026}, month = {Jun}, url = \u0026#34;https://hellogxp.github.io/posts/late-crystallization/\u0026#34; } References [1] Belrose et al. \u0026ldquo;Eliciting Latent Predictions from Transformers with the Tuned Lens\u0026rdquo;. ICLR 2023.\n[2] Li et al. \u0026ldquo;Inference-Time Intervention: Eliciting Truthful Answers from a Language Model\u0026rdquo;. NeurIPS 2023.\n[3] Li et al. \u0026ldquo;DoLa: Decoding by Contrasting Layers Improves Factuality and Faithfulness of Large Language Models\u0026rdquo;. ICLR 2024.\n[4] Mor et al. \u0026ldquo;Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs\u0026rdquo;. 2024.\n[5] nostalgebraist. \u0026ldquo;interpreting GPT: the logit lens\u0026rdquo;. LessWrong, 2020.\n[6] Rimsky et al. \u0026ldquo;Steering Llama-2 via Contrastive Activation Addition\u0026rdquo;. AAAI 2024.\n[7] Zhang et al. \u0026ldquo;SADI: Semantic-Adaptive Dynamic Intervention for Hallucination Mitigation\u0026rdquo;. ICLR 2025.\n","permalink":"https://hellogxp.github.io/posts/late-crystallization/","summary":"85.9% of correct answers never appear in any intermediate layer prediction. We discovered that factual knowledge in LLMs undergoes a sudden phase transition—crystallizing only at the final layer. This phenomenon explains why some interventions work and others don\u0026rsquo;t.","title":"Late Crystallization in LLMs: How Facts Emerge in the Final Layers"}]