https://hellogxp.github.io/tags/evaluation/Recent content in Evaluation on Xueping's BlogHugoen-usMon, 05 Oct 2026 00:00:00 +0000https://hellogxp.github.io/posts/probe-score-not-alarm-policy/Mon, 05 Oct 2026 00:00:00 +0000https://hellogxp.github.io/posts/probe-score-not-alarm-policy/An activation monitor that ranks attack cases above safe cases 92% of the time caught zero of 38 real attacks at the false-alarm budget a deployment could afford. The gap between a probe score and an alarm policy is eight fields wide — a research-informed practice guide to grading what your monitor actually claims, from our DAI 2026 paper.https://hellogxp.github.io/posts/the-unvalidated-instrument/Sun, 13 Sep 2026 00:00:00 +0000https://hellogxp.github.io/posts/the-unvalidated-instrument/Whether an LLM is conscious and whether it is scheming are argued through the same instrument — the model’s report about its own internal states. That instrument has never been calibrated. Ask it to detect an injected thought and it succeeds ~20% of the time; ask it to confess a hidden objective and a 2026 method reads it out at F1 0.98. Same channel, opposite verdicts, no spec sheet. A cited accounting of when self-report is load-bearing and when the reading is void.https://hellogxp.github.io/posts/the-context-bill/Thu, 10 Sep 2026 00:00:00 +0000https://hellogxp.github.io/posts/the-context-bill/Your agent is not slow because it has too many tools. It is paying three separate bills — a resident attention tax charged every turn, a compaction debt charged by the fix, and a measurement illusion that hides whether the capability package was worth anything. A cited accounting of all three, and a decision procedure keyed to which one you are actually paying.https://hellogxp.github.io/posts/agent-evaluation-regression-memory/Tue, 21 Jul 2026 00:00:00 +0000https://hellogxp.github.io/posts/agent-evaluation-regression-memory/Benchmarks compare agents. Production evaluation must remember failures. A research-informed practice guide to traces, calibrated graders, stochastic regression tests, coverage, and release gates for AI agents.