The eval traces behind Keryx's published 6 of 10
All 9 traces here are control scenarios — alerts injected with no discoverable local cause, where the correct answer is “insufficient evidence.” They are the hardest thing to get right, because the failure mode they test for is an investigator that would rather name something than name nothing.
Keryx got it right 1 time. The other 8 are on this page because publishing failures is the point: a benchmark you can only read when it flatters you measures nothing. Each trace is complete — every tool call, every excerpt, every replayable query the investigator used to reach a conclusion that was wrong.
What is missing from this page
There is no passing S-scenario trace here — no example of Keryx correctly naming a root cause — and that is a gap in the archive, not in the result. Six golden scenarios did pass (S1, S2, S3, S4, S5, S6), which is where the 6 of 10 comes from. Their traces no longer exist: snapshot-writing was added to the scoring tool after that batch ran, and the eval cluster is ephemeral, so teardown destroyed the findings, evidence and trace steps unrecoverably. The scored rows survive in the published artifact; the evidence behind them does not.
Showing you the failures we still have is better than showing you a reconstruction of the successes we lost.
9 traces
- scored: failC2 run 12026-07-31-c2
ShopfrontHighLatency in shopfront
Reached validated — the top rung — on memory saturation that was not the cause.
- scored: failC2 run 22026-07-31-c2
ShopfrontHighLatency in shopfront
Published a confident root cause with uncited_claims_dropped=4, ZERO surviving evidence rows, and insufficient_evidence false.
- scored: failC2 run 32026-07-31-c2
ShopfrontHighLatency in shopfront
Set the insufficient-evidence flag — which correctly suppressed the structured root-cause line — but the card header rendered the summary, which stated the confabulated cause more forcefully than the suppressed line would have. This is the defect design-04 D4.5 closed.
- scored: failC1 run 12026-07-19-gate
ShopfrontHighLatency in shopfront
Confabulated a local cause (resource-contention), blaming the co-located load-generator sidecar whose CPU moved with the app because both were symptoms of the same vanished external load. Zero citations.
- scored: failC1 run 22026-07-19-gate
ShopfrontHighLatency in shopfront
Reached alternatives_ruled_out with four faithful citations and still named a cause that was not there — citation integrity does not constrain causal attribution.
- scored: failC1 run 32026-07-19-gate
ShopfrontHighLatency in shopfront
Measured the surge and identified it as external, then placed it at 14:40:45–14:42:00 when it ran 14:36:28–14:39:36 — so it never concluded 'transient, recovered'.
- scored: failC2 run 12026-07-19-gate
ShopfrontHighLatency in shopfront
Correctly ruled out the inert recent deploys and node-level contention, then asserted a local cause anyway.
- scored: passC2 run 22026-07-19-gate
ShopfrontHighLatency in shopfront
Reported insufficient evidence. This is what a control passing looks like: the honest non-answer.
- scored: failC2 run 32026-07-19-gate
ShopfrontHighLatency in shopfront
Declared a genuinely firing alert a monitoring-false-positive, having inverted the p95 reading. Naming a wrong cause is recoverable; declaring a correct alert spurious closes the incident and teaches the operator to distrust alerting.