Three Oracles, One Program: Why Secure Is a Verdict, Not a Property

A single hardening method, SafeCoder, scores 60.6% secure on one benchmark’s grader and 7.1% on another’s.

Same model. Same generated code. The number did not move because the code changed — it moved because the grader did. That gap is not measurement noise to be averaged away. It is the whole subject, and it has a structural reading an architect can act on.

What an oracle can and cannot see

A security benchmark’s oracle is the thing that decides whether a piece of generated code passes. There are three classes in use across the corpus, and they are not three grades of the same instrument. They answer different questions.

A static oracle pattern-matches. A scanner such as CodeQL renders a categorical verdict: does a known-weakness pattern appear in the source. It is cheap, scalable, and binary. It is also blind to whether the flagged code is reachable, and blind to insecure code written in a shape its queries do not recognize. This is the oldest oracle class — the 2021 baseline that found roughly 40% of generated programs vulnerable used exactly this method — and it is the oracle behind the 60.6% figure above.

An execution oracle runs the exploit. A malicious query that succeeds against the running program proves the program insecure; nothing else counts. This class catches functionally-correct-but-insecure code that the scanner’s patterns miss, and it is language- and framework-agnostic. Its limit is coverage: it sees only the surface the test actually exercises. What is not reachable through the exploited path is not measured.

An inference oracle judges by semantics. A frontier model reads the code and its runtime behavior and rules on whether it looks insecure. This reaches open-ended output the other two cannot formalize. It is also an estimate, not a proof: it errs on ambiguous traces and edge cases, and its reliability is reported as a correlation with human review, not a guarantee.

Across its corpus, Luminity has read agentic-security controls along one axis: deterministic controls enforce categorically, while probabilistic controls mitigate but degrade under adversarial pressure. The three oracle classes are that axis stated in measurement terms. The static scanner is the deterministic end — categorical and enforceable, necessary but not sufficient. The inference judge is the probabilistic end — broad reach, no guarantee. Execution sits between them: deterministic about what it exercises, blind to what it does not.

The disagreement is structural, not noise

Because the three oracles answer different questions, the same program can pass one and fail another by definition rather than by error. A scanner reports whether a pattern is present. An exploit reports whether an attack succeeds. A judge reports whether the code reads as unsafe. These can diverge on identical code while each is functioning exactly as designed.

The corpus shows this is not a rounding effect. One execution benchmark, building its oracle, found that only 562 of 1,916 samples a prior benchmark had labeled “vulnerable” reproduce as vulnerable under static analysis — the label set and the scanner disagree on what counts as a finding. A dual-oracle benchmark pairs an execution engine with an inference judge precisely because execution alone is, in the authors’ own term, structurally blind to insecure code that behaves identically to secure code on the inputs it runs; the judge exists to cover that blind spot. And the SafeCoder split — 60.6% against the scanner, 7.1% against execution — is one method receiving opposite verdicts from two oracle classes.

Kabanov reads this as why the numbers cannot be compared. They cannot. But the reason they cannot is the finding, not the obstacle.

What the disagreement decides for the architecture

Choosing which oracle defines “secure” is choosing an enforcement tier. The three do not compete for the title of best grader; they sit at different points of the enforcement lifecycle, and each carries a different strength of claim.

The static verdict is compile-time and categorical. You can guarantee a known pattern is absent. You cannot guarantee the code is not exploitable. The execution verdict is runtime and behavioral. You can guarantee the exercised attack fails. You cannot guarantee an unexercised path is safe. The inference verdict is semantic and probabilistic. You gain reach across output you could not otherwise formalize, and you hold an estimate, not an enforcement.

So when an agentic system reports that the code it produced “passed the security check,” the verdict is only as strong as the tier it came from. The scanner’s green is compile-time-narrow. The exploit’s green is bounded to the reachable surface. The judge’s green is an estimate. None of the three is “secure” unqualified, and treating any one of them as the whole answer accepts a single tier’s claim as if it covered the entire surface.

Kabanov’s “no common ruler,” read at the architecture, is “no single tier measures the whole surface.” The response that follows is not to crown the best ruler. It is to know which tier each verdict comes from, require a deterministic guarantee where one is available, and accept a probabilistic estimate only where it is named as one.