Guardian earns each capability by passing an exam of real faults, graded automatically by fixed rules. The rules are only as good as the person who wrote them, and on day one we caught ourselves out.
In the DHCP scenario, a model fixed the problem by widening the address range into an unused block. A senior engineer might well do the same. Our grader didn’t accept it: we’d written the check with one particular fix in mind and quietly ruled out an equally good one.
What we did about it
We fixed the check, then went back through every scenario and compared its grading rules with the written spec, line by line, looking for the same kind of mistake. We found it again this week in a different form: our playbook told the AI that 10% free addresses was enough, while the exam wanted 15%. A model followed the playbook and failed the exam. The two now share a single number, and a test fails if they ever drift apart.
If an exam punishes good engineering, it isn’t measuring engineering.
It would be easy to only publish tests that make the product look good. We’d rather publish the ones that caught us out, because that’s how you know the passing scores, when they come, mean something. See how we test.