One of the faults in Guardian’s exam goes like this. A new starter plugs in their laptop and gets no network. The office Wi-Fi has run out of addresses, because it’s full of visitors’ phones that left days ago but still hold eight-day leases. A good engineer spots the phones, clears their stale leases, shortens the lease time and checks the laptop gets an address. Ten minutes’ work.
Every model we tried failed it. Not some of the time: every time, across two full experiments. Our first thought was the obvious one: the models aren’t good enough yet.
Reading what the AI actually saw
So we stopped looking at scores and read the transcripts, line by line, exactly as the model saw them. And there it was. When the model asked for the list of leases, our own tool returned the first 20, all of them office PCs, and summarised the rest to keep the conversation short. The 180 phones were never on screen. The model was being asked to remove leases it had never been shown.
We’d set an exam question and hidden the page with the answer on it.
We fixed it properly: the lease tool now filters (show me only devices that aren’t on the network) and pages through long lists, so every one of the 180 phones can reach the model. We also changed our own scripted “perfect engineer” test so it can only use what the model can see, so this kind of blind spot can’t creep back in.
Then it got worse
The next run was worse. Now the models were guessing field names like AddressState and
ClientId. Our tool rejected them with an unhelpful “wrong format”, and one model spiralled into
gibberish trying again. Two more mistakes of ours: the tool’s description never listed the valid names, and our own
playbook used the PowerShell names instead.
All fixed now. The valid names are in the description, a wrong one is refused with the full list, and a test checks every tool so none can hide its options again.
What we learned
When an AI keeps failing at something a person finds easy, check what it was actually shown before you judge its reasoning. Half the failure here was ours. The other half is real: the models can now see every phone, and they still don’t always draw the right conclusion. That’s what the next experiment measures.