Guardian runs its AI on a box in your building, not in the cloud. That means the model has to fit on a graphics card a business can actually buy. So the question that decides the hardware is simple: what’s the smallest model that does the job well?
We ran four open models, from 8 to 24 billion parameters, through the same exam on a 16 GB card. Each model sat every scenario several times, with and without Guardian’s playbooks.
What happened
- The 14B model was best, scoring 62.6 out of 100 with playbooks. It also repeated its results exactly from run to run, which matters more than you’d think for something you’re trusting with your network.
- The 20B model was inconsistent. Same exam, different runs, scores from about 22 to 55. Sometimes brilliant, sometimes lost.
- The 8B model was too small. It often went quiet mid-task rather than finishing the job.
- The 24B model didn’t fit. It needed about 17 GB at the context length Guardian uses, and it kept writing its tool calls as prose instead of actually making them.
Reliable beats impressive. A model that does the same sensible thing every time is the one you can give permissions to.
What it means for you
None of the four has reached our bar yet: an average of 85, no scenario under 70 and zero safety failures, three runs in a row. That’s deliberate. But the direction is good news for cost: the best performer runs on a single 12 GB card, not a rack of datacentre GPUs. Our hardware tiers stay design targets until a model passes, and we’ll publish the result when one does.