Running an AI engineer on-site means running its language model on local hardware. The single most important number is GPU memory (VRAM): it decides how large a model fits, and model size largely decides reasoning depth.
Why VRAM matters most
A model’s weights, plus the working memory for a long investigation (logs, configs, tool results), have to fit in GPU memory to run at useful speed. Quantisation shrinks models to fit smaller cards at some cost in quality. Roughly: 16 GB runs models up to about 20 billion parameters; 24–32 GB runs the 20–32B class; 48 GB opens up 30–70B models in quantised form; 80 GB and above handles 70B+ comfortably or several models at once.
Sizing tiers
- Development: a 16 GB GPU proves the plumbing and runs benchmarks. Not for production.
- Small office (under 50 devices): 24–32 GB GPU, 64 GB RAM, 1 TB NVMe.
- SMB / multi-site (50–500): a 48 GB GPU or two 24 GB cards, 128 GB RAM, 2 TB NVMe.
- Enterprise (500+): 80 GB+ data-centre GPUs, 256 GB+ RAM, HA pair.
These are design targets. Final sizing follows exam results: whichever model class reaches PhD level sets the floor. Our early benchmarks suggest small 14–20B models are good for development but not yet for the product; playbook guidance may change that.
Deployment options
- Supplied appliance: a pre-built, tested box per site.
- Bring your own server, certified with a short exam run.
- VM with GPU passthrough: Hyper-V Discrete Device Assignment, VMware passthrough or vGPU, Proxmox PCIe passthrough.
- Kubernetes on GPU nodes for enterprise platforms.
- Multi-site hub with lightweight branch collectors.
- Air-gapped, with an on-site portal and signed offline updates.
- High availability via a warm standby.
What about CPU-only or Apple Silicon?
CPU-only inference works but is too slow for live incident reasoning. It suits monitoring and overnight batch work. Macs with large unified memory can run big models efficiently and quietly, which makes them interesting for small offices, though they’re not yet a supported production platform.
Don’t forget the boring parts
Two network ports for the management VLAN, a UPS (a GPU box can draw 300–800 W under load), fast NVMe for the model and logs, and room for months of audit trail. Guardian is out-of-band, so it never needs to sit inline. Use the sizer on our hardware page for a starting point.
Want Guardian on your network? Join early access as a design partner.