CPU Discipline: Where the Work Runs

A hosted model runs on someone else's machines and your text leaves the device. A local model runs on yours and nothing leaves, at a real cost in capability.

That is the whole trade, and it is a per-task decision rather than an identity. A sensitive letter and a general research question do not deserve the same answer.

The hardware is less mysterious than it sounds. As a rule of thumb, a model's weights need about two gigabytes of video memory per billion parameters at sixteen-bit precision, so a seventy-billion-parameter model needs about a hundred and forty gigabytes. That means workstation hardware or spilling into system memory and waiting.

Quantized models are what run on consumer machines. In Hugging Face's own measurement, a fifteen-billion-parameter model that needed about thirty gigabytes at sixteen-bit ran in a little over nine at four-bit, on the kind of graphics card many people can get. Those numbers move with every release, so treat them as the shape rather than the specification.

Match the model to the memory you have, then judge it on your own tasks. A small local model that answers well on your own documents beats a large one you cannot load.

Sustained inference pins the processor and the memory bus, which on a laptop means heat, throttling and battery, and on a phone means a device that becomes uncomfortable to hold. There is a security dividend in knowing this. Learn what your machine sounds like when idle: a device running hot while you are asking nothing of it is information — a runaway agent, something mining, or something you did not install.

Then scope your assistants. A working folder rather than the whole drive. A separate browser profile rather than the one holding your signed-in accounts. No standing permission to send, pay or delete.

Every capability you grant is a capability that an instruction hidden in a web page can reach. That is where this lesson and the injection lesson become the same lesson.

Finally, close the loop yourself. Read the intermediate work, not only the conclusion. If you would not sign it, do not ship it.