GPUFits
CPU Offload: Is Partial GPU Offloading Worth It?
updated 2026-08-04 · verified 2026-08-04
Your GPU is 2GB short of fitting the model. llama.cpp’s -ngl flag promises a solution: put most layers on the GPU, the rest on CPU. Here’s what that actually costs.
How partial offload works
llama-cli -ngl 20 model.gguf puts 20 layers on the GPU; the remaining layers and part of the computation run on the CPU in system RAM. Every token still passes through all layers — so every token waits for the slowest part of the pipeline: your system memory.
The memory wall
| Path | Bandwidth | Share of each token’s time |
|---|---|---|
| GPU VRAM (4090) | 1,008 GB/s | fast |
| PCIe 4.0 x16 | ~32 GB/s (transfers) | moderate |
| System RAM (DDR5 dual-channel) | 60–100 GB/s | the bottleneck |
A 4090 reads weights ~12× faster than your CPU reads system RAM. Our speed model reflects this:
Offloaded speed = min(GPU bandwidth, ~80 GB/s) × efficiency ÷ per-token weight size
Real numbers (24B model, Q4_K_M)
| Setup | Est. speed | Feel |
|---|---|---|
| Fully on RTX 4090 | ~50 tok/s | instant |
| 60% layers offloaded | ~6 tok/s | reading speed |
| Fully on CPU (DDR5) | ~4 tok/s | painful |
Two caveats in offload’s favor: MoE models offload better (only the active expert path is read per token — gpt-oss-20b stays usable partially offloaded), and batch/offline jobs don’t care about tok/s the way chat does.
When offload IS worth it
- Evaluation and one-off tasks: “can this 32B model do X at all?” — yes, run it overnight offloaded.
- Batch pipelines: summarizing 500 documents where 4 tok/s × parallel streams is fine.
- Capacity emergencies: the model you need is 2GB over, and the alternative is a cloud bill.
When it’s NOT
- Interactive chat — reading-speed output gets old in minutes.
- Agent loops — agents chain dozens of calls; a 10× slowdown compounds into unusable.
- When a smaller fully-resident model exists — and in 2026, one almost always does. Qwen3-14B fully on-GPU will out-think a 32B gasping through DDR5 in practice, because you’ll actually iterate with it.
The better escape hatches
- Drop a quant level (Q4→Q3): usually smaller quality hit than the offload speed hit.
- q8 KV cache + shorter context: frees real GBs without touching the speed path.
- A second used GPU: two $180 RTX 3060s (24GB combined, layer-split) outclass any offload scheme on speed. See our multi-GPU guide.
FAQ
How slow is CPU offload really?
Expect a 5–10× slowdown. A 24B model that generates at 35 tok/s fully on a 4090 drops to roughly 4–8 tok/s with half its layers offloaded to a typical DDR5 system (60–100 GB/s vs 1,008 GB/s bandwidth).
Is offload better on a Mac?
There is no offload penalty on Apple Silicon — CPU and GPU share the same memory pool, so nothing crosses a bus. 'Offload' is only a PC/discrete-GPU concept.
Offload or smaller model — which should I choose?
Almost always the smaller model. A 13B at Q6 fully on-GPU beats a 32B at Q4 half-offloaded on both speed and quality-per-watt. Offload makes sense only when you specifically need the bigger model's capability and can tolerate reading-speed output.