GPU memory is expensive and finite, yet most AI serving setups waste a large fraction of it. Here are the usual causes and what to do about them.
Full precision you do not need
Serving weights in FP32 or even FP16 when a lower precision would do wastes memory. Quantizing to INT8 or FP4, with accuracy validation, can cut weight memory by half or more, leaving room for larger batches.
Fragmentation from ad hoc allocation
Repeated allocation and freeing of differently sized buffers fragments GPU memory, so a request fails for lack of a contiguous block even though total free memory is sufficient. A preallocated buffer pool eliminates this.
An unmanaged KV cache
For LLMs, the attention KV cache often uses more memory than the weights. Naive implementations reserve worst-case space per request. Paged attention, as used by vLLM, allocates the cache in small pages on demand, dramatically improving memory efficiency and concurrency.
Oversized static batches
Reserving memory for a maximum batch that rarely fills wastes it the rest of the time. Dynamic batching sizes memory to actual demand.
Leaks and retained tensors
Holding references to intermediate tensors, or keeping the computation graph alive during inference, quietly consumes memory. Inference should run without gradient tracking and release intermediates promptly.
Key takeaways
- Quantize weights to the lowest precision that preserves accuracy
- Use a buffer pool to avoid fragmentation
- Adopt paged attention to tame the KV cache
- Size batches dynamically instead of reserving the maximum
- Disable gradient tracking and release intermediates during inference
