Estimate how much GPU memory a model needs for inference or training, based on parameter count and precision.
Inference: VRAM ≈ params × bytes_per_param × 1.2, weights plus a rough activation/KV-cache buffer. Training with Adam: VRAM ≈ params × (2 × bytes_per_param + 8), since Adam keeps weights, gradients at your chosen precision, and two momentum terms per parameter typically stored in fp32 at 4 bytes each, 8 bytes total. These are the same rules of thumb published in Hugging Face's and EleutherAI's model-memory references, not a figure specific to any one framework.
| Scenario | Estimated VRAM |
|---|---|
| Inference, fp16 | 15.65 GiB |
| Inference, int4 | 3.91 GiB |
| Training, fp16 mixed precision (Adam) | 78.23 GiB |
| Training, full fp32 (Adam) | 104.31 GiB |
A 7B model fits comfortably on a single 24GB consumer card for fp16 inference, and int4 leaves enough headroom for a decent context window on top. Full fine-tuning at fp16 mixed precision needs around 80GB, past what one high-end consumer GPU offers. LoRA and similar methods that update a small slice of parameters instead of all of them exist largely to dodge that 80GB number.
Batch size and sequence length both push activation memory up in ways the flat 1.2x buffer does not capture, especially at long context where the KV cache can dominate. Gradient checkpointing, optimizer choice beyond Adam, and framework-specific fragmentation all shift the training number too. Size hardware from this, but confirm the actual footprint with a real run before signing a GPU rental invoice.
Rule-of-thumb math, not a spec sheet. Your actual mileage depends on the framework, batch size, and sequence length you run with.