Add layers, configure their dimensions, and see the total parameter count and memory footprint update live.
How to count parameters in PyTorch
The canonical one-liner counts all learnable parameters across every layer:
import torch
import torch.nn as nn
model = ... # your model
# Total trainable parameters
total = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total:,}")
# Trainable vs frozen (useful after freezing layers for fine-tuning)
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in model.parameters() if not p.requires_grad)
print(f"Trainable: {trainable:,}")
print(f"Frozen: {frozen:,}")
You can also use torchinfo for a Keras-style summary with input/output shapes per layer:
# pip install torchinfo
from torchinfo import summary
summary(model, input_size=(1, 3, 224, 224))
# Output:
# ==================================================================
# Layer (type) Output Shape Param #
# ==================================================================
# Conv2d-1 [1, 64, 112, 112] 9,408
# BatchNorm2d-2 [1, 64, 112, 112] 128
# ...
# Total params: 25,557,032
# Trainable params: 25,557,032
Trainable vs frozen parameters
When fine-tuning a pre-trained model you often freeze the backbone and only train the head. Frozen parameters are still stored in VRAM (they are needed for the forward pass and gradients through them), but their gradient buffers are not allocated, saving roughly 8 bytes × frozen_params in mixed-precision training (2 bytes weights + 2 bytes gradients + 4 bytes optimizer state).
# Freeze all parameters (e.g., backbone)
for param in model.backbone.parameters():
param.requires_grad = False
# Only the head is updated
for param in model.head.parameters():
param.requires_grad = True
Memory formula: from parameters to GPU VRAM
Storing weights alone is just the start. During training you also need:
- Weights: P × dtype_bytes (e.g. 4 for float32, 2 for float16)
- Gradients: same size as weights ≈ P × dtype_bytes
- Optimizer state (Adam): 2 × P × 4 bytes (first and second moment, stored in float32)
- Activations: depends on batch size and sequence length — often 2–5× the weight memory
# Rule of thumb for training with Adam in float32:
# VRAM ≈ P × 4 (weights)
# + P × 4 (gradients)
# + P × 8 (Adam m1 + m2)
# = P × 16 bytes — ignoring activations
# Mixed-precision (AMP) training — weights in fp16, optimizer in fp32:
# VRAM ≈ P × 2 (fp16 weights)
# + P × 2 (fp16 gradients)
# + P × 4 (fp32 master weights)
# + P × 8 (fp32 Adam state)
# = P × 16 bytes — same total, but faster compute
# BERT-Base (110M params) in float32:
params = 110_000_000
training_vram_gb = params * 16 / 1e9
print(f"~{training_vram_gb:.1f} GB for weights + optimizer — plus activations")
Rule of thumb: model size vs GPU VRAM
| Model | Params | Inference (fp16) | Training (fp32 Adam) |
|---|
| ResNet-50 | 25.6M | ~50 MB | ~410 MB |
| BERT-Base | 110M | ~220 MB | ~1.8 GB |
| GPT-2 Small | 117M | ~234 MB | ~1.9 GB |
| LLaMA-7B | 7B | ~14 GB | ~112 GB |
| LLaMA-70B | 70B | ~140 GB | ~1.1 TB |
Training estimates include weights + gradients + Adam optimizer state. Activations depend on batch size and are not included.