Neural Network Parameter Counter
Add layers, configure their dimensions, and see the total parameter count and memory footprint update live.
🔒100% Client-Side. Everything runs in your browser — no data is sent to any server.
LOAD EXAMPLE ARCHITECTURE
Layers
1
Layer type
num_emb
emb_dim
23,440,896
params
Embedding(30522, 768)
2
Layer type
embed_dim
num_heads
2,362,368
params
MultiheadAttention(embed_dim=768, num_heads=12)
3
Layer type
norm_shape
1,536
params
LayerNorm(768)
4
Layer type
in_features
out_features
bias
2,362,368
params
Linear(768, 3072, bias=True)
5
Layer type
in_features
out_features
bias
2,360,064
params
Linear(3072, 768, bias=True)
Summary
Total Parameters
30,527,232
30.5M params
Memory — float32
122.11 MB
4 bytes / param
Memory — float16 / bf16
61.05 MB
2 bytes / param
Memory — int8
30.53 MB
1 byte / param
Size comparison (log scale)
ResNet-50
25.6M
BERT-Base
110.0M
GPT-2 Small
117.0M
Your model
30.5M
GPT-4 (est.)
1.8T
Per-layer breakdown
| # | Layer | Parameters | % of total |
|---|---|---|---|
| 1 | Embedding(30522, 768) | 23,440,896 | 76.8% |
| 2 | MultiheadAttention(embed_dim=768, num_heads=12) | 2,362,368 | 7.7% |
| 3 | LayerNorm(768) | 1,536 | 0.0% |
| 4 | Linear(768, 3072, bias=True) | 2,362,368 | 7.7% |
| 5 | Linear(3072, 768, bias=True) | 2,360,064 | 7.7% |
| Total | 30,527,232 | 100% | |
How to count parameters in PyTorch
The canonical one-liner counts all learnable parameters across every layer:
import torch
import torch.nn as nn
model = ... # your model
# Total trainable parameters
total = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total:,}")
# Trainable vs frozen (useful after freezing layers for fine-tuning)
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in model.parameters() if not p.requires_grad)
print(f"Trainable: {trainable:,}")
print(f"Frozen: {frozen:,}")
You can also use torchinfo for a Keras-style summary with input/output shapes per layer:
# pip install torchinfo
from torchinfo import summary
summary(model, input_size=(1, 3, 224, 224))
# Output:
# ==================================================================
# Layer (type) Output Shape Param #
# ==================================================================
# Conv2d-1 [1, 64, 112, 112] 9,408
# BatchNorm2d-2 [1, 64, 112, 112] 128
# ...
# Total params: 25,557,032
# Trainable params: 25,557,032
Trainable vs frozen parameters
When fine-tuning a pre-trained model you often freeze the backbone and only train the head. Frozen parameters are still stored in VRAM (they are needed for the forward pass and gradients through them), but their gradient buffers are not allocated, saving roughly 8 bytes × frozen_params in mixed-precision training (2 bytes weights + 2 bytes gradients + 4 bytes optimizer state).
# Freeze all parameters (e.g., backbone)
for param in model.backbone.parameters():
param.requires_grad = False
# Only the head is updated
for param in model.head.parameters():
param.requires_grad = True
Memory formula: from parameters to GPU VRAM
Storing weights alone is just the start. During training you also need:
- Weights: P × dtype_bytes (e.g. 4 for float32, 2 for float16)
- Gradients: same size as weights ≈ P × dtype_bytes
- Optimizer state (Adam): 2 × P × 4 bytes (first and second moment, stored in float32)
- Activations: depends on batch size and sequence length — often 2–5× the weight memory
# Rule of thumb for training with Adam in float32:
# VRAM ≈ P × 4 (weights)
# + P × 4 (gradients)
# + P × 8 (Adam m1 + m2)
# = P × 16 bytes — ignoring activations
# Mixed-precision (AMP) training — weights in fp16, optimizer in fp32:
# VRAM ≈ P × 2 (fp16 weights)
# + P × 2 (fp16 gradients)
# + P × 4 (fp32 master weights)
# + P × 8 (fp32 Adam state)
# = P × 16 bytes — same total, but faster compute
# BERT-Base (110M params) in float32:
params = 110_000_000
training_vram_gb = params * 16 / 1e9
print(f"~{training_vram_gb:.1f} GB for weights + optimizer — plus activations")
Rule of thumb: model size vs GPU VRAM
| Model | Params | Inference (fp16) | Training (fp32 Adam) |
|---|---|---|---|
| ResNet-50 | 25.6M | ~50 MB | ~410 MB |
| BERT-Base | 110M | ~220 MB | ~1.8 GB |
| GPT-2 Small | 117M | ~234 MB | ~1.9 GB |
| LLaMA-7B | 7B | ~14 GB | ~112 GB |
| LLaMA-70B | 70B | ~140 GB | ~1.1 TB |
Training estimates include weights + gradients + Adam optimizer state. Activations depend on batch size and are not included.