You are training a model on GPU and suddenly everything stops. The terminal shows a wall of stack frames and at the bottom, a message that tells you almost nothing:
Traceback (most recent call last):
File "train.py", line 47, in <module>
loss = criterion(logits, labels)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/loss.py", line 1179, in forward
return F.cross_entropy(input, target, weight=self.weight,
File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 3029, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_RTLD_GLOBAL=YES` if you get duplicate key errors too.
That is the entire error. No line number that is actually wrong, no variable names, no hint about what value was bad. CUDA silently ate the real diagnostic and handed you a crash receipt instead.
This guide explains why that happens and gives you five concrete fixes in the order you should apply them, starting with the one that immediately reveals the real problem.
The GPU executes thousands of threads in parallel. When PyTorch launches a CUDA kernel — say, the cross-entropy kernel — it does not wait for that kernel to finish before returning control to Python. Execution is asynchronous. Python keeps running, queuing more operations, while the GPU works in the background.
When a thread on the GPU hits an invalid memory access or a bounds check
failure, the CUDA runtime sets an error flag internally. That flag is not
checked until the next synchronization point — usually the next
CUDA API call that forces a sync (like copying a tensor to CPU, calling
.item(), or explicitly calling
torch.cuda.synchronize()).
By the time Python sees the error, several more operations may have been
queued. The Python stack frame that surfaces the
RuntimeError: CUDA error: device-side assert triggered
message is typically not the frame where the actual mistake occurred. The
real offending operation was several steps earlier.
This is why the traceback is useless. The fix is to force synchronous execution so that the error surfaces exactly where the bad kernel runs.
This is the most important step in the entire guide. Before you change anything in your model code, reproduce the crash on CPU. CPU operations are synchronous and eager: when a bounds check fails the error is raised immediately with a Python traceback that points at the exact line responsible.
CUDA_LAUNCH_BLOCKING=1Run your script with the environment variable set. This forces every CUDA kernel launch to block until the kernel completes, which makes GPU errors surface at the correct call site.
CUDA_LAUNCH_BLOCKING=1 python train.py
Or inside a Python script before any CUDA calls:
import os
os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
import torch
# ... rest of your code
With blocking enabled the traceback will now point at the correct line. Read that traceback carefully — it will usually name the operation that triggered the assert (cross-entropy, embedding lookup, etc.) and from that you can identify which of the fixes below you need.
If CUDA_LAUNCH_BLOCKING=1 still does not give a clean message
(rare, but happens with some CUDA versions), move everything to CPU for a
single forward pass:
import torch
import torch.nn as nn
# Assume you have these from your normal training setup
model = MyModel()
inputs = next(iter(train_loader)) # whatever your loader returns
# Move to CPU for diagnosis
model_cpu = model.cpu()
# Unpack your batch — adjust to your actual data format
x, labels = inputs
x = x.cpu()
labels = labels.cpu()
# Single forward + loss on CPU — the real error will appear here
logits = model_cpu(x)
loss = criterion(logits, labels)
print("Forward pass succeeded on CPU — error may be GPU-only")
When this raises an exception, read the full traceback. The message on CPU
will say something like
IndexError: Target 15 is out of bounds.
or
IndexError: index out of range in self.
Those are the real errors. The sections below fix each one.
This is the single most common cause of
CUDA error: device-side assert triggered.
nn.CrossEntropyLoss expects each label
to be an integer in the range
[0, num_classes). If any label is equal
to num_classes or higher — or if it is
negative and not the special
ignore_index value — the CUDA kernel
triggers the device-side assert.
import torch
import torch.nn as nn
num_classes = 10
model = nn.Linear(64, num_classes).cuda()
criterion = nn.CrossEntropyLoss()
# Batch of 4 samples — labels should be 0..9
logits = model(torch.randn(4, 64).cuda())
# BUG: one label equals num_classes (10), which is out of range [0, 10)
labels = torch.tensor([3, 7, 10, 1]).cuda() # 10 is invalid
# On GPU: RuntimeError: CUDA error: device-side assert triggered
# On CPU: IndexError: Target 10 is out of bounds.
loss = criterion(logits, labels)
def check_labels(labels, num_classes, ignore_index=-100):
mask = labels != ignore_index
valid = labels[mask]
if valid.min() < 0:
raise ValueError(
f"Label contains negative value {valid.min().item()} "
f"(not equal to ignore_index={ignore_index})"
)
if valid.max() >= num_classes:
raise ValueError(
f"Label {valid.max().item()} is out of range "
f"[0, {num_classes}). Did you accidentally include num_classes as a label?"
)
print(f"Labels OK: min={valid.min().item()}, max={valid.max().item()}, "
f"num_classes={num_classes}")
check_labels(labels, num_classes=10)
import torch
import torch.nn as nn
num_classes = 10
model = nn.Linear(64, num_classes).cuda()
criterion = nn.CrossEntropyLoss()
logits = model(torch.randn(4, 64).cuda())
# Correct: labels in [0, num_classes) = [0, 9]
labels = torch.tensor([3, 7, 9, 1]).cuda()
loss = criterion(logits, labels)
print(f"loss = {loss.item():.4f}")
The mistake is nearly always an off-by-one. Datasets that are 1-indexed (1
through N) instead of 0-indexed (0 through N-1) will pass label
N when the model has
N output classes, triggering the assert.
Subtract 1 from every label when loading data from such a dataset:
labels = labels - 1 # convert 1-indexed to 0-indexed
Also watch for datasets that use a special "background" or "ignore" class
encoded as 255 in segmentation tasks.
Pass that value as
ignore_index=255 to
nn.CrossEntropyLoss rather than leaving
it as a real label:
criterion = nn.CrossEntropyLoss(ignore_index=255)
nn.Embedding(vocab_size, embedding_dim)
creates a lookup table with indices
0 through
vocab_size - 1. Passing an index that
equals or exceeds vocab_size triggers the
same device-side assert.
import torch
import torch.nn as nn
vocab_size = 1000
embedding = nn.Embedding(vocab_size, 64).cuda()
# Token IDs in a tokenized sentence — one ID exceeds vocabulary
token_ids = torch.tensor([[12, 45, 1000, 7]]).cuda()
# 1000 is out of range: valid indices are 0..999
# On GPU: RuntimeError: CUDA error: device-side assert triggered
# On CPU: IndexError: index out of range in self
out = embedding(token_ids)
def check_embedding_indices(token_ids, vocab_size):
if token_ids.min() < 0:
raise ValueError(
f"Embedding index contains negative value: {token_ids.min().item()}"
)
if token_ids.max() >= vocab_size:
bad_ids = token_ids[token_ids >= vocab_size].unique().tolist()
raise ValueError(
f"Embedding index out of range. "
f"vocab_size={vocab_size}, offending indices: {bad_ids}"
)
print(f"Embedding indices OK: min={token_ids.min().item()}, "
f"max={token_ids.max().item()}, vocab_size={vocab_size}")
check_embedding_indices(token_ids, vocab_size=1000)
import torch
import torch.nn as nn
vocab_size = 1000
embedding = nn.Embedding(vocab_size, 64).cuda()
# All indices in [0, vocab_size)
token_ids = torch.tensor([[12, 45, 999, 7]]).cuda()
out = embedding(token_ids)
print(f"Embedding output shape: {out.shape}")
[PAD],
[CLS],
[SEP] to the tokenizer but did not call
model.resize_token_embeddings(len(tokenizer)).
UNK index inside the range.
# After adding special tokens to a HuggingFace tokenizer:
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
model.resize_token_embeddings(len(tokenizer)) # expand embedding table
Some CUDA kernels assert when they receive non-finite floating point values
as inputs. A common example is log-softmax or cross-entropy receiving
logits that contain NaN or
Inf, or a loss function that explodes
during training and propagates infinite gradients into the next forward pass.
import torch
import torch.nn as nn
# Simulate logits that have gone to infinity (e.g., after gradient explosion)
logits = torch.tensor([[1.0, float("inf"), -2.0],
[0.5, 0.1, float("nan")]]).cuda()
labels = torch.tensor([1, 2]).cuda()
criterion = nn.CrossEntropyLoss()
# Can trigger device-side assert depending on CUDA version and kernel path
loss = criterion(logits, labels)
def check_finite(tensor, name="tensor"):
if torch.isnan(tensor).any():
nan_count = torch.isnan(tensor).sum().item()
raise ValueError(f"{name} contains {nan_count} NaN value(s)")
if torch.isinf(tensor).any():
inf_count = torch.isinf(tensor).sum().item()
raise ValueError(f"{name} contains {inf_count} Inf value(s)")
print(f"{name}: finite, min={tensor.min().item():.4f}, "
f"max={tensor.max().item():.4f}")
check_finite(logits, "logits")
check_finite(inputs, "inputs")
Add these checks immediately before the loss call when debugging. You can also register a forward hook on any module to check all its outputs automatically during a diagnostic run:
def nan_hook(module, inputs, output):
if isinstance(output, torch.Tensor):
if torch.isnan(output).any() or torch.isinf(output).any():
raise RuntimeError(
f"NaN/Inf detected in output of {module.__class__.__name__}"
)
# Register on every submodule
for name, module in model.named_modules():
module.register_forward_hook(nan_hook)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0).
torch.log(x) where
x can be zero (e.g., after a ReLU or
sigmoid for very negative activations). Add a small epsilon:
torch.log(x + 1e-8).
float16 saturates at ~65 504. Activations
above that clip to Inf. Use
torch.cuda.amp.GradScaler and check
whether the scaler is consistently reducing the scale factor
(scaler.get_scale() trends toward zero
is a warning sign).
import torch
from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
for batch in train_loader:
optimizer.zero_grad()
with autocast():
logits = model(batch["input_ids"].cuda())
loss = criterion(logits, batch["labels"].cuda())
scaler.scale(loss).backward()
# Clip gradients BEFORE the optimizer step
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
scaler.step(optimizer)
scaler.update()
print(f"loss={loss.item():.4f}, amp_scale={scaler.get_scale():.1f}")
nn.CrossEntropyLoss and other
classification losses require labels to be
torch.long (64-bit integer). If your
labels are torch.float32 — which
happens when you load them from a NumPy float array or a pandas column
without casting — the CUDA kernel receives unexpected data and triggers the
assert.
import torch
import torch.nn as nn
import numpy as np
num_classes = 5
model = nn.Linear(32, num_classes).cuda()
criterion = nn.CrossEntropyLoss()
logits = model(torch.randn(8, 32).cuda())
# Labels come from a numpy float array (common when reading a CSV)
raw_labels = np.array([0.0, 2.0, 4.0, 1.0, 3.0, 0.0, 2.0, 1.0])
labels = torch.tensor(raw_labels).cuda() # dtype = torch.float32
# On GPU: RuntimeError: CUDA error: device-side assert triggered
# On CPU: RuntimeError: expected scalar type Long but found Float
loss = criterion(logits, labels)
def check_label_dtype(labels):
if labels.dtype != torch.long:
raise TypeError(
f"Labels have dtype {labels.dtype}. "
f"CrossEntropyLoss requires torch.long (torch.int64). "
f"Fix: labels = labels.long()"
)
print(f"Label dtype OK: {labels.dtype}")
check_label_dtype(labels)
import torch
import torch.nn as nn
import numpy as np
num_classes = 5
model = nn.Linear(32, num_classes).cuda()
criterion = nn.CrossEntropyLoss()
logits = model(torch.randn(8, 32).cuda())
raw_labels = np.array([0.0, 2.0, 4.0, 1.0, 3.0, 0.0, 2.0, 1.0])
# Cast explicitly to long before creating the tensor
labels = torch.tensor(raw_labels, dtype=torch.long).cuda()
# Or, if you already have a float tensor: labels = labels.long()
loss = criterion(logits, labels)
print(f"loss = {loss.item():.4f}")
A similar dtype issue appears with
nn.BCELoss for binary classification:
it expects labels to be
torch.float32, not
torch.long. If you switch between
multi-class and binary setups, double-check both the loss function and the
label dtype.
# Multi-class: nn.CrossEntropyLoss — labels must be torch.long
# Binary: nn.BCELoss / nn.BCEWithLogitsLoss — labels must be torch.float
# Binary example
bce = nn.BCEWithLogitsLoss()
logits_binary = model_binary(x).squeeze(1) # shape [B]
labels_binary = labels_01.float() # cast to float32
loss = bce(logits_binary, labels_binary)
When you hit
RuntimeError: CUDA error: device-side assert triggered,
work through this list in order:
| # | Step | What to look for |
|---|---|---|
| 0 | Run with CUDA_LAUNCH_BLOCKING=1 or move to CPU |
Get the real traceback and the real error message |
| 1 | Check label range | labels.min() >= 0 and labels.max() < num_classes |
| 2 | Check embedding indices | token_ids.max() < vocab_size |
| 3 | Check for NaN / Inf | torch.isnan(x).any() and torch.isinf(x).any() |
| 4 | Check label dtype | labels.dtype == torch.long for CrossEntropyLoss |
| 5 | Check for device mismatch | All tensors on the same device — see PyTorch device mismatch fix |
Run these checks as assertions at the top of your training loop during debugging. Once you have confirmed the fix, you can remove them or gate them behind a debug flag to avoid the overhead in production training runs.
DEBUG = True # set False for production
def debug_check_batch(inputs, labels, logits, num_classes):
if not DEBUG:
return
assert not torch.isnan(inputs).any(), "NaN in inputs"
assert not torch.isinf(inputs).any(), "Inf in inputs"
assert labels.dtype == torch.long, f"Expected torch.long, got {labels.dtype}"
assert labels.min() >= 0, f"Negative label: {labels.min().item()}"
assert labels.max() < num_classes, (
f"Label {labels.max().item()} out of range [0, {num_classes})"
)
assert not torch.isnan(logits).any(), "NaN in logits"
assert not torch.isinf(logits).any(), "Inf in logits"
# Inside training loop:
for batch in train_loader:
inputs, labels = batch
inputs = inputs.cuda()
labels = labels.cuda()
logits = model(inputs)
debug_check_batch(inputs, labels, logits, num_classes=NUM_CLASSES)
loss = criterion(logits, labels)
loss.backward()
optimizer.step()
optimizer.zero_grad()
import os, torch
# 1. Force synchronous CUDA (do this first when debugging)
os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
# 2. Check labels
assert labels.dtype == torch.long
assert labels.min() >= 0 and labels.max() < num_classes
# 3. Check embedding indices
assert token_ids.min() >= 0 and token_ids.max() < vocab_size
# 4. Check for NaN / Inf
assert not torch.isnan(x).any() and not torch.isinf(x).any()
# 5. Check devices match (see also: /blog/pytorch-device-mismatch-fix)
assert inputs.device == labels.device == next(model.parameters()).device
CUDA error: device-side assert triggered
is always a symptom of an actual logic error in your data or model. The GPU
catches the error asynchronously, so the stack trace points at the wrong
line. The single most effective debugging step is forcing synchronous
execution with
CUDA_LAUNCH_BLOCKING=1 or reproducing on
CPU — both strategies make the real error visible immediately.
Once you see the real error, the fix is almost always one of four things: a label index that is out of range for your number of classes, an embedding index that exceeds the vocabulary size, a NaN or Inf value that crept into your activations, or labels that have the wrong integer dtype. Add the corresponding assertion to your training loop, confirm it catches the bad batch, fix the root cause in your data pipeline or learning-rate schedule, and the cryptic CUDA error will not come back.