Single-sequence tokenization almost never breaks. The moment you batch more than one input through a HuggingFace tokenizer, though, you run into one of a handful of padding and truncation issues — and the frustrating part is that most of them either fail with an error message that points at the wrong line, or don't fail at all and just quietly produce wrong output. This post walks through the ones that come up constantly, roughly in the order you're likely to hit them.
pip install transformers torchThis is usually the first wall people hit, and it's specific to decoder-only models — GPT-2, LLaMA, Mistral, and most instruction-tuned variants were pretrained without a dedicated pad token, because during pretraining nobody batches multiple documents together with padding.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
batch = tokenizer(["short text", "a somewhat longer piece of text"], padding=True, return_tensors="pt")
ValueError: Asking to pad but the tokenizer does not have a padding token.
Please select a token to use as `pad_token`, e.g. by:
`tokenizer.pad_token = tokenizer.eos_token`.
The error message already gives you the pragmatic fix, and it's the one most people reach for:
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token
batch = tokenizer(["short text", "a somewhat longer piece of text"], padding=True, return_tensors="pt")
Reusing eos_token as the pad token works fine for most fine-tuning and inference workflows, because the attention mask still tells the model which positions are real. The one case where it bites you: if you're training the model to predict eos_token as a meaningful signal (say, to learn when to stop generating), and your loss function isn't masking out padded positions, the model can end up seeing the same token ID mean two different things. If that's your setup, add a genuinely separate token instead:
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
model.resize_token_embeddings(len(tokenizer))
Don't skip resize_token_embeddings — adding a token to the tokenizer without resizing the model's embedding matrix leaves an index that has no corresponding row, and you'll trade this error for a much less obvious IndexError deep inside the forward pass.
No exception here, which is what makes it worse. You fix the pad-token error above, batch a few prompts through model.generate(), and the outputs for anything except the first row in the batch are nonsense — repeated tokens, truncated mid-word, or just unrelated to the prompt.
The cause is padding side. tokenizer.padding_side defaults to "right", which is correct for encoder models (BERT-style classification, where the model reads the whole sequence at once) but wrong for decoder-only generation. A causal LM generates by looking at the last token of each sequence in the batch — with right-padding, the "last token" of any sequence shorter than the longest one in the batch is a pad token, not real content, so the model is effectively being asked to continue from nothing.
tokenizer.padding_side = "left"
tokenizer.pad_token = tokenizer.eos_token # still needed
inputs = tokenizer(
["Translate to French: Hello", "Summarize: The quick brown fox..."],
padding=True, return_tensors="pt"
)
outputs = model.generate(**inputs, max_new_tokens=50)
Set padding_side = "left" for any batched generation call. It's harmless for training and classification, so if you're unsure which code path a tokenizer instance feeds, left-padding by default for a decoder-only model is the safer choice.
This one is the most annoying to debug because the traceback lies about where the problem is:
RuntimeError: The expanded size of the tensor (47) must match the existing size (63)
at non-singleton dimension 1
That error comes from torch.stack or a DataLoader's default collate function trying to build a batch tensor out of sequences of different lengths — it has nothing to do with tensor "expansion" in any meaningful sense, and the numbers in the message are sequence lengths, not the dimension you'd guess from the wording. Almost always, the actual bug is one call earlier: the tokenizer was never told to pad.
# Breaks: sequences come back as different-length lists, not a rectangular batch
batch = tokenizer(texts, return_tensors="pt")
# Fixed
batch = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
If you're inside a custom Dataset/DataLoader setup and tokenizing per-example rather than per-batch, pad in the collate function instead, using the tokenizer's own pad() method so special tokens and attention masks stay consistent:
def collate_fn(batch):
return tokenizer.pad(batch, padding=True, return_tensors="pt")
You'll see this as a warning, not an exception, which is exactly why it causes problems downstream instead of where you'd notice it:
UserWarning: Truncation was not explicitly activated but `max_length` is provided a specific value,
please use `truncation=True` to explicitly truncate examples to max length.
Setting max_length alone only controls padding length in this state — sequences longer than max_length pass through completely untouched. They then hit the model's positional embedding table, which has a fixed size, and fail with a shape error or a CUDA device-side assert that's several layers removed from the tokenizer call that actually caused it.
# Silently does nothing to long sequences
batch = tokenizer(texts, max_length=512, padding=True, return_tensors="pt")
# Correct
batch = tokenizer(texts, max_length=512, padding=True, truncation=True, return_tensors="pt")
max_length and truncation=True are a pair. If you're setting one, set both, and if you ever see the truncation warning in your logs, treat it as a real bug report, not noise — something in your pipeline is going to fail on the first input that exceeds the model's context window.
def safe_tokenize(tokenizer, texts, max_length=512, for_generation=False):
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
if for_generation:
tokenizer.padding_side = "left"
return tokenizer(
texts,
padding=True,
truncation=True,
max_length=max_length,
return_tensors="pt",
)
Most of what shows up above traces back to one of two habits: assuming a tokenizer ships fully configured for batching (it usually doesn't for decoder-only models), or assuming padding and truncation are independent flags rather than a pair that both need to be set together. Once those two assumptions are gone, the rest of a tokenization pipeline tends to just work.