← Glossary
Glossary
Tokenization
Splitting text into the smaller units (tokens — often sub-words, not whole words) that a language model actually operates on. What you’re charged for by most LLM APIs, and a common source of padding/length errors.
Related articles
Fix RuntimeError: CUDA error device-side assert triggered in PyTorchFix FAISS nprobe/nlist Slowdowns, IndexIVFPQ Training Asserts, and StandardGpuResources Alloc FailFix OSError: [Errno 28] No space left on Device — HuggingFace Model CacheFix "cannot import name 'cached_download'" from huggingface_hubFix Hugging Face CUDA Out of Memory When Loading Transformer ModelsFix HuggingFace Tokenizer Padding Errors: No pad_token, Truncation, Left vs Right
Related tools