← Glossary
Glossary
Tokenization
Splitting text into the smaller units (tokens — often sub-words, not whole words) that a language model actually operates on. What you’re charged for by most LLM APIs, and a common source of padding/length errors.
Related articles
Fix CUDA Error: Device-Side Assert Triggered in PyTorchFAISS nprobe, IndexIVFPQ and GPU Memory Errors FixedHugging Face No Space Left on Device (Errno 28) Fixhuggingface_hub ImportError: is_offline_mode, cached_downloadFix Hugging Face CUDA Out of Memory at Model LoadFix HuggingFace Tokenizer Padding and pad_token Errors
Related tools