UnicodeDecodeError: 'utf-8' codec can't decode byte"This one usually shows up the first time a CSV comes from Excel or another Windows tool instead of being generated by your own UTF-8-only pipeline. The instinct is to reach for errors="replace" and move on, but that quietly deletes data instead of fixing anything. This article reproduces the error, explains why it's almost always a single specific encoding mismatch, and shows how to detect and fix it properly.
Any CSV containing accented or non-ASCII characters, saved in a single-byte encoding other than UTF-8, triggers it the moment read_csv() tries to decode those bytes as UTF-8:
import pandas as pd
# A CSV saved as Windows-1252/Latin-1, e.g. exported from Excel
# name,city
# José,Bogotá
# François,Zürich
df = pd.read_csv("data.csv")
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 13: invalid continuation byteThe byte position in the error message points at the first accented character pandas' C parser hit while decoding, not necessarily the first one in the file (the parser reads in chunks), which is why the position often looks like it's "in the middle" of a file that has non-ASCII characters near the top.
pd.read_csv() defaults to encoding="utf-8". UTF-8 encodes any character outside plain ASCII using two or more bytes, following a specific bit-pattern rule for continuation bytes. Windows-1252 and Latin-1 (ISO-8859-1) instead encode common Western European accented characters as a single byte in the upper range (0x80-0xFF). A UTF-8 decoder reading one of those single bytes sees an invalid continuation-byte pattern and raises, rather than silently misreading it as some other character:
"é".encode("utf-8") # b'\xc3\xa9' — two bytes, valid UTF-8 continuation sequence
"é".encode("cp1252") # b'\xe9' — one byte, exactly the 0xe9 the error above namesThis is why the fix is almost never "the file is corrupted" — it's a single, identifiable encoding mismatch, and Windows-1252 (or its close relative Latin-1) is the encoding behind the overwhelming majority of real-world cases, because it's what Excel and most Windows software write by default.
df = pd.read_csv("data.csv", encoding="utf-8", encoding_errors="replace")
print(df)
# name city
# 0 Jos<0xEF><0xBF><0xBD> Bogot<0xEF><0xBF><0xBD>
# 1 Fran<0xEF><0xBF><0xBD>ois Z<0xEF><0xBF><0xBD>richNo exception, but every byte pandas couldn't decode as UTF-8 was replaced with the Unicode replacement character (U+FFFD) and is now permanently gone — "José" became "Jos" plus a placeholder glyph, unrecoverable from this DataFrame. This is fine only if you've deliberately decided lossy data is acceptable; reached for as a quick fix for the exception itself, it's a silent data-corruption bug that looks like a successful read.
Pass the correct encoding directly. If you already know the file's source (Excel/Windows export), cp1252 or latin-1 resolves it immediately:
df = pd.read_csv("data.csv", encoding="cp1252")
# or, near-equivalent for this class of file:
df = pd.read_csv("data.csv", encoding="latin-1")When the source isn't known, detect it instead of guessing — charset-normalizer (pip install charset-normalizer) inspects the file's actual byte statistics:
from charset_normalizer import from_path
result = from_path("data.csv").best()
df = pd.read_csv("data.csv", encoding=result.encoding)to_csv()) so the encoding mismatch doesn't propagate to every downstream consumer of the cleaned file.
Situation Fix
───────────────────────────────────────────────────── ──────────────────────────────────────
Known Excel/Windows-exported CSV encoding="cp1252" (or "latin-1")
Source/encoding unknown charset-normalizer's from_path().best()
Need the error gone but data loss is unacceptable never encoding_errors="replace"
Re-sharing the cleaned file downstream re-save with to_csv() (defaults to UTF-8)