IndexFlatL2 errors are mostly about shapes and dtypes. Once you move to IVF indexes for speed, product quantization for memory, or GPU indexes for scale, a different class of problem shows up — and the first one on this list isn't even an error. It's a silent performance trap that looks like everything is working fine.
The instinct when recall looks bad is to raise nprobe — more inverted lists probed per query, more candidates considered, better recall. That instinct is correct right up until nprobe reaches nlist, at which point it stops meaning anything:
import faiss
import numpy as np
d = 128
nlist = 100
quantizer = faiss.IndexFlatL2(d)
index = faiss.IndexIVFFlat(quantizer, d, nlist)
vectors = np.random.rand(50000, d).astype('float32')
index.train(vectors)
index.add(vectors)
index.nprobe = 500 # 5x nlist — no error, no warning
distances, indices = index.search(vectors[:10], k=5)
Nothing throws. FAISS silently clips the search to the nlist lists that actually exist, which means you're now visiting every list in the index — an exhaustive scan. You get exactly the recall of IndexFlatL2, but at a latency that's worse than either a properly-tuned IVF search or just using a flat index directly, because you're paying the IVF routing overhead on top of a full scan. There's no message telling you this happened; the only symptom is a query time that stopped scaling the way you'd expect. If you find yourself setting nprobe anywhere close to nlist to hit a recall target, the actual fix is usually a smaller nlist in the first place, not a larger nprobe.
Product quantization compresses each vector by splitting it into M equal-length sub-vectors and quantizing each one independently. That splitting step is where training fails if the numbers don't line up:
d = 100
M = 8 # 100 is not divisible by 8
quantizer = faiss.IndexFlatL2(d)
index = faiss.IndexIVFPQ(quantizer, d, 100, M, 8)
vectors = np.random.rand(50000, d).astype('float32')
index.train(vectors) # AssertionError
AssertionError: d % M == 0
There's no rounding or padding happening automatically — d has to divide evenly by M, full stop. For a 100-dimensional embedding, valid subquantizer counts are limited to the actual divisors of 100 (1, 2, 4, 5, 10, 20, 25, 50, 100). In practice, most people hit this after switching embedding models — an old config with M=8 tuned for a 768-dim model doesn't survive a move to a 100-dim or 384-dim one without recalculating M.
def valid_subquantizer_counts(d, max_m=64):
return [m for m in range(1, max_m + 1) if d % m == 0]
d = 100
print(valid_subquantizer_counts(d))
# [1, 2, 4, 5, 10, 20, 25, 50]
M = 10 # a real divisor of 100
index = faiss.IndexIVFPQ(quantizer, d, 100, M, 8)
A related assertion shows up on the nbits parameter (the last argument above): most FAISS builds cap it at 8, meaning 256 codewords per subquantizer. Pass anything higher and training fails the same way. There's rarely a reason to go past 8 anyway — the memory savings from PQ come mostly from the number of subquantizers, not from packing more codewords into each one.
One more note on IVFPQ training specifically: if you also see a warning about insufficient training points for the coarse quantizer's k-means step, that's the same class of issue covered in the IndexFlatL2 error post's training-assertion section — not a PQ-specific problem, and the fix (more training vectors relative to nlist) is the same. Worth knowing: that particular warning has been reported to occasionally fire even when the training set is actually large enough, so don't treat it as gospel if your numbers already look reasonable — check index.is_trained directly instead of trusting the warning alone.
This is the one that confuses people who check nvidia-smi, see plenty of free memory, and still get an allocation failure:
res = faiss.StandardGpuResources()
gpu_index = faiss.index_cpu_to_gpu(res, 0, cpu_index)
distances, indices = gpu_index.search(large_query_batch, k=100)
RuntimeError: Error in ... StandardGpuResources: alloc fail type TemporaryMemoryBuffer
FAISS's GPU backend pre-allocates a fixed pool of temporary/scratch memory — roughly 1.5GB by default — for use during search and training, separate from the memory holding your actual index data. This error means an operation needed more scratch space than that pool has, regardless of how much total GPU memory is free. It's a FAISS-internal bookkeeping limit, not a real out-of-memory condition, which is exactly why nvidia-smi doesn't corroborate it.
res = faiss.StandardGpuResources()
# Option 1: give it a bigger scratch pool (bytes)
res.setTempMemory(2 * 1024 * 1024 * 1024) # 2GB
# Option 2: disable pre-allocation, allocate on demand instead
res.setTempMemory(0)
gpu_index = faiss.index_cpu_to_gpu(res, 0, cpu_index)
Bumping the pool works for most cases and keeps the performance benefit of pre-allocation. Disabling it entirely (setTempMemory(0)) is the fallback when you genuinely don't know how large your batches will get — you trade a small amount of per-call allocation overhead for never hitting this wall.