FAISS Beyond IndexFlatL2: nprobe, IndexIVFPQ Training, and GPU Memory Errors

Tested with: faiss-cpu 1.15.1, numpy 2.5.3, Python 3.12.3 (venv), Ubuntu 24.04, Intel i5-7500 (4 threads). CPU only: the GPU section is not reproduced here, because current faiss-gpu wheels abort on this machine's Pascal GTX 1070 (see the FAISS index error post, Error 4). Last run 2026-09-27.

IndexFlatL2 errors are mostly about shapes and dtypes. Once you move to IVF, product quantization or a GPU index, a different class of problem shows up, and the first one on this list is not even an error. I reran every CPU case below on the setup above; outputs and numbers are copied from those runs.

Reproduced

What I ran	Exact result	Fix that worked
IndexIVFFlat, nlist=100, nprobe = 500	No error. index.nprobe reads back 500; results identical to nprobe=100; 1,374 ms per 1,000 queries vs 89.5 ms for IndexFlatL2	Sweep nprobe against flat ground truth
nlist=1024, 50,000 vectors, nprobe=1, k=100	989 of 1,000 queries contained -1 (distance 3.4028235e+38)	nprobe=8: zero -1
IndexIVFPQ(quantizer, 100, 100, 8, 8)	RuntimeError: ... Error: '!(d % M == 0)' failed: The dimension of the vector (d) should be a multiple of the number of subquantizers (M)	M=10 for d=100
IndexIVFPQ with nbits=9, 10, 12	Trains fine (code_size 18, 20, 24 bytes at M=16)	No fix needed
nbits=16 with 50,000 training vectors	'nx >= static_cast<idx_t>(k)' failed: Number of training points (50000) should be at least as large as number of clusters (65536)	More training data or fewer bits
nbits=25	Error: 'nbits > 24' failed: nbits larger than 24 is not practical.	nbits of 24 or less
IVFPQ, nlist=100, trained on 200 vectors	Number of training points (200) should be at least as large as number of clusters (256)	1,000 vectors trained (with warnings)
IVF256,PQ16 on 100,000 x 128	26.6 bytes/vector vs 512.0 for flat; recall@10 stuck at 0.282 from nprobe=4 up	IndexRefineFlat, k_factor=50: recall@10 0.999
StandardGpuResources alloc fail	Not reproduced (Pascal GPU unsupported by current faiss-gpu wheels)	Documented behaviour, cited below

nprobe set higher than nlist is silently a full scan

The instinct when recall looks bad is to raise nprobe: more inverted lists probed per query, more candidates considered, better recall. That holds until nprobe reaches nlist. After that it changes nothing:

import faiss
import numpy as np

d = 128
nlist = 100
rng = np.random.default_rng(0)
xb = rng.random((50000, d), dtype='float32')
xq = rng.random((1000, d), dtype='float32')

quantizer = faiss.IndexFlatL2(d)
index = faiss.IndexIVFFlat(quantizer, d, nlist)
index.train(xb)
index.add(xb)

index.nprobe = 500  # 5x nlist: no error, no warning
print(index.nprobe)  # 500
distances, indices = index.search(xq, k=10)

The value is stored as given, but the search can only visit the 100 lists that exist, so results were identical to nprobe = 100 (np.array_equal returned True). Here is the full sweep on that data, recall@10 measured against IndexFlatL2, timings for one batch of 1,000 queries:

nprobe	recall@10	time (ms)
1	0.058	14.5
8	0.271	108.1
32	0.647	422.7
64	0.895	847.4
100 (= nlist)	1.000	1,328.7
500	1.000	1,374.3
IndexFlatL2	1.000	89.5

Past nlist you pay the same full-scan price for the same results. And for batched queries, a full IVF scan is far slower than IndexFlatL2, which runs batches through BLAS. For single queries the gap reverses: on 100,000 clustered vectors, one query at a time took 8.57 ms on IndexFlatL2 and 3.95 ms on IndexIVFFlat with nprobe = nlist = 256, so measure your own access pattern.

Uniform random vectors are also the worst case for IVF, since they have no cluster structure. On clustered data (200 Gaussian blobs, nlist=256) recall@10 was already 0.893 at nprobe=1 and 1.000 at nprobe=4. The workable nprobe depends on your data, which is why the fix is a measured sweep, not a rule of thumb:

flat = faiss.IndexFlatL2(d)
flat.add(xb)
_, gt = flat.search(xq, 10)

def recall_at_k(I, gt, k=10):
    return np.mean([len(set(I[i]) & set(gt[i])) / k for i in range(len(gt))])

for nprobe in [1, 2, 4, 8, 16, 32, 64]:
    index.nprobe = nprobe
    _, I = index.search(xq, 10)
    print(nprobe, round(recall_at_k(I, gt), 3))

If you only hit your recall target with nprobe close to nlist, the index is not buying you anything; use a smaller nlist or a flat index.

-1 indices when nprobe is too small for k

The opposite mistake: many small lists, nprobe=1, and a large k. With nlist=1024 on 50,000 vectors the lists held between 1 and 114 vectors (mean 48.8). Searching with k=100 and nprobe=1, 989 of 1,000 queries came back with -1 padding, 38,975 of the 100,000 result slots in total, each with distance 3.4028235e+38 (float32 max). No error is raised. At nprobe=8 the count dropped to 0. Filter -1 before using the indices, because docs[-1] on a Python list quietly returns the last document.

IndexIVFPQ: '!(d % M == 0)' failed

Product quantization compresses each vector by splitting it into M equal-length sub-vectors and quantizing each one independently. If d does not divide by M, current FAISS refuses at construction time, before you get to train():

d = 100
M = 8  # 100 is not divisible by 8

quantizer = faiss.IndexFlatL2(d)
index = faiss.IndexIVFPQ(quantizer, d, 100, M, 8)  # raises here

RuntimeError: Error in void faiss::ProductQuantizer::set_derived_values() at /project/faiss/impl/ProductQuantizer.cpp:64: Error: '!(d % M == 0)' failed: The dimension of the vector (d) should be a multiple of the number of subquantizers (M)

faiss.IndexPQ(100, 8, 8) and faiss.index_factory(100, "IVF100,PQ8") raise the same message. It is a RuntimeError, so an except AssertionError will not catch it. Most people hit this after switching embedding models, when an M tuned for one dimension no longer divides the new one.

def valid_subquantizer_counts(d, max_m=64):
    return [m for m in range(1, max_m + 1) if d % m == 0]

print(valid_subquantizer_counts(100))
# [1, 2, 4, 5, 10, 20, 25, 50]
print(valid_subquantizer_counts(768))
# [1, 2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64]

M = 10  # a real divisor of 100
index = faiss.IndexIVFPQ(faiss.IndexFlatL2(100), 100, 100, M, 8)

With M=10, the index trained and added 20,000 vectors with a code size of 10 bytes.

nbits above 8 and the training-size errors

The last constructor argument, nbits, is often described as capped at 8. It is not in faiss 1.15.1: IndexIVFPQ with nbits 9, 10 and 12 trained without error. The real limits are these two:

# nbits=25
Error: 'nbits > 24' failed: nbits larger than 24 is not practical.

# nbits=16, 50,000 training vectors
Error: 'nx >= static_cast<idx_t>(k)' failed: Number of training points (50000) should be at least as large as number of clusters (65536)

Each sub-quantizer runs k-means with 2^nbits centroids, so it needs at least that many training vectors. The same error hits the default nbits=8 with a small sample: training an IVFPQ with nlist=100 on 200 vectors passed the coarse quantizer but failed on the PQ step with Number of training points (200) should be at least as large as number of clusters (256). With 50 vectors it failed earlier, at the coarse quantizer ((50) ... (100)).

Clearing the error is not the same as training well. With 1,000 vectors, training succeeded but printed WARNING clustering 1000 points to 100 centroids: please provide at least 3900 training points and WARNING clustering 1000 points to 256 centroids: please provide at least 9984 training points. FAISS asks for 39 points per centroid; give it that and the warnings go away. The FAISS index error post covers the coarse-quantizer side in more detail.

IVFPQ memory per vector and the recall ceiling

PQ is where the memory savings come from, and also where recall is lost. I built each index with index_factory on 100,000 clustered 128-dim vectors and measured the size with len(faiss.serialize_index(index)):

Index	bytes / vector	recall@10, nprobe=4	recall@10, nprobe=256
IndexFlatL2	512.0	1.000	1.000
IVF256,Flat	521.3	1.000	1.000
IVF256,PQ64	74.6	0.803	0.803
IVF256,PQ32	42.6	0.527	0.527
IVF256,PQ16	26.6	0.282	0.282
OPQ16,IVF256,PQ16	27.3	0.279	0.279

The size is roughly M code bytes plus 8 bytes for the stored ID, plus a small share of the centroids and codebooks. The recall column is the useful part: for every PQ index, recall stopped moving after nprobe=4. When raising nprobe no longer helps, the lists are not the problem; the compressed codes cannot tell close neighbours apart. More nprobe only costs latency (IVF256,PQ16 went from 14.5 ms to 381.6 ms per 1,000 queries for the same 0.282).

OPQ did not help here (0.279 vs 0.282). The within-cluster noise in this test data is isotropic, so there is no correlation for the rotation to remove; on real embeddings OPQ often does better, but check it on your data rather than assuming. What did work was reranking with the original vectors:

base = faiss.index_factory(d, "IVF256,PQ16")
base.train(xb)
index = faiss.IndexRefineFlat(base)  # wrap BEFORE adding
index.add(xb)

faiss.extract_index_ivf(base).nprobe = 4
params = faiss.IndexRefineSearchParameters(k_factor=50)
D, I = index.search(xq, 10, params=params)

recall@10 went from 0.282 (k_factor=1) to 0.790 (k_factor=10) and 0.999 (k_factor=50). IndexRefineFlat keeps the full float32 vectors, so the serialized index grew to 538.6 bytes per vector. Keep those vectors on disk or in a separate store if memory was the reason you chose PQ. Wrapping an index that already holds vectors fails with Error: 'base_index->ntotal == refine_index->ntotal' failed, which is why the wrap comes before add().

GPU indexes: "StandardGpuResources: alloc fail"

Not reproduced here: current faiss-gpu wheels do not run on this Pascal GTX 1070 (the first GPU search aborts with CUDA error 209 no kernel image is available for execution on the device). What follows comes from the FAISS source and wiki, linked below, not from a run.

res = faiss.StandardGpuResources()
gpu_index = faiss.index_cpu_to_gpu(res, 0, cpu_index)

distances, indices = gpu_index.search(large_query_batch, k=100)

A common explanation is that this error means FAISS's scratch pool is too small while the GPU still has free memory, and that raising setTempMemory fixes it. The source says otherwise. StandardGpuResources reserves a temporary memory stack up front: 512 MiB on GPUs with 4 GiB or less, 1 GiB on GPUs up to 8 GiB, and at most 1.5 GiB (StandardGpuResources.cpp). When a temporary request does not fit, FAISS retries it as an ordinary device allocation with cudaMalloc. The exception is thrown only if that cudaMalloc fails, and the message is built as:

StandardGpuResources: alloc fail <request details> (cudaMalloc error <CUDA error string> [<code>])

So it is a real out-of-memory condition on the device. The Faiss on the GPU wiki page describes the same fallback ("falls back to the heap (cudaMalloc) when the stack size is exhausted") and says large adds and searches should be done in batches, typically powers of two around 8192, because FAISS does not batch them for you.

res = faiss.StandardGpuResources()
# Optional: shrink the reserved stack if the index itself barely fits.
# Oversized requests then go to cudaMalloc (slower, per the wiki).
res.setTempMemory(256 * 1024 * 1024)

gpu_index = faiss.index_cpu_to_gpu(res, 0, cpu_index)

batch = 8192
results = [gpu_index.search(large_query_batch[i:i + batch], 100)
           for i in range(0, len(large_query_batch), batch)]
D = np.vstack([r[0] for r in results])
I = np.vstack([r[1] for r in results])

Raising setTempMemory does not fix a failing cudaMalloc; the reserved stack is taken from the same GPU memory, so a bigger stack leaves less for the index. Smaller batches, a smaller k, freeing memory held by other processes (a PyTorch model on the same card, for example), or a more compact index such as IVFPQ are the levers that reduce the peak.

Related articles

Error Fix

Fix FAISS IndexError and Dimension Mismatch

Error Fix

Fix CUDA Out of Memory in PyTorch

Error Fix

Fix HuggingFace Tokenizer Padding Errors