Fix joblib's "Can't pickle <function <lambda>>" PicklingError

A scikit-learn pipeline with a small inline transform can run for weeks under cross_val_score or GridSearchCV without incident, then fail the moment someone sets backend='multiprocessing' — often copied from an old answer, or picked deliberately to sidestep a different problem. Reproduced below with joblib 1.6.0 and scikit-learn 1.9.0 (Python 3.11), including why the exact same code works under joblib's own default backend.

The Error

The plain version, stripped down to just joblib, with an explicit backend='multiprocessing':

from joblib import Parallel, delayed

def make_worker():
    def worker(x):
        return x * 2
    return worker

w = make_worker()
results = Parallel(n_jobs=2, backend="multiprocessing")(delayed(w)(i) for i in range(4))

# Traceback (most recent call last):
#   ...
# AttributeError: Can't pickle local object 'make_worker.<locals>.worker'

With a bare lambda instead of a nested closure, the same backend raises a slightly different, equally direct message:

import joblib
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

# ... pipeline built with FunctionTransformer(lambda x: ...), see below ...

# joblib.externals.loky.process_executor._RemoteTraceback:
# PicklingError: Can't pickle <function <lambda> at 0x7f4203d984a0>:
# attribute lookup <lambda> on __main__ failed

Both boil down to the same constraint: standard pickle can only serialize a function by recording where to re-import it from — module name plus a plain top-level attribute name. A lambda, a nested closure, or any function defined inside another function has no such dotted path, so pickle has nothing to write down.

Why the Default Backend Doesn't Hit This

Leave backend unset and the identical closure-based example above runs with no error at all:

from joblib import Parallel, delayed

def make_worker():
    def worker(x):
        return x * 2
    return worker

w = make_worker()
results = Parallel(n_jobs=2)(delayed(w)(i) for i in range(4))
print(results)   # [0, 2, 4, 6] — no error, default backend is loky

The Same Failure Inside a Real scikit-learn Pipeline

scikit-learn's own n_jobs parameter is a thin wrapper over joblib, so the same backend switch breaks a Pipeline the same way — and this is the shape it actually takes in practice, since scikit-learn code rarely calls Parallel directly:

import joblib
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import numpy as np

X = np.random.rand(50, 4)
y = np.random.randint(0, 2, 50)

pipe = Pipeline([
    ('log', FunctionTransformer(lambda x: np.log1p(np.abs(x)))),
    ('clf', LogisticRegression()),
])

with joblib.parallel_backend('multiprocessing'):
    scores = cross_val_score(pipe, X, y, cv=3, n_jobs=2)

# PicklingError: Can't pickle <function <lambda> at 0x...>:
# attribute lookup <lambda> on __main__ failed

Remove the with joblib.parallel_backend('multiprocessing'): block and run the exact same pipeline and it completes normally — confirmed on the same environment, same lambda, same n_jobs=2. Nothing about the pipeline itself is wrong; only the backend context around it decides whether the lambda is a problem.

The Fix

Give the function a real dotted import path by defining it at module level instead of inline:

import joblib
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import numpy as np

def log1p_abs(x):
    return np.log1p(np.abs(x))

if __name__ == "__main__":
    X = np.random.rand(50, 4)
    y = np.random.randint(0, 2, 50)

    pipe = Pipeline([
        ('log', FunctionTransformer(log1p_abs)),
        ('clf', LogisticRegression()),
    ])

    with joblib.parallel_backend('multiprocessing'):
        scores = cross_val_score(pipe, X, y, cv=3, n_jobs=2)
    print(scores)   # runs cleanly under multiprocessing now

The other equally valid fix, when nothing actually depends on the multiprocessing backend specifically, is to just not set it — joblib's own loky default already handles lambdas and closures via cloudpickle, per the previous section.

If the Function Needs an Argument

A module-level function still needs to be pickled with whatever extra arguments it was bound to. functools.partial over a module-level function pickles correctly under multiprocessing, same as a plain function:

from functools import partial
from joblib import Parallel, delayed

def scaled_worker(factor, x):
    return x * factor

w = partial(scaled_worker, 3)
results = Parallel(n_jobs=2, backend="multiprocessing")(delayed(w)(i) for i in range(4))
print(results)   # [0, 3, 6, 9] — no error