ValueError: Input Contains NaN — Fix in sklearn Pipelines

ValueError: Input Contains NaN — Fix in sklearn Pipelines

You call model.fit(X, y) and get:

ValueError: Input X contains NaN.
LogisticRegression does not accept missing values encoded as NaN natively.
For supervised learning, you might want to consider
sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept
missing values encoded as NaNs natively.
Alternatively, it is possible to preprocess the data,
for instance by using an imputer transformer in a pipeline or drop samples
with missing values. See https://scikit-learn.org/stable/modules/impute.html

The fix is never to just drop NaN rows and hope for the best. Here is how to find where the NaNs are, which imputation strategy to use, and how to do it correctly without leaking test data into training.

Step 1: Find Which Columns Have NaN

Before fixing, locate the problem. A model error tells you NaN exists but not where.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'age': [25, np.nan, 35, 40, np.nan],
    'income': [50000, 60000, np.nan, 80000, 70000],
    'category': ['A', 'B', 'A', np.nan, 'B'],
    'target': [0, 1, 0, 1, 0]
})

# Count NaNs per column
print(df.isnull().sum())
# age         2
# income      1
# category    1
# target      0

# Percentage missing
print(df.isnull().mean().round(3) * 100)
# age         40.0
# income      20.0
# category    20.0
# target       0.0

# Which rows have any NaN
print(df[df.isnull().any(axis=1)])

If you already have a NumPy array instead of a DataFrame:

import numpy as np

X = np.array([[1, 2], [np.nan, 3], [4, np.nan]])

# Which columns have NaN
print(np.isnan(X).any(axis=0))  # [True, True]

# Count per column
print(np.isnan(X).sum(axis=0))  # [1, 1]

# Which rows
print(np.where(np.isnan(X).any(axis=1))[0])  # [1, 2]

Step 2: Choose an Imputation Strategy

sklearn's SimpleImputer covers three strategies for most cases:

  • mean — replace NaN with the column mean. Works for numeric features that are roughly normally distributed. Sensitive to outliers.
  • median — replace NaN with the column median. Better than mean when the feature has outliers or a skewed distribution.
  • most_frequent — replace NaN with the most common value. Use for categorical features encoded as strings or integers.
  • constant — replace NaN with a specific value you provide via fill_value. Use when domain knowledge dictates the fill (e.g., 0 for "no purchases", "Unknown" for missing category).
from sklearn.impute import SimpleImputer
import numpy as np

X = np.array([[1, 2], [np.nan, 3], [4, np.nan], [6, 8]])

# Mean imputation
imputer_mean = SimpleImputer(strategy='mean')
X_filled = imputer_mean.fit_transform(X)
print(X_filled)
# [[1.  2. ]
#  [3.67 3. ]   ← NaN replaced with column mean (1+4+6)/3 = 3.67
#  [4.  4.33]   ← NaN replaced with (2+3+8)/3 = 4.33
#  [6.  8. ]]

# Median imputation
imputer_median = SimpleImputer(strategy='median')

# Constant imputation
imputer_const = SimpleImputer(strategy='constant', fill_value=0)

Step 3: Use SimpleImputer Inside a Pipeline — Not Before

This is the most important part. The most common mistake is fitting the imputer on the full dataset before the train/test split. This leaks information from the test set into the imputer's statistics (mean, median) and inflates your model's apparent performance.

The Wrong Way

from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import numpy as np

X = np.random.randn(1000, 10)
X[np.random.rand(*X.shape) < 0.1] = np.nan
y = np.random.randint(0, 2, 1000)

# BAD: imputer sees test data before split
imputer = SimpleImputer(strategy='mean')
X_filled = imputer.fit_transform(X)  # ← uses entire X including future test rows

X_train, X_test, y_train, y_test = train_test_split(X_filled, y, test_size=0.2)

model = LogisticRegression()
model.fit(X_train, y_train)
# Model has been trained on data contaminated by test set statistics

The Correct Way: Pipeline

from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
import numpy as np

X = np.random.randn(1000, 10)
X[np.random.rand(*X.shape) < 0.1] = np.nan
y = np.random.randint(0, 2, 1000)

# Split FIRST, before any fitting
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Pipeline: imputer fitted only on training data
pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
    ('model', LogisticRegression(max_iter=500))
])

pipeline.fit(X_train, y_train)

# When predicting, the pipeline applies the same imputation
# (fitted on train stats only) to the test set
score = pipeline.score(X_test, y_test)
print(f"Test accuracy: {score:.3f}")

Inside a Pipeline, fit() on training data calls fit_transform() on each step. predict() or score() on test data calls only transform() using the statistics learned from training. The test set never influences the imputer's mean or median.

make_pipeline Shorthand

For quick pipelines where you do not need to name the steps, make_pipeline auto-names them and is less verbose:

from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Equivalent to the Pipeline above, steps named automatically
pipeline = make_pipeline(
    SimpleImputer(strategy='median'),
    StandardScaler(),
    LogisticRegression(max_iter=500)
)

pipeline.fit(X_train, y_train)
print(pipeline.score(X_test, y_test))

# Step names are lowercased class names
print(pipeline.named_steps.keys())
# dict_keys(['simpleimputer', 'standardscaler', 'logisticregression'])

The named steps are useful for accessing fitted imputer statistics after training:

# Check what values the imputer learned from training data
imputer_step = pipeline.named_steps['simpleimputer']
print(imputer_step.statistics_)  # array of per-column medians learned from X_train

Handling Mixed Types: Numeric and Categorical Together

When your DataFrame has both numeric and categorical columns with NaN, use ColumnTransformer to apply different imputation strategies to different columns:

import pandas as pd
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

df = pd.DataFrame({
    'age': [25, np.nan, 35, 40, np.nan, 30],
    'income': [50000, 60000, np.nan, 80000, 70000, 55000],
    'category': ['A', 'B', 'A', np.nan, 'B', 'A'],
    'target': [0, 1, 0, 1, 0, 1]
})

X = df.drop('target', axis=1)
y = df['target']

numeric_cols = ['age', 'income']
categorical_cols = ['category']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

preprocessor = ColumnTransformer(transformers=[
    ('num', Pipeline([
        ('imputer', SimpleImputer(strategy='median')),
        ('scaler', StandardScaler())
    ]), numeric_cols),
    ('cat', Pipeline([
        ('imputer', SimpleImputer(strategy='most_frequent')),
        ('encoder', OneHotEncoder(handle_unknown='ignore'))
    ]), categorical_cols)
])

full_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('model', LogisticRegression(max_iter=200))
])

full_pipeline.fit(X_train, y_train)
print(full_pipeline.score(X_test, y_test))

When to Drop vs Impute

Dropping rows (df.dropna()) is only acceptable when:

  • The fraction of missing rows is very small (under ~1–2%) and you have enough data.
  • You have reason to believe the values are not missing at random (MNAR) and imputing them would introduce bias.

In all other cases, impute. A model trained on fewer samples due to dropped rows will generalize worse than one trained on imputed data, even with imperfect imputation. For a complete overview of how preprocessing decisions interact with model selection, see the top 10 ML algorithms guide.

For the related issue of ConvergenceWarning that appears after imputation when your features have very different scales, see the ConvergenceWarning fix guide — always pair imputation with scaling in your Pipeline.