End-to-End Data Science Pipeline with Python Scikit-Learn and Pandas

By · · Technology

One of the most common mistakes in data science is jumping straight to model training without properly understanding or preparing the data. A solid pipeline forces discipline: load the data, explore it, engineer features, train models, and evaluate them systematically. We will build a full pipeline using Pandas and Scikit-Learn to predict customer churn, covering each stage with code you can adapt for your own projects.

The Dataset

We will use a synthetic telecom customer churn dataset. Each row represents a customer with features like tenure, monthly charges, contract type, and whether they churned. Let us start by loading and inspecting the data.

import pandas as pd
import numpy as np

df = pd.read_csv("telecom_churn.csv")

print(f"Shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
print(df.head())

Shape: (7043, 21)
Columns: ['customerID', 'gender', 'SeniorCitizen', 'Partner', 'Dependents',
           'tenure', 'PhoneService', 'MultipleLines', 'InternetService',
           'OnlineSecurity', 'TechSupport', 'Contract', 'PaperlessBilling',
           'PaymentMethod', 'MonthlyCharges', 'TotalCharges', 'Churn']

Exploratory Data Analysis

Before building anything, we need to understand what we are working with. Start with the basics: data types, missing values, and class distribution.

print(df.dtypes)
print(f"\nMissing values:\n{df.isnull().sum()[df.isnull().sum() > 0]}")
print(f"\nChurn distribution:\n{df['Churn'].value_counts(normalize=True)}")

A common issue with this dataset is that TotalCharges is stored as a string due to blank values. Let us fix that:

df["TotalCharges"] = pd.to_numeric(df["TotalCharges"], errors="coerce")
print(f"TotalCharges nulls after conversion: {df['TotalCharges'].isnull().sum()}")

For numerical features, summary statistics reveal the distribution:

print(df[["tenure", "MonthlyCharges", "TotalCharges"]].describe())

Visualizing the churn rate across different segments helps identify patterns. Customers on month-to-month contracts churn at a much higher rate than those on one or two-year contracts. High monthly charges also correlate with higher churn.

Data Cleaning

With the exploration complete, we can address the issues we found:

# Drop the customer ID column since it has no predictive value
df = df.drop(columns=["customerID"])

# Fill missing TotalCharges with the median
df["TotalCharges"] = df["TotalCharges"].fillna(df["TotalCharges"].median())

# Convert the target variable to binary
df["Churn"] = df["Churn"].map({"Yes": 1, "No": 0})

print(f"Clean shape: {df.shape}")
print(f"Remaining nulls: {df.isnull().sum().sum()}")

Feature Engineering

Raw features rarely tell the whole story. Let us create a few derived features that capture business logic:

# Average monthly spend across tenure
df["AvgMonthlySpend"] = df["TotalCharges"] / (df["tenure"] + 1)

# Whether the customer has any security or support add-ons
df["HasSupport"] = ((df["OnlineSecurity"] == "Yes") |
                    (df["TechSupport"] == "Yes")).astype(int)

# Tenure buckets
df["TenureBucket"] = pd.cut(
    df["tenure"],
    bins=[0, 12, 24, 48, 72],
    labels=["0-12m", "12-24m", "24-48m", "48-72m"],
)

Preparing Features for Modeling

Scikit-Learn works with numerical arrays, so we need to encode categorical variables and scale numerical ones. The ColumnTransformer lets us apply different transformations to different columns in a single step.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

# Separate features and target
X = df.drop(columns=["Churn"])
y = df["Churn"]

# Identify column types
numerical_cols = X.select_dtypes(include=["int64", "float64"]).columns.tolist()
categorical_cols = X.select_dtypes(include=["object", "category"]).columns.tolist()

# Build the preprocessor
preprocessor = ColumnTransformer(
    transformers=[
        ("num", StandardScaler(), numerical_cols),
        ("cat", OneHotEncoder(drop="first", sparse_output=False), categorical_cols),
    ]
)

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"Train: {X_train.shape}, Test: {X_test.shape}")
print(f"Train churn rate: {y_train.mean():.3f}")
print(f"Test churn rate: {y_test.mean():.3f}")

The stratify parameter ensures the churn ratio is preserved in both splits, which is important for imbalanced datasets.

Model Training with a Pipeline

Scikit-Learn pipelines chain the preprocessor and model together so you never accidentally fit the scaler on test data or forget a transformation step:

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

# Define models to compare
models = {
    "Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=200, random_state=42),
    "Gradient Boosting": GradientBoostingClassifier(n_estimators=200, random_state=42),
}

results = {}

for name, model in models.items():
    pipeline = Pipeline([
        ("preprocessor", preprocessor),
        ("classifier", model),
    ])
    pipeline.fit(X_train, y_train)
    train_score = pipeline.score(X_train, y_train)
    test_score = pipeline.score(X_test, y_test)
    results[name] = {"train": train_score, "test": test_score, "pipeline": pipeline}
    print(f"{name}: Train={train_score:.4f}, Test={test_score:.4f}")

Logistic Regression: Train=0.8050, Test=0.8012
Random Forest: Train=0.9982, Test=0.7893
Gradient Boosting: Train=0.8410, Test=0.8076

Model Evaluation

Accuracy alone is misleading for imbalanced datasets. We need to look at precision, recall, and the confusion matrix:

from sklearn.metrics import classification_report, confusion_matrix

best_model_name = max(results, key=lambda k: results[k]["test"])
best_pipeline = results[best_model_name]["pipeline"]

y_pred = best_pipeline.predict(X_test)

print(f"\nBest model: {best_model_name}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=["No Churn", "Churn"]))

print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))

Best model: Gradient Boosting

Classification Report:
              precision    recall  f1-score   support
   No Churn       0.84      0.91      0.87      1036
       Churn       0.67      0.52      0.59       373

Confusion Matrix:
[[941  95]
 [178 195]]

The model is much better at identifying customers who will stay than those who will churn. In a real scenario, you might address this by adjusting class weights, using SMOTE for oversampling, or tuning the decision threshold.

Cross-Validation

A single train-test split can be misleading. Cross-validation gives a more robust estimate:

from sklearn.model_selection import cross_val_score

pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", GradientBoostingClassifier(n_estimators=200, random_state=42)),
])

cv_scores = cross_val_score(pipeline, X, y, cv=5, scoring="f1")
print(f"Cross-validated F1 scores: {cv_scores}")
print(f"Mean F1: {cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})")

Saving the Pipeline

Once you are satisfied with the model, save the entire pipeline so it can be loaded and used in production without re-fitting:

import joblib

joblib.dump(best_pipeline, "churn_model_pipeline.joblib")

# Later, load and predict
loaded_pipeline = joblib.load("churn_model_pipeline.joblib")
new_predictions = loaded_pipeline.predict(X_test[:5])
print(f"Predictions: {new_predictions}")

Lessons Learned

Building this pipeline reinforces a few important principles:

This pipeline is a template you can adapt to any tabular classification problem. Swap in your dataset, adjust the feature engineering to match your domain, and iterate on the model selection. The structure stays the same.