Modern Data Science with Polars and Python

By · · Technology

For over a decade, Pandas has been the default choice for data manipulation in Python. It is everywhere: tutorials, production code, data science notebooks. But if you have ever watched a Pandas operation grind to a halt on a multi-gigabyte dataset, you know it has limitations. Polars offers a fundamentally different approach. Built in Rust with a query engine designed from the ground up for performance, Polars is now mature enough to be your primary data manipulation library in 2025.

This is not a "Polars vs Pandas" hot take. This is a practical guide to using Polars effectively for data science workflows, with honest comparisons where they matter.

Why Polars Exists

Pandas was designed in 2008 when datasets fit in memory and single-core performance was sufficient. Its API evolved organically, leading to inconsistencies and performance pitfalls that are hard to fix without breaking backward compatibility.

Polars was built with modern hardware in mind:

Getting Started

pip install polars

Let us start with basic operations and build up to more advanced patterns:

import polars as pl

# Read a CSV file
df = pl.read_csv("sales_data.csv")

# Basic inspection
print(df.shape)
print(df.schema)
print(df.head())

The first thing you will notice is that Polars DataFrames have no index. Every row is addressed by its position, and every operation returns a new DataFrame. This eliminates an entire class of bugs related to index alignment and mutation.

Expressions: The Core of Polars

Polars expressions are its most powerful feature. They are composable, lazy, and optimizable. Think of them as declarative descriptions of what you want, not imperative instructions for how to compute it.

# Pandas style (works but not idiomatic Polars)
# df["revenue"] = df["price"] * df["quantity"]

# Polars expression style
df = df.with_columns(
    (pl.col("price") * pl.col("quantity")).alias("revenue"),
    pl.col("date").str.to_date("%Y-%m-%d").alias("sale_date"),
    pl.col("category").str.to_lowercase().alias("category_normalized"),
)

# Multiple aggregations in a single group_by
summary = df.group_by("category_normalized").agg(
    pl.col("revenue").sum().alias("total_revenue"),
    pl.col("revenue").mean().alias("avg_revenue"),
    pl.col("quantity").sum().alias("total_units"),
    pl.len().alias("transaction_count"),
    pl.col("revenue").quantile(0.95).alias("p95_revenue"),
)

print(summary.sort("total_revenue", descending=True))

Notice how multiple operations are expressed declaratively within with_columns and agg. Polars can optimize and parallelize these because it understands the full computation graph before executing.

Lazy Evaluation: Let the Optimizer Work

Lazy evaluation is where Polars pulls ahead. Instead of executing operations immediately, you build a logical plan that Polars optimizes before running.

# Eager execution: each step runs immediately
result_eager = (
    pl.read_csv("large_dataset.csv")
    .filter(pl.col("status") == "active")
    .select("user_id", "amount", "category")
    .group_by("category")
    .agg(pl.col("amount").sum())
)

# Lazy execution: builds a plan, then optimizes and executes
result_lazy = (
    pl.scan_csv("large_dataset.csv")
    .filter(pl.col("status") == "active")
    .select("user_id", "amount", "category")
    .group_by("category")
    .agg(pl.col("amount").sum())
    .collect()
)

The lazy version using scan_csv instead of read_csv only reads the columns it needs from the CSV file. The optimizer applies predicate pushdown (filtering early), projection pushdown (reading only required columns), and common subexpression elimination automatically.

You can inspect the optimized plan:

lazy_plan = (
    pl.scan_csv("large_dataset.csv")
    .filter(pl.col("status") == "active")
    .select("user_id", "amount", "category")
    .group_by("category")
    .agg(pl.col("amount").sum())
)

print(lazy_plan.explain(optimized=True))

Window Functions

Polars has excellent support for window functions, which are essential for time-series analysis and feature engineering:

df = df.with_columns(
    # Running total within each category
    pl.col("revenue")
    .cum_sum()
    .over("category")
    .alias("cumulative_revenue"),

    # Rank within each category
    pl.col("revenue")
    .rank(method="dense", descending=True)
    .over("category")
    .alias("revenue_rank"),

    # Rolling 7-day average
    pl.col("revenue")
    .rolling_mean(window_size=7)
    .over("category")
    .alias("rolling_7d_avg"),

    # Percentage of category total
    (pl.col("revenue") / pl.col("revenue").sum().over("category") * 100)
    .alias("pct_of_category"),
)

The over method is the Polars equivalent of SQL window functions. It partitions the data by the specified columns and applies the expression within each partition. This runs in parallel across partitions.

Practical Comparison with Pandas

Here is a realistic data transformation pipeline in both libraries. The task: read a large CSV, filter rows, create derived columns, aggregate, and join with a reference table.

# ---- Pandas ----
import pandas as pd

pdf = pd.read_csv("transactions.csv", parse_dates=["timestamp"])
ref = pd.read_csv("categories.csv")

result_pd = (
    pdf[pdf["amount"] > 0]
    .assign(
        month=lambda x: x["timestamp"].dt.to_period("M"),
        amount_bucket=lambda x: pd.cut(
            x["amount"], bins=[0, 100, 500, 1000, float("inf")]
        ),
    )
    .groupby(["category_id", "month"])
    .agg(total=("amount", "sum"), count=("amount", "count"))
    .reset_index()
    .merge(ref, on="category_id", how="left")
)


# ---- Polars ----
result_pl = (
    pl.scan_csv("transactions.csv")
    .filter(pl.col("amount") > 0)
    .with_columns(
        pl.col("timestamp").str.to_datetime().dt.month().alias("month"),
        pl.col("amount")
        .cut([100, 500, 1000])
        .alias("amount_bucket"),
    )
    .group_by("category_id", "month")
    .agg(
        pl.col("amount").sum().alias("total"),
        pl.col("amount").count().alias("count"),
    )
    .join(
        pl.scan_csv("categories.csv"),
        on="category_id",
        how="left",
    )
    .collect()
)

On a dataset with 10 million rows, the Polars version typically runs 5 to 10 times faster than Pandas, while using significantly less memory. The lazy evaluation means Polars only reads the columns it needs and applies filters before any joins.

Working with Large Datasets

Polars handles datasets larger than memory through its streaming engine:

# Process a dataset that doesn't fit in memory
result = (
    pl.scan_csv("huge_dataset.csv")
    .filter(pl.col("year") >= 2024)
    .group_by("region", "product")
    .agg(
        pl.col("sales").sum().alias("total_sales"),
        pl.col("sales").mean().alias("avg_sale"),
    )
    .sort("total_sales", descending=True)
    .collect(streaming=True)
)

The streaming=True flag tells Polars to process the data in batches rather than loading it all at once. This works well with the lazy API.

Integration with the Python Ecosystem

Polars plays well with the broader ecosystem. It supports conversion to and from Pandas, NumPy, and Arrow:

# Polars to Pandas (when you need a Pandas-only library)
pandas_df = df.to_pandas()

# Pandas to Polars
polars_df = pl.from_pandas(pandas_df)

# Direct Arrow interop (zero-copy)
arrow_table = df.to_arrow()

# Works with scikit-learn via .to_numpy()
from sklearn.linear_model import LinearRegression

X = df.select("feature_1", "feature_2", "feature_3").to_numpy()
y = df.select("target").to_numpy().ravel()

model = LinearRegression().fit(X, y)

When to Still Use Pandas

Polars is not a universal replacement. Use Pandas when:

For everything else, especially any workflow involving datasets with more than a few hundred thousand rows, Polars is the better choice.

Getting Started Recommendations

If you are coming from Pandas, start by replacing your pd.read_csv calls with pl.scan_csv and chain operations using the expression API. Resist the urge to index into DataFrames with square brackets. Instead, use filter, select, and with_columns. Once you internalize the expression-based approach, you will find that Polars code is more readable and significantly faster than its Pandas equivalent.

The Polars documentation is excellent, and the community has grown rapidly. It is a mature library ready for production workloads.