Polars

https://docs.pola.rs/stable/
Admin
Polars is a fast DataFrame library for data manipulation and analysis, written in Rust with Python...
Tokens:12,216
Snippets:48
Trust Score:10
Update:1 year ago
Show doc for...
Tokens:
Raw
### Polars Column Selectors

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Explains how to use Polars selectors (`polars.selectors`) for dynamic column selection based on patterns (starts with, ends with), data types (numeric, string, temporal), and set operations (AND, OR, NOT).

```python
import polars as pl
import polars.selectors as cs

df = pl.DataFrame({
    "customer_id": [1, 2, 3],
    "sales_2023": [1000, 1500, 2000],
    "sales_2024": [1200, 1800, 2200],
    "revenue_2023": [5000, 7500, 10000],
    "revenue_2024": [6000, 9000, 11000],
    "signup_date": ["2022-01-01", "2022-06-15", "2023-02-20"],
    "status": ["active", "active", "inactive"]
})

# Select all numeric columns
numeric_only = df.select(cs.numeric())
# Returns: customer_id, sales_2023, sales_2024, revenue_2023, revenue_2024

# Select columns by name pattern
sales_cols = df.select(cs.starts_with("sales_"))
# Returns: sales_2023, sales_2024

revenue_cols = df.select(cs.ends_with("_2024"))
# Returns: sales_2024, revenue_2024

# Select by data type
string_cols = df.select(cs.string())
# Returns: signup_date, status

# Combine selectors with set operations
# All numeric columns except customer_id
metrics = df.select(cs.numeric() & ~cs.by_name("customer_id"))

# All columns starting with "sales" or "revenue"
all_metrics = df.select(cs.starts_with("sales_") | cs.starts_with("revenue_"))

# Apply transformations to selected columns
normalized = df.with_columns(
    (cs.starts_with("sales_") / 1000).name.suffix("_k")
)

# Select all temporal columns (if dates were properly typed)
df_typed = df.with_columns(
    pl.col("signup_date").str.strptime(pl.Date, "%Y-%m-%d")
)
temporal = df_typed.select(cs.temporal())
# Returns: signup_date (as Date type)

```

--------------------------------

### Test Series Equality with assert_series_equal

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates using `assert_series_equal` to verify the equality of two Polars Series. This is useful for testing operations that return a Series, ensuring both the data and the name match. The example tests a filtering operation on a Series.

```python
import polars as pl
from polars.testing import assert_series_equal

# Test Series equality
def test_series_operations():
    s1 = pl.Series("values", [1, 2, 3, 4, 5])

    result = s1.filter(s1 > 2)
    expected = pl.Series("values", [3, 4, 5])

    assert_series_equal(result, expected)

test_series_operations()

```

--------------------------------

### Register and Use Custom Series Stats Namespace

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Shows how to register a custom namespace named 'stats' for Polars Series. This namespace includes methods for calculating z-scores and detecting outliers. The example demonstrates creating a Series and then applying these custom methods.

```python
import polars as pl

@pl.api.register_series_namespace("stats")
class StatsNamespace:
    def __init__(self, s: pl.Series):
        self._s = s

    def zscore(self) -> pl.Series:
        """Calculate z-score normalization"""
        mean = self._s.mean()
        std = self._s.std()
        return (self._s - mean) / std

    def outliers(self, threshold: float = 3.0) -> pl.Series:
        """Detect outliers using z-score"""
        z = self.zscore()
        return z.abs() > threshold

# Use custom Series namespace
values = pl.Series("values", [10, 12, 11, 13, 100, 9, 12])
z_scores = values.stats.zscore()
is_outlier = values.stats.outliers(threshold=2.5)

print(f"Z-scores: {z_scores}")
print(f"Is outlier: {is_outlier}")
```

--------------------------------

### Test Aggregation with Tolerance using assert_frame_equal

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Shows how to use `assert_frame_equal` with an absolute tolerance (`atol`) for comparing DataFrames containing floating-point numbers. This is crucial for testing aggregations where minor precision differences might occur. The example tests the mean calculation of a series of floats.

```python
import polars as pl
from polars.testing import assert_frame_equal

# Test with tolerance for floating point comparison
def test_aggregation():
    df = pl.DataFrame({"value": [1.0, 2.0, 3.0, 4.0, 5.0]})

    result = df.select(pl.col("value").mean())
    expected = pl.DataFrame({"value": [3.0]})

    # Check equality with absolute tolerance
    assert_frame_equal(result, expected, atol=1e-6)

test_aggregation()

```

--------------------------------

### Register and Use Custom Expression Text Namespace

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Illustrates registering a custom 'text' namespace for Polars expressions. This namespace provides methods to clean phone numbers by keeping only digits and to mask email addresses. The example shows applying these expression methods to DataFrame columns.

```python
import polars as pl

@pl.api.register_expr_namespace("text")
class TextNamespace:
    def __init__(self, expr: pl.Expr):
        self._expr = expr

    def clean_phone(self) -> pl.Expr:
        """Clean phone number to digits only"""
        return (
            self._expr
            .str.replace_all(r"[^0-9]", "")
        )

    def mask_email(self) -> pl.Expr:
        """Mask email addresses"""
        return (
            self._expr
            .str.split("@")
            .list.first()
            .str.slice(0, 2) + "***@" +
            self._expr.str.split("@").list.last()
        )

# Use custom expression namespace
df_contact = pl.DataFrame({
    "phone": ["(555) 123-4567", "555.987.6543", "555-111-2222"],
    "email": ["alice@example.com", "bob@test.com", "charlie@demo.com"]
})

cleaned = df_contact.with_columns([
    pl.col("phone").text.clean_phone().alias("phone_cleaned"),
    pl.col("email").text.mask_email().alias("email_masked")
])

print(cleaned)

```

--------------------------------

### Test DataFrame Transformation with assert_frame_equal

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates using `assert_frame_equal` to verify DataFrame transformations in unit tests. This function checks for equality between two DataFrames, including data, schema, and column order. The example tests a transformation that adds a 'grade' column based on a 'score' column.

```python
import polars as pl
from polars.testing import assert_frame_equal

# Test DataFrame transformations
def test_data_transformation():
    # Input data
    input_df = pl.DataFrame({
        "name": ["Alice", "Bob", "Charlie"],
        "score": [85, 92, 78]
    })

    # Expected output
    expected = pl.DataFrame({
        "name": ["Alice", "Bob", "Charlie"],
        "score": [85, 92, 78],
        "grade": ["B", "A", "C"]
    })

    # Transformation to test
    result = input_df.with_columns([
        pl.when(pl.col("score") >= 90)
          .then(pl.lit("A"))
          .when(pl.col("score") >= 80)
          .then(pl.lit("B")
          .otherwise(pl.lit("C"))
          .alias("grade")
    ])

    # Assert equality
    assert_frame_equal(result, expected)
    print("Test passed!")

test_data_transformation()

```

--------------------------------

### Polars Configuration

Source: https://docs.pola.rs/stable/api/python/stable/reference

How to configure Polars settings and options.

```APIDOC
## Polars Configuration

### Description
Instructions on managing Polars configuration options, including loading, saving, and using configuration contexts.

### Features
*   **Config options**: Details on available configuration parameters.
*   **Config load, save, state**: Management of configuration settings.
*   **Use as a context manager**: Applying configuration temporarily.
*   **Use as a decorator**: Applying configuration to specific functions.
*   **Multiple Config instances**: Managing different configuration sets.
```

--------------------------------

### Polars Input/Output (I/O)

Source: https://docs.pola.rs/stable/api/python/stable/reference

Documentation for reading and writing data from various file formats and data sources.

```APIDOC
## Polars Input/Output (I/O)

### Description
Provides interfaces for reading data from and writing data to a wide range of file formats and data storage systems.

### Supported Formats/Sources
*   Avro
*   Clipboard
*   CSV
*   Database (SQL)
*   Delta Lake
*   Excel / ODS
*   Feather / IPC
*   Iceberg
*   JSON
*   Partitioned Datasets
*   Parquet
*   PyArrow Datasets
*   Cloud Credentials management for cloud storage access.
```

--------------------------------

### Polars Extensibility and Plugins

Source: https://docs.pola.rs/stable/api/python/stable/reference

Guidance on extending Polars functionality and using plugins.

```APIDOC
## Polars Extensibility and Plugins

### Description
Details on how to add new functionality to Polars and utilize external plugins.

### Topics
*   **Providing new functionality**: Methods for contributing custom operations.
*   **Available registrations**: Overview of extension points.
*   **Examples**: Practical demonstrations of extending the API.
*   **Plugins**: Information on using and developing Polars plugins.
```

--------------------------------

### Polars Exceptions and Testing

Source: https://docs.pola.rs/stable/api/python/stable/reference

Information on error handling and testing utilities within Polars.

```APIDOC
## Polars Exceptions and Testing

### Description
Covers the exceptions raised by Polars operations and the utilities provided for testing code.

### Components
*   **Exceptions**: List and explanation of Polars-specific exceptions.
*   **Testing**: Utilities for asserting correctness and performing parametric testing.
```

--------------------------------

### Polars Parquet Reading and Writing

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates reading from and writing to Parquet files, including reading multiple files, lazy scanning, selecting specific columns for efficiency (projection pushdown), and applying compression during writes.

```python
import polars as pl

# Basic Parquet read
df = pl.read_parquet("data.parquet")

# Read multiple Parquet files with glob pattern
df_multi = pl.read_parquet("data/*.parquet")

# Lazy scan for large Parquet datasets
lf = pl.scan_parquet("large_dataset/*.parquet")
result = (
    lf
    .filter(pl.col("year") == 2024)
    .select(["customer_id", "revenue", "date"])
    .group_by("customer_id")
    .agg(pl.col("revenue").sum())
    .collect()
)

# Read specific columns (projection pushdown)
df_cols = pl.read_parquet(
    "data.parquet",
    columns=["name", "age", "salary"]
)

# Write Parquet with compression
df.write_parquet(
    "output.parquet",
    compression="snappy"  # Options: snappy, gzip, lz4, zstd, uncompressed
)

```

--------------------------------

### Apply Window Functions in Polars

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates partition-based window functions like rank, sum, mean, and percentage of partition. Also covers rolling window operations (mean, sum, max, std) and cumulative operations (sum, product, max, min). Uses `over` for partitioning and `rolling_` for window calculations.

```python
df = pl.DataFrame({
    "date": ["2024-01-01", "2024-01-02", "2024-01-03", "2024-01-04", "2024-01-05"],
    "category": ["A", "A", "B", "B", "A"],
    "value": [100, 150, 200, 175, 125],
    "quantity": [10, 15, 20, 18, 12]
})

# Partition-based window functions
df_windowed = df.with_columns([
    pl.col("value").rank().over("category").alias("rank_in_category"),
    pl.col("value").sum().over("category").alias("category_total"),
    pl.col("value").mean().over("category").alias("category_avg"),
    (pl.col("value") / pl.col("value").sum().over("category") * 100).alias("pct_of_category")
])

# Rolling window operations
df_rolling = df.with_columns([
    pl.col("value").rolling_mean(window_size=3).alias("3day_avg"),
    pl.col("value").rolling_sum(window_size=3).alias("3day_sum"),
    pl.col("value").rolling_max(window_size=3).alias("3day_max"),
    pl.col("value").rolling_std(window_size=3).alias("3day_std")
])

# Cumulative operations
df_cumulative = df.with_columns([
    pl.col("value").cum_sum().alias("cumulative_value"),
    pl.col("quantity").cum_prod().alias("cumulative_product"),
    pl.col("value").cum_max().alias("running_max"),
    pl.col("value").cum_min().alias("running_min")
])

# Lead and lag operations
df_shifted = df.with_columns([
    pl.col("value").shift(1).alias("prev_value"),
    pl.col("value").shift(-1).alias("next_value"),
    (pl.col("value") - pl.col("value").shift(1)).alias("value_change")
])
```

--------------------------------

### Mixed Workflow: Pandas to Polars and back

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates a common data processing workflow: reading data with Pandas, converting to Polars for efficient transformations (filtering, aggregation), and then converting back to Pandas for compatibility with legacy systems or further analysis.

```python
import polars as pl
import pandas as pd

# Assume data.csv exists
# pandas_raw = pd.read_csv("data.csv")

# Placeholder for actual pandas_raw if data.csv is not available
pandas_raw = pd.DataFrame({
    "category": ["A", "B", "A", "B", "A"],
    "amount": [150, 50, 200, 120, 80]
})

polars_fast = pl.from_pandas(pandas_raw)

processed = (
    polars_fast
    .filter(pl.col("amount") > 100)
    .group_by("category")
    .agg(pl.col("amount").sum())
)

pandas_final = processed.to_pandas()

# Optionally convert to NumPy
# np_final = processed.to_numpy()
```

--------------------------------

### Optimize Queries with LazyFrames and Streaming in Polars

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Illustrates building optimized query plans using Polars' LazyFrame for lazy evaluation. It covers scanning CSV and Parquet files, applying transformations, aggregations, filtering, sorting, and executing queries with optional streaming for large datasets. The `explain()` method for viewing query plans is also demonstrated.

```python
import polars as pl

# Create LazyFrame from CSV (doesn't load data yet)
lf = pl.scan_csv("sales_data.csv")

# Build query plan with chained operations
optimized_query = (
    lf
    .filter(pl.col("date").is_between("2024-01-01", "2024-12-31"))
    .select(["customer_id", "product", "amount", "date"])
    .with_columns([
        pl.col("date").str.strptime(pl.Date, "%Y-%m-%d"),
        pl.col("amount").cast(pl.Float64)
    ])
    .group_by("customer_id")
    .agg([
        pl.col("amount").sum().alias("total_spent"),
        pl.col("product").n_unique().alias("unique_products"),
        pl.col("date").min().alias("first_purchase")
    ])
    .filter(pl.col("total_spent") > 1000)
    .sort("total_spent", descending=True)
)

# View the optimized query plan
print(optimized_query.explain())
# Shows: predicate pushdown, projection pushdown, etc.

# Execute the query
result = optimized_query.collect()

# For large datasets, use streaming
large_result = optimized_query.collect(streaming=True)

# Scan multiple Parquet files with glob pattern
lf_parquet = (
    pl.scan_parquet("data/*.parquet")
    .filter(pl.col("status") == "active")
    .collect()
)

```

--------------------------------

### Polars SQL Interface

Source: https://docs.pola.rs/stable/api/python/stable/reference

Information on using SQL queries directly with Polars DataFrames and LazyFrames.

```APIDOC
## Polars SQL Interface

### Description
Enables users to query Polars DataFrames and LazyFrames using standard SQL syntax.

### Components
*   **Python API**: Functions and methods to execute SQL queries.
*   **SQL Clauses**: Supported SQL clauses (e.g., SELECT, FROM, WHERE).
*   **SQL Functions**: Built-in SQL functions available for use.
*   **Set Operations**: Support for SQL set operations like UNION, INTERSECT.
*   **Table Operations**: Operations related to managing tables within the SQL context.
```

--------------------------------

### Write Parquet with Statistics and Partitioning in Polars

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates writing DataFrames to Parquet format with options for generating statistics for better query performance and partitioning large datasets for efficient storage and retrieval. Uses `write_parquet` method.

```python
import polars as pl

# Write Parquet with statistics for better query performance
df.write_parquet(
    "output.parquet",
    statistics=True,
    use_pyarrow=False
)

# Partitioned Parquet write for large datasets
df.write_parquet(
    "partitioned_data",
    partition_by=["year", "month"]
)
# Creates directory structure: partitioned_data/year=2024/month=01/data.parquet
```

--------------------------------

### Create and Manipulate DataFrames with Eager Evaluation in Polars

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates creating a Polars DataFrame from a dictionary and performing basic operations like selecting columns, filtering rows, adding computed columns with expressions, and chaining multiple operations. It showcases Polars' eager evaluation model.

```python
import polars as pl

# Create DataFrame from dictionary
df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie", "Diana"],
    "age": [25, 30, 35, 28],
    "city": ["NYC", "LA", "Chicago", "NYC"],
    "salary": [70000, 80000, 90000, 75000]
})

# Select specific columns
result = df.select("name", "age")
# shape: (4, 2)
# ┌─────────┬─────┐
# │ name    │ age │
# ├─────────┼─────┤
# │ Alice   │ 25  │
# │ Bob     │ 30  │
# │ Charlie │ 35  │
# │ Diana   │ 28  │
# └─────────┴─────┘

# Filter rows with expressions
filtered = df.filter(pl.col("age") > 28)
# Returns: Charlie (35), Bob (30)

# Add computed columns
df_with_bonus = df.with_columns([
    (pl.col("salary") * 0.1).alias("bonus"),
    pl.when(pl.col("age") > 30)
      .then(pl.lit("senior"))
      .otherwise(pl.lit("junior"))
      .alias("level")
])

# Chain operations
result = (
    df
    .filter(pl.col("city") == "NYC")
    .select(["name", "salary"])
    .sort("salary", descending=True)
)
# Returns: Diana (75000), Alice (70000)

```

--------------------------------

### Database Operations with Polars using SQL

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Details how to read data from and write data to SQL databases using Polars. Supports reading entire tables or specific queries, filtering with query pushdown, parameterized queries, and using SQLAlchemy connections. Also covers writing DataFrames to tables with different `if_exists` strategies. Uses `read_database`, `read_database_uri`, and `write_database`.

```python
import polars as pl
from sqlalchemy import create_engine

# Read from database with connection string
connection_uri = "postgresql://user:password@localhost:5432/database"

# Read entire table
df = pl.read_database(
    query="SELECT * FROM customers",
    connection=connection_uri
)

# Read with filtering (pushdown to database)
df_filtered = pl.read_database(
    query="SELECT customer_id, name, email FROM customers WHERE status = 'active'",
    connection=connection_uri
)

# Read with parameterized query
df_params = pl.read_database(
    query="SELECT * FROM orders WHERE order_date >= ? AND amount > ?",
    connection=connection_uri,
    # Note: Actual parameter passing depends on the connection library
)

# Read using SQLAlchemy connection
engine = create_engine("postgresql://user:password@localhost:5432/database")
df_alchemy = pl.read_database(
    query="SELECT * FROM sales WHERE year = 2024",
    connection=engine
)

# Write DataFrame to database table
df.write_database(
    table_name="new_customers",
    connection=connection_uri,
    if_exists="replace"  # Options: fail, replace, append
)

# Append to existing table
df.write_database(
    table_name="customers",
    connection=connection_uri,
    if_exists="append"
)

# Use with various database engines
# PostgreSQL
df_postgres = pl.read_database_uri(
    query="SELECT * FROM table",
    uri="postgresql://localhost/db"
)

# MySQL
df_mysql = pl.read_database_uri(
    query="SELECT * FROM table",
    uri="mysql://localhost/db"
)

# SQLite
df_sqlite = pl.read_database_uri(
    query="SELECT * FROM table",
    uri="sqlite:///path/to/database.db"
)
```

--------------------------------

### Polars SQL Interface for DataFrame Querying

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Enables querying Polars DataFrames using SQL syntax. It allows users to register DataFrames as tables within a SQL context and execute standard SQL queries, including joins, aggregations, CTEs, and window functions. Requires the Polars library.

```python
import polars as pl

# Create sample DataFrames
customers = pl.DataFrame({
    "customer_id": [1, 2, 3, 4],
    "name": ["Alice", "Bob", "Charlie", "Diana"],
    "age": [25, 30, 35, 28],
    "city": ["NYC", "LA", "Chicago", "NYC"]
})

orders = pl.DataFrame({
    "order_id": [101, 102, 103, 104],
    "customer_id": [1, 2, 1, 3],
    "amount": [250.50, 180.00, 320.75, 150.25],
    "order_date": ["2024-01-15", "2024-01-16", "2024-01-17", "2024-01-18"]
})

# Create SQL context with named tables
ctx = pl.SQLContext({
    "customers": customers,
    "orders": orders
})

# Execute basic SQL query
result = ctx.execute("""
    SELECT name, age, city
    FROM customers
    WHERE age > 25
    ORDER BY age DESC
""")

# Joins in SQL
result_join = ctx.execute("""
    SELECT
        c.name,
        c.city,
        o.order_id,
        o.amount,
        o.order_date
    FROM customers c
    INNER JOIN orders o ON c.customer_id = o.customer_id
    WHERE o.amount > 200
""")

# Aggregations in SQL
result_agg = ctx.execute("""
    SELECT
        c.city,
        COUNT(*) as customer_count,
        AVG(o.amount) as avg_order_amount,
        SUM(o.amount) as total_revenue
    FROM customers c
    LEFT JOIN orders o ON c.customer_id = o.customer_id
    GROUP BY c.city
    HAVING total_revenue > 200
    ORDER BY total_revenue DESC
""")

# CTE (Common Table Expression)
result_cte = ctx.execute("""
    WITH high_value_orders AS (
        SELECT customer_id, amount
        FROM orders
        WHERE amount > 200
    )
    SELECT
        c.name,
        hvo.amount
    FROM customers c
    JOIN high_value_orders hvo ON c.customer_id = hvo.customer_id
""")

# Window functions in SQL
result_window = ctx.execute("""
    SELECT
        customer_id,
        amount,
        order_date,
        ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY amount DESC) as rank,
        SUM(amount) OVER (PARTITION BY customer_id) as total_spent
    FROM orders
""")

# Register DataFrame dynamically
new_df = pl.DataFrame({"id": [1, 2], "value": [100, 200]})
ctx.register("new_table", new_df)
result_new = ctx.execute("SELECT * FROM new_table WHERE value > 100")

```

--------------------------------

### Polars Core Objects

Source: https://docs.pola.rs/stable/api/python/stable/reference

This section covers the fundamental data structures in Polars, including DataFrame, LazyFrame, and Series.

```APIDOC
## Polars Core Objects

### Description
Core data structures for representing and manipulating tabular data in Polars.

### Objects
*   **DataFrame**: Represents a 2D, tabular, fixed-size in each dimension collection that is optimized into a columnar format.
*   **LazyFrame**: Represents a query plan that is lazily evaluated.
*   **Series**: Represents a 1D collection of data.
```

--------------------------------

### Polars Data Types

Source: https://docs.pola.rs/stable/api/python/stable/reference

An overview of the various data types supported by Polars for column representation.

```APIDOC
## Polars Data Types

### Description
Describes the comprehensive set of data types available in Polars for defining schemas and column properties.

### Types
*   **DataType**: Base class for all Polars data types.
*   **Numeric**: Integer and floating-point types.
*   **Temporal**: Date, time, and timestamp types.
*   **Nested**: Struct, List, and Tensor types.
*   **String**: UTF-8 encoded string type.
*   **Other**: Boolean, Binary, Categorical, etc.
```

--------------------------------

### Polars Expressions API

Source: https://docs.pola.rs/stable/api/python/stable/reference

Details on how to construct and manipulate expressions for data transformations and selections within Polars.

```APIDOC
## Polars Expressions API

### Description
Defines how to build and use expressions for selecting, transforming, and aggregating data within Polars DataFrames and LazyFrames.

### Categories
*   **Selectors**: Mechanisms for selecting columns.
*   **DataType Expressions**: Expressions related to data type operations.
*   **Functions**: Various functions for expression manipulation, including conversion, miscellaneous, multiple queries, and random operations. StringCache related functions are also included.
```

--------------------------------

### Join DataFrames in Polars

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates how to combine two DataFrames using different join strategies. Covers inner join (only matching rows) and left join (all rows from the left DataFrame with matching or nulls from the right). Uses the `join` method with `on` and `how` parameters.

```python
import polars as pl

customers = pl.DataFrame({
    "customer_id": [1, 2, 3, 4],
    "name": ["Alice", "Bob", "Charlie", "Diana"],
    "region": ["North", "South", "East", "West"]
})

orders = pl.DataFrame({
    "order_id": [101, 102, 103, 104, 105],
    "customer_id": [1, 2, 1, 3, 5],
    "amount": [250, 180, 320, 150, 400],
    "date": ["2024-01-15", "2024-01-16", "2024-01-17", "2024-01-18", "2024-01-19"]
})

# Inner join (only matching records)
inner = customers.join(orders, on="customer_id", how="inner")

# Left join (all customers, nulls for no orders)
left = customers.join(orders, on="customer_id", how="left")
```

--------------------------------

### Configure Polars Display and Behavior Settings

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Allows customization of global display and behavior settings for Polars DataFrames and operations. This includes table display dimensions, numeric formatting, verbosity, and boolean representation. Configuration can be applied globally, temporarily using context managers, or via decorators. Requires the Polars library.

```python
import polars as pl

# Set table display options
pl.Config.set_tbl_rows(20)  # Number of rows to display
pl.Config.set_tbl_cols(10)  # Number of columns to display
pl.Config.set_tbl_width_chars(120)  # Terminal width in characters
pl.Config.set_fmt_str_lengths(50)  # Max string display length

# Set numeric formatting
pl.Config.set_float_precision(3)  # Decimal places for floats
pl.Config.set_decimal_separator(",")  # Use comma as decimal separator
pl.Config.set_thousands_separator("_")  # Thousands separator

# Set verbose output
pl.Config.set_verbose(True)  # Show additional info

# Configure boolean display
pl.Config.set_fmt_table_cell_list_len(3)  # Max list elements to show

# Use as context manager (temporary config)
df = pl.DataFrame({"a": range(100), "b": range(100, 200)})

with pl.Config(tbl_rows=50, tbl_cols=15):
    print(df)  # Shows 50 rows, 15 columns
# Config reverts after context

print(df)  # Back to default settings

# Use as decorator
@pl.Config(tbl_rows=100, float_precision=5)
def analyze_data():
    df = pl.DataFrame({"value": [1.123456, 2.234567, 3.345678]})
    print(df)
    return df

analyze_data()  # Uses custom config

# Save and restore configuration state
saved_state = pl.Config.save()

# Make changes
pl.Config.set_tbl_rows(5)
pl.Config.set_float_precision(2)

# Restore previous state
pl.Config.load(saved_state)

# Reset to defaults
pl.Config.restore_defaults()

# Check current settings
state = pl.Config.state()
print(state)  # Shows all current config values

```

--------------------------------

### Apply Conditional Logic to DataFrame Columns

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates how to apply conditional logic to create new columns based on existing data. Uses `when`, `then`, `otherwise`, and `alias` for creating segmented or transformed columns. Handles multiple conditions sequentially.

```python
df_segments = df.with_columns([
    pl.when(pl.col("purchases") > 10)
      .then(pl.lit("high")),
    pl.when(pl.col("purchases") > 5)
      .then(pl.lit("medium")),
    .otherwise(pl.lit("low"))
      .alias("customer_segment"),
    (pl.col("revenue") / pl.col("purchases")).alias("avg_order_value")
])
```

--------------------------------

### Register and Use DataFrame Business Namespace

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates registering a custom business namespace for Polars DataFrames to perform operations like calculating ROI and customer lifetime value. This requires defining a class with methods that operate on the DataFrame. It showcases how to call these custom methods after registration.

```python
import polars as pl

# Assume pl.business is registered elsewhere
# For demonstration purposes, this shows the usage pattern
df = pl.DataFrame({
    "customer_id": [1, 1, 2, 2, 3],
    "revenue": [100, 150, 200, 175, 300],
    "cost": [80, 120, 160, 140, 240]
})

# Example usage (assuming the 'business' namespace is correctly registered)
# roi_df = df.business.calculate_roi("revenue", "cost")
# clv_df = df.business.customer_lifetime_value("revenue", "customer_id")

# Placeholder for actual registration and usage demonstration if registration code was available
print("DataFrame business namespace usage demonstrated.")
```

--------------------------------

### Polars DataFrame Joins

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates various join types (inner, outer, anti, semi) between two Polars DataFrames, illustrating how records are combined or filtered based on matching keys or lack thereof. Also shows joining on columns with different names.

```python
import polars as pl

customers = pl.DataFrame({
    "customer_id": [1, 2, 3, 4, 5],
    "name": ["Alice", "Bob", "Charlie", "Diana", "Eve"],
    "city": ["NY", "LA", "NY", "LA", "SF"]
})
orders = pl.DataFrame({
    "order_id": [101, 102, 103, 104],
    "customer_id": [1, 2, 1, 3],
    "order_date": ["2023-01-10", "2023-01-12", "2023-01-15", "2023-01-20"]
})

# Inner join (only matching records)
inner = customers.join(orders, on="customer_id", how="inner")
# Returns: 4 records (Alice, Bob, Charlie)

# Left join (all from left, matching from right)
left = customers.join(orders, on="customer_id", how="left")
# Returns: 5 records (Diana has null order values)

# Outer join (all records from both)
outer = customers.join(orders, on="customer_id", how="outer")
# Returns: 6 records (includes customer_id=5 with null customer info)

# Anti join (records from left not in right)
no_orders = customers.join(orders, on="customer_id", how="anti")
# Returns: Diana (customer_id=4)

# Semi join (records from left that are in right)
with_orders = customers.join(orders, on="customer_id", how="semi")
# Returns: Alice, Bob, Charlie

# Join on different column names
products = pl.DataFrame({
    "prod_id": [1, 2, 3],
    "product_name": ["Laptop", "Phone", "Tablet"]
})
order_items = pl.DataFrame({
    "item_id": [1, 2, 3],
    "product_code": [1, 2, 1],
    "quantity": [2, 5, 1]
})
joined = products.join(
    order_items,
    left_on="prod_id",
    right_on="product_code",
    how="inner"
)

```

--------------------------------

### Polars CSV Reading and Writing

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Details how to read from and write to CSV files using Polars, covering basic operations and advanced options like specifying separators, headers, data types, null values, encoding, and lazy scanning for large files.

```python
import polars as pl

# Basic CSV read
df = pl.read_csv("data.csv")

# Read with specific options
df = pl.read_csv(
    "data.csv",
    separator=",",
    has_header=True,
    skip_rows=0,
    n_rows=1000,  # Read only first 1000 rows
    columns=["name", "age", "salary"],  # Select specific columns
    dtypes={"age": pl.Int32, "salary": pl.Float64},  # Specify types
    null_values=["NA", "NULL", ""],  # Custom null values
    ignore_errors=True,  # Skip malformed rows
    encoding="utf8",
    low_memory=False  # Faster parsing for smaller files
)

# Lazy CSV scan for large files
lf = pl.scan_csv(
    "large_data.csv",
    dtypes={"customer_id": pl.Int64}
)
result = (
    lf
    .filter(pl.col("amount") > 100)
    .select(["customer_id", "amount", "date"])
    .collect()
)

# Write CSV with options
df.write_csv(
    "output.csv",
    separator=",",
    has_header=True,
    quote_char='"',
    datetime_format="%Y-%m-%d %H:%M:%S",
    float_precision=2
)

# Read from compressed CSV
df_compressed = pl.read_csv("data.csv.gz")  # Auto-detects compression

# Read CSV with custom schema
schema = {
    "name": pl.Utf8,
    "age": pl.Int32,
    "score": pl.Float64,
    "active": pl.Boolean
}
df_typed = pl.read_csv("data.csv", schema=schema)

```

--------------------------------

### Read and Write JSON/NDJSON Files with Polars

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Covers reading and writing standard JSON and newline-delimited JSON (NDJSON) files. Includes lazy scanning for large NDJSON files, handling nested structures, pretty-printing, and reading JSON from strings. Uses `read_json`, `read_ndjson`, `scan_ndjson`, and `write_json`, `write_ndjson`.

```python
import polars as pl

# Read standard JSON
df = pl.read_json("data.json")

# Read NDJSON (newline-delimited JSON) - one JSON object per line
df_ndjson = pl.read_ndjson("data.ndjson")

# Lazy scan for large NDJSON files
lf = pl.scan_ndjson("large_data.ndjson")
result = (
    lf
    .filter(pl.col("status") == "active")
    .select(["id", "name", "value"])
    .collect()
)

# Read JSON with nested structures
json_data = '''
{"name": "Alice", "age": 25, "addresses": [{"city": "NYC", "zip": "10001"}]}
{"name": "Bob", "age": 30, "addresses": [{"city": "LA", "zip": "90001"}]}
'''
# Polars automatically handles nested structures as List/Struct types

# Write to JSON
df.write_json("output.json")

# Write to NDJSON (more efficient for large datasets)
df.write_ndjson("output.ndjson")

# Pretty-print JSON
df.write_json("output.json", pretty=True)

# Read JSON from string
json_string = '{"a": [1, 2, 3], "b": [4, 5, 6]}'
df_from_str = pl.read_json(json_string.encode())

# Handle nested JSON structures
df_nested = pl.DataFrame({
    "id": [1, 2],
    "data": [
        {"value": 100, "tags": ["a", "b"]},
        {"value": 200, "tags": ["c", "d"]}
    ]
})
# Access nested fields
df_unnested = df_nested.with_columns([
    pl.col("data").struct.field("value").alias("value"),
    pl.col("data").struct.field("tags").alias("tags")
])
```

--------------------------------

### Read Parquet Files and Cloud Storage with Polars

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Shows how to read Parquet files, including lazy scanning with parallel processing and filtering, and reading directly from cloud storage services like S3, Azure, and GCS. Utilizes `scan_parquet` and `read_parquet`.

```python
import polars as pl

# Read Parquet with row group filtering
lf = pl.scan_parquet(
    "data.parquet",
    parallel="auto"  # Control parallelism
)

# Read from cloud storage
df_s3 = pl.read_parquet("s3://bucket/data.parquet")
df_azure = pl.read_parquet("az://container/data.parquet")
df_gcs = pl.read_parquet("gs://bucket/data.parquet")
```

--------------------------------

### Test Expected Polars Errors with pytest.raises

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Demonstrates integrating Polars testing with pytest, specifically using `pytest.raises` to assert that certain operations raise expected Polars exceptions. This is essential for testing error handling and validation logic, such as attempting operations on non-existent columns or invalid types.

```python
import polars as pl
import pytest

def test_invalid_operation():
    df = pl.DataFrame({"value": [1, 2, 3]})

    with pytest.raises(pl.ColumnNotFoundError):
        df.select("nonexistent_column")

def test_type_error():
    df = pl.DataFrame({"text": ["a", "b", "c"]})

    with pytest.raises(pl.InvalidOperationError):
        df.select(pl.col("text").mean())  # Can't compute mean of strings

# To run these tests, you would typically save this code in a file (e.g., test_polars.py)
# and run 'pytest' from your terminal in the same directory.
print("Pytest error tests defined. Run with pytest.")

```

--------------------------------

### Test DataFrame Equality with Relative Tolerance and Options

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Illustrates advanced usage of `assert_frame_equal`, including testing with relative tolerance (`rtol`) for floating-point comparisons and controlling other checking options like `check_column_order`. This allows for flexible and precise DataFrame comparisons in tests.

```python
import polars as pl
from polars.testing import assert_frame_equal

# Test with specific checks
def test_with_options():
    df1 = pl.DataFrame({
        "a": [1, 2, 3],
        "b": [4.0001, 5.0001, 6.0001]
    })

    df2 = pl.DataFrame({
        "a": [1, 2, 3],
        "b": [4.0, 5.0, 6.0]
    })

    # Compare with relative tolerance
    assert_frame_equal(df1, df2, rtol=1e-3)

    # Check only values (ignore column names)
    assert_frame_equal(df1, df2, check_column_order=False, rtol=1e-3)

test_with_options()

```

--------------------------------

### Perform Grouped Aggregations in Polars

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Shows how to perform aggregations on grouped data. Supports single and multiple grouping keys. Includes various aggregation functions like sum, mean, count, and unique counts. Uses `group_by` and `agg`.

```python
df = pl.DataFrame({
    "category": ["Electronics", "Electronics", "Clothing", "Clothing", "Electronics"],
    "region": ["East", "West", "East", "West", "East"],
    "product": ["Laptop", "Phone", "Shirt", "Pants", "Tablet"],
    "sales": [1200, 800, 50, 80, 600],
    "quantity": [2, 5, 10, 8, 3],
    "date": ["2024-01-01", "2024-01-02", "2024-01-01", "2024-01-02", "2024-01-03"]
})

# Single group aggregation
by_category = df.group_by("category").agg([
    pl.col("sales").sum().alias("total_sales"),
    pl.col("sales").mean().alias("avg_sales"),
    pl.col("quantity").sum().alias("total_quantity"),
    pl.col("product").count().alias("num_transactions"),
    pl.col("product").n_unique().alias("unique_products")
])

# Multiple group keys
by_cat_region = df.group_by(["category", "region"]).agg([
    pl.col("sales").sum().alias("total_sales"),
    pl.col("quantity").sum().alias("total_quantity")
]).sort("total_sales", descending=True)

# Statistical aggregations
stats = df.group_by("category").agg([
    pl.col("sales").min().alias("min_sale"),
    pl.col("sales").max().alias("max_sale"),
    pl.col("sales").median().alias("median_sale"),
    pl.col("sales").std().alias("std_dev"),
    pl.col("sales").quantile(0.25).alias("q25"),
    pl.col("sales").quantile(0.75).alias("q75")
])
```

--------------------------------

### Extend Polars API with Custom Namespaces

Source: https://context7.com/context7/pola_rs_stable/llms.txt

Provides a mechanism to extend the Polars API with custom functionalities by registering new namespaces for DataFrames. This allows for the creation of reusable, domain-specific methods that operate on DataFrames. Requires the Polars library and type aliases for type hinting.

```python
import polars as pl
from polars.type_aliases import PolarsDataType

# Register custom DataFrame namespace
@pl.api.register_dataframe_namespace("business")
class BusinessNamespace:
    def __init__(self, df: pl.DataFrame):
        self._df = df

    def calculate_roi(self, revenue_col: str, cost_col: str) -> pl.DataFrame:
        """Calculate return on investment"""
        return self._df.with_columns([
            ((pl.col(revenue_col) - pl.col(cost_col)) / pl.col(cost_col) * 100)
            .alias("roi_percent")
        ])

    def customer_lifetime_value(self, revenue_col: str, customer_col: str) -> pl.DataFrame:
        """Calculate CLV by customer"""
        return self._df.group_by(customer_col).agg([
            pl.col(revenue_col).sum().alias("total_revenue"),
            pl.col(revenue_col).count().alias("num_purchases"),
            pl.col(revenue_col).mean().alias("avg_purchase")
        ])

# Example Usage:
data = {
    "revenue": [1000, 1500, 1200, 1800, 2000],
    "cost": [500, 600, 700, 800, 900],
    "customer_id": [1, 2, 1, 3, 2]
}
df = pl.DataFrame(data)

# Using the custom namespace
df_with_roi = df.business.calculate_roi("revenue", "cost")
print(df_with_roi)

clv_df = df.business.customer_lifetime_value("revenue", "customer_id")
print(clv_df)

```
Tokens:
Raw