### Install Polars DS with Plotting Support

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/docs/index.md

Installs the Polars DS library with optional plotting capabilities.

```bash
pip install "polars_ds[plot]"
```

--------------------------------

### Compile and Develop with Maturin

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/docs/index.md

Builds and installs the package in development mode using Maturin, optimizing for release.

```bash
maturin develop --release
```

--------------------------------

### Run Tests Locally

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/README.md

Install test requirements and run pytest to verify the build locally.

```bash
pip install -r requirements-test.txt
pytest tests/test_*
```

--------------------------------

### Create DataFrame with Polars and Polars DS

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Initializes a Polars DataFrame with synthetic data and adds new columns using polars_ds random functions. This setup is used for subsequent examples.

```python
import polars as pl
import polars_ds as pds
import numpy as np
```

```python
size = 10_000
df = (
    pl.DataFrame(
        {
            "f": np.sin(list(range(size))),
            "time_idx": range(size),
            "dummy": ["a"] * (size // 2) + ["b"] * (size // 2),
            "actual": np.round(np.random.random(size=size)).astype(np.int32),
            "predicted": np.random.random(size=size),
            "dummy_groups": ["a"] * (size // 2) + ["b"] * (size // 2),
        }
    )
    .with_columns(
        pds.random(0.0, 1.0).alias("x1"),
        pds.random(0.0, 1.0).alias("x2"),
        pds.random(0.0, 1.0).alias("x3"),
        pds.random(0.0, 1.0).alias("a"),
        pds.random(0.0, 1.0).alias("b"),
    )
    .with_columns(
        y=pl.col("x1") * 0.15 + pl.col("x2") * 0.3 - pl.col("x3") * 1.5 + pds.random() * 0.0001,
        y2=pl.col("x1") * 0.13 + pl.col("x2") * 0.45 - pl.col("x3") * 0.1 + pds.random() * 0.0001,
    )
)
df.head()
```

--------------------------------

### Import Libraries and Check Version

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/benchmarks/linear_regression.ipynb

Imports necessary libraries including polars, pandas, and polars_ds. Checks the installed version of polars_ds.

```python
import polars as pl
import pandas as pd
import polars_ds as pds
import polars_ds.linear_models as pds_linear

# Requires version >= v0.5.1
print(pds.__version__)
```

--------------------------------

### Materialize and Transform DataFrame using Blueprint

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

This snippet demonstrates how to materialize a defined Blueprint into a pipeline and then apply that pipeline to transform a DataFrame. Use this after defining your data preparation steps to get the processed data.

```python
pipe2 = bp2.materialize()
df_transformed2 = pipe2.transform(df)
df_transformed2.head()
```

--------------------------------

### Integrating Polars DS Transformer into Sklearn Pipeline

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/SKLEARN_COMPATIBILITY.md

Demonstrates how to load a Parquet file, instantiate the custom Polars DS transformer, and integrate it into a Scikit-learn Pipeline. This example requires the 'dependency.parquet' file to be present.

```python
df = pl.read_parquet("../examples/dependency.parquet")
df.head()

pipe = Pipeline(
    steps=[
        ("CustomPDSTransformer", CustomPDSTransformer())
    ]
)
df_transformed = pipe.fit_transform(df)
df_transformed
```

--------------------------------

### Run Pytest Tests

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/docs/index.md

Executes the test suite for the package using pytest. Assumes test requirements are installed.

```bash
pytest tests/test_*
```

--------------------------------

### Import necessary libraries for Polars DS Extension

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Imports Polars and specific components for pipeline creation and testing. Ensure these libraries are installed.

```python
import polars as pl
import polars.selectors as cs
from polars_ds.pipeline import Pipeline, Blueprint
from polars.testing import assert_frame_equal
```

--------------------------------

### Get Partition Names

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Retrieves the names of the partitions created by PartitionHelper. Requires polars_ds.

```python
parts.names()
```

--------------------------------

### Integrating Custom Transformer into Sklearn Pipeline

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Instantiate the custom transformer and integrate it into a Scikit-learn Pipeline. This example shows how to read data, create the pipeline with the custom transformer, and then apply fit_transform to process the DataFrame.

```python
df = pl.read_parquet("../examples/dependency.parquet")

pipe = Pipeline(steps=[("CustomPDSTransformer", CustomPDSTransformer())])
df_transformed = pipe.fit_transform(df)
df_transformed
```

--------------------------------

### Create Sample DataFrame for Nearest Neighbors

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Generates a Polars DataFrame with random data for 'var1', 'var2', 'var3', 'r', and 'rh' columns, along with an 'id' column. This setup is for demonstrating nearest neighbor queries.

```python
import polars_ds as pds

size = 2000
df = pl.DataFrame(
    {
        "id": range(size),
    }
).with_columns(
    pds.random().alias("var1"),
    pds.random().alias("var2"),
    pds.random().alias("var3"),
    pds.random().alias("r"),
    (pds.random() * 10).alias("rh"),
    pl.col("id").cast(pl.UInt32),
)
```

--------------------------------

### Linear Regression with Prediction and Residuals

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Use `lin_reg` to get predictions and residuals. Ensure `return_pred=True` and `add_bias=False` are set.

```python
df.select(
    "x1",
    "x2",
    "y",
    pds.lin_reg("x1", pl.col("x2"), target="y", add_bias=False, return_pred=True).alias(
        "prediction"
    ),
).unnest("prediction").head()
```

--------------------------------

### PCA with Singular Values and Principal Components

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Perform Principal Component Analysis (PCA) using `pca` to get both singular values and weight vectors (principal components). The result is unnested.

```python
# Singular values + The principal components
df.select(pds.pca("a", "b")).unnest("a")
```

--------------------------------

### Create and Print a Data Preparation Blueprint

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

This snippet shows how to initialize a Blueprint with a DataFrame, apply various transformations like filtering, feature generation, and aggregation, and then print the blueprint's summary. It's useful for defining complex data preparation steps.

```python
bp2 = (
    Blueprint(df, name="example", target="approved", lowercase=True)  # You can optionally put target of the ML model here
    .filter(
        "city_category is not null"  # or equivalently, you can do: pl.col("city_category").is_not_null()
    )
    .with_columns(  # generate some features
        pl.col("existing_emi").log1p().alias("existing_emi_log1p"),
        pl.col("loan_amount").log1p().alias("loan_amount_log1p"),
        pl.col("loan_amount")
        .clip(lower_bound=0, upper_bound=1000)
        .alias("loan_amount_log1p_clipped"),
        pl.col("loan_amount").sqrt().alias("loan_amount_sqrt"),
        pl.col("loan_amount").shift(-1).alias("loan_amount_lag_1"),  # any kind of lag transform
    )
    .group_by_agg(
        by="city_category",
        agg=[
            pl.col("loan_amount").sqrt().mean().alias("loan_amount_sqrt_mean"),
            pl.col("loan_amount").min().alias("loan_amount_min"),
            pl.col("loan_amount").max().alias("loan_amount_max"),
        ],
    )
    .sort(by=["city_category"], descending=True)
)

print(bp2)
```

--------------------------------

### Create a pipeline blueprint

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Initializes a blueprint, which serves as a plan for a pipeline. Transformations are fitted only when the blueprint is materialized. The 'target' parameter excludes specific columns from fitting.

```python
# Create a blueprint first.
# A blueprint is a plan for a pipeline. No hard work will be done until the blueprint is materialized, which
# is when the tranforms are fitted (e.g. scale learns the mean and std from base data)
# If target is specified for the blueprint, target will be excluded from all transformations that require a fit,

```

--------------------------------

### Accessing Individual Splits

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/sample_and_split.ipynb

Provides guidance on how to work with the individual splits after they have been created, suggesting filtering or using `partition_by`.

```python
# If you need to do work with the splits individually
# you can filter, or use .partition_by("my_splits") to get separated dataframes
```

--------------------------------

### Append Custom Steps from Dictionary and Transform Data

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Demonstrates how to initialize an `ExtendedBlueprint`, append custom imputation steps using dictionaries, materialize the pipeline, and transform a DataFrame. It also shows how to add an extra column for imputation value comparison and filter for null values.

```python
bp = ExtendedBlueprint(df, name="example", target="approved")

# Takes in a dict with 3 fields: `name`, `args`, and `kwargs`.
# Args and kwargs are optional depending on whether the method call needs certain arguments.
step_dict_1 = {"name": "smallest_abs_impute", "kwargs": {"cols": ["Existing_EMI"], "epsilon": 0.01}}

step_dict_2 = {
    "name": "smallest_abs_impute2",
    "kwargs": {"cols": ["Existing_EMI"], "epsilon": 0.01},
}

bp.append_step_from_dict(step_dict_1).append_step_from_dict(step_dict_2)

pipe = bp.materialize()
df_transformed = pipe.transform(df)

df_transformed.with_columns(impute_value=pl.col("Existing_EMI").abs().min() + 0.01).filter(
    pl.col("Existing_EMI").is_null()
).select(
    pl.col("Existing_EMI"),
    pl.col("Existing_EMI_imputed"),
    pl.col("Existing_EMI_imputed2"),
    pl.col("impute_value"),
)
```

--------------------------------

### Get Specific Partition Data

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Retrieves and displays the head of a specific partition from the DataFrame. Requires polars_ds and polars.

```python
parts.get("virginica").head()
```

--------------------------------

### Sample Distributions Using Column Statistics

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Samples from normal and uniform distributions using column statistics (mean, std, max) and respecting nulls. Also demonstrates adding a random perturbation.

```python
df.with_columns(
    # Sample from a normal distribution, using reference column "a" 's mean and std
    pds.random_normal(pl.col("a").mean(), pl.col("a").std()).alias("test1"),
    # Sample from uniform distribution, with low = 0 and high = "a"'s max, and respect the nulls in "a"
    pl.when(pl.col("a").is_null())
    .then(None)
    .otherwise(pds.random(lower=0.0, upper=pl.col("a").max()).alias("test2")),
).with_columns(
    # Add a random pertubation to test1
    pds.perturb("test1", epsilon=0.001).alias("test1_perturbed")
).head()
```

--------------------------------

### Build from Source with Maturin

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/README.md

Compile the package from source using maturin in release mode.

```bash
maturin develop --release
```

--------------------------------

### Generate and Display Sample DataFrame

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Creates a Polars DataFrame with random data and displays the first few rows. Imports necessary libraries.

```python
import pandas as pd
import numpy as np
import polars as pl
import polars_ds as pds
from polars_ds.compat import compat as pds2

df = pds.frame(size=100_000).select(
    pds.random(0.0, 1.0).round().alias("actual"),
    pds.random(0.0, 1.0).alias("predicted"),
    pds.random_int(0, 3).alias("0-2"),
    pds.random_int(0, 10).alias("0-9"),
    pds.random_str(min_size=1, max_size=2).alias("s1"),
    pds.random_str(min_size=1, max_size=2).alias("s2"),
)
df.head()
```

--------------------------------

### Sample DataFrame by Ratio

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/sample_and_split.ipynb

Use the `sample` method to select a fraction of rows from a DataFrame. This is useful for creating smaller, representative datasets for analysis or testing.

```python
ss.sample(df, 0.6)  # by ratio
```

--------------------------------

### Get DataFrame Metadata

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Retrieve metadata about the DataFrame, including lists of columns categorized by their data types (numerics, floats, strings, etc.).

```python
dia.meta()
```

--------------------------------

### Get IDs of K Nearest Neighbors

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Retrieves the IDs of the k nearest neighbors for each point in the dataset. This is a fundamental operation for many machine learning algorithms.

```python
# Get ids of the k nearest neighbors.
```

--------------------------------

### Import Sklearn Metrics

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/benchmarks/benchmarks.ipynb

Imports common metrics from scikit-learn for evaluation.

```python
from sklearn.metrics import roc_auc_score, log_loss, brier_score_loss
```

--------------------------------

### Filter DataFrame for Null City Category

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Filters a transformed DataFrame to select rows where the 'city_category' column is null. This example demonstrates a filtering operation on a DataFrame that has already been processed.

```python
# Empty. Because we filtered this to not null.
df_transformed.filter(pl.col("city_category").is_null())
```

--------------------------------

### Configure Linear Regression for f32

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Set the global configuration to use f32 for linear regression expressions. This can improve performance when input data is already in f32 format.

```python
pds.config.LIN_REG_EXPR_F64 = False
df.select(
    pds.lin_reg(pl.col("x1"), pl.col("x2"), target=[pl.col("y"), pl.col("y2")], add_bias=False)
).unnest("coeffs")
```

--------------------------------

### Apply Levenshtein Distance in Narwhals

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Applies the Levenshtein distance calculation within a Narwhals DataFrame using `map_batches`. This example demonstrates using a Polars literal as input to the mapped function.

```python
# If you are using Narwhals, well, Narwhal expressions are not Polars expressions.
# Using the pds2 module, you can run pds functions in map_batches, but this is limited to 1 input column.

import narwhals as nw

df_nw = nw.from_native(df_pd)
df_nw.with_columns(
    nw_levenshtein_dist=nw.col("s1").map_batches(
        lambda s: pds2.str_leven(s.to_numpy(), pl.lit("k9"))
    )
).head()
```

--------------------------------

### Downsample a Specific Group

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/sample_and_split.ipynb

Use the `downsample` function to reduce the number of rows in a specific group within a DataFrame. This example downsamples rows where the 'flags' column is 0 to 50% of its original count.

```python
# Downsample on one group
sa1 = ss.downsample(df, [(pl.col("flags") == 0, 0.5)])
sa1.group_by("flags").len().sort("flags")
```

--------------------------------

### Build a Data Processing Blueprint

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Constructs a data processing blueprint with various transformations including selection, SQL transform, imputation, feature engineering, scaling, and encoding. The target field is specified during initialization.

```python
bp = (
    Blueprint(
        df, name="example", target="approved", lowercase=True
    )  # You can optionally put target of the ML model here
    .select_by_std(
        min_=0.001, max_=30000
    )  # keep only numeric features with std between min and max. This will drop `loan_amount` in this example
    .sql_transform(sql)  # Run a SQL transform on the df
    # Say you want to remove a population for your data pipeline.
    .filter(
        "city_category is not null"  # or equivalently, you can do: pl.col("city_category").is_not_null()
    )
    # Here we explicitly put target, since this is not the target for prediction.
    # Use a linear regression with x1 = var1, x2=existing_emi to predict missing values in loan_period
    # This transform should not be used when memory is an issue.
    .linear_impute(features=["var1", "existing_emi"], target="loan_period")
    .impute(["existing_emi"], method="median")
    .with_columns(  # generate some features
        pl.col("existing_emi").log1p().alias("existing_emi_log1p"),
        pl.col("interest_rate").log1p().alias("interest_rate_log1p"),
        pl.col("interest_rate")
        .clip(lower_bound=0, upper_bound=1000)
        .alias("interest_rate_log1p_clipped"),
        pl.col("interest_rate").sqrt().alias("interest_rate_sqrt"),
        pl.col("interest_rate").shift(-1).alias("interest_rate_lag_1"),  # any kind of lag transform
    )
    .scale(  # target is numerical, but will be excluded automatically because bp is initialzied with a target
        cs.numeric().exclude(["var1", "existing_emi_log1p"])
    )  # Scale the columns up to this point. The columns below won't be scaled
    .with_columns(
        # Add missing flags
        pl.col("employer_category1").is_null().cast(pl.UInt8).alias("employer_category1_is_missing")
    )
    .ordinal_encode(cols=["city_category"])
    .woe_encode(
        cols=pl.exclude("id")
    )  # No need to specify target because we initialized bp with a target. None means encode all str columns
    # .sort(by = "monthly_income", descending=True)
    # .one_hot_encode(cols=None, drop_first=True) # None means all str columns, or you can provide a list of columns
    # .target_encode("employer_category1", min_samples_leaf = 20, smoothing = 10.0) # same as above
)

print(bp)
```

--------------------------------

### Extend Blueprint with Custom Imputation Steps

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Extends the `Blueprint` class to include custom imputation methods like `smallest_abs_impute` and `smallest_abs_impute2`. These methods wrap the imputation logic into `FitStep` objects, allowing them to be added to the pipeline.

```python
class ExtendedBlueprint(Blueprint):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def smallest_abs_impute(self, cols: List[str], epsilon: float = 0.01) -> "ExtendedBlueprint":
        # bind all arguments, except df and cols.
        # If you don't want to use partial from functool, you can define an inner function
        partial_func = partial(smallest_abs_impute, epsilon=epsilon)
        self._steps.append(FitStep(partial_func, cols, self.exclude))
        return self

    def smallest_abs_impute2(self, cols: List[str], epsilon: float = 0.01) -> "ExtendedBlueprint":
        # bind all arguments, except df and cols.
        # Example of using an inner function
        def inner_func(df: Union[pl.DataFrame, pl.LazyFrame], cols: List[str]) -> List[pl.Expr]:
            temp = df.lazy().select(pl.col(cols).abs().min() + epsilon).collect().row(0)
            return [pl.col(c).fill_null(m).name.suffix("_imputed2") for c, m in zip(cols, temp)]

        self._steps.append(FitStep(inner_func, cols, self.exclude))
        return self
```

--------------------------------

### Print Blueprint Details

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Prints the details of the constructed blueprint, including its name, column name handling, number of steps, and expected features.

```python
print(bp)
```

--------------------------------

### Create Time-Series Aggregation Blueprint

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

This snippet defines a Blueprint for time-series data, sorting by date, and performing dynamic group-by aggregations on a yearly basis. It calculates the square root mean, minimum, and maximum of the 'Close' price. This is useful for summarizing stock data over time.

```python
bp3 = (
    Blueprint(df_app)
    .sort(by=["Date"], descending=False)
    .group_by_dynamic_agg(
        index_column="Date",
        every="1y",
        agg=[
            pl.col("Close").sqrt().mean().alias("sqrt_mean"),
            pl.col("Close").min().alias("min"),
            pl.col("Close").max().alias("max"),
        ],
    )
)

pipe3 = bp3.materialize()
df_transformed3 = pipe3.transform(df_app)
df_transformed3.head()
```

--------------------------------

### Custom Polars DS Transformer for Sklearn

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/SKLEARN_COMPATIBILITY.md

Defines a custom Scikit-learn Transformer that integrates a Polars DS pipeline. This class can be used within Scikit-learn pipelines for data transformation tasks. Ensure Polars and Polars DS are installed.

```python
from typing import Optional
import polars_ds.pipeline as pds_pipe
import polars.selectors as cs
from sklearn.pipeline import Pipeline
from sklearn.base import BaseEstimator, TransformerMixin

class CustomPDSTransformer(BaseEstimator, TransformerMixin):

    def __init__(self):
        self.pipe: Optional[pds_pipe.Pipeline] = None

    def fit(self, df: pl.DataFrame, y: Optional[pl.Series] = None) -> "CustomPDSTransformer":
        # specify all the rules for the transform here
        bp = (
            pds_pipe.Blueprint(df, name = "example", target = "approved", lowercase=True) 
            .filter(pl.col("city_category").is_not_null())
            .select(cs.numeric() | cs.by_name(["gender", "employer_category1", "city_category", "test_col"]))
            .linear_impute(features = ["var1", "existing_emi"], target = "loan_period") 
            .impute(["existing_emi"], method = "median")
        )
        self.pipe = bp.materialize()
        return self

    def transform(self, df, y=None):
        return self.pipe.transform(df)
```

--------------------------------

### Quick Linear Regression with Predictions

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/README.md

Run a linear regression on the fly and return predictions. Set `add_bias=False` to exclude the bias term.

```python
df.select(pds.lin_reg(pl.col("x1"), pl.col("x2"), target=pl.col("y"), add_bias=False, return_pred=True))
```

--------------------------------

### Linear Regression with Pandas Compatibility

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/README.md

Performs linear regression using the Polars DS Extension's compatibility module with Pandas DataFrames. This example demonstrates how to add regression predictions as a new column to an existing Pandas DataFrame.

```python
from polars_ds.compat import compat as pds2

df_pd["linear_regression_result"] = pds2.lin_reg(
    df_pd["x1"], df_pd["x2"], df_pd["x3"],
    target = df_pd["y"],
    return_pred = True
)
df_pd
```

--------------------------------

### Generate Sample DataFrame for Benchmarking

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/benchmarks/benchmarks.ipynb

Creates a Polars DataFrame with random predicted values, actual targets, and dates for benchmarking ML metric evaluations. Converts to Pandas DataFrame for comparison.

```python
from datetime import date

dates = pl.date_range(date(2001, 1, 1), date(2025, 5, 1), "1d", eager=True)
df = pds.frame(size=len(dates)).select(
    pds.random().alias("predicted"),
    (pds.random() > 0.25).cast(pl.UInt8).alias("actual_target"),
    dates=dates,
)
df_pd = df.to_pandas()
```

--------------------------------

### Generate Various Random Samples

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Generates random samples from a uniform distribution [0, 1), a normal distribution using column statistics, and random integers within a range. This version does not respect nulls.

```python
df.with_columns(
    pds.random().alias("[0, 1)"),
    pds.random_normal(pl.col("a").mean(), pl.col("a").std()).alias("Normal"),
    pds.random_int(0, 10).alias("Int from [0, 10)"),
).head()
```

--------------------------------

### Enable Linear Regression for f64

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Enable the use of f64 for linear regression expressions. This is the default behavior and ensures higher precision.

```python
pds.Config.LIN_REG_EXPR_F64 = True  # pds.Config or pds.config will both work
```

--------------------------------

### Load and prepare data for pipeline processing

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Reads data from a Parquet file and selects specific columns, excluding sensitive or irrelevant ones. Use this to load your initial dataset.

```python
df = pl.read_parquet("../examples/dependency.parquet").select(
    pl.exclude(["DOB", "Source", "Lead_Creation_Date", "City_Code", "Employer_Code"])
)
df.head()
```

--------------------------------

### Sample DataFrame by Count

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/sample_and_split.ipynb

Use the `sample` method with an integer argument to select a specific number of rows from a DataFrame. This is useful when a fixed-size sample is required.

```python
ss.sample(df, 30_000)  # by count
```

--------------------------------

### Configure Display Options

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Set various display options for dataframes, such as padding, colors, and fonts. These options affect how data is rendered in the output.

```python
from polars.config import Options

options = Options(
    column_line_width=OptionsInfo(scss=True, category='column', type='px', value='100px'),
    column_wrap_width=OptionsInfo(scss=True, category='column', type='px', value='100px'),
    column_spacing=OptionsInfo(scss=True, category='column', type='px', value='10px'),
    column_set_width=OptionsInfo(scss=True, category='column', type='px', value='100px'),
    column_set_wrap_width=OptionsInfo(scss=True, category='column', type='px', value='100px'),
    column_set_spacing=OptionsInfo(scss=True, category='column', type='px', value='10px'),
    row_height=OptionsInfo(scss=True, category='row', type='px', value='25px'),
    row_lines_width=OptionsInfo(scss=True, category='row', type='px', value='1px'),
    row_lines_color=OptionsInfo(scss=True, category='row', type='value', value='#D3D3D3'),
    source_notes_padding=OptionsInfo(scss=True, category='source_notes', type='px', value='4px'),
    source_notes_padding_horizontal=OptionsInfo(scss=True, category='source_notes', type='px', value='5px'),
    source_notes_background_color=OptionsInfo(scss=True, category='source_notes', type='value', value=None),
    source_notes_font_size=OptionsInfo(scss=True, category='source_notes', type='px', value='90%'),
    source_notes_border_bottom_style=OptionsInfo(scss=True, category='source_notes', type='value', value='none'),
    source_notes_border_bottom_width=OptionsInfo(scss=True, category='source_notes', type='px', value='2px'),
    source_notes_border_bottom_color=OptionsInfo(scss=True, category='source_notes', type='value', value='#D3D3D3'),
    source_notes_border_lr_style=OptionsInfo(scss=True, category='source_notes', type='value', value='none'),
    source_notes_border_lr_width=OptionsInfo(scss=True, category='source_notes', type='px', value='2px'),
    source_notes_border_lr_color=OptionsInfo(scss=True, category='source_notes', type='value', value='#D3D3D3'),
    source_notes_multiline=OptionsInfo(scss=False, category='source_notes', type='boolean', value=True),
    source_notes_sep=OptionsInfo(scss=False, category='source_notes', type='value', value=' '),
    row_striping_background_color=OptionsInfo(scss=True, category='row', type='value', value='rgba(128,128,128,0.05)'),
    row_striping_include_stub=OptionsInfo(scss=False, category='row', type='boolean', value=False),
    row_striping_include_table_body=OptionsInfo(scss=False, category='row', type='boolean', value=False),
    container_width=OptionsInfo(scss=False, category='container', type='px', value='auto'),
    container_height=OptionsInfo(scss=False, category='container', type='px', value='auto'),
    container_padding_x=OptionsInfo(scss=False, category='container', type='px', value='0px'),
    container_padding_y=OptionsInfo(scss=False, category='container', type='px', value='10px'),
    container_overflow_x=OptionsInfo(scss=False, category='container', type='overflow', value='auto'),
    container_overflow_y=OptionsInfo(scss=False, category='container', type='overflow', value='auto'),
    quarto_disable_processing=OptionsInfo(scss=False, category='quarto', type='logical', value=False),
    quarto_use_bootstrap=OptionsInfo(scss=False, category='quarto', type='logical', value=False))

options.set_global()
```

```

--------------------------------

### Batched ML Data Transformation with Polars

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/README.md

Demonstrates using the materialized pipeline for batched machine learning model updates. This approach is suitable for large datasets where processing can be done in batches, leveraging `collect_batches()` for efficiency.

```Python
for df_batch in pipe.transform(df, return_lazy=True).collect_batches():
    X_batch, y_batch = your_function_to_turn_df_batch_into_model_inputs(df_batch)
    ml_model.update(X_batch, y_batch)
```

--------------------------------

### Import Sklearn Pipeline Components

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/benchmarks/benchmarks.ipynb

Imports necessary classes from scikit-learn for building data processing pipelines.

```python
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
```

--------------------------------

### Import Polars DS Libraries

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/sample_and_split.ipynb

Imports necessary libraries for data manipulation and sampling with Polars.

```python
import polars as pl
import polars_ds as pds
import polars_ds.sample_and_split as ss
```

--------------------------------

### Plot Column Dependencies

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Visualizes the inferred dependencies between columns using a directed graph. This helps in understanding complex relationships within the data. The visualization is generated using the graphviz library. Be aware that some columns might be excluded from the analysis.

```python
dia.plot_dependency()
```

--------------------------------

### Create and Populate DataFrame

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/sample_and_split.ipynb

Generates a large DataFrame with various types of random data, including uniform, exponential, normal distributions, integers, and categorical data.

```python
df = pds.frame(size=100_000).with_columns(
    pds.random(0.0, 12.0).alias("uniform_1"),
    pds.random(0.0, 1.0).alias("urandom_colsniform_2"),
    pds.random_exp(0.5).alias("exp"),
    pds.random_normal(0.0, 1.0).alias("normal"),
    pds.random_normal(0.0, 1000.0).alias("fat_normal"),
    (pds.random_int(0, 3)).alias("flags"),
    pl.Series(["A"] * 30_000 + ["B"] * 30_000 + ["C"] * 40_000).alias("category"),
)
df.head()
```

--------------------------------

### Build a Streamable ML Data Transformation Pipeline

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/README.md

Constructs a data transformation pipeline using `polars_ds.pipeline.Blueprint`. This includes filtering, imputation, feature engineering (log, clip, sqrt, shift), scaling, one-hot encoding, and target encoding. The pipeline can be materialized for transformation.

```Python
import polars as pl
import polars.selectors as cs
from polars_ds.pipeline import Pipeline, Blueprint

bp = (
    Blueprint(df, name = "example", target = "approved", lowercase=True) # You can optionally 
    .filter(pl.col("city_category").is_not_null())
    .linear_impute(features = ["var1", "existing_emi"], target = "loan_period") 
    .impute(["existing_emi"], method = "median")
    .append_expr( # generate some features
        pl.col("existing_emi").log1p().alias("existing_emi_log1p"),
        pl.col("loan_amount").log1p().alias("loan_amount_log1p"),
        pl.col("loan_amount").clip(lower_bound = 0, upper_bound = 1000).alias("loan_amount_clipped"),
        pl.col("loan_amount").sqrt().alias("loan_amount_sqrt"),
        pl.col("loan_amount").shift(-1).alias("loan_amount_lead_1") # shift(-1) is a lead transform
    )
    .scale( # target is numerical, but will be excluded automatically because bp is initialzied with a target
        cs.numeric().exclude(["var1", "existing_emi_log1p"]), method = "standard"
    ) # Scale the columns up to this point. The columns below won't be scaled
    .append_expr(
        # Add missing flags
        pl.col("employer_category1").is_null().cast(pl.UInt8).alias("employer_category1_is_missing")
    )
    .one_hot_encode("gender", drop_first=True)
    .woe_encode("city_category") # No need to specify target because we initialized bp with a target
    .target_encode("employer_category1", min_samples_leaf = 20, smoothing = 10.0) # same as above
)

print(bp)

pipe:Pipeline = bp.materialize()
# Check out the result in our example notebooks! (examples/pipeline.ipynb)
df_transformed = pipe.transform(df)
df_transformed.head()
```

--------------------------------

### Time Series Pipeline Construction

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Constructs a time series pipeline using Blueprint for scaling, grouping, aggregating, and transforming time-series data. Requires Polars and NumPy.

```python
import numpy as np
import random

df_ts = pl.DataFrame(
    {
        "id": [1] * 9 + [2] * 15 + [3] * 6,
        "timestamp": list(range(9)) + list(range(15)) + list(range(6)),
        "var_1": np.random.rand(30),
        "var_2": np.random.rand(30),
        "target": random.choices([False, True], k=30),
    }
)

bp4 = (
    Blueprint(df_ts)
    .scale(["var_1", "var_2"], method="standard")
    .group_by_agg(
        by="id",
        maintain_order=True,
        agg=[
            "timestamp",
            pl.col("var_1").mul(10),
            pl.col("var_2").truediv(10),
            "target",
        ],
    )
    .explode(columns=pl.exclude("id"))
    .group_by_dynamic_agg(
        index_column="timestamp",
        every="3i",
        group_by="id",
        start_by="datapoint",
        agg=[
            pl.concat_list("var_1", "var_2").alias("features"),
            pl.col("target").sum(),
        ],
    )
    .with_columns(pl.col("features").cast(pl.Array(pl.Float64, (3, 2))))
)

pipe4 = bp4.materialize()
df_transformed4 = pipe4.transform(df_ts)
df_transformed4.head()
```

--------------------------------

### Load and Prepare Iris Dataset

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Loads the Iris dataset from scikit-learn, converts it to a Polars DataFrame, and maps the target integer labels to species names.

```python
import polars as pl
import polars_ds as pds
import altair as alt

# Only used to get dataset.
from sklearn import datasets

dataset = datasets.load_iris()
df = (
    pl.from_numpy(dataset.data, schema=dataset.feature_names)
    .with_columns(pl.Series(values=dataset.target).alias("species"))
    .with_columns(
        pl.when(pl.col("species") == 0)
        .then(pl.lit("setosa"))
        .when(pl.col("species") == 1)
        .then(pl.lit("versicolor"))
        .when(pl.col("species") == 2)
        .then(pl.lit("virginica"))
        .alias("species")
    )
)
df.head()
```

--------------------------------

### Append Step from Dictionary

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Appends a transformation step to a Blueprint using a dictionary configuration. The dictionary must contain a 'name' and optionally 'args' and 'kwargs'.

```python
bp = Blueprint(df, name="example", target="approved")

# Takes in a dict with 3 fields: `name`, `args`, and `kwargs`.
# Args and kwargs are optional depending on whether the method call needs certain arguments.
step_dict_1 = {"name": "impute", "kwargs": {"cols": ["Existing_EMI"], "method": "median"}}

# step_dict_2 = {"name": "does_not_exist", "kwargs": {"test": 1}}

# filter_step = {
#     "name": "filter",
#     "args": ["Employer_Category1 is not null"]
# }

bp.append_step_from_dict(step_dict_1)

# .append_step_from_dict(
#     filter_step
# )

# bp.append_step_from_dict(step_dict_2) # Will error
pipe = bp.materialize()
#
df_transformed = pipe.transform(df)
df_transformed.select(
    pl.col("Existing_EMI").null_count()  # Imputed. So 0
)
```

--------------------------------

### Define Polars DS Pipeline using Blueprint

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/benchmarks/benchmarks.ipynb

Constructs a data transformation pipeline using Polars DS Blueprint for imputation and scaling, then materializes it.

```python
bp = (
    Blueprint(df_pl, name="example_pipeline")
    .impute(["x4", "x5"], method="median")
    .scale(pl.all(), method="standard")
)

pipe = bp.materialize()  # bp.fit() also works
pipe.transform(df_pl).head(10)
```

--------------------------------

### Import Polars DS Pipeline Components

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/benchmarks/benchmarks.ipynb

Imports the Pipeline and Blueprint classes from the polars_ds.modeling.pipeline module.

```python
from polars_ds.modeling.pipeline import Pipeline, Blueprint
```

--------------------------------

### Generate Sample DataFrame for Pipeline Benchmarking

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/benchmarks/benchmarks.ipynb

Creates a Polars DataFrame with random features and introduces missing values for testing data transformation pipelines. Converts to Pandas DataFrame for comparison.

```python
# A random Dataframe with 50k records
size = 50_000
df_pl = (
    pds.frame(size=size)
    .select(
        pds.random(0.0, 1.0).alias("x1"),
        pds.random(0.0, 1.0).alias("x2"),
        pds.random(0.0, 1.0).alias("x3"),
    )
    .with_columns(
        x4=pl.when(pl.col("x3") > 0.3).then(None).otherwise(pl.col("x3")),
        x5=pl.when(pl.col("x2") > 0.5).then(None).otherwise(pl.col("x2")),
    )
)
df_pd = df_pl.to_pandas()
```

--------------------------------

### Import Polars and Polars DS

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/benchmarks/benchmarks.ipynb

Imports the necessary libraries for using Polars DataFrames and the Polars DS extension.

```python
import polars as pl
import polars_ds as pds
```

--------------------------------

### Read Parquet and Display Head

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Reads a Parquet file into a Polars DataFrame and displays the first 5 rows. Ensure the 'dependency.parquet' file is accessible.

```python
df = pl.read_parquet("dependency.parquet")
df.head()
```

--------------------------------

### Set RUSTFLAGS for Native CPU Features

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/README.md

Configure RUSTFLAGS to automatically detect and use native CPU features for compilation.

```bash
RUSTFLAGS=-C target-cpu=native
```

--------------------------------

### Generate Numerical Profile with Histogram

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Use `numeric_profile` to generate a detailed profile for numerical columns. Set `histogram=True` to include histogram data in the output.

```python
dia.numeric_profile(histogram=True)
```

--------------------------------

### Transform DataFrame with Performance Optimizations

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Applies a transformation to a DataFrame using a pipeline, with an option to tune performance during the collect phase. This snippet demonstrates how to use `pl.QueryOptFlags` for optimization.

```python
df_transformed = pipe.transform(
    df,
    # This is an optional parameter, which can be used to tune performance
    # during collect
    optimizations=pl.QueryOptFlags(),
)
df_transformed.head()
```

--------------------------------

### Load Data and Perform Numeric Profile

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Reads a parquet file into a Polars DataFrame and then uses the DIA library to generate a numeric profile. The `iqr_multiplier` parameter controls the sensitivity for outlier detection.

```python
df = pl.read_parquet("dependency.parquet")
df.head()

dia = DIA(df)
dia.numeric_profile(iqr_multiplier=2)
```

--------------------------------

### Materialize and Inspect Pipeline

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Materializes the data processing blueprint into a Pipeline object, optionally specifying Polars optimizations. The resulting pipeline's text representation is then displayed.

```python
# Materialize the blueprint
pipe: Pipeline = bp.materialize(
    # This is an optional parameter, which will be passed to .collect() 
    # when there is a fit step. User may decide which Polars optimization to use
    optimizations=pl.QueryOptFlags()
)
# Text representation of the pipeline
pipe
```

--------------------------------

### Materialize Blueprint and Return Lazy DataFrame

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Materializes a blueprint and returns the entire query plan for a DataFrame as a lazy object. This is useful for inspecting the query plan before execution.

```python
_df_lazy, _pipe = bp.materialize(return_df=True)
_df_lazy
```

--------------------------------

### Create a Sample Polars DataFrame

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Generates a large Polars DataFrame with various distributions (uniform, exponential, normal) and a concatenated list column. Use this to create sample data for EDA.

```python
df = (
    pds.frame(size=1_000_000)
    .select(
        pds.random(0.0, 12.0).alias("uniform_1"),
        pds.random(0.0, 1.0).alias("uniform_2"),
        pds.random_exp(0.5).alias("exp"),
        pds.random_normal(0.0, 1.0).alias("normal"),
        pds.random_normal(0.0, 1000.0).alias("fat_normal"),
    )
    .with_columns(pl.concat_list("uniform_2", 1 - pl.col("uniform_2")).alias("list_prob"))
)
df.head()
```

--------------------------------

### Create and Materialize a Data Transformation Pipeline

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/docs/index.md

Defines a data transformation pipeline using Blueprint for feature engineering, imputation, scaling, encoding, and then materializes it into a Pipeline object for transforming dataframes.

```Python
import polars as pl
import polars.selectors as cs
from polars_ds.pipeline import Pipeline, Blueprint

bp = (
    Blueprint(df, name = "example", target = "approved", lowercase=True) # You can optionally 
    .filter(pl.col("city_category").is_not_null())
    .linear_impute(features = ["var1", "existing_emi"], target = "loan_period") 
    .impute(["existing_emi"], method = "median")
    .append_expr( # generate some features
        pl.col("existing_emi").log1p().alias("existing_emi_log1p"),
        pl.col("loan_amount").log1p().alias("loan_amount_log1p"),
        pl.col("loan_amount").clip(lower_bound = 0, upper_bound = 1000).alias("loan_amount_clipped"),
        pl.col("loan_amount").sqrt().alias("loan_amount_sqrt"),
        pl.col("loan_amount").shift(-1).alias("loan_amount_lead_1") # shift(-1) is a lead transform
    )
    .scale( # target is numerical, but will be excluded automatically because bp is initialzied with a target
        cs.numeric().exclude(["var1", "existing_emi_log1p"])
    ) # Scale the columns up to this point. The columns below won't be scaled
    .append_expr(
        # Add missing flags
        pl.col("employer_category1").is_null().cast(pl.UInt8).alias("employer_category1_is_missing")
    )
    .one_hot_encode("gender", drop_first=True)
    .woe_encode("city_category") # No need to specify target because we initialized bp with a target
    .target_encode("employer_category1", min_samples_leaf = 20, smoothing = 10.0) # same as above
)

print(bp)

pipe:Pipeline = bp.materialize()
# Check out the result in our example notebooks! (examples/pipeline.ipynb)
df_transformed = pipe.transform(df)
df_transformed.head()
```

--------------------------------

### Configure Sklearn for Pandas Output

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/benchmarks/benchmarks.ipynb

Sets scikit-learn's configuration to output transformed data as Pandas DataFrames.

```python
from sklearn import set_config

set_config(transform_output="pandas")
```

--------------------------------

### Generate Linear Regression Report with Formulaic Input (f64)

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Use `lin_reg_report` with formulaic string inputs for features and a target column. This function provides a detailed statistical report of the regression. Configuration is set to use f64.

```python
df.select(
    pds.lin_reg_report(
        # formulaic input is also available for lstsq related queries,
        # or you can always use polars expressions, e.g. pl.col('x1') + 1, pl.col('x2').exp(), pl.col('x3').sin()
        "ln(x1+1)",
        "exp(x2)",
        "sin(x3)",
        target="y",
        add_bias=True,
    ).alias("report")
).unnest("report")
```

--------------------------------

### Import Polars DS and DIA

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/eda.ipynb

Imports the necessary Polars library and the DIA (Data Inspection Assistant) class from the polars_ds.eda.diagnosis module.

```python
import polars as pl
import polars_ds as pds
from polars_ds.eda.diagnosis import DIA
```

--------------------------------

### Check Initial Null Counts

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/pipeline.ipynb

Checks the initial null count for a specific column before any imputation or transformation is applied.

```python
df.select(pl.col("Existing_EMI").null_count())
```

--------------------------------

### Set RUSTFLAGS for SSE Features

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/README.md

Configure RUSTFLAGS to enable specific CPU target features for optimized compilation.

```bash
RUSTFLAGS=-C target-feature=+sse3,+ssse3,+sse4.1,+sse4.2,+popcnt,+cmpxchg16b
```

--------------------------------

### Compute Full-Length Real Fast Fourier Transform (RFFT)

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/examples/basics.ipynb

Computes the real Fast Fourier Transform (RFFT) and returns the full-length output, unlike the default compact representation.

```python
# FFT. But return the full length
df.select(pds.rfft("f", return_full=True)).shape
```

--------------------------------

### Import Polars DS Library

Source: https://github.com/abstractqqq/polars_ds_extension/blob/main/docs/index.md

Imports the Polars DS library for use in your Python scripts.

```python
import polars_ds as pds
```