### Launch PostgreSQL Instance

Source: https://github.com/xorq-labs/xorq/blob/main/examples/README.md

This command starts a PostgreSQL database instance, which is required for certain project functionalities or examples.

```bash
just up postgres
```

--------------------------------

### Setup Dev Environment with uv

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Instructions for setting up a development environment using uv, including cloning the repo, configuring uv to not sync, installing dependencies and the package, activating the venv, and setting up pre-commit hooks.

```bash
# fetch this repo
git clone git@github.com:xorq-labs/xorq.git
# set uv run command to not sync 
export UV_NO_SYNC=1
# prepare development environment and install dependencies (including current package)
uv sync --all-extras --all-groups
# activate the venv
source venv/bin/activate
# set up the git hook scripts
uv run pre-commit install
```

--------------------------------

### Setup Dev Environment with Pip and Venv

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Steps to clone the repository, create and activate a Python virtual environment using venv, install dependencies from requirements-dev.txt, and set up pre-commit hooks.

```bash
# fetch this repo
git clone git@github.com:xorq-labs/xorq.git
# prepare development environment (used to build wheel / install in development)
python3 -m venv venv
# activate the venv
source venv/bin/activate
# update pip itself if necessary
python -m pip install -U pip
# install dependencies 
python -m pip install -r requirements-dev.txt
# install current package in editable mode
python -m pip install -e .
# set up the git hook scripts
pre-commit install
```

--------------------------------

### Setup Dependencies with Pixi

Source: https://github.com/xorq-labs/xorq/blob/main/python/xorq/vendor/ibis/examples/README.md

Enter a shell environment with the required dependencies to run Ibis example generation scripts. This command ensures all necessary tools and libraries are available.

```shell
pixi shell
```

--------------------------------

### Install xorq using pip

Source: https://github.com/xorq-labs/xorq/blob/main/docs/bank-marketing.mdx

Installs the xorq library using pip, the standard Python package installer. This is the primary method for adding xorq to your Python environment.

```bash
pip install xorq
```

--------------------------------

### Install Xorq with Examples

Source: https://github.com/xorq-labs/xorq/blob/main/README.md

Installs the Xorq Python package along with necessary dependencies for running examples. This command sets up the environment for exploring Xorq's capabilities.

```bash
pip install xorq[examples]
```

--------------------------------

### Run xorq with Nix

Source: https://github.com/xorq-labs/xorq/blob/main/docs/bank-marketing.mdx

Executes xorq within a Nix shell environment. This is useful for reproducible development and testing without global installations.

```bash
nix run github:xorq-labs/xorq
```

--------------------------------

### XORQ CLI Run Command

Source: https://github.com/xorq-labs/xorq/blob/main/docs/bank-marketing.mdx

Demonstrates how to execute a previously saved XORQ pipeline execution plan using the command-line interface. This command points to the generated build artifact.

```bash
xorq run builds/3a81bd906a6d
```

--------------------------------

### Download Test Data

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Command to download the necessary example data required for successfully running the project's test suite.

```bash
just download-data
```

--------------------------------

### Download Project Data

Source: https://github.com/xorq-labs/xorq/blob/main/examples/README.md

This command downloads the necessary testing data for the project. It is a prerequisite for running other examples.

```bash
just download-data
```

--------------------------------

### Quick Start ML Pipeline with xorq

Source: https://github.com/xorq-labs/xorq/blob/main/docs/bank-marketing.mdx

Builds a simple machine learning pipeline to predict bank deposit subscriptions. It demonstrates deferred data loading, transformation, train-test splitting, and fitting a scikit-learn RandomForestClassifier.

```python
import xorq as xo
from xorq.common.utils.defer_utils import deferred_read_csv
from xorq.expr.ml import deferred_fit_predict_sklearn, train_test_splits
import xorq.expr.ibis.expr.datatypes as dt
from sklearn.ensemble import RandomForestClassifier

con = xo.connect()

dataset_name = "bank-marketing"
csv_path = xo.options.pins.get_path(dataset_name)
bank_data = deferred_read_csv(path=csv_path, con=con)

bank_data = bank_data.mutate(
    row_number=xo.row_number(),
    deposit=(bank_data["deposit"] == "yes").cast("int")
)

train, test = train_test_splits(
    bank_data, 
    unique_key="row_number", 
    test_sizes=[0.7, 0.3],
    random_seed=42
)

deferred_model, model_udaf, predict_fn = deferred_fit_predict_sklearn(
    expr=train,
    target="deposit",
    features=["age", "balance", "duration"],
    cls=RandomForestClassifier,
    return_type=dt.float64,
    name="rf_prediction"
)

results = test.mutate(prediction=predict_fn.on_expr(test))

results.select(["deposit", "prediction"]).execute()
```

--------------------------------

### Run Test Suite

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Instructions to run the project's test suite, which may involve starting PostgreSQL and then executing pytest. Ensure the development virtual environment is activated first.

```bash
# make sure you activate the venv using "source venv/bin/activate" first
just up postgres # some of the tests use postgres
python -m pytest # or pytest
```

--------------------------------

### Release Flow: Create Release Branch

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Steps for maintainers to create a new release branch, starting from upstream main, and preparing for version updates and tagging.

```git
git switch main && git pull
# Compute the new version number ($version_number) according to [Semantic Versioning](https://semver.org/) rules.
git switch --create=release-$version_number
```

--------------------------------

### XORQ CLI Build Command

Source: https://github.com/xorq-labs/xorq/blob/main/docs/bank-marketing.mdx

Shows how to use the XORQ command-line interface to build a pipeline and save its execution plan. The command specifies the Python script, the expression to build, and the directory for build artifacts.

```bash
xorq build examples/bank_marketing.py -e "predictions_expr" --builds-dir builds
```

--------------------------------

### Run Local Cache Script

Source: https://github.com/xorq-labs/xorq/blob/main/examples/README.md

This command executes the local cache Python script. Ensure all dependencies and prerequisite services are running before execution.

```python
python local_cache.py
```

--------------------------------

### Initialize Xorq and Import Utilities

Source: https://github.com/xorq-labs/xorq/blob/main/python/xorq/tests/fixtures/pipeline.ipynb

Demonstrates importing the xorq library and essential utility functions for deferred reading and backend integration. Sets xorq to interactive mode.

```python
import xorq as xo
from xorq.common.utils.defer_utils import deferred_read_parquet
from xorq.expr.relations import into_backend


xo.options.interactive = True
```

--------------------------------

### Deferred One-Hot Encoding with xorq

Source: https://github.com/xorq-labs/xorq/blob/main/docs/bank-marketing.mdx

Demonstrates creating a fully deferred one-hot encoding transformation using scikit-learn's OneHotEncoder and xorq's deferred execution capabilities. This allows for lazy application of the encoding process.

```python
import functools
import xorq as xo
import xorq.selectors as s
import xorq.vendor.ibis.expr.datatypes as dt
from xorq.expr.ml import deferred_fit_transform, train_test_splits
from sklearn.preprocessing import OneHotEncoder
import toolz
import pandas as pd

# Create a fully deferred one-hot encoding transformation
def fit(
    df,
    cls=functools.partial(OneHotEncoder, handle_unknown="ignore", drop="first"),
    features=slice(None),
):
    model = cls().fit(df[features])
    return model

@toolz.curry
def transform(model, df, features=slice(None)):
    names = model.get_feature_names_out()
    return pd.Series(
        (
            tuple({"key": key, "value": float(value)} for key, value in zip(names, row))
            for row in model.transform(df[features]).toarray()
        )
    )

# Define the return type for our encoding transformation
return_type = dt.Array(dt.Struct({"key": str, "value": float}))
deferred_one_hot = deferred_fit_transform(
    fit=fit,
    transform=transform,
    return_type=return_type,
)
```

--------------------------------

### Generate Ibis Registry and Examples

Source: https://github.com/xorq-labs/xorq/blob/main/python/xorq/vendor/ibis/examples/README.md

Execute the Python script to generate the Ibis registry and examples. The `-d` flag can be used to prevent uploading results to the examples bucket, useful for testing new functions.

```python
python ibis/examples/gen_registry.py
```

```python
python ibis/examples/gen_registry.py -d
```

--------------------------------

### Train/Test Splitting Utility

Source: https://github.com/xorq-labs/xorq/blob/main/docs/bank-marketing.mdx

Demonstrates the usage of the `train_test_splits` function for creating robust train/test splits in a dataset. It specifies parameters for unique identification, split proportions, number of buckets, and reproducibility.

```python
train, test = train_test_splits(
    data, 
    unique_key="id",          # Column to uniquely identify rows
    test_sizes=[0.7, 0.3],    # Split proportions
    num_buckets=2,            # Number of splits
    random_seed=42            # For reproducibility
)
```

--------------------------------

### Release Flow: Push Branch and Open PR

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Steps for maintainers to push the newly created release branch to the upstream repository and open a pull request for it.

```git
git push --set-upstream upstream "release-$version_number"
# Open a PR for the new branch `release-$version_number`
```

--------------------------------

### Release Flow: Create GitHub Release

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Final step for maintainers to create a GitHub release from the newly pushed tag, which triggers the publishing workflow.

```git
# Create a [GitHub release](https://github.com/xorq-labs/xorq/releases/new) to trigger the publishing workflow.
```

--------------------------------

### Deferred Model Training Utility

Source: https://github.com/xorq-labs/xorq/blob/main/docs/bank-marketing.mdx

Illustrates how to use `deferred_fit_predict` to create models that train on demand. This function takes training data, target variable, features, model class, and a name for the prediction column.

```python
deferred_model, model_udaf, predict = deferred_fit_predict(
    expr=train_data,          # Training data
    target="target_column",   # Target variable
    features=feature_list,    # List of features
    cls=ModelClass,           # Scikit-learn compatible model class
    name="prediction_column"  # Name for prediction column
)
```

--------------------------------

### Build ML Pipeline Function

Source: https://github.com/xorq-labs/xorq/blob/main/docs/bank-marketing.mdx

Defines a Python function to construct a complete machine learning pipeline using XORQ. It handles data loading, train/test splitting, one-hot encoding, model fitting (XGBoost), and prediction generation.

```python
import functools
import xorq as xo
import xorq.selectors as s
import ibis.expr.types as expr
import ibis.expr.datatypes as dt

# Assuming XGBoostModelExplodeEncoded is defined elsewhere
# class XGBoostModelExplodeEncoded:
#     def __init__(self, encoded_col):
#         self.encoded_col = encoded_col
#     # ... other methods

def make_pipeline_exprs(dataset_name, target_column, predicted_col):
    ROW_NUMBER = "row_number"
    ENCODED = "encoded"

    # Create connection and load data
    con = xo.connect()
    
    # Read CSV and create train/test split
    train_table, test_table = (
        expr.drop(ROW_NUMBER)
        for expr in (
            xo.deferred_read_csv(
                path=xo.options.pins.get_path(dataset_name),
                con=con,
            )
            .mutate(
                **{
                    target_column: (xo._[target_column] == "yes").cast("int"),
                    ROW_NUMBER: xo.row_number(),
                }
            )
            .pipe(
                xo.train_test_splits, # Assuming train_test_splits is part of xo
                unique_key=ROW_NUMBER,
                test_sizes=[0.5, 0.5],
                num_buckets=2,
                random_seed=42,
            )
        )
    )

    # Apply one-hot encoding to string columns
    deferred_encoder, encoder_udaf, transform_fn = xo.deferred_one_hot(
        train_table,
        features=train_table.select(s.of_type(str)).columns,
    )
    
    # Encode both train and test data
    encoded_train, encoded_test = (
        expr.mutate(**{ENCODED: transform_fn.on_expr(expr)})
        for expr in (train_table, test_table)
    )

    # Get numeric features for model training
    numeric_features = [
        col
        for col in encoded_train.select(s.numeric()).columns
        if col != target_column and col != target_column + "_yes"
    ]
    
    # Create and train the model
    deferred_model, model_udaf, predict_fn = xo.deferred_fit_predict_sklearn(
        expr=encoded_train,
        target=target_column,
        features=numeric_features + [ENCODED],
        cls=functools.partial(XGBoostModelExplodeEncoded, encoded_col=ENCODED),
        return_type=dt.float64,
        name="xgb_prediction",
    )
    
    # Apply predictions to test data
    predictions = encoded_test.mutate(
        **{predicted_col: predict_fn.on_expr(encoded_test)}
    ).drop(ENCODED)

    return {
        "encoded_train": encoded_train,
        "encoded_test": encoded_test,
        "predictions": predictions,
        "encoder": deferred_encoder,
        "model": deferred_model,
    }

# Example usage:
dataset_name = "bank-marketing"
target_column = "deposit"
predicted_col = "predicted"
results = make_pipeline_exprs(dataset_name, target_column, predicted_col)

# Execute the pipeline and evaluate results
# predictions_df = results["predictions"].execute()

```

--------------------------------

### Conventional Commits Structure Example

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Illustrates the Conventional Commits structure for commit messages, including common types like 'fix', 'feat', 'docs', 'style', and how to reference GitHub issues in the commit description.

```git
fix(types): make all floats doubles

This commit addresses an issue where float types were not consistently handled, ensuring they are all treated as doubles for better precision.

fixes #4242
```

--------------------------------

### Connect to PostgreSQL and Load Table

Source: https://github.com/xorq-labs/xorq/blob/main/python/xorq/tests/fixtures/pipeline.ipynb

Shows how to establish a connection to a PostgreSQL database using xorq and load a specific table into a DataFrame-like object for further operations.

```python
pg = xo.postgres.connect(
    host="localhost",
    port=5432,
    user="postgres",
    password="postgres",
    database="ibis_testing",
)
batting = pg.table("batting")
batting
```

--------------------------------

### Release Flow: Trigger Pre-release Action

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Instructions for maintainers to trigger the CI pre-release action from the created release branch to run tests and checks before merging.

```git
# Trigger the [ci-pre-release action](https://github.com/xorq-labs/xorq/actions/workflows/ci-pre-release.yml) from the branch created: Run workflow -> Use workflow from -> Branch `$version_number`
```

--------------------------------

### Release Flow: Update Version and Changelog

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Steps for maintainers to update the project version in pyproject.toml, generate the CHANGELOG using git cliff, and commit these changes.

```git
# Update the version number in pyproject.toml: version = "$version_number"
# Update the CHANGELOG using `git cliff --github-repo xorq-labs/xorq -p CHANGELOG.md --tag v$version_number -u`, manually add any additional notes (links to blogposts, etc.).
git add --update && git commit -m "release: $version_number"
```

--------------------------------

### Release Flow: Merge PR and Tag Release

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Steps for maintainers to squash and merge the release PR, then tag the updated main branch with the new version number and push the tags.

```git
# "Squash and merge" the PR
git fetch && git tag v$version_number origin/main && git push --tags
```

--------------------------------

### Configure Test Environment Variables

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Sets essential environment variables for the test suite, specifically configuring PostgreSQL connection details such as database name, host, user, password, and port.

```bash
export POSTGRES_DATABASE=ibis_testing
export POSTGRES_HOST=localhost
export POSTGRES_USER=postgres
export POSTGRES_PASSWORD=postgres
export POSTGRES_PORT=5432
```

--------------------------------

### Filter PostgreSQL Table by Year

Source: https://github.com/xorq-labs/xorq/blob/main/python/xorq/tests/fixtures/pipeline.ipynb

Filters the 'batting' table (connected via PostgreSQL) to select rows where the 'yearID' column equals 2015.

```python
left = batting.filter(batting.yearID == 2015)
left
```

--------------------------------

### Initialize Xorq Project

Source: https://github.com/xorq-labs/xorq/blob/main/README.md

Initializes a new Xorq project using a specified template. The '-t penguins' argument indicates the use of the 'penguins' dataset template for project setup.

```bash
xorq init -t penguins
```

--------------------------------

### Configure Local Dependency Path in pyproject.toml

Source: https://github.com/xorq-labs/xorq/blob/main/CONTRIBUTING.md

Adds a local path configuration for the 'xorq-datafusion' dependency within the pyproject.toml file, enabling development with a local copy of the dependency repository.

```toml
[tool.uv.sources]
xorq-datafusion = { path = "local/path/to/xorq-datafusion-repo" }
```