### Install DataSieve Library

Source: https://github.com/emergentmethods/datasieve/blob/main/README.md

Provides instructions for installing the DataSieve library via pip or by cloning the repository for development.

```bash
pip install datasieve

git clone https://github.com/emergentmethods/datasieve.git
cd datasieve
poetry install
```

--------------------------------

### Create Custom Transforms with BaseTransform

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Illustrates how to create a custom data transformation by extending the BaseTransform class. The example implements an IQR-based outlier remover that can be integrated into DataSieve pipelines.

```Python
from datasieve.transforms.base_transform import BaseTransform
from datasieve.utils import remove_outliers
import numpy as np

class IQROutlierRemover(BaseTransform):
    """Custom transform that removes outliers using IQR method."""

    def __init__(self, multiplier=1.5, **kwargs):
        self.multiplier = multiplier
        self.lower_bounds = None
        self.upper_bounds = None

    def fit(self, X, y=None, sample_weight=None, feature_list=None, **kwargs):
        Q1 = np.percentile(X, 25, axis=0)
        Q3 = np.percentile(X, 75, axis=0)
        IQR = Q3 - Q1
        self.lower_bounds = Q1 - self.multiplier * IQR
        self.upper_bounds = Q3 + self.multiplier * IQR
        return X, y, sample_weight, feature_list

    def transform(self, X, y=None, sample_weight=None, feature_list=None,
                  outlier_check=False, **kwargs):
        # Identify inliers (all features within bounds)
        inliers = np.all((X >= self.lower_bounds) & (X <= self.upper_bounds), axis=1)
        inliers = inliers.astype(int)

        if not outlier_check:
            X, y, sample_weight = remove_outliers(X, y, sample_weight, inliers)
        else:
            y += inliers
            y -= 1

        return X, y, sample_weight, feature_list

# Use the custom transform in a pipeline
from datasieve.pipeline import Pipeline

X = np.random.randn(100, 3)
X[0:5] = 10  # Add outliers
y = np.arange(100)

pipeline = Pipeline([
    ("iqr_outlier", IQROutlierRemover(multiplier=1.5))
])

X_clean, y_clean, _ = pipeline.fit_transform(X, y)
print(f"Removed {len(X) - len(X_clean)} outliers using IQR method")
```

--------------------------------

### Build and Execute DataSieve Pipeline

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Demonstrates how to initialize a DataSieve Pipeline with multiple transformations, including custom wrappers and built-in outlier extractors. It shows the fit_transform, transform, and inverse_transform workflow for consistent data processing.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
from sklearn.preprocessing import MinMaxScaler
import pandas as pd
import numpy as np

X = pd.DataFrame(np.random.randn(100, 5), columns=['f1', 'f2', 'f3', 'f4', 'f5'])
y = pd.DataFrame(np.random.randn(100, 1), columns=['target'])
sample_weight = np.ones(100)

feature_pipeline = Pipeline([
    ("detect_constants", dst.VarianceThreshold(threshold=0)),
    ("pre_svm_scaler", dst.SKLearnWrapper(MinMaxScaler(feature_range=(-1, 1)))),
    ("svm", dst.SVMOutlierExtractor()),
    ("pca", dst.PCA(n_components=0.95)),
    ("post_pca_scaler", dst.SKLearnWrapper(MinMaxScaler(feature_range=(-1, 1))))
])

X_transformed, y_transformed, sw_transformed = feature_pipeline.fit_transform(X, y, sample_weight)
X_new = pd.DataFrame(np.random.randn(20, 5), columns=['f1', 'f2', 'f3', 'f4', 'f5'])
X_new_transformed, _, _ = feature_pipeline.transform(X_new)
X_recovered, _, _ = feature_pipeline.inverse_transform(X_new_transformed)
```

--------------------------------

### Dynamically Append Pipeline Steps

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Illustrates how to add transformation steps to an existing pipeline instance at runtime and verify their existence within the pipeline structure.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst

pipeline = Pipeline([])
pipeline.append(("variance", dst.VarianceThreshold(threshold=0.01)))
pipeline.append(("pca", dst.PCA(n_components=3)))
pipeline.append(("noise", dst.Noise(sigma=0.05)))

if "pca" in pipeline:
    pca_step = pipeline["pca"]
```

--------------------------------

### Fit and Transform Data with DataSieve Pipeline

Source: https://github.com/emergentmethods/datasieve/blob/main/README.md

Shows how to fit a DataSieve pipeline to input data (X, y, sample_weight) and then transform it. This process applies the defined sequence of transformations, including outlier removal and feature manipulation, ensuring consistent dimensions and feature names in the output.

```python
X, y, sample_weight = feature_pipeline.fit_transform(X, y, sample_weight)
```

--------------------------------

### Build DataSieve Pipeline with SKLearn Compatibility

Source: https://github.com/emergentmethods/datasieve/blob/main/README.md

Demonstrates how to construct a DataSieve Pipeline, incorporating both custom DataSieve transforms and standard scikit-learn transforms using SKLearnWrapper. The pipeline processes X, y, and sample_weight arrays, handling transformations like variance thresholding, outlier extraction, PCA, and scaling.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
from sklearn.preprocessing import MinMaxScaler

feature_pipeline = Pipeline([
    ("detect_constants", dst.VarianceThreshold(threshold=0)),
    ("pre_svm_scaler", dst.SKlearnWrapper(MinMaxScaler(feature_range=(-1, 1)))),
    ("svm", dst.SVMOutlierExtractor()),
    ("pca", dst.PCA(n_components=0.95)),
    ("post_pca_scaler", dst.SKlearnWrapper(MinMaxScaler(feature_range=(-1, 1))))
])
```

--------------------------------

### Transform New Data with Fitted DataSieve Pipeline

Source: https://github.com/emergentmethods/datasieve/blob/main/README.md

Illustrates how to use a previously fitted DataSieve pipeline to transform new datasets (X2). The transform method applies the learned transformations without refitting, returning the transformed data while optionally ignoring y and sample_weight.

```python
Xprime, _, _ = feature_pipeline.transform(X2)
```

--------------------------------

### Inverse Transform Data with DataSieve Pipeline

Source: https://github.com/emergentmethods/datasieve/blob/main/README.md

Demonstrates the inverse transformation capability of a DataSieve pipeline. This allows for reverting the transformations applied to a dataset (Xprime) back to its original or a prior state, useful for analysis or comparison.

```python
X2, _ ,_ = feature_pipeline.inverse_transform(Xprime)
```

--------------------------------

### Fit Pipeline Without Transformation

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Shows how to fit a pipeline to training data without applying the transformations immediately. This is useful for preparing the pipeline state before processing test data.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
import numpy as np

X_train = np.random.randn(100, 5)
y_train = np.random.randn(100)

pipeline = Pipeline([
    ("scaler", dst.SKLearnWrapper(MinMaxScaler())),
    ("svm", dst.SVMOutlierExtractor())
])

pipeline.fit(X_train, y_train)
X_test = np.random.randn(20, 5)
X_test_transformed, _, _ = pipeline.transform(X_test)
```

--------------------------------

### Create Training Data with Dissimilarity Filter

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Generates training and test data, then applies a DissimilarityIndex transform to filter out dissimilar points from the test set. This is useful for anomaly detection or focusing on data points similar to the training distribution.

```Python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
import numpy as np

X_train = np.random.randn(100, 5)
y_train = np.arange(100)

# Create test data - some similar, some very different
X_similar = np.random.randn(20, 5)  # Similar to training
X_dissimilar = np.random.randn(10, 5) * 10  # Very different
X_test = np.vstack([X_similar, X_dissimilar])
y_test = np.arange(len(X_test))

pipeline = Pipeline([
    ("di", dst.DissimilarityIndex(di_threshold=1.5, n_jobs=-1))
])

# Fit on training data
pipeline.fit(X_train, y_train)

# Transform test data - dissimilar points will be removed
X_filtered, y_filtered, _ = pipeline.transform(X_test, y_test)

print(f"Test samples: {len(X_test)}, After DI filter: {len(X_filtered)}")
print(f"Removed {len(X_test) - len(X_filtered)} dissimilar points")
```

--------------------------------

### Add Gaussian Noise to Training Data

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Demonstrates how to add Gaussian noise to training features during the fit_transform process using the Noise transform. Test data remains unaffected during the transform step. This is useful for data augmentation or simulating noisy environments.

```Python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
import numpy as np

X_train = np.random.randn(100, 5)
X_test = np.random.randn(20, 5)

pipeline = Pipeline([
    ("noise", dst.Noise(mu=0, sigma=0.1))
])

# Noise is added only during fit_transform
X_train_noisy, _, _ = pipeline.fit_transform(X_train.copy())

# No noise added during transform (test data stays clean)
X_test_clean, _, _ = pipeline.transform(X_test.copy())

print(f"Training data modified: {not np.allclose(X_train, X_train_noisy)}")
print(f"Test data unchanged: {np.allclose(X_test, X_test_clean)}")
print(f"Noise std applied: ~{np.std(X_train_noisy - X_train):.3f}")
```

--------------------------------

### SKLearnWrapper: Wrap Scikit-learn Transformers in DataSieve Pipeline

Source: https://context7.com/emergentmethods/datasieve/llms.txt

The SKLearnWrapper allows any scikit-learn transformer to be used within a DataSieve pipeline. It supports parallel processing configuration for transformers that benefit from it. It handles fitting, transforming, and inverse transforming data, ensuring compatibility with DataSieve's pipeline structure.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
import numpy as np

X = np.random.randn(100, 5)
y = np.random.randn(100)
sample_weight = np.ones(100)

# Wrap multiple sklearn transformers
pipeline = Pipeline([
    ("standard_scaler", dst.SKLearnWrapper(StandardScaler())),
    ("minmax_scaler", dst.SKLearnWrapper(MinMaxScaler(feature_range=(0, 1)))),
    ("robust_scaler", dst.SKLearnWrapper(RobustScaler(), n_jobs=4, backend="loky"))
])

X_transformed, y_out, sw_out = pipeline.fit_transform(X, y, sample_weight)

# Inverse transform works for invertible sklearn transforms
X_recovered, _, _ = pipeline.inverse_transform(X_transformed)
print(f"Recovery error: {np.mean(np.abs(X - X_recovered)):.6f}")
```

--------------------------------

### DBSCAN: Clustering-based Outlier Detection with Automatic Parameter Fitting

Source: https://context7.com/emergentmethods/datasieve/llms.txt

DBSCAN transform performs outlier detection using a density-based clustering approach. It automatically determines optimal epsilon and min_samples parameters by analyzing k-nearest neighbor distances, making it robust for various datasets. It effectively removes noise points that do not belong to any cluster.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
import numpy as np

# Create clustered data with outliers
np.random.seed(42)
X_cluster1 = np.random.randn(50, 3) + [0, 0, 0]
X_cluster2 = np.random.randn(50, 3) + [5, 5, 5]
X_outliers = np.random.randn(10, 3) * 10  # Outliers far from clusters
X = np.vstack([X_cluster1, X_cluster2, X_outliers])
y = np.arange(len(X))

pipeline = Pipeline([
    ("dbscan", dst.DBSCAN(n_jobs=-1, backend="loky"))
])

# DBSCAN automatically computes optimal eps and min_samples
X_clean, y_clean, _ = pipeline.fit_transform(X, y)

print(f"Original samples: {len(X)}, After DBSCAN: {len(X_clean)}")
print(f"Removed {len(X) - len(X_clean)} outliers detected by clustering")
```

--------------------------------

### SKLearnWrapper

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Wraps standard scikit-learn transformers to integrate them into DataSieve pipelines with support for parallel processing.

```APIDOC
## SKLearnWrapper

### Description
Wraps any scikit-learn transformer to work with the DataSieve pipeline, with optional parallel backend configuration.

### Method
N/A (Class-based Transformer)

### Parameters
#### Constructor Parameters
- **transformer** (object) - Required - An initialized scikit-learn transformer.
- **n_jobs** (int) - Optional - Number of jobs for parallel processing.
- **backend** (str) - Optional - Parallel backend (e.g., 'loky').

### Request Example
```python
from datasieve.transforms import SKLearnWrapper
from sklearn.preprocessing import StandardScaler
wrapper = SKLearnWrapper(StandardScaler(), n_jobs=4)
```
```

--------------------------------

### Identify Outliers with Outlier Check

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Demonstrates the use of the outlier_check parameter to detect outliers without removing them from the dataset. Returns a binary vector where 0 indicates an outlier and 1 indicates an inlier.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
from sklearn.preprocessing import MinMaxScaler
import numpy as np

X_train = np.random.randn(100, 5)
pipeline = Pipeline([
    ("pre_svm_scaler", dst.SKLearnWrapper(MinMaxScaler())),
    ("svm", dst.SVMOutlierExtractor())
])

pipeline.fit(X_train)
X_test = np.random.randn(50, 5)
X_result, outlier_flags, _ = pipeline.transform(X_test, outlier_check=True)
```

--------------------------------

### DBSCAN

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Clustering-based outlier detection that automatically fits epsilon and min_samples parameters.

```APIDOC
## DBSCAN

### Description
Clustering-based outlier detection with automatic epsilon and min_samples parameter fitting using the elbow method on k-nearest neighbor distances.

### Parameters
#### Constructor Parameters
- **n_jobs** (int) - Optional - Number of parallel jobs.
- **backend** (str) - Optional - Parallel backend.
```

--------------------------------

### Implement Custom Outlier Extractor Transform

Source: https://github.com/emergentmethods/datasieve/blob/main/README.md

Defines a custom transform class inheriting from BaseTransform that wraps SGDOneClassSVM. It implements fit and transform methods to identify and remove outliers from X, y, and sample_weight.

```python
class SVMOutlierExtractor(BaseTransform):
    def __init__(self, **kwargs):
        self._skl = SGDOneClassSVM(**kwargs)

    def fit_transform(self, X, y=None, sample_weight=None, feature_list=None, **kwargs):
        self.fit(X, y, sample_weight=sample_weight)
        return self.transform(X, y, sample_weight, feature_list)

    def fit(self, X, y=None, sample_weight=None, feature_list=None, **kwargs):
        self._skl.fit(X, y=y, sample_weight=sample_weight)
        return X, y, sample_weight, feature_list

    def transform(self, X, y=None, sample_weight=None, feature_list=None,
                  outlier_check=False, **kwargs):
        y_pred = self._skl.predict(X)
        y_pred = np.where(y_pred == -1, 0, y_pred)
        if not outlier_check:
            X, y, sample_weight = remove_outliers(X, y, sample_weight, y_pred)
        else:
            y += y_pred
            y -= 1
        return X, y, sample_weight, feature_list
```

--------------------------------

### Perform Outlier Detection without Removal

Source: https://github.com/emergentmethods/datasieve/blob/main/README.md

Demonstrates how to use a fitted DataSieve pipeline to flag outliers using the outlier_check parameter. This returns a vector indicating inliers and outliers without modifying the input dataset.

```python
pipeline = Pipeline([
    ("pre_svm_scaler", transforms.DataSieveMinMaxScaler()),
    ("svm", transforms.SVMOutlierExtractor())
])

pipeline.fit(X)

X, outliers, _ = pipeline.transform(X, outlier_check=True)
```

--------------------------------

### Utility Function for Removing Outliers

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Demonstrates the usage of the `remove_outliers` utility function, which synchronizes the removal of rows across multiple arrays (X, y, sample_weight) based on a provided inlier mask.

```Python
from datasieve.utils import remove_outliers
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6], [100, 100], [7, 8]])
y = np.array([0, 1, 2, 3, 4])
sample_weight = np.array([1.0, 1.0, 1.0, 1.0, 1.0])

# Binary mask: 1 = inlier, 0 = outlier
inliers = np.array([1, 1, 1, 0, 1])  # Mark row 3 as outlier

X_clean, y_clean, sw_clean = remove_outliers(X, y, sample_weight, inliers)

print(f"X_clean:\n{X_clean}")
print(f"y_clean: {y_clean}")
print(f"Removed indices: {np.where(inliers == 0)[0]}")
```

--------------------------------

### PCA: Principal Component Analysis with Feature Renaming

Source: https://context7.com/emergentmethods/datasieve/llms.txt

This PCA transform performs Principal Component Analysis and automatically renames the resulting components to PC0, PC1, etc. This ensures that feature names are maintained correctly throughout the pipeline, especially when working with pandas DataFrames. It supports specifying the number of components or the variance to retain.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
import pandas as pd
import numpy as np

# Create DataFrame with named features
X = pd.DataFrame(
    np.random.randn(100, 10),
    columns=[f'feature_{i}' for i in range(10)]
)
y = pd.DataFrame(np.random.randn(100, 1), columns=['target'])

pipeline = Pipeline([
    ("pca", dst.PCA(n_components=0.95))  # Keep 95% variance
])

X_transformed, y_out, _ = pipeline.fit_transform(X, y)

print(f"Original features: {X.shape[1]}, PCA components: {X_transformed.shape[1]}")
print(f"New feature names: {list(X_transformed.columns)}")
# Output: ['PC0', 'PC1', 'PC2', ...]

# Inverse transform to recover original feature space
X_recovered, _, _ = pipeline.inverse_transform(X_transformed)
print(f"Recovered shape: {X_recovered.shape}")
```

--------------------------------

### SVMOutlierExtractor: Detect and Remove Outliers with SGDOneClassSVM

Source: https://context7.com/emergentmethods/datasieve/llms.txt

SVMOutlierExtractor utilizes scikit-learn's SGDOneClassSVM to identify and remove outliers from datasets. It ensures that corresponding entries in X, y, and sample_weight are removed consistently. This transform is useful for cleaning data before further analysis or model training.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
from sklearn.preprocessing import MinMaxScaler
import numpy as np

# Create data with some outliers
X = np.random.randn(100, 5)
X[0:5] = X[0:5] * 10  # Add outliers
y = np.arange(100)
sample_weight = np.ones(100)

pipeline = Pipeline([
    ("scaler", dst.SKLearnWrapper(MinMaxScaler(feature_range=(-1, 1)))),
    ("svm_outlier", dst.SVMOutlierExtractor(nu=0.1, tol=1e-4))
])

X_clean, y_clean, sw_clean = pipeline.fit_transform(X, y, sample_weight)

print(f"Original samples: {len(X)}, After outlier removal: {len(X_clean)}")
print(f"Removed {len(X) - len(X_clean)} outliers")
# y and sample_weight have the same rows removed
print(f"y shape matches X: {len(y_clean) == len(X_clean)}")
```

--------------------------------

### PCA

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Principal Component Analysis that maintains and renames feature columns to PC0, PC1, etc.

```APIDOC
## PCA

### Description
Principal Component Analysis that properly renames feature columns to PC0, PC1, etc., maintaining feature names throughout the pipeline.

### Parameters
#### Constructor Parameters
- **n_components** (float/int) - Optional - Number of components to keep or variance ratio.
```

--------------------------------

### SVMOutlierExtractor

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Detects and removes outliers using SGDOneClassSVM, ensuring consistent row removal across X, y, and sample_weight.

```APIDOC
## SVMOutlierExtractor

### Description
Detects and removes outliers using scikit-learn's SGDOneClassSVM, propagating row removals across X, y, and sample_weight arrays.

### Parameters
#### Constructor Parameters
- **nu** (float) - Optional - An upper bound on the fraction of training errors and a lower bound of the fraction of support vectors.
- **tol** (float) - Optional - Tolerance for stopping criterion.

### Response
- **X_clean** (array) - Filtered feature matrix.
- **y_clean** (array) - Filtered target array.
```

--------------------------------

### DissimilarityIndex: Filter Data Points Based on Pairwise Distances

Source: https://context7.com/emergentmethods/datasieve/llms.txt

The DissimilarityIndex transform filters out data points that are excessively dissimilar from the main distribution of the training data. It achieves this by calculating pairwise distances and removing points that fall outside a defined dissimilarity threshold. This is useful for removing anomalous or distant data points.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
import numpy as np
```

--------------------------------

### VarianceThreshold

Source: https://context7.com/emergentmethods/datasieve/llms.txt

Removes low-variance features from the dataset while tracking feature names.

```APIDOC
## VarianceThreshold

### Description
Removes low-variance features and properly tracks feature names through the pipeline.

### Parameters
#### Constructor Parameters
- **threshold** (float) - Optional - Features with variance lower than this threshold will be removed.
```

--------------------------------

### VarianceThreshold: Remove Low-Variance Features While Tracking Names

Source: https://context7.com/emergentmethods/datasieve/llms.txt

VarianceThreshold removes features with variance below a specified threshold. It is designed to work seamlessly with pandas DataFrames, properly tracking and updating feature names after removal. This is crucial for maintaining interpretability and ensuring correct column alignment in subsequent pipeline steps.

```python
from datasieve.pipeline import Pipeline
import datasieve.transforms as dst
import pandas as pd
import numpy as np

# Create data with some constant/low-variance features
X = pd.DataFrame({
    'constant': np.ones(100),              # Zero variance - will be removed
    'low_var': np.random.randn(100) * 0.001,  # Low variance
    'high_var_1': np.random.randn(100),
    'high_var_2': np.random.randn(100) * 2,
    'high_var_3': np.random.randn(100) * 3
})

pipeline = Pipeline([
    ("variance_filter", dst.VarianceThreshold(threshold=0.01))
])

X_filtered, _, _ = pipeline.fit_transform(X)

print(f"Original features: {list(X.columns)}")
print(f"Remaining features: {list(X_filtered.columns)}")
print(f"Removed {X.shape[1] - X_filtered.shape[1]} low-variance features")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.