Kedro

https://docs.kedro.org/en/1.0.0/
Admin
Kedro is a toolbox for production-ready data science that provides a framework for building modular,...
Tokens:171,075
Snippets:1,853
Trust Score:7.7
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install and run Kedro-Viz

Source: https://docs.kedro.org/en/1.0.0/create/new_project

Install the Kedro-Viz package and start the visualization server.

```bash
uv pip install kedro-viz
```

```bash
kedro viz run
```

--------------------------------

### Install and launch MLflow UI

Source: https://docs.kedro.org/en/1.0.0/integrations-and-plugins/mlflow

Install the MLflow package and start the tracking server to record metadata to a local directory.

```bash
(.venv) $ pip install mlflow
(.venv) $ mlflow ui --backend-store-uri ./mlflow_runs
```

--------------------------------

### Include example pipeline

Source: https://docs.kedro.org/en/1.0.0/create/new_project

The CLI prompts to include starter example code after tool selection.

```text
Would you like to include an example pipeline? :
 (no):
```

--------------------------------

### Commented out setup cells

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/databricks/databricks_notebooks_development_workflow

Example of how to comment out setup cells to prevent redundant execution.

```python
#dbutils.fs.cp(
#    "file:///Workspace/Repos/<databricks_username>/iris-databricks/data",
#    "dbfs:/FileStore/iris-databricks/data",
#    recurse=True,
#)
```

```python
#dbutils.fs.ls("dbfs:/FileStore/iris-databricks/data")
```

--------------------------------

### Install Kedro-Viz

Source: https://docs.kedro.org/en/1.0.0/tutorials/create_a_pipeline

Install the Kedro-Viz package into your virtual environment.

```bash
pip install kedro-viz
```

--------------------------------

### Select example code directly via command line

Source: https://docs.kedro.org/en/1.0.0/create/new_project_tools

Use the --example flag to automatically accept or decline the inclusion of the spaceflights starter pipeline.

```bash
uvx kedro new --example=y
```

--------------------------------

### setup(metadata, /, args, **kwargs)

Source: https://docs.kedro.org/en/1.0.0/api/framework/kedro.framework.cli

Initializes the Jupyter Kernel for a Kedro project.

```APIDOC
## setup(metadata, /, args, **kwargs)

### Description
Initializes the Jupyter Kernel for a Kedro project, creating a kernel named based on the project package name.

### Parameters
- **metadata** (ProjectMetadata) - Required - The project metadata object.
- **args** (Any) - Required - Arguments for the setup process.
- **kwargs** (Any) - Optional - Additional keyword arguments.
```

--------------------------------

### Create spaceflights project (interactive vs single-line)

Source: https://docs.kedro.org/en/1.0.0/create/new_project

Create a project named spaceflights with test setup and example code.

```bash
kedro new ⮐
spaceflights ⮐
2 ⮐
yes ⮐
```

```bash
uvx kedro new --name=spaceflights --tools=test --example=y
```

--------------------------------

### Refactored Notebook Code Example

Source: https://docs.kedro.org/en/1.0.0/integrations-and-plugins/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook

A complete example showing the setup of Kedro configuration, data loading, data processing functions, and model training workflow within a single notebook context.

```python
# Kedro setup for data management and configuration
from kedro.config import OmegaConfigLoader
from kedro.io import DataCatalog

conf_loader = OmegaConfigLoader(conf_source=".")
conf_catalog = conf_loader["catalog"]
conf_params = conf_loader["parameters"]

# Create the DataCatalog instance from the configuration
catalog = DataCatalog.from_config(conf_catalog)

# Load the datasets
companies = catalog.load("companies")
reviews = catalog.load("reviews")
shuttles = catalog.load("shuttles")

# Load the configuration data
test_size = conf_params["model_options"]["test_size"]
random_state = conf_params["model_options"]["random_state"]


####################
# Data processing  #
####################
import pandas as pd


def _is_true(x: pd.Series) -> pd.Series:
    return x == "t"


def _parse_percentage(x: pd.Series) -> pd.Series:
    x = x.str.replace("%", "")
    x = x.astype(float) / 100
    return x


def _parse_money(x: pd.Series) -> pd.Series:
    x = x.str.replace("$", "").str.replace(",", "")
    x = x.astype(float)
    return x


def preprocess_companies(companies: pd.DataFrame) -> pd.DataFrame:
    companies["iata_approved"] = _is_true(companies["iata_approved"])
    companies["company_rating"] = _parse_percentage(companies["company_rating"])
    return companies


def preprocess_shuttles(shuttles: pd.DataFrame) -> pd.DataFrame:
    shuttles["d_check_complete"] = _is_true(shuttles["d_check_complete"])
    shuttles["moon_clearance_complete"] = _is_true(shuttles["moon_clearance_complete"])
    shuttles["price"] = _parse_money(shuttles["price"])
    return shuttles


def create_model_input_table(
    shuttles: pd.DataFrame, companies: pd.DataFrame, reviews: pd.DataFrame
) -> pd.DataFrame:
    rated_shuttles = shuttles.merge(reviews, left_on="id", right_on="shuttle_id")
    model_input_table = rated_shuttles.merge(
        companies, left_on="company_id", right_on="id"
    )
    model_input_table = model_input_table.dropna()
    return model_input_table


##################################
# Model training and evaluation  #
##################################

from typing import Dict, Tuple
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split


def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
    X = data[parameters["features"]]
    y = data["price"]
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=parameters["test_size"], random_state=parameters["random_state"]
    )
    return X_train, X_test, y_train, y_test


def train_model(X_train: pd.DataFrame, y_train: pd.Series) -> LinearRegression:
    regressor = LinearRegression()
    regressor.fit(X_train, y_train)
    return regressor


def evaluate_model(
    regressor: LinearRegression, X_test: pd.DataFrame, y_test: pd.Series
):
    y_pred = regressor.predict(X_test)
    print(r2_score(y_test, y_pred))


# Call data processing functions
preprocessed_companies = preprocess_companies(companies)
preprocessed_shuttles = preprocess_shuttles(shuttles)
model_input_table = create_model_input_table(
    preprocessed_shuttles, preprocessed_companies, reviews
)

# Call model evaluation functions
X_train, X_test, y_train, y_test = split_data(
    model_input_table, conf_params["model_options"]
)
regressor = train_model(X_train, y_train)
evaluate_model(regressor, X_test, y_test)
```

--------------------------------

### Start Prefect Server

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/prefect

Initializes the local Prefect Server instance.

```bash
prefect server start
```

--------------------------------

### Install project dependencies

Source: https://docs.kedro.org/en/1.0.0/create/new_project

Install the required dependencies for the project using uv.

```bash
uv pip install -r requirements.txt
```

--------------------------------

### Run Kedro-Viz

Source: https://docs.kedro.org/en/1.0.0/tutorials/create_a_pipeline

Start the Kedro-Viz server to visualise the project in a browser.

```bash
kedro viz run
```

--------------------------------

### Install ruff individually

Source: https://docs.kedro.org/en/1.0.0/develop/linting

Install ruff directly using pip.

```bash
pip install ruff
```

--------------------------------

### Install a packaged Kedro project

Source: https://docs.kedro.org/en/1.0.0/deploy/package_a_project

Use pip to install the generated .whl file on a machine without Kedro installed.

```bash
pip install <path-to-wheel-file>
```

--------------------------------

### Example base parameters configuration

Source: https://docs.kedro.org/en/1.0.0/configure/parameters

Sample YAML structure for base environment parameters.

```yaml
# base/parameters.yml
model_options:
  model_params:
    learning_date: "2023-11-01"
    training_date: "2023-11-01"
    data_ratio: 14

data_options:
  step_size: 123123
```

--------------------------------

### Install kedro-mlflow

Source: https://docs.kedro.org/en/1.0.0/integrations-and-plugins/mlflow

Install the plugin to enable MLflow tracking in Kedro projects.

```bash
pip install kedro-mlflow
```

--------------------------------

### Install project dependencies

Source: https://docs.kedro.org/en/1.0.0/develop/linting

Install the project with development dependencies including linting tools.

```bash
pip install ."[dev]"
```

--------------------------------

### Install dependencies with uv

Source: https://docs.kedro.org/en/1.0.0/getting-started/install

Commands for syncing environments or installing from requirements files using uv.

```bash
uv sync
```

```bash
uv pip install -r requirements.txt
```

--------------------------------

### Install project dependencies

Source: https://docs.kedro.org/en/1.0.0/develop/automated_testing

Commands to install the project with development dependencies or as an editable package.

```bash
pip install ."[dev]"
```

```bash
pip install -e .
```

```bash
pip install pytest
```

--------------------------------

### Context project path output

Source: https://docs.kedro.org/en/1.0.0/tutorials/notebooks_tutorial

Example output showing the project path.

```text
PosixPath('/Users/username/kedro_projects/spaceflights')
```

--------------------------------

### Example local parameters configuration

Source: https://docs.kedro.org/en/1.0.0/configure/parameters

Sample YAML structure for local environment parameters.

```yaml
# local/parameters.yml
features:
    rate: 123
```

--------------------------------

### Install pip-tools

Source: https://docs.kedro.org/en/1.0.0/develop/dependencies

Install the pip-tools package to enable dependency pinning and environment reproducibility.

```bash
pip install pip-tools
```

--------------------------------

### Load parameters output

Source: https://docs.kedro.org/en/1.0.0/tutorials/notebooks_tutorial

Example output from loading parameters.

```text
INFO     Loading data from 'parameters' (MemoryDataset)...

{
    'model_options': {
        'test_size': 0.2,
        'random_state': 3,
        'features': [
            'engines',
            'passenger_capacity',
            'crew',
            'd_check_complete',
            'moon_clearance_complete',
            'iata_approved',
            'company_rating',
            'review_scores_rating'
        ]
    }
}
```

--------------------------------

### View Deployment Output

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/databricks/databricks_ide_databricks_asset_bundles_workflow

Example of the expected console output after a successful deployment.

```text
Uploading databrick_iris-0.1-py3-none-any.whl...
Uploading bundle files to /Workspace/Users/xxxxxxx.com/.bundle/databrick_iris/local/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```

--------------------------------

### CSV loading output example

Source: https://docs.kedro.org/en/1.0.0/tutorials/set_up_data

Example console output after loading a CSV dataset.

```text
INFO     Loading data from 'companies' (CSVDataset)
Out[1]:
      id company_rating       company_location  total_fleet_count iata_approved
0  35029           100%                   Niue                4.0             f
1  30292            67%               Anguilla                6.0             f
2  19032            67%     Russian Federation                4.0             f
3   8238            91%               Barbados               15.0             t
4  30342            NaN  Sao Tome and Principe                2.0             t
```

--------------------------------

### Excel loading output example

Source: https://docs.kedro.org/en/1.0.0/tutorials/set_up_data

Example console output after loading an Excel dataset.

```text
INFO     Loading data from 'shuttles' (ExcelDataset)
Out[1]:
      id       shuttle_location shuttle_type engine_type  ... d_check_complete  moon_clearance_complete     price company_id
0  63561                   Niue      Type V5     Quantum  ...                f                        f  $1,325.0      35029
1  36260               Anguilla      Type V5     Quantum  ...                t                        f  $1,780.0      30292
2  57015     Russian Federation      Type V5     Quantum  ...                f                        f  $1,715.0      19032
3  14035               Barbados      Type V5      Plasma  ...                f                        f  $4,770.0       8238
4  10036  Sao Tome and Principe      Type V2      Plasma  ...                f                        f  $2,820.0      30342
```

--------------------------------

### Install Kedro and Databricks CLI

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/databricks/databricks_deployment_workflow

Install the necessary dependencies within the active virtual environment.

```bash
pip install kedro databricks-cli
```

--------------------------------

### Set up local environment

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/databricks/databricks_notebooks_development_workflow

Commands to create and activate a Conda environment and install Kedro.

```bash
conda create --name iris-databricks python=3.10
```

```bash
conda activate iris-databricks
```

```bash
pip install kedro
```

--------------------------------

### Start Local Airflow Cluster

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/airflow

Navigate to the project folder and start the local Astronomer environment.

```bash
cd kedro-airflow-spaceflights
astro dev start
```

--------------------------------

### Configure Checkpoint YAML

Source: https://docs.kedro.org/en/1.0.0/extend/hooks/examples

Example configuration for a SimpleCheckpoint.

```yaml
yaml_config = f"""
name: {my_checkpoint_name}
config_version: 1.0
class_name: SimpleCheckpoint
run_name_template: "%Y%m%d-%H%M%S-my-run-name-template"
validations:
  - batch_request:
      datasource_name: {my_datasource_name}
      data_connector_name: default_runtime_data_connector_name
      data_asset_name: my_runtime_asset_name
      data_connector_query:
        index: -1
    expectation_suite_name: {my_expectation_suite_name}
"""
```

--------------------------------

### Example configuration file structure

Source: https://docs.kedro.org/en/1.0.0/build/run_a_pipeline

Define pipeline execution parameters in a YAML configuration file.

```yaml
run:
  tags: tag1, tag2, tag3
  pipeline: pipeline1
  runner: ParallelRunner
  node_names: node1, node2
  env: env1
```

--------------------------------

### Viewing Pipeline Structure

Source: https://docs.kedro.org/en/1.0.0/integrations-and-plugins/notebooks_and_ipython/kedro_and_notebooks

Example output format when inspecting the default pipeline.

```python
{'__default__': Pipeline([
Node(create_confusion_matrix, 'companies', 'dummy_confusion_matrix', None),
Node(preprocess_companies, 'companies', ['preprocessed_companies', 'companies_columns'], 'preprocess_companies_node'),
Node(preprocess_shuttles, 'shuttles', 'preprocessed_shuttles', 'preprocess_shuttles_node'),
...

```

--------------------------------

### Create a Kedro project with a starter

Source: https://docs.kedro.org/en/1.0.0/create/starters

Use the --starter flag with kedro new to initialize a project from a template path.

```bash
uvx kedro new --starter=<path-to-starter>
```

--------------------------------

### Create a project with a specific starter version

Source: https://docs.kedro.org/en/1.0.0/create/starters

Use the --checkout flag to specify a branch, tag, or commit for the starter repository.

```bash
kedro new --starter=pandas-iris --checkout=0.18.14
```

```bash
uvx kedro new --starter=spaceflights-pandas --checkout=0.1.0
```

--------------------------------

### Initialize Project Directory

Source: https://docs.kedro.org/en/1.0.0/create/minimal_kedro_project

Create and navigate into the project directory.

```bash
mkdir minikedro
```

```bash
cd minikedro
```

--------------------------------

### Sample project dependencies

Source: https://docs.kedro.org/en/1.0.0/getting-started/install

Example content of a pyproject.toml dependencies section for a Kedro project.

```toml
# code quality packages
ipython~=8.10; python_version >= '3.8'
ruff==0.1.8

# notebook tooling
jupyter~=1.0
jupyterlab_server>=2.11.1
jupyterlab~=3.0

# Pytest + useful extensions
pytest-cov~=3.0
pytest-mock>=1.7.1, <2.0
pytest~=7.2

# Kedro dependencies and datasets to work with different data formats (including CSV, Excel, and Parquet)
kedro~=0.19.0
kedro-datasets[pandas-csvdataset, pandas-exceldataset, pandas-parquetdataset]>=3.0
kedro-telemetry>=0.3.1
kedro-viz~=6.0 # Visualise pipelines

# For modeling in the data science pipeline
scikit-learn~=1.0
```

--------------------------------

### Install and Verify AWS CDK

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/aws_step_functions

Commands to install the AWS Cloud Development Kit globally and verify the installation.

```bash
$ npm install -g aws-cdk
# to verify that the cdk command has been installed
$ cdk -h
```

--------------------------------

### Start Prefect Agent

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/prefect

Launches the agent to subscribe to the specified work queue and pool.

```bash
prefect agent start --pool <work_pool_name> --work-queue <work_queue_name>
```

--------------------------------

### Create a project from a custom starter

Source: https://docs.kedro.org/en/1.0.0/create/starters

Initialize a project from a custom local or remote starter path into a specific directory.

```bash
uvx kedro new --starter=<path-to-starter> --directory <directory>
```

--------------------------------

### Install Great Expectations

Source: https://docs.kedro.org/en/1.0.0/extend/hooks/examples

Install the required dependency for data validation.

```bash
pip install great-expectations
```

--------------------------------

### Create a project with a configuration file

Source: https://docs.kedro.org/en/1.0.0/create/customise_project

Use the --config flag to pass a YAML file that defines project details like folder and package names.

```bash
kedro new --config=config.yml
```

--------------------------------

### Initialize Plugins

Source: https://docs.kedro.org/en/1.0.0/api/framework/kedro.framework.cli

Loads and executes all registered 'init' entry points.

```python
def _init_plugins() -> None:
    init_hooks = load_entry_points("init")
    for init_hook in init_hooks:
        init_hook()
```

--------------------------------

### Install Kedro

Source: https://docs.kedro.org/en/1.0.0/deploy/single_machine

Install the Kedro package using pip or conda.

```bash
pip install kedro
```

```bash
conda install -c conda-forge kedro
```

--------------------------------

### Verify Git Installation

Source: https://docs.kedro.org/en/1.0.0/deploy/single_machine

Check if Git is correctly installed on the server.

```bash
git --version
```

--------------------------------

### Create a project using a starter alias

Source: https://docs.kedro.org/en/1.0.0/create/starters

Use a shorthand alias for official Kedro starters to avoid specifying full paths.

```bash
uvx kedro new --starter=spaceflights-pandas
```

--------------------------------

### Install memory_profiler dependency

Source: https://docs.kedro.org/en/1.0.0/extend/hooks/examples

Install the required memory_profiler package before implementing the hook.

```bash
pip install memory_profiler
```

--------------------------------

### Example configuration file structure

Source: https://docs.kedro.org/en/1.0.0/create/customise_project

A sample YAML configuration file defining the output directory, project name, repository name, and Python package.

```yaml
output_dir: ~/code
project_name: My First Kedro Project
repo_name: testing-kedro
python_package: test_kedro
```

--------------------------------

### Install kedro-databricks

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/databricks/databricks_ide_databricks_asset_bundles_workflow

Install the kedro-databricks package to enable Databricks CLI integration.

```bash
pip install kedro-databricks
```

--------------------------------

### Initialize a new Kedro project

Source: https://docs.kedro.org/en/1.0.0/integrations-and-plugins/mlflow

Create a new project using the spaceflights-pandas-viz starter and set up the virtual environment.

```bash
$ kedro new --starter=spaceflights-pandas-viz --name spaceflights-mlflow
$ cd spaceflights-mlflow
$ python -m venv && source .venv/bin/activate
(.venv) $ pip install -r requirements.txt
```

--------------------------------

### Install pre-commit via pip

Source: https://docs.kedro.org/en/1.0.0/develop/linting

Install the pre-commit package using pip.

```bash
pip install pre-commit
```

--------------------------------

### List supported starter aliases

Source: https://docs.kedro.org/en/1.0.0/create/starters

Display all available starter aliases maintained by the Kedro team.

```bash
kedro starter list
```

--------------------------------

### Example pytest output

Source: https://docs.kedro.org/en/1.0.0/develop/automated_testing

Expected output when running the example test successfully.

```text
============================= test session starts ==============================
...
collected 1 item

tests/test_run.py .                                                  [100%]

============================== 1 passed in 0.38s ===============================
```

--------------------------------

### Install cookiecutter

Source: https://docs.kedro.org/en/1.0.0/extend/create_a_starter

Install the cookiecutter package required to manage Kedro starter templates.

```bash
pip install cookiecutter
```

--------------------------------

### Install pytest-cov

Source: https://docs.kedro.org/en/1.0.0/develop/automated_testing

Install the plugin using pip within an active virtual environment.

```bash
pip install pytest-cov
```

--------------------------------

### Accessing Configurations with OmegaConfigLoader

Source: https://docs.kedro.org/en/1.0.0/api/config/kedro.config.OmegaConfigLoader

Example showing how to initialize the loader and access specific configuration sections like catalog and parameters.

```python
import logging.config
from kedro.config import OmegaConfigLoader
from kedro.framework.project import settings

conf_path = str(project_path / settings.CONF_SOURCE)
conf_loader = OmegaConfigLoader(conf_source=conf_path, env="local")

conf_catalog = conf_loader["catalog"]
conf_params = conf_loader["parameters"]
```

--------------------------------

### View help for Kedro project tools

Source: https://docs.kedro.org/en/1.0.0/create/new_project_tools

Run this command to see the available tool options and usage examples for the --tools flag.

```text
...
-t, --tools TEXT    Select which tools you'd like to include. By default,
                      none are included.

                      Tools

                      1) Linting: Provides a basic linting setup with Ruff

                      2) Testing: Provides basic testing setup with pytest

                      3) Custom Logging: Provides more logging options

                      4) Documentation: Basic documentation setup with Sphinx

                      5) Data Structure: Provides a directory structure for
                      storing data

                      6) PySpark: Provides set up configuration for working
                      with PySpark

                      Example usage:

                      kedro new --tools=lint,test,log,docs,data,pyspark
                      (or any subset of these options)

                      kedro new --tools=all

                      kedro new --tools=none
...
```

--------------------------------

### Install Kedro and dbx

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/databricks/databricks_dbx_workflow

Install the necessary packages within the activated Conda environment.

```bash
pip install kedro dbx --upgrade
```

--------------------------------

### Define starter repository specifications

Source: https://docs.kedro.org/en/1.0.0/api/framework/kedro.framework.cli

Constants and structures for official starter templates.

```python
_STARTERS_REPO = 'git+https://github.com/kedro-org/kedro-starters.git'
```

```python
_OFFICIAL_STARTER_SPECS = [KedroStarterSpec('astro-airflow-iris', _STARTERS_REPO, 'astro-airflow-iris'), KedroStarterSpec('spaceflights-pandas', _STARTERS_REPO, 'spaceflights-pandas'), KedroStarterSpec('spaceflights-pyspark', _STARTERS_REPO, 'spaceflights-pyspark'), KedroStarterSpec('databricks-iris', _STARTERS_REPO, 'databricks-iris')]
```

```python
_OFFICIAL_STARTER_SPECS_DICT = {alias: specfor spec in _OFFICIAL_STARTER_SPECS}
```

--------------------------------

### Manage KedroSession Lifecycle

Source: https://docs.kedro.org/en/1.0.0/api/framework/kedro.framework.cli

Use bootstrap_project to configure the environment and create a session as a context manager to execute runs.

```python
from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
from pathlib import Path

# If you are creating a session outside of a Kedro project (i.e. not using
# `kedro run` or `kedro jupyter`), you need to run `bootstrap_project` to
# let Kedro find your configuration.
bootstrap_project(Path("<project_root>"))
with KedroSession.create() as session:
    session.run()
```

--------------------------------

### Archive Deployment Files

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/airflow

Create a plugins.zip archive containing the wheel, configuration, and logging files.

```bash
zip -j plugins.zip dist/new_kedro_project-0.1-py3-none-any.whl dist/conf-new_kedro_project.tar.gz conf/logging.yml
```

--------------------------------

### Install Jinja2 dependency

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/argo

Install the required Jinja2 package to process the workflow template.

```bash
$ pip install Jinja2
```

--------------------------------

### Initialize project settings

Source: https://docs.kedro.org/en/1.0.0/api/framework/kedro.framework.cli

Global settings instance for the project.

```python
settings = _ProjectSettings()
```

--------------------------------

### Install a specific Kedro version

Source: https://docs.kedro.org/en/1.0.0/create/starters

Install a particular version of Kedro using pip.

```bash
pip install kedro==0.18.14
```

--------------------------------

### Initialize OmegaConfigLoader for standalone configuration

Source: https://docs.kedro.org/en/1.0.0/configure/advanced_configuration

Demonstrates loading parameters from a local directory without a full Kedro project structure.

```yaml
learning_rate: 0.01
train_test_ratio: 0.7
```

```python
from kedro.config import OmegaConfigLoader
config_loader = OmegaConfigLoader(conf_source=".")

# Optionally, you can also use environments
# config_loader = OmegaConfigLoader(conf_source=".", base_env="base", default_run_env="local")

print(config_loader["parameters"])
```

--------------------------------

### Initialize a new Kedro project

Source: https://docs.kedro.org/en/1.0.0/getting-started/install

Use uvx to create a new project from the spaceflights-pandas starter.

```bash
uvx kedro new --starter spaceflights-pandas --name spaceflights
```

--------------------------------

### Create a new Kedro project

Source: https://docs.kedro.org/en/1.0.0/integrations-and-plugins/notebooks_and_ipython/kedro_and_notebooks

Initialize a new project using the spaceflights starter with visualization tools enabled.

```bash
kedro new -n spaceflights --tools=viz --example=yes
```

--------------------------------

### Create default project (interactive vs single-line)

Source: https://docs.kedro.org/en/1.0.0/create/new_project

Create a project named My-Project with no tools and no example code.

```bash
kedro new ⮐
My-Project ⮐
none ⮐
no ⮐
```

```bash
uvx kedro new --name=My-Project --tools=none --example=n
```

--------------------------------

### Install Kedro in an environment

Source: https://docs.kedro.org/en/1.0.0/getting-started/install

Commands to install the Kedro package using various dependency managers.

```bash
uv pip install kedro
```

```bash
uv add kedro
```

```bash
pip install kedro
```

```bash
poetry add kedro
```

```bash
conda install -c conda-forge kedro
```

--------------------------------

### Initialize a Pipeline

Source: https://docs.kedro.org/en/1.0.0/api/pipeline/kedro.pipeline.Pipeline

Demonstrates how to define a pipeline using nodes and input/output mappings.

```python
from kedro.pipeline import Pipeline
from kedro.pipeline import node

# In the following scenario first_ds and second_ds
# are datasets provided by io. Pipeline will pass these
# datasets to first_node function and provides the result
# to the second_node as input.


def first_node(first_ds, second_ds):
    return dict(third_ds=first_ds + second_ds)


def second_node(third_ds):
    return third_ds


pipeline = Pipeline(
    [
        node(first_node, ["first_ds", "second_ds"], ["third_ds"]),
        node(second_node, dict(third_ds="third_ds"), "fourth_ds"),
    ]
)
```

--------------------------------

### Install Kedro with OmegaConf

Source: https://docs.kedro.org/en/1.0.0/configure/config_loader_migration

Commands to install or update Kedro to a version that includes OmegaConf support.

```bash
pip install kedro==0.18.5
```

```bash
pip install -U kedro
```

--------------------------------

### Kedro Starter Command Usage

Source: https://docs.kedro.org/en/1.0.0/getting-started/commands_reference

Usage pattern for managing project starters.

```bash
kedro starter [OPTIONS] COMMAND [ARGS]...
```

--------------------------------

### Install pipeline dependencies

Source: https://docs.kedro.org/en/1.0.0/build/modular_pipelines

Use this command to install dependencies listed in a local requirements.txt file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Kedro Starter List Command

Source: https://docs.kedro.org/en/1.0.0/getting-started/commands_reference

Command to list available project starters.

```bash
kedro starter list [OPTIONS]
```

--------------------------------

### Install ptvsd library

Source: https://docs.kedro.org/en/1.0.0/ide/set_up_vscode

Install the ptvsd library on both local and remote machines to enable remote debugging.

```bash
python -m pip install --upgrade ptvsd
```

--------------------------------

### Install Pillow dependency

Source: https://docs.kedro.org/en/1.0.0/extend/how_to_create_a_custom_dataset

Install the Pillow library to enable generic image processing functionality for the custom dataset.

```bash
pip install Pillow
```

--------------------------------

### Register entry point in pyproject.toml

Source: https://docs.kedro.org/en/1.0.0/extend/create_a_starter

Register the starters list in the project's pyproject.toml file under the kedro.starters entry point.

```toml
[project.entry-points."kedro.starters"]
starter = "plugin:starters"
```

--------------------------------

### Install Kedro without Git

Source: https://docs.kedro.org/en/1.0.0/getting-started/faq

Use this workaround to initialize a new Kedro project if Git is not installed on your system.

```bash
uvx kedro new -s https://github.com/kedro-org/kedro-starters/archive/1.0.0.zip --directory=spaceflights-pandas
```

--------------------------------

### Install Data Catalog dependencies by type

Source: https://docs.kedro.org/en/1.0.0/develop/dependencies

Install dependencies limited to a specific data type within a group.

```bash
pip install "kedro-datasets[<group>-<dataset>]"
```

--------------------------------

### Prepare project directory for Airflow

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/airflow

Navigate to the project directory and set up the Airflow configuration folder.

```bash
cd new-kedro-project
mkdir conf/airflow
cp conf/base/catalog.yml conf/airflow/catalog.yml
```

--------------------------------

### Install project in editable mode

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/dask

Install the Kedro project as an editable package to ensure dependencies are available to workers.

```bash
pip install -e .
```

--------------------------------

### Load credentials using OmegaConfigLoader

Source: https://docs.kedro.org/en/1.0.0/configure/credentials

Demonstrates the standard approach for initializing the configuration loader and retrieving credentials.

```python
from pathlib import Path

from kedro.config import OmegaConfigLoader
from kedro.framework.project import settings

# Substitute <project_root> with the [root folder for your project](https://docs.kedro.org/en/stable/tutorial/spaceflights_tutorial.html#terminology)
conf_path = str(Path(<project_root>) / settings.CONF_SOURCE)
conf_loader = OmegaConfigLoader(conf_source=conf_path)
credentials = conf_loader["credentials"]
```

--------------------------------

### Install Data Catalog dependencies by group

Source: https://docs.kedro.org/en/1.0.0/develop/dependencies

Install all dependencies associated with a specific data type group, such as pandas or spark.

```bash
pip install "kedro-datasets[<group>]"
```

--------------------------------

### Create an example test with KedroSession

Source: https://docs.kedro.org/en/1.0.0/develop/automated_testing

Example of a test file that programmatically executes a Kedro run using the KedroSession class.

```python
from pathlib import Path
from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project

class TestKedroRun:
    def test_kedro_run(self):
        bootstrap_project(Path.cwd())

        with KedroSession.create(project_path=Path.cwd()) as session:
            assert session.run() is not None
```

--------------------------------

### Create Initial Version Directory Structure

Source: https://docs.kedro.org/en/1.0.0/extend/how_to_create_a_custom_dataset

Manually move the data file into a versioned directory structure to initialize the dataset.

```bash
$ mv data/01_raw/pokemon-images-and-types/images/images/pikachu.png data/01_raw/pokemon-images-and-types/images/images/pikachu.png.backup
$ mkdir -p data/01_raw/pokemon-images-and-types/images/images/pikachu.png/2020-02-22T00.00.00.000Z/
$ mv data/01_raw/pokemon-images-and-types/images/images/pikachu.png.backup data/01_raw/pokemon-images-and-types/images/images/pikachu.png/2020-02-22T00.00.00.000Z/pikachu.png
```

--------------------------------

### Install Project Dependencies in Databricks Notebook

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/databricks/databricks_dbx_workflow

Install requirements from the project's requirements.txt file using the %pip magic command.

```python
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/requirements.txt"
```

--------------------------------

### Initialize OmegaConfigLoader

Source: https://docs.kedro.org/en/1.0.0/api/config/kedro.config.OmegaConfigLoader

Constructor logic for setting up configuration paths, resolvers, and filesystem protocols.

```python
    self.base_env = base_env or ""
    self.default_run_env = default_run_env or ""
    self.merge_strategy = merge_strategy or {}
    self._globals_oc: DictConfig | None = None
    self._runtime_params_oc: DictConfig | None = None
    self.config_patterns = {
        "catalog": ["catalog*", "catalog*/**", "**/catalog*"],
        "parameters": ["parameters*", "parameters*/**", "**/parameters*"],
        "credentials": ["credentials*", "credentials*/**", "**/credentials*"],
        "globals": ["globals.yml"],
    }
    self.config_patterns.update(config_patterns or {})

    # Deactivate oc.env built-in resolver for OmegaConf
    OmegaConf.clear_resolver("oc.env")
    # Register user provided custom resolvers
    self._custom_resolvers = custom_resolvers
    if custom_resolvers:
        self._register_new_resolvers(custom_resolvers)
    # Register globals resolver
    self._register_globals_resolver()

    # Setup file system and protocol
    self._fs, self._protocol = self._initialise_filesystem_and_protocol(conf_source)

    # Store remote root path if using cloud protocol
    if self._protocol in CLOUD_PROTOCOLS or self._protocol in HTTP_PROTOCOLS:
        options = _parse_filepath(conf_source)
        self._remote_root_path = options["path"].rstrip("/")

    super().__init__(
        conf_source=conf_source,
        env=env,
        runtime_params=runtime_params,
    )
    try:
        self._globals = self["globals"]
    except MissingConfigException:
        self._globals = {}
```

--------------------------------

### Create a project from a remote VCS repository

Source: https://docs.kedro.org/en/1.0.0/create/starters

Specify a git URL and target directory to initialize a project from a remote starter.

```bash
uvx kedro new --starter git+https://github.com/kedro-org/kedro-starters.git --directory spaceflights-pandas
```

--------------------------------

### Install Kedro Dependency

Source: https://docs.kedro.org/en/1.0.0/create/minimal_kedro_project

Add Kedro to the project dependencies.

```bash
uv add kedro
```

--------------------------------

### Define setup.py template

Source: https://docs.kedro.org/en/1.0.0/api/framework/kedro.framework.cli

Template string used for generating setup.py files for modular pipelines.

```python
_SETUP_PY_TEMPLATE = '# -*- coding: utf-8 -*-\nfrom setuptools import setup, find_packages\n\nsetup(\n    name="{name}",\n    version="{version}",\n    description="Modular pipeline `{name}`",\n    packages=find_packages(),\n    include_package_data=True,\n    install_requires={install_requires},\n)\n'
```

--------------------------------

### Initialize Kedro project

Source: https://docs.kedro.org/en/1.0.0/deploy/supported-platforms/databricks/databricks_ide_databricks_asset_bundles_workflow

Create a new project using the databricks-iris starter template.

```bash
kedro new --starter=databricks-iris
```

--------------------------------

### Catalog list output

Source: https://docs.kedro.org/en/1.0.0/tutorials/notebooks_tutorial

Example output from running catalog.list().

```text
[
    'companies',
    'reviews',
    'shuttles',
    'preprocessed_companies',
    'preprocessed_shuttles',
    'model_input_table',
    'regressor',
    'metrics',
    'companies_columns',
    'shuttle_passenger_capacity_plot_exp',
    'shuttle_passenger_capacity_plot_go',
    'dummy_confusion_matrix',
    'parameters',
    'params:model_options',
    'params:model_options.test_size',
    'params:model_options.random_state',
    'params:model_options.features'
]
```
Tokens:
Raw