Pathway

https://pathway.com/developers/user-guide/
Admin
Pathway is a Python data processing framework with a scalable Rust engine for building real-time...
Tokens:201,782
Snippets:1,237
Trust Score:9.4
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install Pathway and Dependencies

Source: https://pathway.com/developers/user-guide/temporal-data/asof-join

Installs the necessary Python libraries: pandas, yahooquery, and pathway. This is a prerequisite for running the tutorial's code examples.

```Python
%%capture --no-display
# !pip install pandas yahooquery -U pathway

```

--------------------------------

### Install Pathway

Source: https://pathway.com/developers/user-guide/introduction/why-pathway

Installs the Pathway framework using pip. This is the primary method for getting started with Pathway, ensuring compatibility with the Python ecosystem.

```bash
pip install pathway
```

--------------------------------

### Install All Pathway Optional Packages

Source: https://pathway.com/developers/user-guide/introduction/installation

Installs Pathway along with all available optional packages, providing comprehensive functionality.

```bash
pip install "pathway[all]"
```

--------------------------------

### Install Pathway with Local LLM Deployment Support

Source: https://pathway.com/developers/user-guide/introduction/installation

Installs Pathway with libraries specifically for deploying and running LLMs locally.

```bash
pip install "pathway[xpack-llm-local]"
```

--------------------------------

### Install Pathway using pip

Source: https://pathway.com/developers/user-guide/introduction/welcome

This snippet shows the command to install the Pathway framework using pip, the Python package installer. It's the first step to getting started with Pathway.

```bash
pip install pathway

```

--------------------------------

### Install Pathway

Source: https://pathway.com/developers/user-guide/introduction/first_realtime_app_with_pathway

Installs the Pathway library using pip. Requires Python 3.10 or higher.

```bash
pip install pathway

```

--------------------------------

### Install Pathway with Multiple Optional Packages

Source: https://pathway.com/developers/user-guide/introduction/installation

Demonstrates how to install Pathway with multiple optional packages simultaneously, such as LLM support and the Airbyte connector.

```bash
pip install "pathway[xpack-llm, airbyte]"
```

--------------------------------

### Install Pathway with Document Parsing Support

Source: https://pathway.com/developers/user-guide/introduction/installation

Installs Pathway with tools necessary for processing various document formats like PDFs and Microsoft Word files.

```bash
pip install "pathway[xpack-llm-docs]"
```

--------------------------------

### Install Pathway with LLM Support

Source: https://pathway.com/developers/user-guide/introduction/installation

Installs Pathway along with common LLM libraries such as OpenAI and Langchain, enabling the development of AI pipelines and Live AI™ systems.

```bash
pip install "pathway[xpack-llm]"
```

--------------------------------

### Install Pathway using pip

Source: https://pathway.com/developers/user-guide/deployment/from-jupyter-to-deploy

This command installs the Pathway library into the current Python runtime environment. It is typically used in environments like Google Colab or within a project's setup script.

```bash
!pip install pathway
```

--------------------------------

### Install Pathway with Airbyte Connector Support

Source: https://pathway.com/developers/user-guide/introduction/installation

Installs Pathway with support for integrating with Airbyte, facilitating data synchronization from various sources.

```bash
pip install "pathway[airbyte]"
```

--------------------------------

### Dockerfile for Pathway Environment Setup

Source: https://pathway.com/developers/user-guide/connect/connectors/database-connectors

A Dockerfile that sets up a Python 3.10 environment, installs the Pathway library and kafka-python, copies a Python script for the pipeline, and sets it as the container's command.

```dockerfile
FROM --platform=linux/x86_64 python:3.10
RUN pip install -U pathway
RUN pip install kafka-python
COPY ./pathway-src/sum.py sum.py
CMD ["python", "-u", "sum.py"]
```

--------------------------------

### Install Pathway with SharePoint Connector Support

Source: https://pathway.com/developers/user-guide/introduction/installation

Installs Pathway with support for the SharePoint connector. Note that this requires a license key.

```bash
pip install "pathway[xpack-sharepoint]"
```

--------------------------------

### Install Pathway with LLM xpack

Source: https://pathway.com/developers/user-guide/llm-xpack/overview

Installs the Pathway framework with LLM xpack functionality, enabling the use of Large Language Models. Alternatively, installs all Pathway functionalities.

```bash
pip install "pathway[xpack-llm]"
```

```bash
pip install "pathway[all]"
```

--------------------------------

### ETL Example Pipeline Description

Source: https://pathway.com/developers/user-guide/deployment/azure-aci-deploy

This provides a description of an example ETL pipeline that can be built and run using the Pathway framework. It serves as a practical demonstration of the framework's capabilities in data processing.

```python
# Example ETL Pipeline Structure (Conceptual)
# This is a placeholder for the actual pipeline code.
# The Pathway framework would be used to define the data sources, transformations, and sinks.

# from pathway import ...

# def define_pipeline():
#     # Define data sources (e.g., Kafka, files)
#     source = ...
#
#     # Define transformations (e.g., joins, aggregations)
#     transformed_data = ...
#
#     # Define data sinks (e.g., database, file)
#     sink = ...
#
#     return source, transformed_data, sink

# if __name__ == "__main__":
#     source, transformed_data, sink = define_pipeline()
#     # Execute the pipeline
#     pass

```

--------------------------------

### Dockerfile for Pathway Deployment

Source: https://pathway.com/developers/user-guide/deployment/docker-deployment

This Dockerfile configures a container to run a Pathway application. It starts from the official Pathway image, sets a working directory, copies and installs project dependencies from requirements.txt, and then copies the application code into the container.

```dockerfile
FROM pathwaycom/pathway:latest
# Set working directory
WORKDIR /app
# Copy requirements file and install dependencies
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
# Copy the rest of the application code
COPY . .
```

--------------------------------

### Pathway MCP Server Setup

Source: https://pathway.com/developers/user-guide/llm-xpack/pathway_mcp_server

This section details the setup and requirements for the Pathway MCP Server. It outlines the necessary steps to get the server running and highlights its key features for integrating LLMs with data sources and tools.

```Python
# Example of setting up Pathway MCP Server
# This is a placeholder and would typically involve importing Pathway modules
# and configuring the server instance.

# from pathway import mcp

# server = mcp.MCP_Server()
# server.start()

print("Pathway MCP Server setup and configuration would go here.")
```

--------------------------------

### AWS CLI Login and Boto3 Client Setup

Source: https://pathway.com/developers/user-guide/deployment/aws-fargate-deploy

This snippet demonstrates how to log in to AWS CLI and initialize Boto3 clients for ECS and ECR services. It requires the AWS CLI to be installed and configured with access keys. The region name can be adjusted based on your AWS setup.

```Python
import boto3
ecs_client = boto3.client("ecs", region_name="eu-central-1")
ecr_client = boto3.client("ecr", region_name="eu-central-1")
```

--------------------------------

### Docker Deployment of Pathway

Source: https://pathway.com/developers/user-guide/deployment/from-jupyter-to-deploy

Provides instructions and examples for deploying Pathway applications using Docker containers. This ensures consistent environments for development and production.

```Dockerfile
FROM pathwayai/pathway:latest

WORKDIR /app
COPY . .

# Install any additional Python dependencies
# RUN pip install -r requirements.txt

# Command to run your Pathway application
# CMD ["python", "your_app.py"]
```

```Python
# Example Python script to run within the Docker container
import pathway as pw

# Define your Pathway pipeline
# ...

# Start the computation
# pw.run()

```

--------------------------------

### Run Pathway Notebook Example

Source: https://pathway.com/developers/user-guide/advanced/consistency

This section demonstrates how to run a notebook example within the Pathway Framework. It focuses on the consistency of the Pathway Data Model, explaining how computations are processed as if static data were loaded, and how streaming changes are handled with timestamps for consistent results.

```Python
# Consistency of the Pathway Data Model
Computations in Pathway are expressed as if static data were loaded into the system. When streaming changes, Pathway produces inputs consistent with the state of all inputs at a given point in time.
Pathway delivers consistent results by explicitly reasoning about time: every processed input message bears a timestamp, and each output message specifies exactly for which input times it was computed. In other words, each output produced by Pathway is the final answer that would have been given if all sources were read up to the indicated cutoff times, and the computation was carried in entirety. No intermediate results are shown. Updates to the outputs will be sent only when new data is input into the system.
This consistency behavior requires specifying an update schedule for each input. For instance, an interactive system may react to user input every 500 milliseconds and update the data to be displayed every 10 seconds. Then, fast interactive manipulations are possible and the data shown lags by at most 10 seconds.
```

--------------------------------

### Install Pathway via Pip

Source: https://pathway.com/developers/user-guide/introduction/installation

Installs the Pathway framework and its core Rust engine using pip. This command ensures all basic dependencies for running Pathway pipelines are included.

```bash
pip install -U pathway
```

--------------------------------

### Install Pathway

Source: https://pathway.com/developers/user-guide/migrating/migrate-from-pandas

Provides the command to install or upgrade the Pathway library using pip. This ensures you have the latest version for development.

```Shell
pip install -U pathway
```

--------------------------------

### Run Pathway Pipeline

Source: https://pathway.com/developers/user-guide/llm-xpack/llm-app-pathway

Executes the main Python script to launch the Pathway pipeline and start processing queries. This is the primary command to get your pipeline running.

```Python
pw.run()
```

--------------------------------

### Install and Import Pathway

Source: https://pathway.com/developers/user-guide/introduction/pathway-overview

Installs the Pathway library using pip and imports it into a Python script for use in data processing pipelines.

```bash
pip install pathway
```

```python
import pathway as pw

```

--------------------------------

### Install Azure SDK Packages

Source: https://pathway.com/developers/user-guide/deployment/azure-aci-deploy

Installs the necessary Azure Python SDK packages for managing container instances and authentication. These packages are required to interact with Azure services programmatically.

```bash
pip install azure-identity
pip install azure-mgmt-containerinstance
```

--------------------------------

### Run Pathway Application

Source: https://pathway.com/developers/user-guide/data-transformation/indexes-in-pathway

Starts the Pathway computation. This command initiates the data processing pipeline.

```bash
pw.run()
```

--------------------------------

### Install Pathway with SQL Support

Source: https://pathway.com/developers/user-guide/development/changelog

To use the `pw.sql` functionality, you now need to install the Pathway package with the SQL extra. This command ensures all necessary dependencies for SQL operations are included.

```Bash
pip install "pathway[sql]"
```

--------------------------------

### Docker Deployment of Pathway

Source: https://pathway.com/developers/user-guide/temporal-data/behaviors

This section provides instructions and code examples for deploying the Pathway framework using Docker. It covers containerization and running Pathway applications in Docker environments.

```Dockerfile
# Example Dockerfile for Pathway deployment
# This is a placeholder and would contain actual Dockerfile instructions.

# FROM pathway/base-image
# WORKDIR /app
# COPY . .
# RUN pathway install
# CMD ["pathway", "run", "your_app.py"]

```

--------------------------------

### Bash: Build and Run Docker Compose Application

Source: https://pathway.com/developers/user-guide/deployment/from-jupyter-to-deploy

Commands to build the Docker images defined in `docker-compose.yml` and then start the services. This prepares and launches the Kafka and Pathway applications in a containerized environment.

```bash
docker compose -f "docker-compose.yml" build
docker compose -f "docker-compose.yml" up
```

--------------------------------

### Install Pathway and dotenv

Source: https://pathway.com/developers/user-guide/llm-xpack/llm-app-pathway

Installs the Pathway library with LLM extensions and the python-dotenv package for managing environment variables. This is a prerequisite for using Pathway's LLM capabilities and loading API keys.

```bash
pip install pathway[xpack-llm] python-dotenv

```

--------------------------------

### Create Example Graph with Pathway

Source: https://pathway.com/developers/user-guide/data-transformation/iterate

Defines an example graph structure using Pathway, including vertices and edges, and initializes a 'repr' column for connected components. It uses markdown tables for data input and assigns IDs from a column.

```Python
vertices = pw.debug.table_from_markdown(
    """
    a
    1
    2
    3
    4
    5
    6
    7
""",
    id_from=["a"],
).with_columns(repr=pw.this.a)
edges = pw.debug.table_from_markdown(
    """
    a | b
    2 | 3
    3 | 4
    4 | 5
    5 | 6
    6 | 1
"""
)
```

--------------------------------

### Start Data Streaming with Pathway

Source: https://pathway.com/developers/user-guide/deployment/from-jupyter-to-deploy

This code snippet initiates the data streaming process for the Pathway dashboard, enabling live updates for the configured widgets.

```Python
pw.run()
```

--------------------------------

### Install Pathway using pip

Source: https://pathway.com/developers/user-guide/introduction/why-pathway

Pathway can be installed as a standard Python library using pip. This command installs the package and makes it available for import in your Python projects.

```bash
pip install pathway
```

--------------------------------

### Install Dependencies in Jupyter Notebook

Source: https://pathway.com/developers/user-guide/deployment/jupyter-docker

Demonstrates how to install Python packages like langchain, langchain_community, and lanchain_openai within a Jupyter notebook using pip install commands prefixed with an exclamation mark.

```Bash
!pip install langchain
!pip install langchain_community
!pip install lanchain_openai

```

--------------------------------

### Setup Tables from Markdown

Source: https://pathway.com/developers/user-guide/temporal-data/interval-join

Defines three tables (vehicle locations, routes, maintenance schedules) from markdown formatted strings using the `pw.debug.table_from_markdown` function. This code is compatible with streaming mode.

```Python
import pathway as pw
vehicle_locations = pw.debug.table_from_markdown(
    """
    | timestamp     | vehicle_id |  route_id    | current_lat  | current_long  | dest_lat | dest_long
  1 |  1682884074   | 1          |  h432af      | 12.8716      | 77.5946       | 12.9800  | 77.5950
  2 |  1682884300   | 1          |  h432af      | 12.8717      | 77.5946       | 12.9800  | 77.5950
  3 |  1682884801   | 1          |  h577dk      | 12.8768      | 77.5947       | 12.9800  | 77.5950
  4 |  1682885003   | 1          |  h577dk      | 12.8829      | 77.5947       | 12.9800  | 77.5950
  5 |  1682992012   | 2          |  h120cc      | 10.4601      | 78.0937       | 10.4532  | 78.0987
  6 |  1682992274   | 2          |  h120cc      | 10.4600      | 78.0948       | 10.4532  | 78.0987
  7 |  1682992674   | 2          |  h120cc      | 10.4595      | 78.0959       | 10.4532  | 78.0987"""
)
routes = pw.debug.table_from_markdown(
    """
        | route_id  | traffic_speed_km_per_h
    1   | h120cc  | 60
    2   | h432af  | 40
    3   | h577dk  | 80
    """
)
# Create a maintenance data table
maintenance = pw.debug.table_from_markdown(
    """
      | route_id    | start_time    | delay_minutes
    1 | h120cc      | 1682992274    | 30
    2 | g392rt      | 1682884441    | 20
    3 | h577dk      | 1682885777    | 10
    """
)
```

--------------------------------

### Read from MQTT

Source: https://pathway.com/developers/user-guide/development/changelog

This example demonstrates reading data from an MQTT broker using `pw.io.mqtt.read`. Configure your MQTT connection details, including host, port, and topic.

```Python
import pathway as pw

# Read from MQTT
mqtt_table = pw.io.mqtt.read(host="your_mqtt_host", port=1883, topic="your_topic")
```

--------------------------------

### Execute Python Script

Source: https://pathway.com/developers/user-guide/llm-xpack/llm-app-pathway

Starts the Python script named 'main.py'. This script is responsible for initializing and running the Pathway pipeline, including launching the web server.

```Bash
python main.py
```

--------------------------------

### Dockerfile for Pathway Application

Source: https://pathway.com/developers/user-guide/connect/connectors/switching-to-redpanda

A Dockerfile to build a Python environment for the Pathway application. It installs Pathway, copies the application script, and sets the command to run the script.

```docker
FROM python:3.10
RUN pip install -U pathway
COPY ./pathway-src/process-stream.py process-stream.py
CMD ["python", "-u", "process-stream.py"]
```

--------------------------------

### Tutorial: From Jupyter to Deploy

Source: https://pathway.com/developers/user-guide/deployment/from-jupyter-to-deploy

A step-by-step tutorial guiding users from developing a Pathway application in a Jupyter notebook to deploying it as a standalone application.

```Python
# Example snippet from a Jupyter notebook
import pathway as pw

# Load data
data = pw.demo.generate_data_stream()

# Perform transformations
processed = data.map(lambda x: x.value * 2)

# To export for deployment, you might use:
# pw.export_notebook("my_notebook.ipynb", "my_app.py")

# For standalone execution:
# pw.run(processed)

```

--------------------------------

### Set Pathway License Key in Python Code

Source: https://pathway.com/developers/user-guide/introduction/installation

Shows how to set the Pathway license key directly within your Python code before running Pathway. The license can be loaded from a file or provided as a string.

```python
import pathway as pw
pw.set_license_key("file:///path/to/license.lic")
# or linlined
pw.set_license_key("-----BEGIN LICENSE FILE-----
...
-----END LICENSE FILE-----")
pw.run()
```

--------------------------------

### Install Dependencies in Dockerfile (requirements.txt)

Source: https://pathway.com/developers/user-guide/deployment/jupyter-docker

Illustrates how to manage Python dependencies in a Dockerfile by using a requirements.txt file. This approach is recommended for projects with a larger number of dependencies.

```Dockerfile
FROM pathwaycom/pathway:latest
# Copy requirements file and install dependencies
COPY requirements.txt ./
RUN pip install -r ./requirements.txt
COPY . .
CMD [ "python", "./your-script.py" ]

```

--------------------------------

### Pathway Dataflow Example

Source: https://pathway.com/developers/user-guide/introduction/streaming-and-static-modes

Demonstrates building a dataflow graph in Pathway by joining two tables and applying transformations. It selects specific columns and computes an 'adult' status based on age.

```Python
T1bis = T1.select(*pw.this, adult=pw.apply(lambda x: x>18, pw.this.age))
T2bis = T2.filter(pw.this.country == "US")
T3 = T1bis.join(T2bis, pw.left.name == pw.right.name).select(
    pw.left.name, pw.left.adult
)
```

--------------------------------

### Access JSON Fields with Default Value

Source: https://pathway.com/developers/user-guide/connect/json_type

Illustrates using the `get()` method to access JSON attributes, providing a default value if the attribute is missing or null. This example retrieves the book title and category, with a default 'Uncategorized' for the category.

```python
sample = table.select(
    author=pw.this.data["author"]["name"],
    title=pw.this.data["books"][0]["title"],
    category=pw.this.data["books"][0].get("category", default=pw.Json("Uncategorized")),
)
sample
```

--------------------------------

### Introduction to the LLM xpack

Source: https://pathway.com/developers/user-guide/temporal-data/asof-join

Introduces the 'LLM xpack' for Pathway, enabling integration with Large Language Models. This section likely covers setting up LLM-powered features within data pipelines.

```Python
import pathway as pw

# Example of initializing the LLM xpack
# This is a placeholder and would contain actual Pathway code
# for setting up LLM functionalities.
print("Pathway LLM xpack introduction")
```

--------------------------------

### Deploy Pathway Web Service with Render

Source: https://pathway.com/developers/user-guide/deployment/cloud-deployment

This section guides users on deploying a Pathway web service using Render, a platform that simplifies the deployment process for web applications. It highlights the ease of deployment with tools like Render.

```N/A
Read the tutorial to learn how to do it.
```

--------------------------------

### Using Airbyte Connector in Pathway

Source: https://pathway.com/developers/user-guide/connect/supported-data-sources

This example demonstrates how to utilize Pathway's Airbyte connector to ingest data from various sources. It highlights the flexibility of connecting to over 350 data sources supported by Airbyte.

```Python
from pathway import InputConnector

# Example of initializing an Airbyte connector (specific connector details would vary)
# This is a conceptual example as the actual connector instantiation would depend on the specific Airbyte source.
# For instance, to connect to a hypothetical 'MyDataSource' Airbyte source:
# connector = InputConnector.airbyte(source='MyDataSource', config={'host': 'localhost', 'port': 8000, ...})

# The actual usage would involve defining a Pathway pipeline that reads from this connector.
# For example:
# data_stream = pipeline.input_stream(connector)

# The list of available connectors is extensive and detailed in the documentation.
# Users should refer to the specific Airbyte connector documentation for configuration details.
print("Pathway Airbyte connector examples are conceptual and depend on specific Airbyte source configurations.")
print("Refer to the Airbyte documentation for detailed setup and usage.")
```

--------------------------------

### Dockerfile: Build Pathway Application Environment

Source: https://pathway.com/developers/user-guide/deployment/from-jupyter-to-deploy

A Dockerfile to set up an environment for running Pathway applications. It uses a Python 3.11 base image, installs the Pathway library via pip, and copies the project files into the container.

```dockerfile
FROM python:3.11

RUN pip install pathway
COPY . .
```

--------------------------------

### Dockerfile for Python Kafka Producer

Source: https://pathway.com/developers/user-guide/connect/connectors/switching-to-redpanda

This Dockerfile sets up a Python 3.10 environment, installs the 'kafka-python' package using pip, and copies the producer script and dataset CSV file into the container. The CMD instruction specifies that the 'create-stream.py' script should be executed when the container starts.

```Dockerfile
FROM python:3.10
RUN pip install kafka-python
COPY ./producer-src/create-stream.py create-stream.py
COPY ./producer-src/dataset.csv dataset.csv
CMD ["python", "-u", "create-stream.py"]
```

--------------------------------

### Install Pathway Package with pip

Source: https://pathway.com/developers/user-guide/introduction/installation

Installs or upgrades the Pathway package using pip. This command can be used with or without a pre-configured private PyPI repository.

```bash
pip install -U pathway
```

```bash
pip install -U pathway --extra-index-url=https://<username>:<password>@<URL>
```

--------------------------------

### Deploy Pathway Web Service from Git Repository

Source: https://pathway.com/developers/user-guide/deployment/render-deploy

This snippet demonstrates the process of deploying a Pathway web service by selecting 'Build and deploy from a Git repository' on Render. It involves providing the Git repository URL and configuring environment variables.

```Text
https://github.com/pathway-labs/realtime-indexer-qa-chat 
```

--------------------------------

### Install Dependencies in Dockerfile (pip install)

Source: https://pathway.com/developers/user-guide/deployment/jupyter-docker

Shows how to install Python dependencies directly within a Dockerfile using RUN pip install commands. This method is suitable for a small number of dependencies.

```Dockerfile
FROM pathwaycom/pathway:latest
RUN pip install langchain
RUN pip install langchain_community
RUN pip install lanchain_openai
COPY . .
CMD [ "python", "./your-script.py" ]

```

--------------------------------

### Deploying Pathway on AWS Fargate

Source: https://pathway.com/developers/user-guide/deployment/aws-fargate-deploy

A step-by-step guide to deploying a Pathway application on AWS Fargate. This includes logging into the AWS CLI, registering a task definition, optionally creating a cluster, and launching the container. It focuses on simplifying the cloud deployment process for Pathway projects.

```bash
# Step 1: Login Into AWS CLI
aws configure

# Step 2: Register Task Definition In AWS
aws ecs register-task-definition --cli-input-json file://task-definition.json

# Step 3 (optional): Create a Cluster Where The Container Will Run
aws ecs create-cluster --cluster-name my-pathway-cluster

# Step 4: Launch The Container
aws ecs run-task --cluster my-pathway-cluster --task-definition my-pathway-task:1 --launch-type FARGATE --network-configuration "awsvpcConfiguration={subnets=[subnet-xxxxxxxxxxxxxxxxx,subnet-yyyyyyyyyyyyyyyyy],securityGroups=[sg-zzzzzzzzzzzzzzzzz],assignPublicIp=ENABLED}"
```

--------------------------------

### Create DocumentStore Instance

Source: https://pathway.com/developers/user-guide/llm-xpack/llm-app-pathway

Initializes the DocumentStore with the loaded documents and the configured components (retriever, parser, splitter). The DocumentStore manages the indexing and retrieval process for the RAG pipeline.

```python
document_store = DocumentStore(
    docs=documents,
    retriever_factory=retriever_factory,
    parser=parser,
    splitter=text_splitter,
)

```

--------------------------------

### Retry PyPI Connector Installation

Source: https://pathway.com/developers/user-guide/development/changelog

This change ensures that `pw.io.airbyte.read` will retry the pip install command if it fails during connector installation. This applies only when using the PyPI version of the connector.

```Python
# This is an internal behavior change in pw.io.airbyte.read
# No direct code snippet to demonstrate, but it improves robustness.
# Example usage:
# pw.io.airbyte.read(connector_name="your_connector", config={...})
```

--------------------------------

### KNN Index with `get_nearest_items_asof_now`

Source: https://pathway.com/developers/user-guide/data-transformation/indexes-in-pathway

Illustrates using `KNNIndex` with the `get_nearest_items_asof_now` method for retrieving k-nearest neighbors. This method returns the closest points at the moment the query is processed and does not update results if the index changes later. It uses the same data setup as the previous example but highlights the difference in output due to the method used.

```Python
index = KNNIndex(data.coords, data, n_dimensions=2, n_and=5)

result_asof_now = queries + index.get_nearest_items_asof_now(
    queries.coords, k=2
).select(nn_ids=pw.this.point_id, nns=pw.this.coords)

pw.debug.compute_and_print(result_asof_now)
```

--------------------------------

### Build and Run Docker Image

Source: https://pathway.com/developers/user-guide/deployment/jupyter-docker

Provides the bash commands to build a Docker image for a Pathway application and then run it as a container.

```Bash
docker build -t my-pathway-app .
docker run -it --rm --name my-pathway-app my-pathway-app

```

--------------------------------

### Install Geopy Package

Source: https://pathway.com/developers/user-guide/temporal-data/interval-join

Installs the 'geopy' package, which is required for calculating geodesic distances. The `--no-display` flag suppresses output.

```shell
%%capture --no-display
#!pip install geopy
```

--------------------------------

### Pull Pathway Docker Image

Source: https://pathway.com/developers/user-guide/introduction/installation

This command pulls the official Pathway Docker image from Docker Hub. Ensure Docker is installed and running on your system.

```bash
docker pull pathwaycom/pathway
```

--------------------------------

### Download Sample Data and Schema Files

Source: https://pathway.com/developers/user-guide/connect/schema-generation

This code snippet downloads sample data and schema files required for schema generation examples in Pathway. It uses `wget` to fetch files from specified S3 URLs and saves them locally as 'data.csv' and 'schema.json'. The `--capture --no-display` flags are used to suppress output during execution.

```python
%%capture --no-display
# !wget https://public-pathway-releases.s3.eu-central-1.amazonaws.com/data/schema-generation-sample-data.csv -O data.csv
# !wget https://public-pathway-releases.s3.eu-central-1.amazonaws.com/data/schema-generation-schema.json -O schema.json

```

--------------------------------

### Real-time Data Pipeline with Azure

Source: https://pathway.com/developers/user-guide/deployment/cloud-deployment

This snippet describes building a real-time data pipeline using Azure Event Hubs, Pathway, and Azure OpenAI. It points to a GitHub repository for a practical tutorial on this integration.

```N/A
You can see how to build a real-time data pipeline with Azure Event Hubs, Pathway, and Azure OpenAI in our associated GitHub repository.
```

--------------------------------

### MacOS: Install libmagic for UnstructuredParser

Source: https://pathway.com/developers/user-guide/development/troubleshooting

This command installs the libmagic library on MacOS using Homebrew, which is required by the `unstructured` library's `UnstructuredParser` to correctly determine file mime types and avoid `FileType.UNK` exceptions.

```bash
brew install libmagic
```

--------------------------------

### Build and Run Pathway Docker Image

Source: https://pathway.com/developers/user-guide/deployment/docker-deployment

These commands demonstrate how to build a Docker image for a Pathway application and then run it. This is a standard Docker workflow for deploying containerized applications.

```bash
docker build -t my-pathway-app .
docker run -it --rm --name my-pathway-app my-pathway-app
```

--------------------------------

### Docker Deployment of Pathway

Source: https://pathway.com/developers/user-guide/temporal-data/asof-join

Provides guidance on deploying Pathway applications using Docker containers. This covers containerizing Pathway applications for consistent and scalable deployment.

```Dockerfile
# Example Dockerfile for Pathway deployment
# This is a placeholder and would contain actual Dockerfile instructions
# for building and running a Pathway application.
FROM python:3.9
WORKDIR /app
COPY . .
RUN pip install pathway
CMD ["python", "your_pathway_app.py"]
```

--------------------------------

### Docker Deployment of Pathway

Source: https://pathway.com/developers/user-guide/introduction/streaming-and-static-modes

This section explains the process of deploying applications built with the Pathway framework using Docker containers. It covers creating Dockerfiles, building images, and running Pathway applications in a containerized environment.

```Dockerfile
# Example Dockerfile for Pathway deployment (conceptual)
# FROM python:3.9-slim
# WORKDIR /app
# COPY requirements.txt .
# RUN pip install --no-cache-dir -r requirements.txt
# COPY . .
# CMD ["python", "your_pathway_app.py"]
```

--------------------------------

### Linux/Colab: Install libmagic for UnstructuredParser

Source: https://pathway.com/developers/user-guide/development/troubleshooting

This command installs the libmagic library on Debian-based Linux distributions or Google Colab, which is required by the `unstructured` library's `UnstructuredParser` to correctly determine file mime types and avoid `FileType.UNK` exceptions.

```bash
sudo apt install libmagic1
```

--------------------------------

### Compact Long Chain Data Joining and Filtering with '*' Notation

Source: https://pathway.com/developers/user-guide/data-transformation/join-manual

This example demonstrates a more compact way to write the long chain of data operations using the '*' notation for selecting all columns from a table. It achieves the same result as the first example but with significantly less code.

```Python
pw.debug.compute_and_print(
    friends.join(likes, friends.id == likes.f_id_ptr)
    .select(*friends, likes.m_id_ptr)
    .join(menu, pw.left.m_id_ptr == menu.id)
    .select(*pw.left.without("m_id_ptr"), *menu)
    .join_left(allergies, pw.left.name == pw.right.person)
    .select(*pw.left, allergies.allergy)
    .join(allergens_in_menu, pw.left.dish == pw.right.dish)
    .select(*pw.left, allergens_in_menu.allergen)
    .filter((pw.this.price <= pw.this.budget) & (pw.this.allergy != pw.this.allergen))
    .groupby(*pw.this)
    .reduce(*pw.this)
)
```

--------------------------------

### LLM Xpack Integrations

Source: https://pathway.com/developers/user-guide/development/changelog

The LLM Xpack now provides integrations with LlamaIndex and LangChain, facilitating the setup and use of the Pathway VectorStore server.

```Python
Pathway VectorStore server
```

--------------------------------

### Docker Deployment of Pathway

Source: https://pathway.com/developers/user-guide/connect/connectors/switching-to-redpanda

Details the steps and considerations for deploying the Pathway framework using Docker. This enables containerized deployment for consistent environments.

```Shell
# Example Dockerfile for Pathway deployment
# FROM pathwayai/pathway:latest
# COPY . /app
# WORKDIR /app
# CMD ["python", "your_script.py"]
```

--------------------------------

### CSV Data Example

Source: https://pathway.com/developers/user-guide/deployment/persistence_restart_with_new_data

This snippet shows an example of CSV data representing user sessions. It includes columns for user ID, session length, start and end times, event count, and time differences. This data format is commonly used as input for data processing tasks.

```csv
user_id,session_length,session_start,session_finish,count,time,diff
"alice",583,1701103035,1701103618,5,1701270112954,-1
"alice",881,1701103035,1701103916,8,1701270112954,1

```

--------------------------------

### Create Debug Tables from Markdown

Source: https://pathway.com/developers/user-guide/data-transformation/join-manual

This example shows how to create sample data tables using Pathway's debug utility, converting markdown formatted strings into Pathway tables. This is useful for testing and demonstrating functionalities like joins.

```Python
friends = pw.debug.table_from_markdown(
    """
    | name  |budget
  1 | Alice | 13
  2 | Bob   | 10
  3 | Carol | 11
  4 | Dan   | 12
 """
)
menu = pw.debug.table_from_markdown(
    """
     | dish                              | price
  11 | pancakes_with_jam                 | 11
  12 | pb_jam_sandwich                   | 9
  13 | jam_pb_pancakes                   | 12
  14 | scrambled_egg                     | 11
 """
)

```

--------------------------------

### CI/CD

Source: https://pathway.com/developers/user-guide/deployment/from-jupyter-to-deploy

Guidance on integrating Pathway applications into Continuous Integration and Continuous Deployment (CI/CD) pipelines. This covers automated testing and deployment strategies.

```Bash
# Example CI/CD script snippet (e.g., for GitHub Actions)
# name: Pathway CI/CD
# 
# on: [push]
# 
# jobs:
#   build-and-deploy:
#     runs-on: ubuntu-latest
#     steps:
#     - uses: actions/checkout@v3
#     - name: Set up Python
#       uses: actions/setup-python@v3
#       with:
#         python-version: '3.9'
#     - name: Install dependencies
#       run: pip install -r requirements.txt
#     - name: Run Pathway tests
#       run: pytest tests/
#     # Add deployment steps here (e.g., build Docker image, deploy to cloud)

```

--------------------------------

### Filter Transformation Example (Stateless)

Source: https://pathway.com/developers/user-guide/introduction/concepts

Illustrates a filter operation, a stateless transformation. Filtering only requires the current data point and does not depend on previous data.

```Python
stream.filter(lambda x: x > 10)
```

--------------------------------

### Docker Deployment of Pathway

Source: https://pathway.com/developers/user-guide/temporal-data/temporal_behavior

Provides guidance on deploying Pathway applications using Docker containers. This section covers creating Dockerfiles, building images, and running Pathway applications in isolated environments.

```dockerfile
# Example Dockerfile for a Pathway application

# Use an official Python runtime as a parent image
FROM python:3.9-slim

# Set the working directory in the container
WORKDIR /app

# Copy the requirements file into the container
COPY requirements.txt .

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code into the container
COPY . .

# Make port 80 available to the world outside this container (if applicable)
# EXPOSE 80

# Define environment variable (optional)
# ENV NAME World

# Run app.py when the container launches
# CMD ["python", "app.py"]

# For Pathway, you might run a specific script or command
# CMD ["pathway", "run", "your_pipeline.py"]

CMD ["echo", "Dockerfile for Pathway deployment."]
```

--------------------------------

### LLM Embedder: Get Embedding Dimension

Source: https://pathway.com/developers/user-guide/development/changelog

Embedders in the LLM xpack now have a `get_embedding_dimension` method to retrieve the number of dimensions used by the selected embedder.

```Python
embedder.get_embedding_dimension()
```

--------------------------------

### Generate Data Streams with demo Module

Source: https://pathway.com/developers/user-guide/temporal-data/asof-join

Explains how to use the 'demo' module within Pathway to generate sample data streams. This is useful for testing and development purposes, allowing users to simulate real-world data scenarios.

```Python
import pathway as pw
from pathway.stdlib.demo import * 

# Example of generating a data stream using the demo module
# This is a placeholder and would contain actual Pathway code
# for creating and manipulating demo data streams.
print("Pathway demo data stream generation")
```

--------------------------------

### Read Kafka from Timestamp

Source: https://pathway.com/developers/user-guide/development/changelog

The `pw.io.kafka.read` function now supports reading entries starting from a specified timestamp, allowing for more targeted data retrieval from Kafka streams.

```Python
pw.io.kafka.read
```

--------------------------------

### Initialize DocumentStore and Retriever

Source: https://pathway.com/developers/user-guide/llm-xpack/docs-indexing

Demonstrates how to set up a DocumentStore for retrieving documents. It involves reading data sources, defining a text splitter, and configuring the retriever factory.

```Python
from pathway.xpacks.llm.document_store import DocumentStore
from pathway.xpacks.llm.splitters import TokenCountSplitter
import pathway as pw
data_sources = pw.io.fs.read(
    "./sample_docs",
    format="binary",
    with_metadata=True,
)
text_splitter = TokenCountSplitter()
store = DocumentStore(
    docs=data_sources,
    retriever_factory=retriever_factory,
    splitter=text_splitter,
)
```

--------------------------------

### Apply AsyncTransformer to Table

Source: https://pathway.com/developers/user-guide/data-transformation/asynchronous-transformations

Applies a defined AsyncTransformer to an input table and retrieves the results. It demonstrates how to use the 'successful' property to get only the successfully processed rows.

```Python
table = pw.debug.table_from_markdown(
    """
    value
      12
       6
       2
       2
       6
"""
)
result = SimpleAsyncTransformer(input_table=table).successful
pw.debug.compute_and_print(result)
```

--------------------------------

### Configure DocumentStore Components

Source: https://pathway.com/developers/user-guide/llm-xpack/llm-app-pathway

Sets up the components required for the DocumentStore: a TokenCountSplitter for chunking text, an OpenAIEmbedder for generating embeddings, a BruteForceKnnFactory for retrieval, and an UnstructuredParser for extracting text from documents.

```python
text_splitter = TokenCountSplitter(
    min_tokens=100, max_tokens=500, encoding_name="cl100k_base"
)
embedder = OpenAIEmbedder(api_key=os.environ["OPENAI_API_KEY"])
retriever_factory = BruteForceKnnFactory(
    embedder=embedder,
)
parser = UnstructuredParser(
    chunking_mode="by_title",
    chunking_kwargs={
        "max_characters": 3000,
        "new_after_n_chars": 2000,
    },
)

```

--------------------------------

### Diff on String Column (Error Example)

Source: https://pathway.com/developers/user-guide/connect/datatypes

Illustrates an attempt to use the `diff` operator on a string column ('values_str'). This operation is not supported because strings cannot be subtracted, resulting in a `TypeError`.

```Python
table3 = table + table.diff(pw.this.values_str, pw.this.values)
print(table3.schema)

pw.debug.compute_and_print(table3, include_id=False)
```

--------------------------------

### Create and Print Salary Table

Source: https://pathway.com/developers/user-guide/data-transformation/indexing-grouped-tables

Initializes a Pathway table from markdown data containing employee salaries and departments, then prints the table to the console. This sets up the data for subsequent analysis.

```Python
import pathway as pw
salaries = pw.debug.table_from_markdown(
    """
 salary   | department
 1800   | Sales
 2000   | Finance
 2300   | Sales
 2700   | Finance
 1900   | Finance
 """
)
pw.debug.compute_and_print(salaries)
```

--------------------------------

### Docker Compose for Stream Producer Service

Source: https://pathway.com/developers/user-guide/connect/connectors/switching-to-redpanda

This Docker Compose configuration defines a service named 'stream-producer'. It specifies the build context and Dockerfile for the producer image and sets up a dependency on the 'kafka' service, ensuring Kafka is available before the producer starts. It also assigns the service to a 'tutorial_network'.

```YAML
  stream-producer:
    build:
      context: .
      dockerfile: ./producer-src/Dockerfile
    depends_on: [kafka]
    networks:
      - tutorial_network
```

--------------------------------

### Set Pathway License Key via Environment Variable

Source: https://pathway.com/developers/user-guide/introduction/installation

Demonstrates how to set the Pathway license key using an environment variable. The key can be provided as a file path or directly as a string.

```bash
PATHWAY_LICENSE_KEY=file:///path/to/license.lic
```

```bash
PATHWAY_LICENSE_KEY="-----BEGIN LICENSE FILE-----
...
-----END LICENSE FILE-----"
```

--------------------------------

### Batch Processing in Python

Source: https://pathway.com/developers/user-guide/temporal-data/asof-join

Demonstrates how to perform batch processing using the Pathway framework in Python. This section likely covers setting up data sources, applying transformations, and outputting results in a batch mode.

```Python
import pathway as pw

# Example of batch processing setup
# This is a placeholder and would contain actual Pathway code
# for reading data, transforming it, and writing it back.
print("Pathway batch processing example")
```

--------------------------------

### Pathway Dataflow Graph Example

Source: https://pathway.com/developers/user-guide/introduction/concepts

Illustrates the dataflow graph concept in Pathway, where tables are nodes linked by transformations. This graph represents the structure of data processing but does not trigger computation itself.

```text
Table A -> Transformation 1 -> Table B -> Transformation 2 -> Table C
```

--------------------------------

### Get Nearest Items As Of Now with KNNIndex

Source: https://pathway.com/developers/user-guide/development/changelog

Introduces `get_nearest_items_asof_now` to the `KNNIndex`. This method allows retrieving the nearest neighbors without updating future queries, useful for time-sensitive lookups.

```Python
import pathway as pw

# Assuming knn_index is a KNNIndex instance
# query_vector = [1.0, 2.0]
# nearest_items = knn_index.get_nearest_items_asof_now(query_vector, k=5)

```

--------------------------------

### Populate Data Directory

Source: https://pathway.com/developers/user-guide/data-transformation/indexes-in-pathway

Creates a directory named 'points' and adds a CSV file 'a.csv' containing sample point data (x, y coordinates). This is a shell command.

```bash
mkdir points && echo -e "x,y\n2,2\n-2,2\n0,6\n2,8" > points/a.csv
```

--------------------------------

### Run Python from GitHub using PATHWAY_SPAWN_ARGS

Source: https://pathway.com/developers/user-guide/deployment/aws-fargate-deploy

This example shows an alternative method to run a Python script from a GitHub repository using the Pathway CLI by leveraging the PATHWAY_SPAWN_ARGS environment variable. This approach simplifies the `pathway spawn-from-env` command by pre-configuring the repository URL and script.

```bash
GITHUB_PERSONAL_ACCESS_TOKEN=YOUR_GITHUB_PERSONAL_ACCESS_TOKEN \
    PATHWAY_LICENSE_KEY=YOUR_PATHWAY_LICENSE_KEY \
    PATHWAY_SPAWN_ARGS='--repository-url https://github.com/pathway-labs/airbyte-to-deltalake python main.py' \
    pathway spawn-from-env
```

--------------------------------

### Configure pip for Private PyPI Repository

Source: https://pathway.com/developers/user-guide/introduction/installation

This snippet shows how to configure pip to use a private PyPI repository by modifying the pip configuration file. Replace placeholders with your actual credentials and URL.

```bash
[global]
extra-index-url = https://<username>:<password>@<URL>
```
Tokens:
Raw