### Run Docker Compose Example Source: https://github.com/pathwaycom/pathway/blob/main/examples/projects/twitter/README.md Command to start a specific docker-compose example, building the images and using environment variables from settings.env. ```bash docker-compose -f --env-file settings.env up --build ``` -------------------------------- ### Building the Hello Example Source: https://github.com/pathwaycom/pathway/blob/main/external/timely-dataflow/mdbook/src/chapter_0/chapter_0_1.md Command to build the 'hello' example specifically. This compiles the example program and its dependencies. ```bash cargo build --example hello ``` -------------------------------- ### Install Pathway and Download Data Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/adaptive_rag_question_answering.ipynb Installs necessary Python packages and downloads sample data for the Adaptive RAG example. Uncomment the pip install lines if Pathway and OpenAI packages are not already installed. The data file `adaptive-rag-contexts.jsonl` is downloaded using `wget`. ```python # Uncomment, if you need to install Pathway and OpenAI packages # !pip install pathway # !pip install openai # Download `adaptive-rag-contexts.jsonl` with ~1000 contexts from SQUAD dataset !wget -q -nc https://public-pathway-releases.s3.eu-central-1.amazonaws.com/data/adaptive-rag-contexts.jsonl # If you want to use cache, set `PATHWAY_PERSISTENT_STORAGE environmental variable # !export PATHWAY_PERSISTENT_STORAGE=".cache" ``` -------------------------------- ### Running the Hello Example (Single Worker) Source: https://github.com/pathwaycom/pathway/blob/main/external/timely-dataflow/mdbook/src/chapter_0/chapter_0_1.md Command to run the 'hello' example with a single worker thread. This executes the compiled example program. ```bash cargo run --example hello ``` -------------------------------- ### Install Pathway Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/80.advanced/20.function_calls_caching.md Installs the Pathway library. Use this command before starting to work with Pathway. ```python %%capture --no-display !pip install pathway ``` -------------------------------- ### Complete Pathway Pipeline Example Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/10.introduction/50.concepts.md This example demonstrates a complete Pathway pipeline, including schema definition, reading from Kafka, filtering data, performing a reduction (summation), writing to Kafka, and finally running the computation. It showcases how transformations and connectors are defined and how `pw.run()` is used to start the processing. ```python import pathway as pw class InputSchema(pw.Schema): name: str = pw.column_definition(primary_key=True) age: int input_table = pw.io.kafka.read(kafka_settings, schema=InputSchema, topic="topic1", format="json") filtered_table = input_table.filter(input_table.age >= 0) result_table = filtered_table.reduce(sum_age = pw.reducers.sum(filtered_table.age)) pw.io.kafka.write(result_table, kafka_settings, topic="topic2", format="json") pw.run() ``` -------------------------------- ### Pathway Dockerfile Setup Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/7.templates/ETL/7.realtime-log-monitoring.md A Dockerfile to set up the Pathway container. It uses a Python base image, installs Pathway and python-dateutil, and copies the alert script. ```Dockerfile FROM --platform=linux/x86_64 python:3.10 RUN pip install -U pathway RUN pip install python-dateutil COPY ./pathway-src/alerts.py alerts.py CMD ["python", "-u", "alerts.py"] ``` -------------------------------- ### Multi-Machine Setup with Multiple Processes on One Host Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/80.advanced/70.running_on_multiple_machines.md Demonstrates starting multiple Pathway processes on the same host by assigning different ports. Process 0 and 1 run on 192.168.1.10 with ports 9000 and 9001 respectively, while process 2 runs on 192.168.1.11. ```bash pathway spawn \ --addresses 192.168.1.10:9000,192.168.1.10:9001,192.168.1.11:9000 \ --process-id 0 \ python pipeline.py ``` ```bash pathway spawn \ --addresses 192.168.1.10:9000,192.168.1.10:9001,192.168.1.11:9000 \ --process-id 1 \ python pipeline.py ``` ```bash pathway spawn \ --addresses 192.168.1.10:9000,192.168.1.10:9001,192.168.1.11:9000 \ --process-id 2 \ python pipeline.py ``` -------------------------------- ### Download Example PDF Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/rag-evaluations.ipynb Downloads the Alphabet 10K report as an example PDF to the specified input folder. ```bash !wget -P "$INPUT_FOLDER" "https://github.com/pathwaycom/llm-app/blob/main/templates/multimodal_rag/data/20230203_alphabet_10K.pdf" ``` -------------------------------- ### Install Pathway and Libraries Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/40.temporal-data/40.asof-join.md Installs the necessary libraries for the tutorial, including pandas, yahooquery, and pathway. Use this command in your environment to set up the prerequisites. ```python %%capture --no-display # !pip install pandas yahooquery -U pathway ``` -------------------------------- ### Install LLM xpack Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/50.llm-xpack/10.overview.md Installs the Pathway library with LLM xpack functionality. Use 'all' to install all available xpacks. ```bash pip install "pathway[xpack-llm]" ``` ```bash pip install "pathway[all]" ``` -------------------------------- ### Install Pathway using pip Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/10.introduction/10.welcome.md Use this command to install the Pathway library. Ensure you have Python and pip installed. ```bash pip install pathway ``` -------------------------------- ### Full Kafka Producer Example Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/20.connect/99.connectors/80.switching-to-redpanda.md A complete Python script demonstrating how to produce messages to a Kafka topic, including initial setup, sending a commit message, and processing data from a CSV file. ```python from kafka import KafkaProducer import csv import time import json topic = "ratings" #We wait for Kafka and ZooKeeper to be ready time.sleep(30) producer = KafkaProducer( bootstrap_servers=["kafka:9092"], security_protocol="PLAINTEXT", api_version=(0, 10, 2), ) producer.send(topic, "*COMMIT*".encode("utf-8")) time.sleep(2) with open("./dataset.csv", newline="") as csvfile: dataset_reader = csv.reader(csvfile, delimiter=",") first_line = True for row in dataset_reader: # We skip the header if first_line: first_line = False continue message_json = { "userId": int(row[0]), "movieId": int(row[1]), "rating": float(row[2]), "timestamp": int(row[3]), } producer.send(topic, (json.dumps(message_json)).encode("utf-8")) time.sleep(0.1) producer.send(topic, "*COMMIT*".encode("utf-8")) time.sleep(2) producer.close() ``` -------------------------------- ### Running the Hello Example (Multiple Workers) Source: https://github.com/pathwaycom/pathway/blob/main/external/timely-dataflow/mdbook/src/chapter_0/chapter_0_1.md Command to run the 'hello' example with multiple worker threads. The '-w2' flag specifies two workers. Arguments for the program are passed after '--'. ```bash cargo run --example hello -- -w2 ``` -------------------------------- ### Install Azure Container Instance Packages Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/60.deployment/25.azure-aci-deploy.md Install the required Azure Python SDK packages for managing container instances. Use pip to install 'azure-identity' and 'azure-mgmt-containerinstance'. ```bash pip install azure-identity pip install azure-mgmt-containerinstance ``` -------------------------------- ### Install airbyte-serverless Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/7.templates/ETL/150.etl-python-airbyte.md Install the airbyte-serverless tool using pip. This tool is required for setting up Airbyte sources. Ensure Docker is installed and running. ```bash pip install airbyte-serverless ``` -------------------------------- ### Install Pathway and Dependencies Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/live_vector_indexing_pipeline.ipynb Installs Pathway with LLM and documentation extensions, along with the unstructured package for advanced document parsing. Also creates a directory for sample documents and downloads a README file. ```bash !pip install "pathway[xpack-llm,xpack-llm-docs]" !pip install unstructured[all-docs] !mkdir -p sample_documents ![ -f sample_documents/repo_readme.md ] || wget 'https://gist.githubusercontent.com/janchorowski/dd22a293f3d99d1b726eedc7d46d2fc0/raw/pathway_readme.md' -O 'sample_documents/repo_readme.md' ``` -------------------------------- ### Example Commit Count Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/60.deployment/20.aws-fargate-deploy.md This is an example output showing the number of commits processed. ```text 664 ``` -------------------------------- ### Build Docker Image for Pathway ACI Example Source: https://github.com/pathwaycom/pathway/blob/main/examples/projects/azure-aci-deploy/README.md Build the Docker image required to run the Pathway example in Azure Container Instances. ```bash docker build . -t pathway-azure-container-instances-example ``` -------------------------------- ### Install Pathway Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/fuzzy_join_part1.ipynb Installs the Pathway library. Use this command in a Linux environment with Python 3.10+. ```bash %%capture --no-display !pip install --prefer-binary pathway ``` -------------------------------- ### Install Pathway and LiteLLM Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/multimodal-rag-using-Gemini.ipynb Installs the Pathway library with all features and a specific version of LiteLLM. Ensure you have the correct versions for compatibility. ```python !pip install 'pathway[all]>=0.14.0' litellm==1.40.0 ``` -------------------------------- ### Run wordcount example with one worker Source: https://github.com/pathwaycom/pathway/blob/main/external/timely-dataflow/mdbook/src/chapter_2/chapter_2_5.md Execute the wordcount example using cargo run. This command builds and runs the example, showing output for a single worker. ```ignore Echidnatron% cargo run --example wordcount Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs Running `target/debug/examples/wordcount` seen: ("round", 1) seen: ("round", 2) seen: ("round", 3) seen: ("round", 4) seen: ("round", 5) seen: ("round", 6) seen: ("round", 7) seen: ("round", 8) seen: ("round", 9) seen: ("round", 10) Echidnatron% ``` -------------------------------- ### Example Type Hints Output Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/20.connect/40.schema.md An example output of `table.typehints()`, showing the inferred data types for columns 'age', 'owner', and 'pet'. ```text {'age': , 'owner': , 'pet': } ``` -------------------------------- ### Install LiteLLM Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/mistral_adaptive_rag_question_answering.ipynb Installs LiteLLM, a library for Python wrappers for LLM calls, enabling easy LLM switching. ```bash !pip install "litellm>=1.35" ``` -------------------------------- ### Install Pathway with SharePoint Connector Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/10.introduction/20.installation.md Installs support for connecting to SharePoint. Note that this feature requires a free license key. ```bash pip install "pathway[xpack-sharepoint]" ``` -------------------------------- ### Install Pathway with All Optional Packages Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/multimodal-rag.ipynb Install the Pathway library along with all its optional packages to ensure full functionality for multimodal RAG applications. This command is essential for setting up the environment. ```python !pip install 'pathway[all]>=0.20.0' ``` -------------------------------- ### Run Docker Container for Pathway ACI Example Source: https://github.com/pathwaycom/pathway/blob/main/examples/projects/azure-aci-deploy/README.md Run the pre-built Docker container to execute the Pathway example within Azure Container Instances. ```bash docker run -t pathway-azure-container-instances-example ``` -------------------------------- ### Benchmarking with Cargo Run --release Source: https://github.com/pathwaycom/pathway/blob/main/external/timely-dataflow/mdbook/src/chapter_1/chapter_1_1.md Use the `--release` flag with `cargo run` for optimized performance. This example demonstrates benchmarking different worker counts for an example program. ```ignore Echidnatron% time cargo run --release --example hello -- -w1 > output1.txt Finished release [optimized] target(s) in 0.0 secs Running `target/release/examples/hello -w1` cargo run --release --example hello -- -w1 > output1.txt 0.78s user 0.06s system 96% cpu 0.881 total Echidnatron% time cargo run --release --example hello -- -w2 > output2.txt Finished release [optimized] target(s) in 0.0 secs Running `target/release/examples/hello -w2` cargo run --release --example hello -- -w2 > output2.txt 0.73s user 0.05s system 165% cpu 0.474 total ``` -------------------------------- ### Install Pathway via Pip Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/10.introduction/20.installation.md Installs the basic Pathway package and its core dependencies, including the Rust engine. Use this for standard usage of the Live Data Framework. ```bash pip install -U pathway ``` -------------------------------- ### Run Naive Delta Query Example Source: https://github.com/pathwaycom/pathway/blob/main/external/differential-dataflow/dogsdogsdogs/README.md Executes the naive delta query example with a specified dataset and batch size. Observe performance characteristics, particularly when encountering nodes with many edges. ```bash Echidnatron% cargo run --release --example delta_query -- ~/Projects/Datasets/livejournal 1 Finished release [optimized] target(s) in 0.07s Running `target/release/examples/delta_query /Users/mcsherry/Projects/Datasets/livejournal 1` 3.429001ms Round 1 complete 32.624059ms Round 2 complete 38.686335ms Round 3 complete 41.348212ms Round 4 complete 44.279022ms Round 5 complete ... 425.859742ms Round 85 complete 426.276961ms Round 86 complete 426.550825ms Round 87 complete 26.317746169s Round 88 complete 26.318698709s Round 89 complete 26.319160175s Round 90 complete ... ``` -------------------------------- ### Prepare File Structure Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/fs_connector.ipynb Creates input and output directories and populates input files with sample data for file system connector examples. ```bash ! mkdir -p plain_input ! mkdir -p plain_output ! echo -e "test1\ndata1" > plain_input/in1.txt ! echo -e "test2\ndata2" > plain_input/in2.txt ``` -------------------------------- ### Install Pathway with LLM Packages Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/10.introduction/20.installation.md Installs Pathway along with common LLM libraries like OpenAI and Langchain. Use this for AI pipelines or building Live AI™ systems. ```bash pip install "pathway[xpack-llm]" ``` -------------------------------- ### Initialize Pathway and Execute SQL Query Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/sql_api.ipynb Initializes Pathway, sets a license key (or uses demo), creates a sample table from markdown, and executes a SQL query with a WHERE clause. The result is then computed and printed. ```python import pathway as pw # To use advanced features with Pathway Scale, get your free license key from # https://pathway.com/features and paste it below. # To use Pathway Community, comment out the line below. pw.set_license_key("demo-license-key-with-telemetry") t = pw.debug.table_from_markdown( """ | a | b 1 | 1 | 2 2 | 4 | 3 3 | 4 | 7 """ ) ret = pw.sql("SELECT * FROM tab WHERE a json_input/json_in1.txt ! echo -e '{"header1":"data5","header2":"data6"}\n{"header1":"data7","header2":"data8"}\n' > json_input/json_in2.txt ! echo -e "json_in1.txt:" ! cat json_input/json_in1.txt ! echo -e "json_in2.txt:" ! cat json_input/json_in2.txt ``` -------------------------------- ### Executing SQL Command Example Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/10.introduction/30.pathway-overview.md Demonstrates how to execute a SQL query against a Pathway table. ```python pw.sql(query, tab=t) ``` -------------------------------- ### Prepare CSV Input Files Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/fs_connector.ipynb Create input directories and sample CSV files for testing. This setup is necessary before reading CSV data. ```bash ! mkdir -p csv_input ! mkdir -p csv_output ! echo -e "header1;header2\ndata1;data2\n\ndata3;data4" > csv_input/csv_in1.txt ! echo -e "header1;header2\ndata5;data6\n\ndata7;data8" > csv_input/csv_in2.txt ! echo -e "csv_in1.txt:" ! cat csv_input/csv_in1.txt ! echo -e "csv_in2.txt:" ! cat csv_input/csv_in2.txt ``` -------------------------------- ### Install LangChain Packages Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/langchain-integration.ipynb Installs necessary LangChain packages for integration with Pathway. Ensure these are installed if you plan to use LangChain components. ```python !pip install langchain !pip install langchain_community !pip install langchain_openai ``` -------------------------------- ### Install Required Packages Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/option-greeks.ipynb Installs necessary Python packages for Pathway, Databento, data manipulation, and numerical operations. Ensure these are installed before proceeding. ```bash !pip install pathway databento pandas scipy numpy python-dotenv ``` -------------------------------- ### Launch Project Containers Source: https://github.com/pathwaycom/pathway/blob/main/examples/projects/realtime-log-monitoring/logstash-pathway-elastic/README.md Starts all necessary Docker containers for the log monitoring system. Ensure you are in the root repository directory. ```bash make ``` -------------------------------- ### Download Sample Data and Schema Files Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/schema-generation.ipynb Downloads sample CSV data and a JSON schema definition file. Uncomment and run these commands to fetch the necessary files for the tutorial. ```bash %%capture --no-display # !wget https://public-pathway-releases.s3.eu-central-1.amazonaws.com/data/schema-generation-sample-data.csv -O data.csv # !wget https://public-pathway-releases.s3.eu-central-1.amazonaws.com/data/schema-generation-schema.json -O schema.json ``` -------------------------------- ### Install Pathway LLM xpack Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/7.templates/20.run-a-template.md Install the Pathway LLM xpack for local RAG pipeline development. Ensure you have Python 3.8+ installed. ```bash pip install pathway[all] ``` -------------------------------- ### Install Pathway and Python-dotenv Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/50.llm-xpack/20.llm-app-pathway.md Install the necessary packages for Pathway's LLM xpack and environment variable management. Ensure you have Python installed. ```bash pip install pathway[xpack-llm] python-dotenv ``` -------------------------------- ### Initialize Document Store Components Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/50.llm-xpack/20.llm-app-pathway.md Sets up the necessary components for the DocumentStore, including text splitting, embedding, retrieval strategy, and parsing. Uses OpenAI embeddings and a brute-force retrieval method. ```python text_splitter = TokenCountSplitter( min_tokens=100, max_tokens=500, encoding_name="cl100k_base" ) embedder = OpenAIEmbedder(api_key=os.environ["OPENAI_API_KEY"]) retriever_factory = BruteForceKnnFactory( embedder=embedder, ) parser = UnstructuredParser( chunking_mode="by_title", chunking_kwargs={ "max_characters": 3000, "new_after_n_chars": 2000, }, ) ``` -------------------------------- ### MCP Client Example Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/50.llm-xpack/40.pathway_mcp_server.md Connect to an MCP server using the fastmcp client, list available tools, and call a specific tool with arguments. ```python import asyncio from fastmcp import Client PATHWAY_MCP_URL = "http://localhost:8123/mcp/" client = Client(PATHWAY_MCP_URL) async def main(): async with client: tools = await client.list_tools() print(tools) async with client: result = await client.call_tool(name="get_constant_value", arguments={}) print(result) asyncio.run(main()) ``` -------------------------------- ### Install Sentence-Transformers Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/mistral_adaptive_rag_question_answering.ipynb Installs Sentence-Transformers for embedding chunked texts. ```bash !pip install sentence-transformers ``` -------------------------------- ### Initialize DocumentStore Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/50.llm-xpack/30.docs-indexing.md Create a DocumentStore to manage document processing and indexing. Requires data sources, a retriever factory, and a text splitter. ```python from pathway.xpacks.llm.document_store import DocumentStore from pathway.xpacks.llm.splitters import TokenCountSplitter import pathway as pw data_sources = pw.io.fs.read( "./sample_docs", format="binary", with_metadata=True, ) text_splitter = TokenCountSplitter() store = DocumentStore( docs=data_sources, retriever_factory=retriever_factory, splitter=text_splitter, ) ``` -------------------------------- ### Initialize Document Store and RAG Question Answerer Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/multimodal-rag.ipynb Creates a document store using a brute-force KNN index and configures the BaseRAGQuestionAnswerer with the LLM, document store, and search parameters. This prepares the system for question answering. ```python index = pw.indexing.BruteForceKnnFactory(embedder=embedder) doc_store = DocumentStore( sources, retriever_factory=index, splitter=splitter, parser=parser, ) app = BaseRAGQuestionAnswerer( llm=chat, indexer=doc_store, search_topk=6, ) app.build_server(host=app_host, port=app_port) ``` -------------------------------- ### Timely Dataflow Example with Progress Tracking Source: https://github.com/pathwaycom/pathway/blob/main/external/timely-dataflow/mdbook/src/chapter_1/chapter_1_3.md This example demonstrates initializing a Timely Dataflow worker, creating an input, exchanging data, inspecting output, and using a probe to track progress. It advances input and steps the worker until the probe indicates completion for a given time. ```rust extern crate timely; use timely::dataflow::InputHandle; use timely::dataflow::operators::{Input, Exchange, Inspect, Probe}; fn main() { // initializes and runs a timely dataflow. timely::execute_from_args(std::env::args(), |worker| { let index = worker.index(); let mut input = InputHandle::new(); // create a new input, exchange data, and inspect its output let probe = worker.dataflow(|scope| scope.input_from(&mut input) .exchange(|x| *x) .inspect(move |x| println!("worker {}:\thello {}", index, x)) .probe() ); // introduce data and watch! for round in 0..10 { if worker.index() == 0 { input.send(round); } input.advance_to(round + 1); worker.step_while(|| probe.less_than(input.time())); } }).unwrap(); } ``` -------------------------------- ### Install Python Dependencies Source: https://github.com/pathwaycom/pathway/blob/main/examples/projects/azure-aci-deploy/README.md Install the necessary Python packages listed in requirements.txt for the Pathway ACI deployment. ```bash pip install -r requirements.txt ``` -------------------------------- ### Initialize Pathway and Define Parameters Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/asof_join.ipynb Set up the Pathway license key and define parameters for fetching financial data, including ticker symbols, start date, and period. ```python import pandas as pd from yahooquery import Ticker import pathway as pw # To use advanced features with Pathway Scale, get your free license key from # https://pathway.com/features and paste it below. # To use Pathway Community, comment out the line below. pw.set_license_key("demo-license-key-with-telemetry") # Define the list of ticker symbols for the 5 major US companies tickers = ["AAPL", "GOOGL", "MSFT", "AMZN", "META"] start = "2023-01-01" fmt = "%Y-%m-%d" period = "1y" ``` -------------------------------- ### Install geopy Package Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/interval_join.ipynb Uncomment and run this cell to install the geopy package if it's not already present. ```python %%capture --no-display #!pip install geopy ``` -------------------------------- ### Initialize Pathway and LLM Components Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/multimodal-rag.ipynb Sets up the Pathway license, imports necessary modules, and initializes LLM components with retry and caching strategies. Use a valid license key for Pathway Scale or comment out the line for Community. ```python import pathway as pw # To use advanced features with Pathway Scale, get your free license key from # https://pathway.com/features and paste it below. # To use Pathway Community, comment out the line below. pw.set_license_key("demo-license-key-with-telemetry") from pathway.udfs import DefaultCache, ExponentialBackoffRetryStrategy from pathway.xpacks.llm import embedders, llms, parsers, prompts, splitters from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer from pathway.xpacks.llm.document_store import DocumentStore ``` -------------------------------- ### Install libmagic on MacOS Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/90.development/10.troubleshooting.md Install the libmagic dependency on MacOS using Homebrew to resolve 'UnstructuredParser UnsupportedFileFormatError'. ```bash brew install libmagic ``` -------------------------------- ### Install Tweepy Library Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/20.connect/99.connectors/30.custom-python-connectors.md Install the Tweepy library using pip. This is required for interacting with the Twitter API. ```bash pip install tweepy ``` -------------------------------- ### Configure Filesystem Persistence Backend Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/60.deployment/60.persistence_recovery.md Sets up a persistence backend using the local filesystem. It specifies the directory (`./PStorage`) where the computation snapshots will be stored. ```python backend = pw.persistence.Backend.filesystem("./PStorage") persistence_config = pw.persistence.Config(backend) ``` -------------------------------- ### Install Pathway in Colab Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/installation_first_steps.ipynb Installs Pathway in a Python 3.8+ Linux runtime within Google Colab. Requires a specific pip install link provided upon beta access. Ensure you are using Python 3.8 or higher. ```python #@title ⚙️ Pathway installer. Please provide the pip install link for Pathway: # Please copy here the installation line: PATHWAY_INSTALL_LINE='pip install https://packages.pathway.com/...' #@param {type:"string"} if PATHWAY_INSTALL_LINE.startswith('pip install '): PATHWAY_INSTALL_LINE=PATHWAY_INSTALL_LINE[len('pip install '):] class InterruptExecution(Exception): def _render_traceback_(self): pass if '...' in PATHWAY_INSTALL_LINE or not PATHWAY_INSTALL_LINE.startswith('https://'): print( "⛔ Please register at https://pathway.com/developers/documentation/introduction/installation-and-first-steps\n" "to Copy & Paste the Linux pip install line for Pathway!" ) raise InterruptExecution if 'macosx_11_0_arm64' in PATHWAY_INSTALL_LINE: PATHWAY_INSTALL_LINE = PATHWAY_INSTALL_LINE.replace('macosx_11_0_arm64','manylinux_2_12_x86_64.manylinux2010_x86_64') DO_INSTALL = False import sys if sys.version_info >= (3, 8): print(f'✅ Python {sys.version} is active.') try: import pathway as pw print('✅ Pathway successfully imported.') except: DO_INSTALL = True else: print("⛔ Pathway requires Python 3.8 or higher.") raise InterruptExecution if DO_INSTALL: !ls $(dirname $(which python))/../lib/python*/*-packages/pathway 1>/dev/null 2>/dev/null || echo "⌛ Installing Pathway. This usually takes a few seconds..." !ls $(dirname $(which python))/../lib/python*/*-packages/pathway 1>/dev/null 2>/dev/null || pip install {PATHWAY_INSTALL_LINE} 1>/dev/null 2>/dev/null !ls $(dirname $(which python))/../lib/python*/*-packages/pathway 1>/dev/null 2>/dev/null || echo "⛔ Installation failed. Don't be shy to reach out to the community at https://pathway.com !" !ls $(dirname $(which python))/../lib/python*/*-packages/pathway 1>/dev/null 2>/dev/null && echo "✅ All installed. Enjoy Pathway!" ``` -------------------------------- ### Import Libraries and Setup Environment Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/multimodal-rag-using-Gemini.ipynb Imports necessary Python libraries for logging, environment variable management, Google Generative AI, LiteLLM, and Pathway. Sets up the Pathway license key and logging level for LiteLLM. ```python import logging import os import google.generativeai as genai import litellm import pathway as pw # To use advanced features with Pathway Scale, get your free license key from # https://pathway.com/features and paste it below. # To use Pathway Community, comment out the line below. pw.set_license_key("demo-license-key-with-telemetry") from pathway.udfs import DiskCache, ExponentialBackoffRetryStrategy from pathway.xpacks.llm import embedders, llms, parsers, prompts, splitters from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer from pathway.xpacks.llm.vector_store import VectorStoreServer # Set the logging level for LiteLLM to DEBUG os.environ["LITELLM_LOG"] = "DEBUG" # to help in debugging ``` -------------------------------- ### Stop Docker Compose Example Source: https://github.com/pathwaycom/pathway/blob/main/examples/projects/twitter/README.md Command to stop and remove containers, networks, and volumes for a specific docker-compose example. ```bash docker-compose -f --env-file settings.env down -v ``` -------------------------------- ### Complete Kafka Integration Example Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/20.connect/99.connectors/30.kafka_connectors.md A full Python script demonstrating reading from a Kafka topic, processing data (calculating a sum), and writing results to another Kafka topic using Pathway. ```python import pathway as pw # Kafka settings rdkafka_settings = { "bootstrap.servers": "server-address:9092", "security.protocol": "sasl_ssl", "sasl.mechanism": "SCRAM-SHA-256", "group.id": "$GROUP_NAME", "session.timeout.ms": "6000", "sasl.username": "username", "sasl.password": "********", } class InputSchema(pw.Schema): value: int # We use the Kafka connector to listen to the "connector_example" topic t = pw.io.kafka.read( rdkafka_settings, topic="connector_example", format="csv", schema=InputSchema, autocommit_duration_ms=1000 ) # We compute the sum (this part is independent of the connectors). t = t.reduce(sum=pw.reducers.sum(t.value)) # We use the Kafka connector to send the resulting output stream containing the sum pw.io.kafka.write(t, rdkafka_settings, topic_name="sum", format="json") # We launch the computation. pw.run() ``` -------------------------------- ### Install Dependencies Source: https://github.com/pathwaycom/pathway/blob/main/examples/projects/ag2-multiagent-rag/README.md Install the necessary Python packages, including Pathway, AG2 with OpenAI support, requests, and python-dotenv. ```bash pip install -U pathway "ag2[openai]>=0.11.4,<1.0" requests python-dotenv ``` -------------------------------- ### Initialize Pathway and File System Connector Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/live_vector_indexing_pipeline.ipynb Initializes Pathway with a license key (or uses a demo key) and sets up a file system connector to monitor a local directory for data changes. This connector is configured for streaming binary data with metadata. ```python import pathway as pw # To use advanced features with Pathway Scale, get your free license key from # https://pathway.com/features and paste it below. # To use Pathway Community, comment out the line below. pw.set_license_key("demo-license-key-with-telemetry") # This creates a connector that tracks files in a given directory. data_sources = [] data_sources.append( pw.io.fs.read( "./sample_documents", format="binary", mode="streaming", with_metadata=True, ) ) # This creates a connector that tracks files in Google Drive. # Please follow the instructions at /developers/user-guide/connect/connectors/gdrive-connector/ to get credentials. # data_sources.append( # pw.io.gdrive.read(object_id="17H4YpBOAKQzEJ93xmC2z170l0bP2npMy", service_user_credentials_file="credentials.json", with_metadata=True)) ``` -------------------------------- ### Python Dockerfile with Pathway Installation Source: https://github.com/pathwaycom/pathway/blob/main/README.md Dockerfile using a standard Python image and installing Pathway via pip. ```dockerfile FROM --platform=linux/x86_64 python:3.10 RUN pip install -U pathway COPY ./pathway-script.py pathway-script.py CMD ["python", "-u", "pathway-script.py"] ``` -------------------------------- ### Initialize Streaming Input Table Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/20.connect/99.connectors/35.python-output-connectors.md Sets up a streaming input table using a demo function. This is the source data for the output connector. ```python import pathway as pw table = pw.demo.range_stream() ``` -------------------------------- ### Install libmagic on Debian/Ubuntu Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/90.development/10.troubleshooting.md Install the libmagic1 dependency on Debian-based Linux distributions or Google Colab to resolve 'UnstructuredParser UnsupportedFileFormatError'. ```bash apt install libmagic1 ``` -------------------------------- ### PostgreSQL Output Connector Example Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/10.introduction/30.pathway-overview.md Shows how to write data to a PostgreSQL table using provided connection settings. ```python pw.io.postgres.write(table, output_postgres_settings, "sum_table") ``` -------------------------------- ### Initialize Pathway and Load Data Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/showcases/langchain-integration.ipynb Initializes Pathway, sets a license key (or uses a demo key), and sets up a streaming read from the './data' directory. This prepares Pathway to process files from the specified location. ```python import pathway as pw # To use advanced features with Pathway Scale, get your free license key from # https://pathway.com/features and paste it below. # To use Pathway Community, comment out the line below. pw.set_license_key("demo-license-key-with-telemetry") from pathway.xpacks.llm.vector_store import VectorStoreServer from langchain_openai import OpenAIEmbeddings from langchain.text_splitter import CharacterTextSplitter data = pw.io.fs.read( "./data", format="binary", mode="streaming", with_metadata=True, ) ``` -------------------------------- ### Example Output of Timely Dataflow Steps Source: https://github.com/pathwaycom/pathway/blob/main/external/differential-dataflow/mdbook/src/chapter_a/chapter_a_3.md Illustrates the typical output format when running a timely dataflow computation with interactive loading and synchronization, showing elapsed time for data loading and individual steps. ```ignore Echidnatron% cargo run --release -- 10000000 Finished release [optimized] target(s) in 0.24s Running `target/release/my_project 10000000` 4.092895186s data loaded 4.092975626s step 1 complete 4.093021676s step 2 complete 4.093041130s step 3 complete 4.093110803s step 4 complete 4.093144075s step 5 complete 4.093187645s step 6 complete 4.093208245s step 7 complete 4.093236460s step 8 complete 4.093281793s step 9 complete ``` ```ignore 21.689493445s step 397525 complete 21.689522815s step 397526 complete 21.689553410s step 397527 complete 21.689593500s step 397528 complete 21.689643055s step 397529 complete ``` -------------------------------- ### Example Delay Output (Release) Source: https://github.com/pathwaycom/pathway/blob/main/external/differential-dataflow/server/README.md Example delay output from the `degr_dist` computation when running with release optimizations. Latencies are generally lower. ```text delays: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 16, 6, 26, 46, 14, 0, 9, 0, 0, 0, 0, 0, 0, 0] ``` ```text delays: [0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ``` ```text delays: [0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ``` ```text delays: [0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ``` ```text delays: [0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ``` ```text delays: [0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ``` ```text delays: [0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ``` -------------------------------- ### Use HFPipelineChat with GPT2 Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/50.llm-xpack/80.llm-chats.md Example of using the HFPipelineChat wrapper for local Hugging Face models. This example uses the 'gpt2' model. ```python from pathway.xpacks.llm import llms model = llms.HFPipelineChat( model="gpt2", # Choose the model you want ) responses = queries.select(result=model(pw.this.questions)) pw.debug.compute_and_print(responses) ``` -------------------------------- ### ImageParser Output Example Source: https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/50.llm-xpack/50.parsers.md Example of the structured output from ImageParser, including a textual description and extracted details based on a Pydantic schema. ```json { "parsed": [ [ "The image shows a happy Corgi dog running in a grassy area. The Corgi has a reddish-brown and white coat, a fluffy tail, and its tongue is out, giving it a cheerful expression. Its ears are perked up, and it appears to be wearing a red collar. The background is slightly blurred, emphasizing the dog in motion.", { "breed": "Pembroke Welsh Corgi", "surroundings": "outdoors in a grassy area", "color": "tan and white" } ] ], } ``` -------------------------------- ### Run the Pathway RAG Server Source: https://github.com/pathwaycom/pathway/blob/main/examples/notebooks/tutorials/rag-evaluations.ipynb Initializes and starts the RAG application server using QASummaryRestServer. The server runs in a separate process to handle incoming requests for question answering. ```python server = QASummaryRestServer(pathway_host, pathway_port, rag_app) server_process = multiprocessing.Process(target=server.run, kwargs=dict(threaded=False)) server_process.start() ```