### Downloading Example Parquet Data (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/README.md

Downloads a sample Parquet file (`prices.parquet`) from the DuckDB website using the wget command. This file serves as example data for the quick start guide. Requires wget to be installed and a network connection.

```bash
wget https://duckdb.org/data/prices.parquet
```

--------------------------------

### Installing smallpond Python Library Bash

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/getstarted.rst

Installs the smallpond library using pip. This command downloads and installs the required packages from PyPI. Requires Python 3.8+ and pip.

```bash
pip install smallpond
```

--------------------------------

### Viewing Ray Dashboard URL Bash

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/getstarted.rst

Shows an example of the output printed by smallpond upon startup, indicating the URL for the Ray Dashboard. The Ray Dashboard is used to monitor task execution within smallpond. Access this URL in a web browser.

```bash
... Started a local Ray instance. View the dashboard at http://127.0.0.1:8008
```

--------------------------------

### Installing smallpond Documentation Dependencies (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/README.md

Installs the smallpond library from the current directory in editable mode (`.`) along with the extra dependencies specified in the `docs` section of the project's setup file. This command is used when setting up the environment to build the project documentation. Requires the project source code and a Python environment.

```bash
pip install .[docs]
```

--------------------------------

### Serving Local Documentation (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/README.md

Starts a simple HTTP server using Python's built-in `http.server` module. It serves files from the specified `--directory`, which is the output location for the built HTML documentation (`build/html`). This allows you to view the generated documentation in a web browser locally. Requires Python to be installed.

```bash
python -m http.server --directory build/html
```

--------------------------------

### Installing smallpond Development Dependencies (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/README.md

Installs the smallpond library from the current directory in editable mode (`.`) along with the extra dependencies specified in the `dev` section of the project's setup file. This command is used when setting up the environment for contributing to the project. Requires the project source code and a Python environment.

```bash
pip install .[dev]
```

--------------------------------

### Initializing Driver and Submitting Job (Python)

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api/execution.rst

Demonstrates using the `smallpond.execution.driver.Driver` as the entry point to submit a job. It shows how to initialize the Driver, add custom command-line arguments (`-i`, `-n`), get arguments, build a logical plan using a placeholder function `my_pipeline`, and finally run the plan using `driver.run()`. Requires the `smallpond` library.

```python
from smallpond.execution.driver import Driver

if __name__ == "__main__":
   driver = Driver()
   # add your own arguments
   driver.add_argument("-i", "--input_paths", nargs="+")
   driver.add_argument("-n", "--npartitions", type=int, default=10)
   # build and run logical plan
   plan = my_pipeline(**driver.get_arguments())
   driver.run(plan)
```

--------------------------------

### Installing smallpond with pip (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/README.md

Installs the smallpond library using the Python package manager, pip. This command downloads and installs the latest version from PyPI. Requires Python 3.8 to 3.12 to be installed.

```bash
pip install smallpond
```

--------------------------------

### Partitioning smallpond DataFrame Python

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/getstarted.rst

Repartition a smallpond DataFrame into a specified number of partitions. Examples show repartitioning by files, by rows, and by hashing a column's value. Required for certain operations in smallpond.

```python
df = df.repartition(3)                 # repartition by files
df = df.repartition(3, by_row=True)    # repartition by rows
df = df.repartition(3, hash_by="host") # repartition by hash of column
```

--------------------------------

### Building HTML Documentation (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/README.md

Executes the `make html` command within the `docs` directory. This command typically uses Sphinx to process the documentation source files (like reStructuredText) and generate static HTML output. Requires a Makefile in the `docs` directory and the documentation dependencies to be installed.

```bash
make html
```

--------------------------------

### Loading Parquet Data with smallpond Python

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/getstarted.rst

Loads data from one or more Parquet files into a smallpond DataFrame. The method takes a path or pattern to the input files. Requires an initialized smallpond session.

```python
df = sp.read_parquet("path/to/dataset/*.parquet")
```

--------------------------------

### Initializing smallpond Session Python

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/getstarted.rst

Initializes a new smallpond session. This is the first step required before performing any data operations. It returns a session object used for subsequent actions like reading or transforming data.

```python
import smallpond

sp = smallpond.init()
```

--------------------------------

### Specifying Custom Platform via CLI Argument (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api/execution.rst

Command line example demonstrating how to instruct Smallpond to use a custom platform class defined in a specific Python file (`path.to.my.platform`) by providing the module path to the `--platform` command-line argument.

```bash
# run with your platform
# if using Driver
python script.py --platform path.to.my.platform
```

--------------------------------

### Running smallpond Unit Tests (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/README.md

Executes the unit tests for the smallpond project using the pytest framework. The `-v` flag provides verbose output, showing details of each test run. Requires pytest and the development dependencies to be installed.

```bash
pytest -v tests/test*.py
```

--------------------------------

### Specifying Built-in Platform (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api/execution.rst

Command line example showing how to execute a Python script (`script.py`) while specifying a built-in Smallpond execution platform (`mpi`) using the `--platform` command-line argument. This is typically used when the script utilizes the `Driver` or processes the `--platform` argument itself.

```bash
# run with your platform
python script.py --platform mpi
```

--------------------------------

### Transforming smallpond Data with Map Python

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/getstarted.rst

Transforms a smallpond DataFrame using the map operation. It supports applying SQL expressions or Python lambda functions to each row. This creates a new DataFrame with the results of the transformation.

```python
df = df.map('a + b as c')
df = df.map(lambda row: {'c': row['a'] + row['b']})
```

--------------------------------

### Saving smallpond Data to Parquet Python

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/getstarted.rst

Writes the content of a smallpond DataFrame to Parquet files at the specified output path. The DataFrame must have been processed or loaded previously. Requires write access to the output directory.

```python
df.write_parquet("path/to/output")
```

--------------------------------

### Specifying Custom Platform via Environment Variable (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api/execution.rst

Command line example demonstrating how to instruct Smallpond to use a custom platform class defined in a specific Python file (`path.to.my.platform`) by setting the `SP_PLATFORM` environment variable before executing the script. This method is useful when the script doesn't parse the `--platform` argument itself, like when using `smallpond.init`.

```bash
# if using smallpond.init
SP_PLATFORM=path.to.my.platform python script.py
```

--------------------------------

### Constructing and Running Low-level Smallpond Logical Plan (Python)

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api.rst

Shows how to define a low-level Smallpond pipeline by manually creating logical nodes (DataSourceNode, DataSetPartitionNode, SqlEngineNode) to build a static data flow graph. It defines a `my_pipeline` function that constructs a `LogicalPlan` and a main execution block that uses a `Driver` to parse command-line arguments and execute the plan using the built-in scheduler.

```python
from smallpond.logical.dataset import ParquetDataSet
from smallpond.logical.node import Context, DataSourceNode, DataSetPartitionNode, SqlEngineNode, LogicalPlan
from smallpond.execution.driver import Driver
from typing import List

def my_pipeline(input_paths: List[str], npartitions: int):
   ctx = Context()
   dataset = ParquetDataSet(input_paths)
   node = DataSourceNode(ctx, dataset)
   node = DataSetPartitionNode(ctx, (node,), npartitions=npartitions)
   node = SqlEngineNode(ctx, (node,), "SELECT * FROM {0}")
   return LogicalPlan(ctx, node)

if __name__ == "__main__":
   driver = Driver()
   driver.add_argument("-i", "--input_paths", nargs="+")
   driver.add_argument("-n", "--npartitions", type=int, default=10)

   plan = my_pipeline(**driver.get_arguments())
   driver.run(plan)
```

--------------------------------

### Initializing Runtime Context and Creating Execution Plan in Smallpond Python

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api/tasks.rst

This Python snippet demonstrates the initial steps to set up a Smallpond job. It involves creating a `RuntimeContext` to manage job resources, generating a `LogicalPlan` (represented here by a placeholder function), and then using a `Planner` to convert the logical plan into an `ExecutionPlan` ready for scheduling by a scheduler.

```python
# create a runtime context
runtime_ctx = RuntimeContext(JobId.new(), data_root)
runtime_ctx.initialize(socket.gethostname(), cleanup_root=True)

# create a logical plan
plan = create_logical_plan()

# create an execution plan
planner = Planner(runtime_ctx)
exec_plan = planner.create_exec_plan(plan)
```

--------------------------------

### Executing Low-level Smallpond Script (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api.rst

Provides the command-line instruction to execute the Python script demonstrating the low-level Smallpond API. It shows how to pass input file paths using the `-i` argument and the desired number of partitions using the `-n` argument to the script.

```bash
python script.py -i "path/to/*.parquet" -n 10
```

--------------------------------

### Performing Basic Data Processing with smallpond (Python)

Source: https://github.com/deepseek-ai/smallpond/blob/main/README.md

Demonstrates a basic data processing workflow using smallpond. It initializes a session, reads data from a Parquet file, repartitions the data, applies a SQL query to aggregate information, saves the results, and prints the final data as a Pandas DataFrame. Requires the `smallpond` library and the `prices.parquet` file.

```python
import smallpond

# Initialize session
sp = smallpond.init()

# Load data
df = sp.read_parquet("prices.parquet")

# Process data
df = df.repartition(3, hash_by="ticker")
df = sp.partial_sql("SELECT ticker, min(price), max(price) FROM {0} GROUP BY ticker", df)

# Save results
df.write_parquet("output/")
# Show results
print(df.to_pandas())
```

--------------------------------

### Implementing Custom Platform Class (Python)

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api/execution.rst

Shows the basic structure for creating a user-defined Smallpond execution platform by subclassing `smallpond.platform.Platform`. Custom platforms allow integrating Smallpond with specific execution environments by implementing methods like `start_job`. Requires the `smallpond` library.

```python
# path/to/my/platform.py
from smallpond.platform import Platform

class MyPlatform(Platform):
   def start_job(self, ...) -> List[str]:
      ...
```

--------------------------------

### Initializing Smallpond and Processing DataFrame (Python)

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api.rst

Demonstrates the basic workflow of the high-level Smallpond API using DataFrames. It initializes Smallpond, reads data from Parquet files, repartitions the DataFrame, applies a simple transformation using `map`, and writes the result to Parquet files. This API supports dynamic graph construction and uses Ray as the backend.

```python
import smallpond

sp = smallpond.init()

df = sp.read_parquet("path/to/dataset/*.parquet")
df = df.repartition(10)
df = df.map("x + 1")
df.write_parquet("path/to/output")
```

--------------------------------

### Building Logical Plan with Smallpond Nodes - Python

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api/nodes.rst

This snippet illustrates the typical workflow for creating a logical plan in Smallpond. It involves initializing a global context, defining a dataset (e.g., Parquet), creating a source node, applying transformations like data partitioning and SQL processing, and finally constructing the logical plan object from the root node.

```python
# Create a global context
ctx = Context()

# Create a dataset
dataset = ParquetDataSet("path/to/dataset/*.parquet")

# Create a data source node
node = DataSourceNode(ctx, dataset)

# Partition the data
node = DataSetPartitionNode(ctx, (node,), npartitions=2)

# Create a SQL engine node to transform the data
node = SqlEngineNode(ctx, (node,), "SELECT * FROM {0}")

# Create a logical plan from the root node
plan = LogicalPlan(ctx, node)
```

--------------------------------

### Changing Directory to Docs (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/README.md

Changes the current working directory to the `docs` subdirectory within the project's root directory. This step is necessary before executing commands that operate specifically on the documentation source files, such as building the documentation. Requires the `docs` directory to exist within the current location.

```bash
cd docs
```

--------------------------------

### Creating a ParquetDataSet in Python

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api/dataset.rst

Shows how to create a ParquetDataSet instance by providing a path pattern to Parquet files. This object represents a collection of Parquet files that can be processed by the smallpond library.

```python
dataset = ParquetDataSet("path/to/dataset/*.parquet")
```

--------------------------------

### Illustrating Typical Workflow with smallpond DataFrame - Python

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/api/dataframe.rst

This snippet demonstrates a common workflow using smallpond. It shows how to initialize a session, read data from parquet files, repartition the DataFrame to a specific number of partitions, apply a simple transformation using the map function, and finally write the processed data back to parquet files. It illustrates the chaining of operations on a lazily computed DataFrame before triggering execution via a write operation.

```Python
import smallpond

sp = smallpond.init()

df = sp.read_parquet("path/to/dataset/*.parquet")
df = df.repartition(10)
df = df.map("x + 1")
df.write_parquet("path/to/output")
```

--------------------------------

### Illustrating Data Root Directory Structure (Bash)

Source: https://github.com/deepseek-ai/smallpond/blob/main/docs/source/internals.rst

This snippet displays the typical directory structure found within Smallpond's `data_root`. It shows how data for a specific job, identified by job time and ID, is organized into subdirectories for configuration (`config`), logs (`log`), message queue (`queue`), output (`output`), intermediate staging data (`staging`), and temporary files (`temp`).

```bash
data_root
└── 2024-12-11-12-00-28.2cc39990-296f-48a3-8063-78cf6dca460b # job_time.job_id
    ├── config  # configuration and state
    │   ├── exec_plan.pickle
    │   ├── logical_plan.pickle
    │   └── runtime_ctx.pickle
    ├── log     # logs
    │   ├── graph.png
    │   └── scheduler.log
    ├── queue   # message queue between scheduler and workers
    ├── output  # output data
    ├── staging # intermediate data
    │   ├── DataSourceTask.000001
    │   ├── EvenlyDistributedPartitionProducerTask.000002
    │   ├── completed_tasks  # output dataset of completed tasks
    │   └── started_tasks    # used for checkpoint
    └── temp    # temporary data
        ├── DataSourceTask.000001
        └── EvenlyDistributedPartitionProducerTask.000002
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.