Apache Spark

https://spark.apache.org/docs/3.5.1/
Admin
Apache Spark is a unified analytics engine for large-scale data processing with high-level APIs in...
Tokens:13,982,091
Snippets:55,107
Trust Score:10
Update:1 year ago
Show doc for...
Tokens:
Raw
### Apache Spark 3.5.1 Installation Guide

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

Details on how to install Apache Spark version 3.5.1. This guide covers prerequisites and steps for setting up Spark on various environments.

```text
Installation:
Refer to the official Apache Spark documentation for the most up-to-date installation instructions for version 3.5.1. This typically involves downloading the pre-built Spark package and configuring your environment variables.
```

--------------------------------

### Apache Spark 3.5.1 Quickstart: Spark Connect

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

A quickstart guide for using Spark Connect with Apache Spark version 3.5.1. This covers connecting to a Spark cluster remotely and executing commands.

```python
# Quickstart: Spark Connect
from pyspark.sql import SparkSession

# Connect to a remote Spark cluster
spark = SparkSession.builder \
    .remote("sc://localhost:15002") \
    .getOrCreate()

# Example DataFrame operation using Spark Connect
data = [("Alice", 1), ("Bob", 2)]
columns = ["name", "id"]
df = spark.createDataFrame(data, columns)

df.show()

spark.stop()
```

--------------------------------

### Apache Spark 3.5.1 Quickstart: DataFrame

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

A quickstart guide for using DataFrames in Apache Spark version 3.5.1. It covers basic operations like creating DataFrames, loading data, and performing transformations.

```python
# Quickstart: DataFrame operations
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataFrameQuickstart").getOrCreate()

# Create a DataFrame
data = [("Alice", 34), ("Bob", 45), ("Charlie", 29)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)

# Show DataFrame
df.show()

# Filter DataFrame
df.filter(df.Age > 30).show()

spark.stop()
```

--------------------------------

### Apache Spark 3.5.1 Quickstart: Pandas API on Spark

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

A quickstart guide for using the Pandas API on Spark with Apache Spark version 3.5.1. This allows users familiar with Pandas to leverage Spark's distributed computing power.

```python
# Quickstart: Pandas API on Spark
import pyspark.pandas as ps

# Create a Pandas DataFrame
pd_df = ps.DataFrame({'col1': [1, 2], 'col2': [3, 4]})

# Convert to Spark DataFrame (ps.DataFrame is already a Spark DataFrame)
spark_df = pd_df.to_spark()

# Perform operations using Pandas API on Spark
ps_df = spark_df.to_pandas_on_spark()
ps_df['col3'] = ps_df['col1'] + ps_df['col2']

print(ps_df)

```

--------------------------------

### Conda Environment Setup for PySpark

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/install

Creates and activates a Conda environment, then installs PySpark along with a specified Python version and other packages.

```bash
# can also add "python=3.8 some_package [etc.]" here
conda install pyspark python=3.9
```

--------------------------------

### PySpark ALS setColdStartStrategy Example

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/ml/recommendation

Example of setting the cold start strategy for the ALS model in PySpark.

```python
from pyspark.ml.recommendation import ALS

algs = ALS()
algs.setColdStartStrategy("nan")
```

--------------------------------

### PySpark Connect Quickstart

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

Instructions for using Spark Connect, a new client-server architecture for Spark. It covers launching a Spark server with Spark Connect and connecting to it from a client.

```python
# Launching Spark server with Spark Connect (command line)
# $SPARK_HOME/sbin/start-connect-server.sh

# Connecting to Spark Connect server (Python client)
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .remote("sc://localhost:15002") \
    .getOrCreate()

# Now you can use Spark APIs, e.g., create a DataFrame
data = [("a", 1), ("b", 2)]
columns = ["col1", "col2"]
df = spark.createDataFrame(data, columns)
df.show()

spark.stop()
```

--------------------------------

### Launch Spark Connect Server

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/quickstart_connect

This snippet shows how to launch the Spark server with support for Spark Connect sessions by executing the start-connect-server.sh script. It requires specifying the Spark version.

```bash
!$HOME/sbin/start-connect-server.sh$SPARK_VERSION

```

--------------------------------

### Getting the Number of Rows in CoordinateMatrix

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/mllib/linalg/distributed

Illustrates how to get the number of rows in a CoordinateMatrix. The example creates a CoordinateMatrix from an RDD of entries and then prints the number of rows.

```python
from pyspark.mllib.linalg.distributed import CoordinateMatrix, MatrixEntry

entries = sc.parallelize([MatrixEntry(0, 0, 1.2),
                          MatrixEntry(1, 0, 2),
                          MatrixEntry(2, 1, 3.7)])

mat = CoordinateMatrix(entries)
# print(mat.numRows())
# 3
```

--------------------------------

### Manual PySpark Download and Setup

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/install

Demonstrates how to extract a Spark distribution tarball and set the SPARK_HOME and PYTHONPATH environment variables to enable PySpark usage.

```bash
tar xzvf spark-master-bin-hadoop3.tgz
cd spark-master-bin-hadoop3
export SPARK_HOME=`pwd`
export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[_]}"):$PYTHONPATH
```

--------------------------------

### Spark Partition Data Setup and Output Retrieval

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/ml/torch/distributor

Handles the setup of Spark partition data and retrieval of output from a framework wrapper. It uses a context manager for setup and then calls a helper function to get the output. Includes cleanup for log streaming.

```python
try:
    with TorchDistributor._setup_spark_partition_data(iterator, schema_json):
        output = TorchDistributor._get_output_from_framework_wrapper(
            framework_wrapper_fn,
            input_params,
            train_object,
            run_pytorch_file_fn,
            *args,
            **kwargs,
        )
finally:
    try:
        LogStreamingClient._destroy()
    except BaseException:
        pass
```

--------------------------------

### PySpark DataFrame Quickstart

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

A guide to performing common operations with PySpark DataFrames, including creation, viewing, selection, applying functions, grouping, data input/output, and SQL integration.

```python
# Example: DataFrame Creation
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataFrameExample").getOrCreate()
data = [("Alice", 1), ("Bob", 2)]
columns = ["name", "id"]
df = spark.createDataFrame(data, columns)
df.show()

# Example: Selecting and Accessing Data
df.select("name").show()
df.select(df.name, df.id + 1).show()

# Example: Grouping Data
# Assuming a DataFrame with a 'department' column
# df.groupBy("department").count().show()

# Example: Working with SQL
df.createOrReplaceTempView("people")
sql_results = spark.sql("SELECT * FROM people WHERE id > 1")
sql_results.show()

spark.stop()
```

--------------------------------

### Building Spark Package

Source: https://spark.apache.org/docs/3.5.1/api/python/development/setting_ide

This snippet indicates the process of building the Spark package after cloning the source code. SBT is recommended for faster builds.

```bash
build package
```

--------------------------------

### Example UDTF Class Implementation

Source: https://spark.apache.org/docs/3.5.1/api/python/user_guide/sql/python_udtf

A practical example of a Python UDTF class named `SquareNumbers`. This UDTF takes a start and end integer, and for each number in the range, it yields a tuple containing the number and its square.

```python
# Define the UDTF class and implement the required `eval` method.
class SquareNumbers:
    def eval(self, start: int, end: int):
        for num in range(start, end + 1):
            yield (num, num * num)

```

--------------------------------

### Get Current Timestamp (now)

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/sql/functions

Returns the current timestamp at the start of query evaluation. This function is available from version 3.5.0.

```python
df = spark.range(1)
df.select(now()).show(truncate=False) # doctest: +SKIP
```

--------------------------------

### PySpark Installation with Release Mirror

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/install

Installs PySpark and specifies a release mirror for faster downloads using PYSPARK_RELEASE_MIRROR and PYSPARK_HADOOP_VERSION environment variables.

```bash
PYSPARK_RELEASE_MIRROR=http://mirror.apache-kr.orgPYSPARK_HADOOP_VERSION=3
```

--------------------------------

### PySpark Installation Options

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

Details various methods for installing PySpark, including using PyPI, Conda, manual downloads, and building from source. It also covers supported Python versions and dependencies.

```python
# Installation using PyPI
pip install pyspark

# Installation using Conda
conda install -c conda-forge pyspark

# Manual Download and Setup (example for Linux/macOS)
# wget <spark_download_url>
# tar -xzf spark-*.tgz
# export SPARK_HOME=/path/to/spark
# export PATH=$SPARK_HOME/bin:$PATH

# Building from Source (requires build tools like sbt, Java, Scala)
# Refer to Spark documentation for detailed build instructions.
```

--------------------------------

### pyspark.pandas.date_range Examples

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/pandas/namespace

Demonstrates the usage of pyspark.pandas.date_range with different 'closed' parameters to control inclusivity of the start and end dates.

```python
import pyspark.pandas as ps

# Default behavior (inclusive of both start and end)
ps.date_range(
    start='2017-01-01', end='2017-01-04'
)  # doctest: +SKIP
# DatetimeIndex(['2017-01-01', '2017-01-02', '2017-01-03', '2017-01-04'],
#                dtype='datetime64[ns]', freq=None)

# Use ``closed='left'`` to exclude `end` if it falls on the boundary.
ps.date_range(
    start='2017-01-01', end='2017-01-04', closed='left'
)  # doctest: +SKIP
# DatetimeIndex(['2017-01-01', '2017-01-02', '2017-01-03'], dtype='datetime64[ns]', freq=None)

# Use ``closed='right'`` to exclude `start` if it falls on the boundary.
ps.date_range(
    start='2017-01-01', end='2017-01-04', closed='right'
)  # doctest: +SKIP
# DatetimeIndex(['2017-01-02', '2017-01-03', '2017-01-04'], dtype='datetime64[ns]', freq=None)
```

--------------------------------

### Get Current Timestamp

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/functions

The `localtimestamp` function returns the current timestamp without time zone at the start of query evaluation.

```python
from pyspark.sql.functions import localtimestamp

# Example usage:
df.withColumn("current_ts", localtimestamp())
```

--------------------------------

### PySpark User Guides

Source: https://spark.apache.org/docs/3.5.1/api/python/user_guide/python_packaging

Guides for using PySpark, covering topics like Python package management, Spark SQL, and the Pandas API on Spark. Includes details on Arrow, UDTFs, and best practices.

```APIDOC
Python Package Management:
  Using PySpark Native Features:
    Details on integrating PySpark with native Python environments.

  Using Conda:
    Instructions for managing PySpark dependencies with Conda.

  Using Virtualenv:
    Guidance on using Python virtual environments with PySpark.

  Using PEX:
    Information on packaging PySpark applications with PEX.

Spark SQL:
  Apache Arrow in PySpark:
    Explains the integration and usage of Apache Arrow for performance improvements in PySpark.

  Python User-defined Table Functions (UDTFs):
    Covers the creation and usage of Python UDTFs in Spark SQL.

Pandas API on Spark:
  Overview:
    Introduction to the Pandas API on Spark, enabling users to leverage familiar Pandas syntax with Spark's distributed computing power.

  Options and settings:
    Details on configuring and managing options for the Pandas API on Spark.

  From/to pandas and PySpark DataFrames:
    Instructions on converting between Pandas DataFrames, PySpark DataFrames, and the Pandas API on Spark DataFrames.

  Transform and apply a function:
    Explains how to use `transform` and `apply` for custom data manipulation within the Pandas API on Spark.

  Type Support in Pandas API on Spark:
    Details on the data type mappings and support between Pandas and Spark.

  Type Hints in Pandas API on Spark:
    Guidance on using type hints for better code clarity and static analysis.

  From/to other DBMSes:
    Information on reading from and writing to various database management systems.

  Best Practices:
    Recommendations for optimizing the use of the Pandas API on Spark.

  Supported pandas API:
    Lists the subset of pandas API functions that are supported in the Pandas API on Spark.

  FAQ:
    Frequently asked questions regarding the Pandas API on Spark.
```

--------------------------------

### Spark Application Setup and Testing

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/mllib/util

This snippet demonstrates how to set up a Spark application context, run doctests, and manage the application lifecycle. It includes setting global variables for Spark and SparkContext, executing tests with specified options, stopping the Spark session, and exiting with an error code if tests fail.

```python
globs["spark"] = spark
globs["sc"] = spark.sparkContext
(failure_count, test_count) = doctest.testmod(globs=globs, optionflags=doctest.ELLIPSIS)
spark.stop()
if failure_count:
    sys.exit(-1)
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.ss/core_classes

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.ss/io

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.StreamingQuery

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### Cloning Spark Source Code

Source: https://spark.apache.org/docs/3.5.1/api/python/development/setting_ide

This snippet shows the command to clone the Spark source code from GitHub. It is essential for setting up the development environment.

```bash
git clone <git_url>
```

--------------------------------

### PySpark Development: Environment Setup

Source: https://spark.apache.org/docs/3.5.1/api/python/development/contributing

Details on setting up the development environment for PySpark, including prerequisites and instructions for using Conda or pip for package management.

```python
# Conda environment setup
conda create -n pyspark-dev python=3.9
conda activate pyspark-dev
conda install -c conda-forge "pyspark=3.5.1"

# pip environment setup
pip install "pyspark==3.5.1"

```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/dataframe

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PandasOnSparkTestCase Setup

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/testing/pandasutils

Sets up the PandasOnSparkTestCase by calling the superclass's setUpClass and enabling the Spark Arrow configuration.

```python
class PandasOnSparkTestCase(ReusedSQLTestCase, PandasOnSparkTestUtils):
    @classmethod
    def setUpClass(cls):
        super(PandasOnSparkTestCase, cls).setUpClass()
        cls.spark.conf.set(SPARK_CONF_ARROW_ENABLED, True)
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.Column

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrameNaFunctions

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### IDE Setup for PySpark

Source: https://spark.apache.org/docs/3.5.1/api/python/development/index

Instructions for configuring Integrated Development Environments (IDEs) for PySpark development, with specific guidance for PyCharm.

```python
# Setting up IDEs
# PyCharm
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.pandas/io

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark Live Notebooks

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

Provides links to live, interactive notebooks for trying out PySpark functionalities without local setup. These notebooks cover DataFrames, Spark Connect, and the Pandas API on Spark.

```python
from IPython.display import display, Markdown

markdown_content = """
There are live notebooks where you can try PySpark out without any other step:
  * [Live Notebook: DataFrame](https://mybinder.org/v2/gh/apache/spark/master?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_df.ipynb)
  * [Live Notebook: Spark Connect](https://mybinder.org/v2/gh/apache/spark/master?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_connect.ipynb)
  * [Live Notebook: pandas API on Spark](https://mybinder.org/v2/gh/apache/spark/master?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_ps.ipynb)
"""

display(Markdown(markdown_content))
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.pandas/frame

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark Development Guide

Source: https://spark.apache.org/docs/3.5.1/api/python/development/index

This section outlines the process for contributing to PySpark, including setting up the development environment, adhering to coding standards, and contributing documentation.

```python
# Contributing to PySpark
# Environment Setup
# Code and Docstring Guide
# Contributing Error and Exception
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.dt

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark Unit Test Setup with unittest

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/testing_pyspark

Sets up a Spark session for PySpark tests using the unittest framework's setUpClass and tearDownClass methods. This ensures a Spark session is available for tests and properly stopped afterward.

```python
import unittest

class PySparkTestCase(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.spark = SparkSession.builder.appName("Testing PySpark Example").getOrCreate()


    @classmethod
    def tearDownClass(cls):
        cls.spark.stop()

```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.mllib.stat

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### Build Spark and Run PySpark Tests

Source: https://spark.apache.org/docs/3.5.1/api/python/development/testing

Instructions on how to build Spark and subsequently run PySpark test cases using the `python/run-tests` script. It also mentions setting an environment variable for Mac OS users.

```bash
# Build Spark (example using Maven)
mvn clean package -Pscala-2.12 -DskipTests

# Run PySpark tests
python/run-tests
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.mllib.evaluation

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### Initialize SparkSession

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/quickstart_df

Initializes the SparkSession, which is the entry point for PySpark applications. If running in the PySpark shell, the session is automatically created.

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.ml

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### Install PySpark with Extra Dependencies

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/install

Demonstrates how to install PySpark with specific extra dependencies for components like Spark SQL or the Pandas API on Spark. It also shows how to include Plotly for data visualization.

```python
pip install pyspark[sql]
pip install pyspark[pandas_on_spark]
# to plot your data, you can install plotly together.
pip install pyspark[pandas_on_spark,plotly]
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.SparkFiles

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### Apache Spark 3.5.1 User Guides

Source: https://spark.apache.org/docs/3.5.1/api/python/development/index

This section contains user guides for Apache Spark version 3.5.1, offering practical instructions and explanations for using Spark effectively in various data processing scenarios.

```APIDOC
Apache Spark 3.5.1 User Guides:

This collection of user guides provides in-depth explanations and practical examples for utilizing Apache Spark's features. It covers topics ranging from basic concepts to advanced techniques for data manipulation, analysis, and application development.

Topics include:
- DataFrame operations
- Spark SQL queries
- Performance tuning
- Working with different data formats (Parquet, ORC, JSON, CSV)
- Cluster deployment and management

These guides are essential for developers and data scientists looking to leverage Spark for their big data needs.
```

--------------------------------

### PySpark Streaming Query Example

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/sql/streaming/readwriter

Demonstrates starting and stopping a basic streaming query using the rate source and console sink.

```python
>>> df = spark.readStream.format("rate").load()
>>> df = df.selectExpr("value % 3 as v")
>>> q = df.writeStream.format("console").start()
>>> time.sleep(3)
>>> q.stop()
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/protobuf

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PEX File Creation for Spark

Source: https://spark.apache.org/docs/3.5.1/api/python/user_guide/python_packaging

Example command to create a .pex file containing Python dependencies for use with Apache Spark. This file is self-contained and executable.

```bash
pex
"-r", "requirements.txt",
"-o", "pyspark_pex_env.pex"
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/grouping

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### Install Spark Release Candidate

Source: https://spark.apache.org/docs/3.5.1/api/python/development/contributing

Installs a specific release candidate of Apache Spark using pip. This is a common method for testing pre-release versions.

```python
pip install pyspark==3.0.0rc1
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.UDTFRegistration

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### SparkSession Initialization and Configuration

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/sql/session

Details the constructor for SparkSession, including how it handles existing sessions, Spark Connect mode, and configuration options.

```python
from pyspark.sql import SparkSession
from pyspark.context import SparkContext
from typing import Dict, Any, Optional
from py4j.java_gateway import JavaObject

# This is a simplified representation of the __init__ method
# Actual implementation involves JVM interaction and error handling.

class SparkSession:
    _instantiatedSession: Optional["SparkSession"] = None
    _activeSession: Optional["SparkSession"] = None

    def __init__(
        self,
        sparkContext: SparkContext,
        jsparkSession: Optional[JavaObject] = None,
        options: Dict[str, Any] = {},
    ):
        self._sc = sparkContext
        self._jsc = self._sc._jsc
        self._jvm = self._sc._jvm

        assert self._jvm is not None

        if jsparkSession is None:
            if (
                self._jvm.SparkSession.getDefaultSession().isDefined()
                and not self._jvm.SparkSession.getDefaultSession().get().sparkContext().isStopped()
            ):
                jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
                getattr(getattr(self._jvm, "SparkSession$"), "MODULE$").applyModifiableSettings(
                    jsparkSession, options
                )
            else:
                jsparkSession = self._jvm.SparkSession(self._jsc.sc(), options)
        else:
            getattr(getattr(self._jvm, "SparkSession$"), "MODULE$").applyModifiableSettings(
                jsparkSession, options
            )
        self._jsparkSession = jsparkSession
        # ... other initialization steps ...

        if (
            SparkSession._instantiatedSession is None
            or SparkSession._instantiatedSession._sc._jsc is None
        ):
            SparkSession._instantiatedSession = self
            SparkSession._activeSession = self
            assert self._jvm is not None
            self._jvm.SparkSession.setDefaultSession(self._jsparkSession)
            self._jvm.SparkSession.setActiveSession(self._jsparkSession)

    @property
    def _jconf(self) -> "JavaObject":
        """Accessor for the JVM SQL-specific configurations"""
        return self._jsparkSession.sessionState().conf()

    def _repr_html_(self) -> str:
        return """
            <div>
                <p><b>SparkSession - {catalogImplementation}</b></p>
{sc_HTML}
            </div>
        """.format(
            catalogImplementation=self.conf.get("spark.sql.catalogImplementation"),
            sc_HTML=self.sparkContext._repr_html_(),
        )

```
Tokens:
Raw