### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.Window

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.SparkSession.builder

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.ml.recommendation

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.mllib.recommendation

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.Catalog

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/index

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.UDFRegistration

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/core_classes

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/udf

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.ml.image

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.mllib.tree

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SparkContext Singleton and Configuration

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/udtf

Provides methods for getting or creating a SparkContext instance and accessing Spark configuration. It also includes properties for Spark user and start time.

```APIDOC
pyspark.SparkContext.getOrCreate(conf=None)
  Get current SparkContext or create a new one if needed.
  Parameters:
    conf: SparkConf object.
  Returns: SparkContext instance.

pyspark.SparkContext.sparkUser
  The user running the Spark application.

pyspark.SparkContext.startTime
  The start time of the SparkContext in milliseconds since epoch.
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.ss/query_management

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SparkContext Utility and Information API

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.protobuf.functions

API documentation for PySpark SparkContext utility methods and information retrieval, including creating an empty RDD, listing files and archives, and accessing Spark user and start time.

```APIDOC
pyspark.SparkContext.dump_profiles(profile_dir: str)
  Dumps the Spark profiling information to the specified directory.
  Parameters:
    profile_dir: The directory to dump profiling information.

pyspark.SparkContext.getOrCreate(conf=None)
  Gets an existing SparkContext or creates a new one if none exists.
  Parameters:
    conf: SparkConf object.
  Returns: The SparkContext instance.

pyspark.SparkContext.listArchives()
  Returns a list of archives that are added to the SparkContext.

pyspark.SparkContext.listFiles()
  Returns a list of files that are added to the SparkContext.

pyspark.SparkContext.show_profiles(profile_dir: str)
  Shows the Spark profiling information from the specified directory.
  Parameters:
    profile_dir: The directory containing profiling information.

pyspark.SparkContext.sparkUser()
  Returns the user of Spark.

pyspark.SparkContext.startTime()
  Returns the start time of the SparkContext.
```

--------------------------------

### Apache Spark 3.5.1 Installation Guide

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

Details on how to install Apache Spark version 3.5.1. This guide covers prerequisites and steps for setting up Spark on various environments.

```text
Installation:
Refer to the official Apache Spark documentation for the most up-to-date installation instructions for version 3.5.1. This typically involves downloading the pre-built Spark package and configuring your environment variables.
```

--------------------------------

### PySpark ALS setColdStartStrategy Example

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/ml/recommendation

Example of setting the cold start strategy for the ALS model in PySpark.

```python
from pyspark.ml.recommendation import ALS

algs = ALS()
algs.setColdStartStrategy("nan")
```

--------------------------------

### Getting the Number of Rows in CoordinateMatrix

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/mllib/linalg/distributed

Illustrates how to get the number of rows in a CoordinateMatrix. The example creates a CoordinateMatrix from an RDD of entries and then prints the number of rows.

```python
from pyspark.mllib.linalg.distributed import CoordinateMatrix, MatrixEntry

entries = sc.parallelize([MatrixEntry(0, 0, 1.2),
                          MatrixEntry(1, 0, 2),
                          MatrixEntry(2, 1, 3.7)])

mat = CoordinateMatrix(entries)
# print(mat.numRows())
# 3
```

--------------------------------

### Spark Partition Data Setup and Output Retrieval

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/ml/torch/distributor

Handles the setup of Spark partition data and retrieval of output from a framework wrapper. It uses a context manager for setup and then calls a helper function to get the output. Includes cleanup for log streaming.

```python
try:
    with TorchDistributor._setup_spark_partition_data(iterator, schema_json):
        output = TorchDistributor._get_output_from_framework_wrapper(
            framework_wrapper_fn,
            input_params,
            train_object,
            run_pytorch_file_fn,
            *args,
            **kwargs,
        )
finally:
    try:
        LogStreamingClient._destroy()
    except BaseException:
        pass
```

--------------------------------

### Apache Spark 3.5.1 Quickstart: Spark Connect

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

A quickstart guide for using Spark Connect with Apache Spark version 3.5.1. This covers connecting to a Spark cluster remotely and executing commands.

```python
# Quickstart: Spark Connect
from pyspark.sql import SparkSession

# Connect to a remote Spark cluster
spark = SparkSession.builder \
    .remote("sc://localhost:15002") \
    .getOrCreate()

# Example DataFrame operation using Spark Connect
data = [("Alice", 1), ("Bob", 2)]
columns = ["name", "id"]
df = spark.createDataFrame(data, columns)

df.show()

spark.stop()
```

--------------------------------

### PySpark SparkContext Utility and Information API

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.WindowSpec

API documentation for PySpark SparkContext utility methods and information retrieval, including creating an empty RDD, listing files and archives, and accessing Spark user and start time.

```APIDOC
pyspark.SparkContext.dump_profiles(profile_dir: str)
  Dumps the Spark profiling information to the specified directory.
  Parameters:
    profile_dir: The directory to dump profiling information.

pyspark.SparkContext.getOrCreate(conf=None)
  Gets an existing SparkContext or creates a new one if none exists.
  Parameters:
    conf: SparkConf object.
  Returns: The SparkContext instance.

pyspark.SparkContext.listArchives()
  Returns a list of archives that are added to the SparkContext.

pyspark.SparkContext.listFiles()
  Returns a list of files that are added to the SparkContext.

pyspark.SparkContext.show_profiles(profile_dir: str)
  Shows the Spark profiling information from the specified directory.
  Parameters:
    profile_dir: The directory containing profiling information.

pyspark.SparkContext.sparkUser()
  Returns the user of Spark.

pyspark.SparkContext.startTime()
  Returns the start time of the SparkContext.
```

--------------------------------

### PySpark SparkContext Utility and Information API

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/configuration

API documentation for PySpark SparkContext utility methods and information retrieval, including creating an empty RDD, listing files and archives, and accessing Spark user and start time.

```APIDOC
pyspark.SparkContext.dump_profiles(profile_dir: str)
  Dumps the Spark profiling information to the specified directory.
  Parameters:
    profile_dir: The directory to dump profiling information.

pyspark.SparkContext.getOrCreate(conf=None)
  Gets an existing SparkContext or creates a new one if none exists.
  Parameters:
    conf: SparkConf object.
  Returns: The SparkContext instance.

pyspark.SparkContext.listArchives()
  Returns a list of archives that are added to the SparkContext.

pyspark.SparkContext.listFiles()
  Returns a list of files that are added to the SparkContext.

pyspark.SparkContext.show_profiles(profile_dir: str)
  Shows the Spark profiling information from the specified directory.
  Parameters:
    profile_dir: The directory containing profiling information.

pyspark.SparkContext.sparkUser()
  Returns the user of Spark.

pyspark.SparkContext.startTime()
  Returns the start time of the SparkContext.
```

--------------------------------

### Apache Spark 3.5.1 Quickstart: DataFrame

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

A quickstart guide for using DataFrames in Apache Spark version 3.5.1. It covers basic operations like creating DataFrames, loading data, and performing transformations.

```python
# Quickstart: DataFrame operations
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataFrameQuickstart").getOrCreate()

# Create a DataFrame
data = [("Alice", 34), ("Bob", 45), ("Charlie", 29)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)

# Show DataFrame
df.show()

# Filter DataFrame
df.filter(df.Age > 30).show()

spark.stop()
```

--------------------------------

### Example UDTF Class Implementation

Source: https://spark.apache.org/docs/3.5.1/api/python/user_guide/sql/python_udtf

A practical example of a Python UDTF class named `SquareNumbers`. This UDTF takes a start and end integer, and for each number in the range, it yields a tuple containing the number and its square.

```python
# Define the UDTF class and implement the required `eval` method.
class SquareNumbers:
    def eval(self, start: int, end: int):
        for num in range(start, end + 1):
            yield (num, num * num)

```

--------------------------------

### PySpark SparkContext Utility and Information API

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/window

API documentation for PySpark SparkContext utility methods and information retrieval, including creating an empty RDD, listing files and archives, and accessing Spark user and start time.

```APIDOC
pyspark.SparkContext.dump_profiles(profile_dir: str)
  Dumps the Spark profiling information to the specified directory.
  Parameters:
    profile_dir: The directory to dump profiling information.

pyspark.SparkContext.getOrCreate(conf=None)
  Gets an existing SparkContext or creates a new one if none exists.
  Parameters:
    conf: SparkConf object.
  Returns: The SparkContext instance.

pyspark.SparkContext.listArchives()
  Returns a list of archives that are added to the SparkContext.

pyspark.SparkContext.listFiles()
  Returns a list of files that are added to the SparkContext.

pyspark.SparkContext.show_profiles(profile_dir: str)
  Shows the Spark profiling information from the specified directory.
  Parameters:
    profile_dir: The directory containing profiling information.

pyspark.SparkContext.sparkUser()
  Returns the user of Spark.

pyspark.SparkContext.startTime()
  Returns the start time of the SparkContext.
```

--------------------------------

### Apache Spark 3.5.1 Quickstart: Pandas API on Spark

Source: https://spark.apache.org/docs/3.5.1/api/python/getting_started/index

A quickstart guide for using the Pandas API on Spark with Apache Spark version 3.5.1. This allows users familiar with Pandas to leverage Spark's distributed computing power.

```python
# Quickstart: Pandas API on Spark
import pyspark.pandas as ps

# Create a Pandas DataFrame
pd_df = ps.DataFrame({'col1': [1, 2], 'col2': [3, 4]})

# Convert to Spark DataFrame (ps.DataFrame is already a Spark DataFrame)
spark_df = pd_df.to_spark()

# Perform operations using Pandas API on Spark
ps_df = spark_df.to_pandas_on_spark()
ps_df['col3'] = ps_df['col1'] + ps_df['col2']

print(ps_df)

```

--------------------------------

### PySpark SparkContext Utility and Information API

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.pandas/indexing

API documentation for PySpark SparkContext utility methods and information retrieval, including creating an empty RDD, listing files and archives, and accessing Spark user and start time.

```APIDOC
pyspark.SparkContext.dump_profiles(profile_dir: str)
  Dumps the Spark profiling information to the specified directory.
  Parameters:
    profile_dir: The directory to dump profiling information.

pyspark.SparkContext.getOrCreate(conf=None)
  Gets an existing SparkContext or creates a new one if none exists.
  Parameters:
    conf: SparkConf object.
  Returns: The SparkContext instance.

pyspark.SparkContext.listArchives()
  Returns a list of archives that are added to the SparkContext.

pyspark.SparkContext.listFiles()
  Returns a list of files that are added to the SparkContext.

pyspark.SparkContext.show_profiles(profile_dir: str)
  Shows the Spark profiling information from the specified directory.
  Parameters:
    profile_dir: The directory containing profiling information.

pyspark.SparkContext.sparkUser()
  Returns the user of Spark.

pyspark.SparkContext.startTime()
  Returns the start time of the SparkContext.
```

--------------------------------

### Get Current Timestamp (now)

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/sql/functions

Returns the current timestamp at the start of query evaluation. This function is available from version 3.5.0.

```python
df = spark.range(1)
df.select(now()).show(truncate=False) # doctest: +SKIP
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.mllib.recommendation

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SparkContext Utility and Information API

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/catalog

API documentation for PySpark SparkContext utility methods and information retrieval, including creating an empty RDD, listing files and archives, and accessing Spark user and start time.

```APIDOC
pyspark.SparkContext.dump_profiles(profile_dir: str)
  Dumps the Spark profiling information to the specified directory.
  Parameters:
    profile_dir: The directory to dump profiling information.

pyspark.SparkContext.getOrCreate(conf=None)
  Gets an existing SparkContext or creates a new one if none exists.
  Parameters:
    conf: SparkConf object.
  Returns: The SparkContext instance.

pyspark.SparkContext.listArchives()
  Returns a list of archives that are added to the SparkContext.

pyspark.SparkContext.listFiles()
  Returns a list of files that are added to the SparkContext.

pyspark.SparkContext.show_profiles(profile_dir: str)
  Shows the Spark profiling information from the specified directory.
  Parameters:
    profile_dir: The directory containing profiling information.

pyspark.SparkContext.sparkUser()
  Returns the user of Spark.

pyspark.SparkContext.startTime()
  Returns the start time of the SparkContext.
```

--------------------------------

### pyspark.pandas.date_range Examples

Source: https://spark.apache.org/docs/3.5.1/api/python/_modules/pyspark/pandas/namespace

Demonstrates the usage of pyspark.pandas.date_range with different 'closed' parameters to control inclusivity of the start and end dates.

```python
import pyspark.pandas as ps

# Default behavior (inclusive of both start and end)
ps.date_range(
    start='2017-01-01', end='2017-01-04'
)  # doctest: +SKIP
# DatetimeIndex(['2017-01-01', '2017-01-02', '2017-01-03', '2017-01-04'],
#                dtype='datetime64[ns]', freq=None)

# Use ``closed='left'`` to exclude `end` if it falls on the boundary.
ps.date_range(
    start='2017-01-01', end='2017-01-04', closed='left'
)  # doctest: +SKIP
# DatetimeIndex(['2017-01-01', '2017-01-02', '2017-01-03'], dtype='datetime64[ns]', freq=None)

# Use ``closed='right'`` to exclude `start` if it falls on the boundary.
ps.date_range(
    start='2017-01-01', end='2017-01-04', closed='right'
)  # doctest: +SKIP
# DatetimeIndex(['2017-01-02', '2017-01-03', '2017-01-04'], dtype='datetime64[ns]', freq=None)
```

--------------------------------

### Get Current Timestamp

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/functions

The `localtimestamp` function returns the current timestamp without time zone at the start of query evaluation.

```python
from pyspark.sql.functions import localtimestamp

# Example usage:
df.withColumn("current_ts", localtimestamp())
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.testing

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.ss/core_classes

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Singleton and Configuration

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.ml.recommendation

Provides methods for getting or creating a SparkContext instance and accessing Spark configuration. It also includes properties for Spark user and start time.

```APIDOC
pyspark.SparkContext.getOrCreate(conf=None)
  Get current SparkContext or create a new one if needed.
  Parameters:
    conf: SparkConf object.
  Returns: SparkContext instance.

pyspark.SparkContext.sparkUser
  The user running the Spark application.

pyspark.SparkContext.startTime
  The start time of the SparkContext in milliseconds since epoch.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.ss/io

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.StreamingQuery

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Singleton and Configuration

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.ml.torch.distributor

Provides methods for getting or creating a SparkContext instance and accessing Spark configuration. It also includes properties for Spark user and start time.

```APIDOC
pyspark.SparkContext.getOrCreate(conf=None)
  Get current SparkContext or create a new one if needed.
  Parameters:
    conf: SparkConf object.
  Returns: SparkContext instance.

pyspark.SparkContext.sparkUser
  The user running the Spark application.

pyspark.SparkContext.startTime
  The start time of the SparkContext in milliseconds since epoch.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/dataframe

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Singleton and Configuration

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.pandas/window

Provides methods for getting or creating a SparkContext instance and accessing Spark configuration. It also includes properties for Spark user and start time.

```APIDOC
pyspark.SparkContext.getOrCreate(conf=None)
  Get current SparkContext or create a new one if needed.
  Parameters:
    conf: SparkConf object.
  Returns: SparkContext instance.

pyspark.SparkContext.sparkUser
  The user running the Spark application.

pyspark.SparkContext.startTime
  The start time of the SparkContext in milliseconds since epoch.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.Column

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrameNaFunctions

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility and Information API

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.conf

API documentation for PySpark SparkContext utility methods and information retrieval, including creating an empty RDD, listing files and archives, and accessing Spark user and start time.

```APIDOC
pyspark.SparkContext.dump_profiles(profile_dir: str)
  Dumps the Spark profiling information to the specified directory.
  Parameters:
    profile_dir: The directory to dump profiling information.

pyspark.SparkContext.getOrCreate(conf=None)
  Gets an existing SparkContext or creates a new one if none exists.
  Parameters:
    conf: SparkConf object.
  Returns: The SparkContext instance.

pyspark.SparkContext.listArchives()
  Returns a list of archives that are added to the SparkContext.

pyspark.SparkContext.listFiles()
  Returns a list of files that are added to the SparkContext.

pyspark.SparkContext.show_profiles(profile_dir: str)
  Shows the Spark profiling information from the specified directory.
  Parameters:
    profile_dir: The directory containing profiling information.

pyspark.SparkContext.sparkUser()
  Returns the user of Spark.

pyspark.SparkContext.startTime()
  Returns the start time of the SparkContext.
```

--------------------------------

### PySpark SQL Miscellaneous Functions

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.UDTFRegistration

Includes utility functions such as `input_file_block_start` for getting the start of the input file block, `user` for the current user, and `version` for the Spark version.

```python
from pyspark.sql.functions import input_file_block_start, user, version

# Example usage:
# print(spark.sparkContext.parallelize([1]).map(input_file_block_start).collect())
# print(user())
# print(version())
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.pandas/io

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.pandas/frame

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility and Information API

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.ml.image

API documentation for PySpark SparkContext utility methods and information retrieval, including creating an empty RDD, listing files and archives, and accessing Spark user and start time.

```APIDOC
pyspark.SparkContext.dump_profiles(profile_dir: str)
  Dumps the Spark profiling information to the specified directory.
  Parameters:
    profile_dir: The directory to dump profiling information.

pyspark.SparkContext.getOrCreate(conf=None)
  Gets an existing SparkContext or creates a new one if none exists.
  Parameters:
    conf: SparkConf object.
  Returns: The SparkContext instance.

pyspark.SparkContext.listArchives()
  Returns a list of archives that are added to the SparkContext.

pyspark.SparkContext.listFiles()
  Returns a list of files that are added to the SparkContext.

pyspark.SparkContext.show_profiles(profile_dir: str)
  Shows the Spark profiling information from the specified directory.
  Parameters:
    profile_dir: The directory containing profiling information.

pyspark.SparkContext.sparkUser()
  Returns the user of Spark.

pyspark.SparkContext.startTime()
  Returns the start time of the SparkContext.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.dt

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Singleton and Configuration

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/pyspark.sql/api/pyspark.sql.Window

Provides methods for getting or creating a SparkContext instance and accessing Spark configuration. It also includes properties for Spark user and start time.

```APIDOC
pyspark.SparkContext.getOrCreate(conf=None)
  Get current SparkContext or create a new one if needed.
  Parameters:
    conf: SparkConf object.
  Returns: SparkContext instance.

pyspark.SparkContext.sparkUser
  The user running the Spark application.

pyspark.SparkContext.startTime
  The start time of the SparkContext in milliseconds since epoch.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.mllib.stat

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Utility Methods

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.mllib.evaluation

Utility methods for SparkContext, including getting the Spark user, start time, and managing profiles.

```APIDOC
pyspark.SparkContext.dump_profiles() -> None
  Dumps the current profiling information.

pyspark.SparkContext.getOrCreate(conf=None, jsc=None)
  Get a SparkContext in the current application, or create a new one if none exists.
  Parameters:
    conf: SparkConf object.
    jsc: Java SparkContext object.

pyspark.SparkContext.listArchives() -> List[str]
  Returns a list of archive URIs that are added to the SparkContext.

pyspark.SparkContext.listFiles(path: str, recursive: bool = False) -> List[str]
  Returns a list of file URIs that are added to the SparkContext.
  Parameters:
    path: The path to list files from.
    recursive: Whether to list files recursively.

pyspark.SparkContext.resources() -> Dict[str, Dict[str, int]]
  Returns the resources associated with the SparkContext.

pyspark.SparkContext.show_profiles() -> None
  Shows the current profiling information.

pyspark.SparkContext.sparkUser() -> str
  Returns the name of the user running the Spark application.

pyspark.SparkContext.startTime() -> int
  Returns the time when the Spark application was started.
```

--------------------------------

### PySpark SparkContext Singleton and Configuration

Source: https://spark.apache.org/docs/3.5.1/api/python/reference/api/pyspark.mllib.tree

Provides methods for getting or creating a SparkContext instance and accessing Spark configuration. It also includes properties for Spark user and start time.

```APIDOC
pyspark.SparkContext.getOrCreate(conf=None)
  Get current SparkContext or create a new one if needed.
  Parameters:
    conf: SparkConf object.
  Returns: SparkContext instance.

pyspark.SparkContext.sparkUser
  The user running the Spark application.

pyspark.SparkContext.startTime
  The start time of the SparkContext in milliseconds since epoch.
```