### SparkContext example from class documentation

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a SparkContext, set the log level, and print the application name. This example is from the SparkContext class documentation and uses the same pattern as the module-level example.

```python
>>> sc = SparkContext("MyApp")
>>> sc.setLogLevel("WARN")
>>> print(sc.appName)
MyApp

```

--------------------------------

### Install Sparkless with documentation tools

Source: https://sparkless.readthedocs.io/en/latest/_sources/installation.rst.txt

Installs Sparkless along with tools for building documentation.

```bash
pip install sparkless[docs]
```

--------------------------------

### Install native pandas backend with pip install .[pandas]

Source: https://sparkless.readthedocs.io/en/latest/_sources/performance/pandas_fallback.md.txt

Installs the optional pandas extra, which brings in pandas and type stubs. NumPy wheels are pulled in transitively where available.

```bash
pip install .[pandas]
```

--------------------------------

### make_date example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Constructs a date from year, month, day integers. The example uses literals.

```python
>>> df.select(F.make_date(F.lit(2024), F.lit(3), F.lit(15)))
```

--------------------------------

### locate example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/string.html

Locates the position of a substring starting from a position (1-indexed). The example is not fully shown in the source.

```python
>>> df.select(F.locate("substr", F.col("text")))
```

--------------------------------

### Example: Using map_keys and showing DataFrame

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Shows a complete example of creating a SparkSession, creating a DataFrame with a map column, and using F.map_keys to extract keys and display the result.

```python
>>> from sparkless.sql import SparkSession, functions as F
>>> spark = SparkSession("test")
>>> data = [{"properties": {"key1": "val1", "key2": "val2"}}]
>>> df = spark.createDataFrame(data)
>>> df.select(F.map_keys(F.col("properties"))).show()
DataFrame[1 rows, 1 columns]
map_keys(properties)
['key1', 'key2']

```

--------------------------------

### SQL Logging Output Example

Source: https://sparkless.readthedocs.io/en/latest/_sources/migration_from_pyspark.md.txt

Example of generated SQL logs when debug mode is enabled.

```text
[DEBUG] Operation: filter
[DEBUG] SQL: SELECT * FROM temp_table_123 WHERE age > 25
```

--------------------------------

### Module docstring with usage example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/spark_types.html

Shows the module's purpose and a usage example for creating a schema with StructType and StructField.

```python
"""
Mock data types and schema system for Sparkless.

This module provides comprehensive mock implementations of PySpark data types
and schema structures that behave identically to the real PySpark types.
Includes primitive types, complex types, schema definitions, and Row objects
for complete type system compatibility.

Key Features:
    - Complete PySpark data type hierarchy
    - Primitive types (String, Integer, Long, Double, Boolean)
    - Complex types (Array, Map, Struct)
    - Schema definition with StructType and StructField
    - Row objects with PySpark-compatible interface
    - Type inference and conversion utilities

Example:
    >>> from sparkless.spark_types import StringType, IntegerType, StructType, StructField
    >>> schema = StructType([
    ...     StructField("name", StringType()),
    ...     StructField("age", IntegerType())
    ... ])
    >>> df = spark.createDataFrame(data, schema)
"""
```

--------------------------------

### Catalog module docstring with usage example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/session/catalog.html

Module-level docstring for the mock Catalog implementation, showing a usage example with Catalog and its methods.

```python
"""
Mock Catalog implementation for Sparkless.

This module provides a mock implementation of PySpark's Catalog
that behaves identically to the real Catalog for testing and development.
It includes database and table management, caching operations, and
catalog queries without requiring a JVM or actual Spark installation.

Key Features:
    - Complete PySpark Catalog API compatibility
    - Database management (create, list, drop)
    - Table management (create, list, drop, cache)
    - Schema validation and error handling
    - Integration with storage manager

Example:
    >>> from sparkless.session import Catalog
    >>> catalog = Catalog(storage_manager)
    >>> catalog.createDatabase("test_db")
    >>> catalog.listDatabases()
    [Database(name='test_db')]
"""
```

--------------------------------

### Module docstring with example usage

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/map.html

The module docstring includes a usage example showing how to create a DataFrame and use map_keys.

```python
"""
Map functions for Sparkless.

This module provides comprehensive map manipulation functions that match PySpark's
map function API. Includes operations for extracting keys, values, entries, and
combining maps for working with map columns in DataFrames.

Key Features:
    - Complete PySpark map function API compatibility
    - Key/value extraction (map_keys, map_values)
    - Entry operations (map_entries)
    - Map combination (map_concat, map_from_arrays)
    - Type-safe operations with proper return types
    - Support for both column references and map literals

Example:
    >>> from sparkless.sql import SparkSession, functions as F
    >>> spark = SparkSession("test")
    >>> data = [{"properties": {"key1": "val1", "key2": "val2"}}]
    >>> df = spark.createDataFrame(data)
    >>> df.select(F.map_keys(F.col("properties"))).show()
    DataFrame[1 rows, 1 columns]
    map_keys(properties)
    ['key1', 'key2']
"""
```

--------------------------------

### WindowFunction.__lt__ example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/window_execution.html

Less than comparison for a window function result. Example: F.row_number().over(window) < 5.

```python
>>> F.row_number().over(window) < 5
```

--------------------------------

### Install Sparkless with dev extras

Source: https://sparkless.readthedocs.io/en/latest/_sources/getting_started.md.txt

Installs Sparkless with development dependencies for testing tools.

```bash
pip install sparkless[dev]
```

--------------------------------

### Example benchmark output

Source: https://sparkless.readthedocs.io/en/latest/_sources/performance/pandas_fallback.md.txt

Example output from the benchmark script on an M3 Pro laptop with macOS 14.6.1, showing mean times for stub and native backends across operations.

```text
Backend    | Create (mean ms)   | to_dict (mean ms) | concat (mean ms)  | iloc (mean ms)
------------------------------------------------------------------------
stub       | 4.812              | 2.337             | 3.921             | 0.118
native     | 7.604              | 1.281             | 2.017             | 0.064
```

--------------------------------

### Module docstring with usage examples

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/dataframe/reader.html

The module docstring includes a usage example showing how to create a SparkSession and read from a table or with format and options.

```python
"""
Mock DataFrameReader implementation for DataFrame read operations.

This module provides DataFrame reading functionality, maintaining compatibility
with PySpark's DataFrameReader interface. Supports reading from various data sources
including tables, files, and custom storage backends.

Key Features:
    - Complete PySpark DataFrameReader API compatibility
    - Support for multiple data formats (parquet, json, csv, table)
    - Flexible options configuration
    - Integration with storage manager
    - Schema inference and validation
    - Error handling for missing data sources

Example:
    >>> from sparkless.sql import SparkSession
    >>> spark = SparkSession("test")
    >>> # Read from table
    >>> df = spark.read.table("my_table")
    >>> # Read with format and options
    >>> df = spark.read.format("parquet").option("header", "true").load("/path")
"""
```

--------------------------------

### WindowFunction.__mul__ example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/window_execution.html

Multiplies a window function result by a value. Example: 100 * F.percent_rank().over(window).

```python
>>> 100 * F.percent_rank().over(window)
```

--------------------------------

### Install Sparkless v3.0.0 with pip

Source: https://sparkless.readthedocs.io/en/latest/_sources/migration_from_v2_to_v3.md.txt

Install the v3.0.0 release of Sparkless. This is the first step in the migration process.

```bash
# Install v3.0.0
pip install sparkless>=3.0.0
```

--------------------------------

### Usage Examples

Source: https://sparkless.readthedocs.io/en/latest/_sources/api/types.rst.txt

Examples of constructing simple and complex schemas using Sparkless data types.

```APIDOC
## Usage Examples

### Description
Demonstrates how to import and use Sparkless data types to create simple and complex schemas.

### Code Example
```python
from sparkless.sql.types import (
    StructType, StructField, StringType, IntegerType,
    ArrayType, MapType, DoubleType
)

# Simple schema
schema = StructType([
    StructField("name", StringType(), True),
    StructField("age", IntegerType(), False)
])

# Complex schema with arrays and maps
complex_schema = StructType([
    StructField("id", IntegerType(), False),
    StructField("tags", ArrayType(StringType()), True),
    StructField("metadata", MapType(StringType(), StringType()), True),
    StructField("score", DoubleType(), True)
])
```
```

--------------------------------

### WindowFunction.__ne__ example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/window_execution.html

Inequality comparison for a window function result. Example: F.row_number().over(window) != 0.

```python
>>> F.row_number().over(window) != 0
```

--------------------------------

### overlay example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Replaces part of a string with another string starting at a position (PySpark 3.0+). The example replaces 3 characters starting at position 5 in 'text' with 'NEW'.

```python
>>> df.select(F.overlay(F.col("text"), F.lit("NEW"), F.lit(5), F.lit(3)))

```

--------------------------------

### Configuration example

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a Configuration instance, set a configuration key, and retrieve it.

```python
>>> from sparkless.session.config import Configuration
>>> conf = Configuration()
>>> conf.set("spark.app.name", "MyApp")
>>> conf.get("spark.app.name")
'MyApp'

```

--------------------------------

### create_map examples

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Creates a map from alternating key-value columns/literals using F.create_map. The second example shows that calling with no arguments returns an empty map {}.

```python
>>> df.select(F.create_map(F.col("k1"), F.col("v1"), F.col("k2"), F.col("v2")))
>>> df.select(F.create_map())  # Returns empty map {}

```

--------------------------------

### dayname example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets the name of the day of the week. The example uses a date column.

```python
>>> df.select(F.dayname(F.col("date")))
```

--------------------------------

### Configuration _config example

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a Configuration instance and set/get a configuration value.

```python
>>> conf = Configuration()
>>> conf.set("spark.app.name", "MyApp")
>>> conf.get("spark.app.name")
'MyApp'

```

--------------------------------

### JSON array length example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Example of using F.json_array_length to get the length of a JSON array.

```python
>>> df.select(F.json_array_length(F.col("json_col"), "$.array"))
```

--------------------------------

### Complete SQL pipeline example

Source: https://sparkless.readthedocs.io/en/latest/_sources/sql_operations_guide.md.txt

A full workflow: create a Spark session, create sample data, parse a SQL query, validate, optimize, and execute. Shows how to handle validation errors.

```python
from sparkless.sql import SparkSession
from sparkless.session.sql import SQLParser, SQLValidator, SQLQueryOptimizer, SQLExecutor

# Create Spark session
spark = SparkSession("SQLExample")

# Create sample data
data = [
    {"name": "Alice", "age": 25, "department": "Engineering"},
    {"name": "Bob", "age": 30, "department": "Marketing"},
    {"name": "Charlie", "age": 35, "department": "Engineering"}
]
df = spark.createDataFrame(data)
df.createOrReplaceTempView("employees")

# SQL Pipeline
parser = SQLParser()
validator = SQLValidator()
optimizer = SQLQueryOptimizer()
executor = SQLExecutor(spark)

# Parse SQL query
query = "SELECT department, COUNT(*) as count FROM employees WHERE age > 25 GROUP BY department ORDER BY count DESC"
ast = parser.parse(query)

# Validate query
if validator.validate(ast):
    # Optimize query
    optimized_ast = optimizer.optimize(ast)
    
    # Execute query
    result = executor.execute(optimized_ast)
    
    # Display results
    result.show()
else:
    print("Query validation failed:")
    for error in validator.get_errors():
        print(f"  - {error}")
```

--------------------------------

### WindowFunction.__eq__ example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/window_execution.html

Equality comparison for a window function result. Example: F.row_number().over(window) == 1.

```python
>>> F.row_number().over(window) == 1
```

--------------------------------

### getItem example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets an item from an array by index or from a map by key. The example shows accessing an array element, a map value, and an out-of-bounds access that returns None.

```python
>>> df.select(F.col("array_col").getItem(0))
>>> df.select(F.col("map_col").getItem("key"))
>>> df.select(F.col("array_col").getItem(999))  # Returns None if out of bounds
```

--------------------------------

### parquet method with example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/dataframe/writer.html

Convenience method to save DataFrame in Parquet format. Example shows usage: df.write.parquet("/path/to/file.parquet").

```python
    def parquet(self, path: str, **options: Any) -> None:
        """Save DataFrame in Parquet format.

        Args:
            path: Path to save the Parquet file.
            **options: Additional options for Parquet format.

        Example:
            >>> df.write.parquet("/path/to/file.parquet")
        """
        self.format("parquet").options(**options).save(path)
```

--------------------------------

### Catalog __init__ method

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/session/catalog.html

Initializes Catalog with storage manager and optional SparkSession.

```python
[docs]
    def __init__(self, storage: IStorageManager, spark: Optional[Any] = None):
        """Initialize Catalog.

        Args:
            storage: Storage manager instance.
            spark: Optional SparkSession instance for SQL-based operations.
                  If provided, createDatabase() will use SQL instead of direct storage calls.
        """
        self._storage = storage
        self.spark = spark
        self._cached_tables: Set[str] = set()  # Track cached tables
```

--------------------------------

### Extract window start time with F.window_time

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Use F.window_time to extract the start time from a window column. The example groups by a 1-hour window and aliases the extracted time as 'window_start'.

```python
>>> df.groupBy(F.window("timestamp", "1 hour")).agg(
...     F.window_time(F.col("window")).alias("window_start")
... )
```

--------------------------------

### WindowFunction.__truediv__ example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/window_execution.html

Divides a window function result by a value. Example: F.row_number().over(window) / 10.

```python
>>> F.row_number().over(window) / 10
```

--------------------------------

### Module docstring with usage example for sparkless.functions.crypto

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/crypto.html

The module docstring includes a usage example showing how to create a SparkSession, create a DataFrame, and call aes_encrypt and aes_decrypt functions. This example is part of the module's documentation and demonstrates the intended usage.

```python
"""
Cryptographic functions for Sparkless.

This module provides cryptographic functions that match PySpark's crypto function API.
Includes encryption and decryption operations for secure data processing in DataFrames.

Key Features:
    - AES encryption and decryption
    - Null-safe cryptographic operations
    - Type-safe operations with proper return types
    - Support for both column references and string literals

Example:
    >>> from sparkless.sql import SparkSession, functions as F
    >>> spark = SparkSession("test")
    >>> data = [{"data": "sensitive information", "key": "secretkey"}]
    >>> df = spark.createDataFrame(data)
    >>> df.select(
    ...     F.aes_encrypt(F.col("data"), F.col("key")),
    ...     F.aes_decrypt(F.col("encrypted"), F.col("key"))
    ... ).show()
"""
```

--------------------------------

### weekday example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets the day of week as an integer (0=Monday, 6=Sunday) using F.weekday.

```python
>>> df.select(F.weekday(F.col("date")))
```

--------------------------------

### slice: Extract array slice

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Extracts a slice of the array starting at a 1-based position for a given length. The example extracts 3 elements starting from position 2 of the 'nums' array.

```python
>>> df.select(F.slice(F.col("nums"), 2, 3))
```

--------------------------------

### Create and query a mock Catalog in Sparkless

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Creates a Catalog instance with a storage manager, creates a database, and lists databases. The output shows the expected list of Database objects.

```python
>>> from sparkless.session import Catalog
>>> catalog = Catalog(storage_manager)
>>> catalog.createDatabase("test_db")
>>> catalog.listDatabases()
[Database(name='test_db')]

```

--------------------------------

### last_day example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Shows how to use F.last_day to get the last day of the month for a date column.

```python
>>> df.select(F.last_day(F.col('date')))

```

--------------------------------

### Create table, insert data, and query with SQL

Source: https://sparkless.readthedocs.io/en/latest/_sources/storage_api_guide.md.txt

Use these SQL commands to create a table with a schema, insert data, and query it. The query result is displayed with `show()`.

```python
# Create table with schema
spark.sql("""
    CREATE TABLE IF NOT EXISTS users (
        id INT,
        name STRING,
        age INT
    )
""")

# Insert data
spark.sql("""
    INSERT INTO users VALUES
    (1, 'Alice', 25),
    (2, 'Bob', 30),
    (3, 'Charlie', 35)
""")

# Query table
result = spark.sql("SELECT * FROM users WHERE age > 25")
result.show()
```

--------------------------------

### array_join examples

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Joins array elements with a delimiter. Two examples show joining with a comma and with a pipe and null replacement.

```python
>>> df.select(F.array_join(F.col("tags"), ", "))
```

```python
>>> df.select(F.array_join(F.col("tags"), "|", "N/A"))
```

--------------------------------

### user example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets the current user name using the user function (PySpark 3.5+).

```python
>>> df.select(F.user())
```

--------------------------------

### FileSchema.__init__(name, base_path)

Source: https://sparkless.readthedocs.io/en/latest/api/backend.html

Initializes a file schema.

```APIDOC
## FileSchema.__init__(name, base_path)

### Description
Initializes a file schema.

### Method
Constructor

### Endpoint
`FileSchema.__init__(name, base_path)`

### Parameters
- **name** (`str`) - Required - Schema name.
- **base_path** (`str`) - Required - Base path for schema files.

### Returns
- **None**

```

--------------------------------

### csv method with example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/dataframe/writer.html

Convenience method to save DataFrame in CSV format. Example shows usage: df.write.csv("/path/to/file.csv").

```python
    def csv(self, path: str, **options: Any) -> None:
        """Save DataFrame in CSV format.

        Args:
            path: Path to save the CSV file.
            **options: Additional options for CSV format.

        Example:
            >>> df.write.csv("/path/to/file.csv")
        """
        self.format("csv").options(**options).save(path)
```

--------------------------------

### overlay example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Replaces part of the 'text' column starting at position 5 with 'NEW' for length 3.

```python
>>> df.select(F.overlay(F.col("text"), F.lit("NEW"), F.lit(5), F.lit(3)))
```

--------------------------------

### ConfigBuilder.build()

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/session/config/configuration.html

Builds and returns the Configuration instance.

```APIDOC
## ConfigBuilder.build()

### Description
Builds and returns the Configuration instance.

### Method
build

### Returns
Configuration - Configuration instance.
```

--------------------------------

### SparkSession usage example

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a SparkSession, create a DataFrame from a list of dictionaries, display it, and execute a SQL command. The output shows the DataFrame contents and the SQL command is executed without visible output.

```python
>>> from sparkless.sql import SparkSession
>>> spark = SparkSession("MyApp")
>>> data = [{"name": "Alice", "age": 25}]
>>> df = spark.createDataFrame(data)
>>> df.show()
DataFrame[1 rows, 2 columns]
age name
25    Alice
>>> spark.sql("CREATE DATABASE test")

```

--------------------------------

### locate example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Returns a ColumnOperation representing locate, which locates the position of a substring starting from a given position (1-indexed).

```python
>>> df.select(F.locate("spark", F.col("text"), 1))
```

--------------------------------

### SparkContext usage example

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a SparkContext, set the log level, and print the application name. The output is 'MyApp'.

```python
>>> from sparkless.session import SparkContext
>>> sc = SparkContext("MyApp")
>>> sc.setLogLevel("WARN")
>>> print(sc.appName)
MyApp

```

--------------------------------

### element_at: Get element at index (1-based, negative for reverse)

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Returns the element at the specified index. Index is 1-based; negative indices count from the end. The example shows getting the first and last elements of the 'nums' array.

```python
>>> df.select(F.element_at(F.col("nums"), 1))  # First element
>>> df.select(F.element_at(F.col("nums"), -1))  # Last element
```

--------------------------------

### Bug report template example

Source: https://sparkless.readthedocs.io/en/latest/_sources/robin_sparkless_issues.md.txt

Minimal Python example for a bug report, showing how to import robin_sparkless. Replace the comment with actual code that triggers the bug.

```python
# Example:
import robin_sparkless
# ...
```

--------------------------------

### array_join examples

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Joins array elements with a delimiter. Two examples show default null replacement and custom null replacement.

```python
>>> df.select(F.array_join(F.col("tags"), ", "))
>>> df.select(F.array_join(F.col("tags"), "|", "N/A"))

```

--------------------------------

### Set and get Spark configuration

Source: https://sparkless.readthedocs.io/en/latest/_sources/api_reference.md.txt

Sets and retrieves a configuration value using the SparkSession's conf object. The example uses the key 'spark.sql.debug'.

```python
# Set configuration
spark.conf.set("spark.sql.debug", "true")

# Get configuration
debug_mode = spark.conf.get("spark.sql.debug")
```

--------------------------------

### FileTable.__init__(name, schema, file_path)

Source: https://sparkless.readthedocs.io/en/latest/api/backend.html

Initializes a file table.

```APIDOC
## FileTable.__init__(name, schema, file_path)

### Description
Initializes a file table.

### Method
Constructor

### Endpoint
`FileTable.__init__(name, schema, file_path)`

### Parameters
- **name** (`str`) - Required - Table name.
- **schema** (`StructType`) - Required - Table schema.
- **file_path** (`str`) - Required - Path to table data file.

### Returns
- **None**

```

--------------------------------

### WindowFunction.__init__

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Initialize WindowFunction.

```APIDOC
## WindowFunction.__init__

### Description
Initialize WindowFunction.

### Method
__init__

### Parameters
- **function** (`Any`) - Required - The window function (e.g., row_number(), rank()).
- **window_spec** (`WindowSpec`) - Required - The window specification.
```

--------------------------------

### For Test Utilities: sparkless Convenience API

Source: https://sparkless.readthedocs.io/en/latest/mock_spark_features.html

Example for test utilities: uses the sparkless convenience API for test setup and returns the Spark session as a pytest fixture.

```python
# Good: Convenient for tests, but sparkless-specific
@pytest.fixture
def setup_test_data(spark):
    spark._storage.create_schema("test")
    schema = StructType([StructField("id", IntegerType())])
    spark._storage.create_table("test", "data", schema)
    return spark

```

--------------------------------

### Configuration via Session Builder

Source: https://sparkless.readthedocs.io/en/latest/_sources/backend_architecture.md.txt

Shows how to configure backend selection using the session builder's .config() method, including default, explicit Polars, memory, and file backends with basePath.

```python
from sparkless.sql import SparkSession

# Default backend (Polars) - v3.0.0+
spark = SparkSession("MyApp")

# Explicit backend selection
spark = SparkSession.builder \
    .config("spark.sparkless.backend", "polars") \
    .getOrCreate()

# Memory backend for lightweight testing
spark = SparkSession.builder \
    .config("spark.sparkless.backend", "memory") \
    .getOrCreate()

# File backend for persistent storage
spark = SparkSession.builder \
    .config("spark.sparkless.backend", "file") \
    .config("spark.sparkless.backend.basePath", "/tmp/sparkless") \
    .getOrCreate()
```

--------------------------------

### aggregate: Reduce array elements to a single value

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Aggregates array elements using an accumulator pattern. The example sums the 'nums' array starting from 0 with a lambda that adds each element to the accumulator.

```python
>>> df.select(F.aggregate(F.col("nums"), F.lit(0), lambda acc, x: acc + x))
```

--------------------------------

### first() example: create DataFrame and get first row

Source: https://sparkless.readthedocs.io/en/latest/api/dataframe.html

Shows how to create a DataFrame with spark.createDataFrame and retrieve the first row using first(). The output is a Row object, and for an empty DataFrame, first() returns None.

```python
>>> df = spark.createDataFrame([{"name": "Alice"}, {"name": "Bob"}])
>>> df.first()
Row(name='Alice')
>>> empty_df.first()
None
```

--------------------------------

### orc method with example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/dataframe/writer.html

Convenience method to save DataFrame in ORC format. Example shows usage: df.write.orc("/path/to/file.orc").

```python
    def orc(self, path: str, **options: Any) -> None:
        """Save DataFrame in ORC format.

        Args:
            path: Path to save the ORC file.
            **options: Additional options for ORC format.

        Example:
            >>> df.write.orc("/path/to/file.orc")
        """
        self.format("orc").options(**options).save(path)
```

--------------------------------

### Adding New Backends - Registration, Configuration, and Detection

Source: https://sparkless.readthedocs.io/en/latest/_sources/backend_architecture.md.txt

Shows code snippets for registering a new backend in create_storage_backend(), adding configuration support, and updating detection logic in get_backend_type(). These are part of the steps to add a new backend implementation.

```python
# In create_storage_backend()
elif backend_type == "new_backend":
    from .new_backend.storage import NewBackendStorageManager
    return NewBackendStorageManager(**kwargs)
```

```python
# Add new config keys for backend-specific options
.config("spark.sparkless.backend.newBackend.option", "value")
```

```python
# In get_backend_type()
elif "new_backend" in module_name:
    return "new_backend"
```

--------------------------------

### Using sparkless .storage API to create schema, table, insert data, and get table

Source: https://sparkless.readthedocs.io/en/latest/_sources/mock_spark_features.md.txt

Use this snippet when writing sparkless-specific test utilities or setting up test fixtures that require convenient programmatic access. It will not work with PySpark. The example creates a schema, creates a table with a schema, inserts rows, and retrieves the table as a DataFrame.

```python
from sparkless.sql import SparkSession
from sparkless.sql.types import StructType, StructField, StringType, IntegerType

spark = SparkSession("MyApp")

# Create schema (database)
spark._storage.create_schema("test_db")

# Create table with schema
schema = StructType([
    StructField("name", StringType(), True),
    StructField("age", IntegerType(), True)
])
spark._storage.create_table("test_db", "users", schema)

# Insert data
spark._storage.insert_data("test_db", "users", [
    {"name": "Alice", "age": 25},
    {"name": "Bob", "age": 30}
])

# Get table as DataFrame
df = spark._storage.get_table("test_db", "users")
```

--------------------------------

### text method with example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/dataframe/writer.html

Convenience method to save DataFrame in text format. Example shows usage: df.write.text("/path/to/file.txt").

```python
    def text(self, path: str, **options: Any) -> None:
        """Save DataFrame in text format.

        Args:
            path: Path to save the text file.
            **options: Additional options for text format.

        Example:
            >>> df.write.text("/path/to/file.txt")
        """
        self.format("text").options(**options).save(path)
```

--------------------------------

### delta method with examples

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/dataframe/writer.html

Convenience method to save DataFrame in Delta Lake format. Examples show usage with and without mergeSchema option.

```python
    def delta(self, path: str, **options: Any) -> None:
        """Save DataFrame in Delta Lake format.

        This is a convenience method equivalent to:
        df.write.format("delta").save(path)

        Args:
            path: Path to save the Delta Lake table.
            **options: Additional options for Delta Lake format.

        Example:
            >>> df.write.delta("/path/to/delta_table")
            >>> df.write.delta("/path/to/delta_table", mergeSchema=True)
        """
        self.format("delta").options(**options).save(path)
```

--------------------------------

### get

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets an element from an array by index or a map by key.

```APIDOC
## get

### Description
Gets element from array by index or map by key.

### Method
static

### Signature
get(col, key)

### Parameters
- **col** (Union[Column, str]) - Required - The column or string representing the array or map.
- **key** (Union[Column, str]) - Required - The index or key to retrieve.

### Return Type
`ColumnOperation`
```

--------------------------------

### Migrating from SQL Commands to .storage API - After (convenience API)

Source: https://sparkless.readthedocs.io/en/latest/_sources/storage_api_guide.md.txt

Shows the equivalent sparkless-specific code using the `.storage` API after migration. Uses `create_schema`, `create_table`, and `insert_data` methods.

```python
spark._storage.create_schema("test_db")
schema = StructType([
    StructField("name", StringType()),
    StructField("age", IntegerType())
])
spark._storage.create_table("test_db", "users", schema)
spark._storage.insert_data("test_db", "users", [{"name": "Alice", "age": 25}])
```

--------------------------------

### date_part example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Extracts a field from a date/timestamp. The example extracts the year.

```python
>>> df.select(F.date_part("YEAR", F.col("date")))
```

--------------------------------

### Migrating from SQL Commands to .storage API - Before (SQL)

Source: https://sparkless.readthedocs.io/en/latest/_sources/storage_api_guide.md.txt

Shows the original SQL commands using `spark.sql` to create a database, table, and insert data. This is the 'before' state for migration to the convenience API.

```python
spark.sql("CREATE DATABASE IF NOT EXISTS test_db")
spark.sql("CREATE TABLE test_db.users (name STRING, age INT)")
spark.sql("INSERT INTO test_db.users VALUES ('Alice', 25)")
```

--------------------------------

### get(col, key)

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/array.html

Gets an element from an array by index or a map by key.

```APIDOC
## get(col, key)

### Description
Gets an element from an array column by index or from a map column by key.

### Parameters
- **col** (Column or str) - Required - Array or map column.
- **key** (Column, str, int, or Any) - Required - Index for arrays or key for maps.

### Returns
ColumnOperation representing the get function.
```

--------------------------------

### case_when example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Example usage of case_when with multiple conditions and an else value.

```python
>>> F.case_when(
...     (F.col("age") > 18, "adult"),
...     (F.col("age") > 12, "teen"),
...     else_value="child"
... )
```

--------------------------------

### date_trunc example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Truncates timestamp to specified unit. The example truncates to month.

```python
>>> df.select(F.date_trunc('month', F.col('timestamp')))
```

--------------------------------

### DataFrameWriter module docstring with usage examples

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/dataframe/writer.html

Module-level docstring for the writer module, showing example usage of DataFrameWriter with SparkSession and DataFrame. The examples demonstrate saving a DataFrame as a table and to a file with options.

```python
"""
Mock DataFrameWriter implementation for DataFrame write operations.

This module provides DataFrame writing functionality, maintaining compatibility
with PySpark's DataFrameWriter interface. Supports writing to various data sinks
including tables, files, and custom storage backends with multiple save modes.

Key Features:
    - Complete PySpark DataFrameWriter API compatibility
    - Support for multiple output formats (parquet, json, csv)
    - Multiple save modes (append, overwrite, error, ignore)
    - Flexible options configuration
    - Integration with storage manager
    - Table and file output support
    - Error handling for invalid configurations

Example:
    >>> from sparkless.sql import SparkSession
    >>> spark = SparkSession("test")
    >>> df = spark.createDataFrame([{"name": "Alice", "age": 25}])
    >>> # Save as table
    >>> df.write.mode("overwrite").saveAsTable("users")
    >>> # Save to file with options
    >>> df.write.format("parquet").option("compression", "snappy").save("/path")
"""
```

--------------------------------

### Module docstring with usage example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/aggregate.html

The module docstring includes a usage example showing how to create a SparkSession, create a DataFrame, group by a column, and apply aggregate functions (count, avg, max). The output shows a DataFrame with one row and four columns.

```python
"""
Aggregate functions for Sparkless.

This module provides comprehensive aggregate functions that match PySpark's
aggregate function API. Includes statistical operations, counting functions,
and data summarization operations for grouped data processing in DataFrames.

Key Features:
    - Complete PySpark aggregate function API compatibility
    - Basic aggregates (count, sum, avg, max, min)
    - Statistical functions (stddev, variance, skewness, kurtosis)
    - Collection aggregates (collect_list, collect_set, first, last)
    - Distinct counting (countDistinct)
    - Type-safe operations with proper return types
    - Support for both column references and expressions
    - Proper handling of null values and edge cases

Example:
    >>> from sparkless.sql import SparkSession, functions as F
    >>> spark = SparkSession("test")
    >>> data = [{"dept": "IT", "salary": 50000}, {"dept": "IT", "salary": 60000}]
    >>> df = spark.createDataFrame(data)
    >>> grouped = df.groupBy("dept")
    >>> result = grouped.agg(
    ...     F.count("*").alias("count"),
    ...     F.avg("salary").alias("avg_salary"),
    ...     F.max("salary").alias("max_salary")
    ... )
    >>> result.show()
    DataFrame[1 rows, 4 columns]
    dept count avg_salary max_salary
    IT 2 55000.0 60000
"""
```

--------------------------------

### Mask example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Example of using F.mask to mask sensitive data with custom characters.

```python
>>> df.select(F.mask(F.col("email"), upperChar='U', lowerChar='l', digitChar='d'))
```

--------------------------------

### explain

Source: https://sparkless.readthedocs.io/en/latest/api/dataframe.html

Explains the execution plan.

```APIDOC
## explain

### Description
Explain the execution plan.

### Parameters
- **extended** (`bool`) - Optional
- **codegen** (`bool`) - Optional
- **cost** (`bool`) - Optional
- **formatted** (`bool`) - Optional
- **mode** (`Optional[str]`) - Optional

### Return Type
`Optional[str]`
```

--------------------------------

### SHA hash example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Example of using F.sha to compute SHA-1 hash of a column.

```python
>>> df.select(F.sha(F.col("text")))
```

--------------------------------

### Run full Robin test suite and failure sample

Source: https://sparkless.readthedocs.io/en/latest/robin_sparkless_ownership_analysis.html

Commands to run the full Robin test suite with parallel execution and to sample failures with long tracebacks.

```bash
# Full Robin run
SPARKLESS_TEST_BACKEND=robin SPARKLESS_BACKEND=robin python -m pytest \
  tests/ --ignore=tests/archive -n 12 --dist loadfile -v --tb=short \
  2>&1 | tee test_results_robin_mode.txt
```

```bash
# Sample with long tracebacks
SPARKLESS_TEST_BACKEND=robin python scripts/run_robin_failure_sample.py --max 50
```

--------------------------------

### parse_url example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Extracts a part from a URL column. The example extracts the HOST part.

```python
>>> df.select(F.parse_url(F.col("url"), "HOST"))

```

--------------------------------

### Example: Using aes_encrypt and aes_decrypt with DataFrame

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Shows how to create a SparkSession, create a DataFrame with sample data, and apply aes_encrypt and aes_decrypt functions to columns. The example uses column references for data and key.

```python
>>> from sparkless.sql import SparkSession, functions as F
>>> spark = SparkSession("test")
>>> data = [{"data": "sensitive information", "key": "secretkey"}]
>>> df = spark.createDataFrame(data)
>>> df.select(
...     F.aes_encrypt(F.col("data"), F.col("key")),
...     F.aes_decrypt(F.col("encrypted"), F.col("key"))
... ).show()

```

--------------------------------

### Migrating from .storage API to SQL Commands - After (PySpark-compatible)

Source: https://sparkless.readthedocs.io/en/latest/_sources/storage_api_guide.md.txt

Shows the equivalent PySpark-compatible SQL commands after migration. Uses `spark.sql` to create a database, table, and insert data.

```python
spark.sql("CREATE DATABASE IF NOT EXISTS test_db")
spark.sql("CREATE TABLE test_db.users (name STRING)")
spark.sql("INSERT INTO test_db.users VALUES ('Alice')")
```

--------------------------------

### WindowFunction.__neg__ example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/window_execution.html

Negates a window function result. Example: -F.row_number().over(window).

```python
>>> -F.row_number().over(window)
```

--------------------------------

### Configuration.getAll

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/session/config/configuration.html

Returns a copy of all configuration values.

```APIDOC
## Configuration.getAll

### Description
Get all configuration values.

### Method
GET

### Endpoint
Configuration.getAll()

### Response
#### Success Response (200)
- **config** (Dict[str, str]) - Dictionary of all configuration values.

#### Response Example
```
{"spark.app.name": "SparklessApp", "spark.master": "local[*]"}
```
```

--------------------------------

### Module docstring with usage example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/datetime.html

The module docstring includes a usage example showing how to create a SparkSession, create a DataFrame, and use datetime functions like year, month, and to_date. The output shows a DataFrame with one row and three columns.

```python
>>> from sparkless.sql import SparkSession, functions as F
>>> spark = SparkSession("test")
>>> data = [{"timestamp": "2024-01-15 10:30:00", "date_str": "2024-01-15"}]
>>> df = spark.createDataFrame(data)
>>> df.select(
...     F.year(F.col("timestamp")),
...     F.month(F.col("timestamp")),
...     F.to_date(F.col("date_str"))
... ).show()
DataFrame[1 rows, 3 columns]
year(timestamp) month(timestamp) to_date(date_str)
2024 1 2024-01-15
```

--------------------------------

### Module docstring with usage example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/math.html

The module docstring includes a usage example showing how to create a SparkSession, create a DataFrame, and apply round, ceil, and sin functions. The expected output is shown in the docstring.

```python
"""
Mathematical functions for Sparkless.

This module provides comprehensive mathematical functions that match PySpark's
math function API. Includes arithmetic operations, rounding functions, trigonometric
functions, and mathematical transformations for numerical processing in DataFrames.

Key Features:
    - Complete PySpark math function API compatibility
    - Arithmetic operations (abs, round, ceil, floor)
    - Advanced math functions (sqrt, exp, log, pow)
    - Trigonometric functions (sin, cos, tan)
    - Type-safe operations with proper return types
    - Support for both column references and numeric literals
    - Proper handling of edge cases and null values

Example:
    >>> from sparkless.sql import SparkSession, functions as F
    >>> spark = SparkSession("test")
    >>> data = [{"value": 3.7, "angle": 1.57}]
    >>> df = spark.createDataFrame(data)
    >>> df.select(
    ...     F.round(F.col("value"), 1),
    ...     F.ceil(F.col("value")),
    ...     F.sin(F.col("angle"))
    ... ).show()
    DataFrame[1 rows, 3 columns]
    <BLANKLINE>
    round(value, 1) CEIL(value) SIN(angle)
    3.7               4.0           0.9999996829318346
"""
```

--------------------------------

### assert_true example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Example usage of assert_true to assert a condition on a column, raising an error if false.

```python
>>> df.select(F.assert_true(F.col("value") > 0))
```

--------------------------------

### str_to_map example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Converts a string to a map using delimiters. The example uses default delimiters (',' and ':').

```python
>>> df.select(F.str_to_map(F.col("string")))
```

--------------------------------

### Basic SQLExecutor usage

Source: https://sparkless.readthedocs.io/en/latest/_sources/sql_operations_guide.md.txt

Shows how to instantiate SQLExecutor with a Spark session and execute an optimized AST. The result is collected and printed.

```python
from sparkless.session.sql import SQLExecutor

executor = SQLExecutor(spark_session)

# Execute a parsed and optimized AST
result = executor.execute(optimized_ast)
print(result.collect())
```

--------------------------------

### datediff example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Returns number of days between two dates. The example uses a column and a literal.

```python
>>> df.select(F.datediff(F.col('end_date'), F.lit('2024-01-01')))
```

--------------------------------

### Before: SQL Commands

Source: https://sparkless.readthedocs.io/en/latest/storage_api_guide.html

SQL commands used before migrating to the .storage API, creating a database, table with name and age, and inserting a row.

```python
spark.sql("CREATE DATABASE IF NOT EXISTS test_db")
spark.sql("CREATE TABLE test_db.users (name STRING, age INT)")
spark.sql("INSERT INTO test_db.users VALUES ('Alice', 25)")

```

--------------------------------

### timestampdiff example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Calculates difference between two timestamps. The example shows day and hour differences.

```python
>>> df.select(F.timestampdiff("DAY", F.col("start_date"), F.col("end_date")))
>>> df.select(F.timestampdiff("HOUR", "2024-01-01", F.col("end_time")))
```

--------------------------------

### timestampadd example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Adds time units to a timestamp. The example shows adding days and hours.

```python
>>> df.select(F.timestampadd("DAY", 7, F.col("created_at")))
>>> df.select(F.timestampadd("HOUR", F.col("offset"), "2024-01-01"))
```

--------------------------------

### Backend Type Detection

Source: https://sparkless.readthedocs.io/en/latest/_sources/backend_architecture.md.txt

Demonstrates creating a storage backend, detecting its type, and listing available backends using BackendFactory methods.

```python
from sparkless.backend.factory import BackendFactory

# Create a storage backend
storage = BackendFactory.create_storage_backend("polars")

# Detect the backend type
backend_type = BackendFactory.get_backend_type(storage)
print(backend_type)  # "polars"

# List available backends
available = BackendFactory.list_available_backends()
print(available)  # ["polars", "memory", "file"]
```

--------------------------------

### Sample run with long tracebacks

Source: https://sparkless.readthedocs.io/en/latest/_sources/robin_sparkless_ownership_analysis.md.txt

Runs a sample of Robin failures with long tracebacks.

```bash
# Sample with long tracebacks
SPARKLESS_TEST_BACKEND=robin python scripts/run_robin_failure_sample.py --max 50
```

--------------------------------

### slice(column, start, length)

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/array.html

Extracts an array slice starting at a position for a given length.

```APIDOC
## slice(column, start, length)

### Description
Extracts an array slice starting at a position for a given length.

### Method
slice

### Signature
slice(column: Union[Column, str], start: int, length: int) -> ColumnOperation

### Parameters
- **column** (Union[Column, str]) - Required - The array column.
- **start** (int) - Required - Starting position (1-based).
- **length** (int) - Required - Number of elements to extract.

### Returns
ColumnOperation representing the slice function.

### Example
```python
df.select(F.slice(F.col("nums"), 2, 3))
```
```

--------------------------------

### transform_values example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Transforms map values using a lambda function. The example multiplies each value by 2.

```python
>>> df.select(F.transform_values(F.col("map"), lambda k, v: v * 2))
```

--------------------------------

### soundex example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Computes Soundex encoding for phonetic matching.

```python
>>> df.select(F.soundex(F.col("name")))

```

--------------------------------

### levenshtein example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Computes the Levenshtein distance between two string columns. The example compares 'word1' and 'word2'.

```python
>>> df.select(F.levenshtein(F.col("word1"), F.col("word2")))

```

--------------------------------

### Basic usage of SQLQueryOptimizer

Source: https://sparkless.readthedocs.io/en/latest/_sources/sql_operations_guide.md.txt

Creates a SQLQueryOptimizer instance, optimizes a parsed AST, and prints the components of the optimized AST.

```python
from sparkless.session.sql import SQLQueryOptimizer

optimizer = SQLQueryOptimizer()

# Optimize a parsed AST
ast = parser.parse("SELECT * FROM users WHERE age > 18 ORDER BY name")
optimized_ast = optimizer.optimize(ast)
print(optimized_ast.components)
```

--------------------------------

### regexp_extract_all example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Extracts all regex matches from a column. The example uses a regex pattern to match digits.

```python
>>> df.select(F.regexp_extract_all(F.col("text"), r"\d+", 0))

```