Sparkless

https://sparkless.readthedocs.io/en/latest/
Admin
Sparkless is a lightweight drop-in PySpark replacement that speeds up tests by eliminating JVM...
Tokens:672,258
Snippets:6,574
Trust Score:9.6
Update:1 week ago
Show doc for...
Tokens:
Raw
### SparkContext example from class documentation

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a SparkContext, set the log level, and print the application name. This example is from the SparkContext class documentation and uses the same pattern as the module-level example.

```python
>>> sc = SparkContext("MyApp")
>>> sc.setLogLevel("WARN")
>>> print(sc.appName)
MyApp

```

--------------------------------

### Install Sparkless with documentation tools

Source: https://sparkless.readthedocs.io/en/latest/_sources/installation.rst.txt

Installs Sparkless along with tools for building documentation.

```bash
pip install sparkless[docs]
```

--------------------------------

### Install native pandas backend with pip install .[pandas]

Source: https://sparkless.readthedocs.io/en/latest/_sources/performance/pandas_fallback.md.txt

Installs the optional pandas extra, which brings in pandas and type stubs. NumPy wheels are pulled in transitively where available.

```bash
pip install .[pandas]
```

--------------------------------

### make_date example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Constructs a date from year, month, day integers. The example uses literals.

```python
>>> df.select(F.make_date(F.lit(2024), F.lit(3), F.lit(15)))
```

--------------------------------

### SQL Logging Output Example

Source: https://sparkless.readthedocs.io/en/latest/_sources/migration_from_pyspark.md.txt

Example of generated SQL logs when debug mode is enabled.

```text
[DEBUG] Operation: filter
[DEBUG] SQL: SELECT * FROM temp_table_123 WHERE age > 25
```

--------------------------------

### WindowFunction.__lt__ example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/window_execution.html

Less than comparison for a window function result. Example: F.row_number().over(window) < 5.

```python
>>> F.row_number().over(window) < 5
```

--------------------------------

### Install Sparkless with dev extras

Source: https://sparkless.readthedocs.io/en/latest/_sources/getting_started.md.txt

Installs Sparkless with development dependencies for testing tools.

```bash
pip install sparkless[dev]
```

--------------------------------

### WindowFunction.__mul__ example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/window_execution.html

Multiplies a window function result by a value. Example: 100 * F.percent_rank().over(window).

```python
>>> 100 * F.percent_rank().over(window)
```

--------------------------------

### overlay example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Replaces part of a string with another string starting at a position (PySpark 3.0+). The example replaces 3 characters starting at position 5 in 'text' with 'NEW'.

```python
>>> df.select(F.overlay(F.col("text"), F.lit("NEW"), F.lit(5), F.lit(3)))

```

--------------------------------

### Configuration example

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a Configuration instance, set a configuration key, and retrieve it.

```python
>>> from sparkless.session.config import Configuration
>>> conf = Configuration()
>>> conf.set("spark.app.name", "MyApp")
>>> conf.get("spark.app.name")
'MyApp'

```

--------------------------------

### Configuration _config example

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a Configuration instance and set/get a configuration value.

```python
>>> conf = Configuration()
>>> conf.set("spark.app.name", "MyApp")
>>> conf.get("spark.app.name")
'MyApp'

```

--------------------------------

### dayname example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets the name of the day of the week. The example uses a date column.

```python
>>> df.select(F.dayname(F.col("date")))
```

--------------------------------

### JSON array length example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Example of using F.json_array_length to get the length of a JSON array.

```python
>>> df.select(F.json_array_length(F.col("json_col"), "$.array"))
```

--------------------------------

### Complete SQL pipeline example

Source: https://sparkless.readthedocs.io/en/latest/_sources/sql_operations_guide.md.txt

A full workflow: create a Spark session, create sample data, parse a SQL query, validate, optimize, and execute. Shows how to handle validation errors.

```python
from sparkless.sql import SparkSession
from sparkless.session.sql import SQLParser, SQLValidator, SQLQueryOptimizer, SQLExecutor

# Create Spark session
spark = SparkSession("SQLExample")

# Create sample data
data = [
    {"name": "Alice", "age": 25, "department": "Engineering"},
    {"name": "Bob", "age": 30, "department": "Marketing"},
    {"name": "Charlie", "age": 35, "department": "Engineering"}
]
df = spark.createDataFrame(data)
df.createOrReplaceTempView("employees")

# SQL Pipeline
parser = SQLParser()
validator = SQLValidator()
optimizer = SQLQueryOptimizer()
executor = SQLExecutor(spark)

# Parse SQL query
query = "SELECT department, COUNT(*) as count FROM employees WHERE age > 25 GROUP BY department ORDER BY count DESC"
ast = parser.parse(query)

# Validate query
if validator.validate(ast):
    # Optimize query
    optimized_ast = optimizer.optimize(ast)
    
    # Execute query
    result = executor.execute(optimized_ast)
    
    # Display results
    result.show()
else:
    print("Query validation failed:")
    for error in validator.get_errors():
        print(f"  - {error}")
```

--------------------------------

### Example: Using map_keys and showing DataFrame

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Shows a complete example of creating a SparkSession, creating a DataFrame with a map column, and using F.map_keys to extract keys and display the result.

```python
>>> from sparkless.sql import SparkSession, functions as F
>>> spark = SparkSession("test")
>>> data = [{"properties": {"key1": "val1", "key2": "val2"}}]
>>> df = spark.createDataFrame(data)
>>> df.select(F.map_keys(F.col("properties"))).show()
DataFrame[1 rows, 1 columns]
map_keys(properties)
['key1', 'key2']

```

--------------------------------

### locate example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/string.html

Locates the position of a substring starting from a position (1-indexed). The example is not fully shown in the source.

```python
>>> df.select(F.locate("substr", F.col("text")))
```

--------------------------------

### getItem example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets an item from an array by index or from a map by key. The example shows accessing an array element, a map value, and an out-of-bounds access that returns None.

```python
>>> df.select(F.col("array_col").getItem(0))
>>> df.select(F.col("map_col").getItem("key"))
>>> df.select(F.col("array_col").getItem(999))  # Returns None if out of bounds
```

--------------------------------

### Catalog __init__ method

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/session/catalog.html

Initializes Catalog with storage manager and optional SparkSession.

```python
[docs]
    def __init__(self, storage: IStorageManager, spark: Optional[Any] = None):
        """Initialize Catalog.

        Args:
            storage: Storage manager instance.
            spark: Optional SparkSession instance for SQL-based operations.
                  If provided, createDatabase() will use SQL instead of direct storage calls.
        """
        self._storage = storage
        self.spark = spark
        self._cached_tables: Set[str] = set()  # Track cached tables
```

--------------------------------

### Extract window start time with F.window_time

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Use F.window_time to extract the start time from a window column. The example groups by a 1-hour window and aliases the extracted time as 'window_start'.

```python
>>> df.groupBy(F.window("timestamp", "1 hour")).agg(
...     F.window_time(F.col("window")).alias("window_start")
... )
```

--------------------------------

### weekday example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets the day of week as an integer (0=Monday, 6=Sunday) using F.weekday.

```python
>>> df.select(F.weekday(F.col("date")))
```

--------------------------------

### slice: Extract array slice

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Extracts a slice of the array starting at a 1-based position for a given length. The example extracts 3 elements starting from position 2 of the 'nums' array.

```python
>>> df.select(F.slice(F.col("nums"), 2, 3))
```

--------------------------------

### Create and query a mock Catalog in Sparkless

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Creates a Catalog instance with a storage manager, creates a database, and lists databases. The output shows the expected list of Database objects.

```python
>>> from sparkless.session import Catalog
>>> catalog = Catalog(storage_manager)
>>> catalog.createDatabase("test_db")
>>> catalog.listDatabases()
[Database(name='test_db')]

```

--------------------------------

### last_day example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Shows how to use F.last_day to get the last day of the month for a date column.

```python
>>> df.select(F.last_day(F.col('date')))

```

--------------------------------

### Create table, insert data, and query with SQL

Source: https://sparkless.readthedocs.io/en/latest/_sources/storage_api_guide.md.txt

Use these SQL commands to create a table with a schema, insert data, and query it. The query result is displayed with `show()`.

```python
# Create table with schema
spark.sql("""
    CREATE TABLE IF NOT EXISTS users (
        id INT,
        name STRING,
        age INT
    )
""")

# Insert data
spark.sql("""
    INSERT INTO users VALUES
    (1, 'Alice', 25),
    (2, 'Bob', 30),
    (3, 'Charlie', 35)
""")

# Query table
result = spark.sql("SELECT * FROM users WHERE age > 25")
result.show()
```

--------------------------------

### user example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets the current user name using the user function (PySpark 3.5+).

```python
>>> df.select(F.user())
```

--------------------------------

### FileSchema.__init__(name, base_path)

Source: https://sparkless.readthedocs.io/en/latest/api/backend.html

Initializes a file schema.

```APIDOC
## FileSchema.__init__(name, base_path)

### Description
Initializes a file schema.

### Method
Constructor

### Endpoint
`FileSchema.__init__(name, base_path)`

### Parameters
- **name** (`str`) - Required - Schema name.
- **base_path** (`str`) - Required - Base path for schema files.

### Returns
- **None**

```

--------------------------------

### overlay example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Replaces part of the 'text' column starting at position 5 with 'NEW' for length 3.

```python
>>> df.select(F.overlay(F.col("text"), F.lit("NEW"), F.lit(5), F.lit(3)))
```

--------------------------------

### ConfigBuilder.build()

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/session/config/configuration.html

Builds and returns the Configuration instance.

```APIDOC
## ConfigBuilder.build()

### Description
Builds and returns the Configuration instance.

### Method
build

### Returns
Configuration - Configuration instance.
```

--------------------------------

### SparkSession usage example

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a SparkSession, create a DataFrame from a list of dictionaries, display it, and execute a SQL command. The output shows the DataFrame contents and the SQL command is executed without visible output.

```python
>>> from sparkless.sql import SparkSession
>>> spark = SparkSession("MyApp")
>>> data = [{"name": "Alice", "age": 25}]
>>> df = spark.createDataFrame(data)
>>> df.show()
DataFrame[1 rows, 2 columns]
age name
25    Alice
>>> spark.sql("CREATE DATABASE test")

```

--------------------------------

### locate example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Returns a ColumnOperation representing locate, which locates the position of a substring starting from a given position (1-indexed).

```python
>>> df.select(F.locate("spark", F.col("text"), 1))
```

--------------------------------

### SparkContext usage example

Source: https://sparkless.readthedocs.io/en/latest/api/session.html

Shows how to create a SparkContext, set the log level, and print the application name. The output is 'MyApp'.

```python
>>> from sparkless.session import SparkContext
>>> sc = SparkContext("MyApp")
>>> sc.setLogLevel("WARN")
>>> print(sc.appName)
MyApp

```

--------------------------------

### element_at: Get element at index (1-based, negative for reverse)

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Returns the element at the specified index. Index is 1-based; negative indices count from the end. The example shows getting the first and last elements of the 'nums' array.

```python
>>> df.select(F.element_at(F.col("nums"), 1))  # First element
>>> df.select(F.element_at(F.col("nums"), -1))  # Last element
```

--------------------------------

### Set and get Spark configuration

Source: https://sparkless.readthedocs.io/en/latest/_sources/api_reference.md.txt

Sets and retrieves a configuration value using the SparkSession's conf object. The example uses the key 'spark.sql.debug'.

```python
# Set configuration
spark.conf.set("spark.sql.debug", "true")

# Get configuration
debug_mode = spark.conf.get("spark.sql.debug")
```

--------------------------------

### FileTable.__init__(name, schema, file_path)

Source: https://sparkless.readthedocs.io/en/latest/api/backend.html

Initializes a file table.

```APIDOC
## FileTable.__init__(name, schema, file_path)

### Description
Initializes a file table.

### Method
Constructor

### Endpoint
`FileTable.__init__(name, schema, file_path)`

### Parameters
- **name** (`str`) - Required - Table name.
- **schema** (`StructType`) - Required - Table schema.
- **file_path** (`str`) - Required - Path to table data file.

### Returns
- **None**

```

--------------------------------

### WindowFunction.__init__

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Initialize WindowFunction.

```APIDOC
## WindowFunction.__init__

### Description
Initialize WindowFunction.

### Method
__init__

### Parameters
- **function** (`Any`) - Required - The window function (e.g., row_number(), rank()).
- **window_spec** (`WindowSpec`) - Required - The window specification.
```

--------------------------------

### For Test Utilities: sparkless Convenience API

Source: https://sparkless.readthedocs.io/en/latest/mock_spark_features.html

Example for test utilities: uses the sparkless convenience API for test setup and returns the Spark session as a pytest fixture.

```python
# Good: Convenient for tests, but sparkless-specific
@pytest.fixture
def setup_test_data(spark):
    spark._storage.create_schema("test")
    schema = StructType([StructField("id", IntegerType())])
    spark._storage.create_table("test", "data", schema)
    return spark

```

--------------------------------

### Configuration via Session Builder

Source: https://sparkless.readthedocs.io/en/latest/_sources/backend_architecture.md.txt

Shows how to configure backend selection using the session builder's .config() method, including default, explicit Polars, memory, and file backends with basePath.

```python
from sparkless.sql import SparkSession

# Default backend (Polars) - v3.0.0+
spark = SparkSession("MyApp")

# Explicit backend selection
spark = SparkSession.builder \
    .config("spark.sparkless.backend", "polars") \
    .getOrCreate()

# Memory backend for lightweight testing
spark = SparkSession.builder \
    .config("spark.sparkless.backend", "memory") \
    .getOrCreate()

# File backend for persistent storage
spark = SparkSession.builder \
    .config("spark.sparkless.backend", "file") \
    .config("spark.sparkless.backend.basePath", "/tmp/sparkless") \
    .getOrCreate()
```

--------------------------------

### aggregate: Reduce array elements to a single value

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Aggregates array elements using an accumulator pattern. The example sums the 'nums' array starting from 0 with a lambda that adds each element to the accumulator.

```python
>>> df.select(F.aggregate(F.col("nums"), F.lit(0), lambda acc, x: acc + x))
```

--------------------------------

### first() example: create DataFrame and get first row

Source: https://sparkless.readthedocs.io/en/latest/api/dataframe.html

Shows how to create a DataFrame with spark.createDataFrame and retrieve the first row using first(). The output is a Row object, and for an empty DataFrame, first() returns None.

```python
>>> df = spark.createDataFrame([{"name": "Alice"}, {"name": "Bob"}])
>>> df.first()
Row(name='Alice')
>>> empty_df.first()
None
```

--------------------------------

### Catalog module docstring with usage example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/session/catalog.html

Module-level docstring for the mock Catalog implementation, showing a usage example with Catalog and its methods.

```python
"""
Mock Catalog implementation for Sparkless.

This module provides a mock implementation of PySpark's Catalog
that behaves identically to the real Catalog for testing and development.
It includes database and table management, caching operations, and
catalog queries without requiring a JVM or actual Spark installation.

Key Features:
    - Complete PySpark Catalog API compatibility
    - Database management (create, list, drop)
    - Table management (create, list, drop, cache)
    - Schema validation and error handling
    - Integration with storage manager

Example:
    >>> from sparkless.session import Catalog
    >>> catalog = Catalog(storage_manager)
    >>> catalog.createDatabase("test_db")
    >>> catalog.listDatabases()
    [Database(name='test_db')]
"""
```

--------------------------------

### Using sparkless .storage API to create schema, table, insert data, and get table

Source: https://sparkless.readthedocs.io/en/latest/_sources/mock_spark_features.md.txt

Use this snippet when writing sparkless-specific test utilities or setting up test fixtures that require convenient programmatic access. It will not work with PySpark. The example creates a schema, creates a table with a schema, inserts rows, and retrieves the table as a DataFrame.

```python
from sparkless.sql import SparkSession
from sparkless.sql.types import StructType, StructField, StringType, IntegerType

spark = SparkSession("MyApp")

# Create schema (database)
spark._storage.create_schema("test_db")

# Create table with schema
schema = StructType([
    StructField("name", StringType(), True),
    StructField("age", IntegerType(), True)
])
spark._storage.create_table("test_db", "users", schema)

# Insert data
spark._storage.insert_data("test_db", "users", [
    {"name": "Alice", "age": 25},
    {"name": "Bob", "age": 30}
])

# Get table as DataFrame
df = spark._storage.get_table("test_db", "users")
```

--------------------------------

### Adding New Backends - Registration, Configuration, and Detection

Source: https://sparkless.readthedocs.io/en/latest/_sources/backend_architecture.md.txt

Shows code snippets for registering a new backend in create_storage_backend(), adding configuration support, and updating detection logic in get_backend_type(). These are part of the steps to add a new backend implementation.

```python
# In create_storage_backend()
elif backend_type == "new_backend":
    from .new_backend.storage import NewBackendStorageManager
    return NewBackendStorageManager(**kwargs)
```

```python
# Add new config keys for backend-specific options
.config("spark.sparkless.backend.newBackend.option", "value")
```

```python
# In get_backend_type()
elif "new_backend" in module_name:
    return "new_backend"
```

--------------------------------

### Migrating from SQL Commands to .storage API - After (convenience API)

Source: https://sparkless.readthedocs.io/en/latest/_sources/storage_api_guide.md.txt

Shows the equivalent sparkless-specific code using the `.storage` API after migration. Uses `create_schema`, `create_table`, and `insert_data` methods.

```python
spark._storage.create_schema("test_db")
schema = StructType([
    StructField("name", StringType()),
    StructField("age", IntegerType())
])
spark._storage.create_table("test_db", "users", schema)
spark._storage.insert_data("test_db", "users", [{"name": "Alice", "age": 25}])
```

--------------------------------

### get

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Gets an element from an array by index or a map by key.

```APIDOC
## get

### Description
Gets element from array by index or map by key.

### Method
static

### Signature
get(col, key)

### Parameters
- **col** (Union[Column, str]) - Required - The column or string representing the array or map.
- **key** (Union[Column, str]) - Required - The index or key to retrieve.

### Return Type
`ColumnOperation`
```

--------------------------------

### Example benchmark output

Source: https://sparkless.readthedocs.io/en/latest/_sources/performance/pandas_fallback.md.txt

Example output from the benchmark script on an M3 Pro laptop with macOS 14.6.1, showing mean times for stub and native backends across operations.

```text
Backend    | Create (mean ms)   | to_dict (mean ms) | concat (mean ms)  | iloc (mean ms)
------------------------------------------------------------------------
stub       | 4.812              | 2.337             | 3.921             | 0.118
native     | 7.604              | 1.281             | 2.017             | 0.064
```

--------------------------------

### date_part example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Extracts a field from a date/timestamp. The example extracts the year.

```python
>>> df.select(F.date_part("YEAR", F.col("date")))
```

--------------------------------

### Module docstring with usage example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/spark_types.html

Shows the module's purpose and a usage example for creating a schema with StructType and StructField.

```python
"""
Mock data types and schema system for Sparkless.

This module provides comprehensive mock implementations of PySpark data types
and schema structures that behave identically to the real PySpark types.
Includes primitive types, complex types, schema definitions, and Row objects
for complete type system compatibility.

Key Features:
    - Complete PySpark data type hierarchy
    - Primitive types (String, Integer, Long, Double, Boolean)
    - Complex types (Array, Map, Struct)
    - Schema definition with StructType and StructField
    - Row objects with PySpark-compatible interface
    - Type inference and conversion utilities

Example:
    >>> from sparkless.spark_types import StringType, IntegerType, StructType, StructField
    >>> schema = StructType([
    ...     StructField("name", StringType()),
    ...     StructField("age", IntegerType())
    ... ])
    >>> df = spark.createDataFrame(data, schema)
"""
```

--------------------------------

### Module docstring with example usage

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/map.html

The module docstring includes a usage example showing how to create a DataFrame and use map_keys.

```python
"""
Map functions for Sparkless.

This module provides comprehensive map manipulation functions that match PySpark's
map function API. Includes operations for extracting keys, values, entries, and
combining maps for working with map columns in DataFrames.

Key Features:
    - Complete PySpark map function API compatibility
    - Key/value extraction (map_keys, map_values)
    - Entry operations (map_entries)
    - Map combination (map_concat, map_from_arrays)
    - Type-safe operations with proper return types
    - Support for both column references and map literals

Example:
    >>> from sparkless.sql import SparkSession, functions as F
    >>> spark = SparkSession("test")
    >>> data = [{"properties": {"key1": "val1", "key2": "val2"}}]
    >>> df = spark.createDataFrame(data)
    >>> df.select(F.map_keys(F.col("properties"))).show()
    DataFrame[1 rows, 1 columns]
    map_keys(properties)
    ['key1', 'key2']
"""
```

--------------------------------

### Migrating from SQL Commands to .storage API - Before (SQL)

Source: https://sparkless.readthedocs.io/en/latest/_sources/storage_api_guide.md.txt

Shows the original SQL commands using `spark.sql` to create a database, table, and insert data. This is the 'before' state for migration to the convenience API.

```python
spark.sql("CREATE DATABASE IF NOT EXISTS test_db")
spark.sql("CREATE TABLE test_db.users (name STRING, age INT)")
spark.sql("INSERT INTO test_db.users VALUES ('Alice', 25)")
```

--------------------------------

### get(col, key)

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/array.html

Gets an element from an array by index or a map by key.

```APIDOC
## get(col, key)

### Description
Gets an element from an array column by index or from a map column by key.

### Parameters
- **col** (Column or str) - Required - Array or map column.
- **key** (Column, str, int, or Any) - Required - Index for arrays or key for maps.

### Returns
ColumnOperation representing the get function.
```

--------------------------------

### case_when example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Example usage of case_when with multiple conditions and an else value.

```python
>>> F.case_when(
...     (F.col("age") > 18, "adult"),
...     (F.col("age") > 12, "teen"),
...     else_value="child"
... )
```

--------------------------------

### Module docstring with usage examples

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/dataframe/reader.html

The module docstring includes a usage example showing how to create a SparkSession and read from a table or with format and options.

```python
"""
Mock DataFrameReader implementation for DataFrame read operations.

This module provides DataFrame reading functionality, maintaining compatibility
with PySpark's DataFrameReader interface. Supports reading from various data sources
including tables, files, and custom storage backends.

Key Features:
    - Complete PySpark DataFrameReader API compatibility
    - Support for multiple data formats (parquet, json, csv, table)
    - Flexible options configuration
    - Integration with storage manager
    - Schema inference and validation
    - Error handling for missing data sources

Example:
    >>> from sparkless.sql import SparkSession
    >>> spark = SparkSession("test")
    >>> # Read from table
    >>> df = spark.read.table("my_table")
    >>> # Read with format and options
    >>> df = spark.read.format("parquet").option("header", "true").load("/path")
"""
```

--------------------------------

### date_trunc example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Truncates timestamp to specified unit. The example truncates to month.

```python
>>> df.select(F.date_trunc('month', F.col('timestamp')))
```

--------------------------------

### DataFrameWriter module docstring with usage examples

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/dataframe/writer.html

Module-level docstring for the writer module, showing example usage of DataFrameWriter with SparkSession and DataFrame. The examples demonstrate saving a DataFrame as a table and to a file with options.

```python
"""
Mock DataFrameWriter implementation for DataFrame write operations.

This module provides DataFrame writing functionality, maintaining compatibility
with PySpark's DataFrameWriter interface. Supports writing to various data sinks
including tables, files, and custom storage backends with multiple save modes.

Key Features:
    - Complete PySpark DataFrameWriter API compatibility
    - Support for multiple output formats (parquet, json, csv)
    - Multiple save modes (append, overwrite, error, ignore)
    - Flexible options configuration
    - Integration with storage manager
    - Table and file output support
    - Error handling for invalid configurations

Example:
    >>> from sparkless.sql import SparkSession
    >>> spark = SparkSession("test")
    >>> df = spark.createDataFrame([{"name": "Alice", "age": 25}])
    >>> # Save as table
    >>> df.write.mode("overwrite").saveAsTable("users")
    >>> # Save to file with options
    >>> df.write.format("parquet").option("compression", "snappy").save("/path")
"""
```

--------------------------------

### Module docstring with usage example

Source: https://sparkless.readthedocs.io/en/latest/_modules/sparkless/functions/aggregate.html

The module docstring includes a usage example showing how to create a SparkSession, create a DataFrame, group by a column, and apply aggregate functions (count, avg, max). The output shows a DataFrame with one row and four columns.

```python
"""
Aggregate functions for Sparkless.

This module provides comprehensive aggregate functions that match PySpark's
aggregate function API. Includes statistical operations, counting functions,
and data summarization operations for grouped data processing in DataFrames.

Key Features:
    - Complete PySpark aggregate function API compatibility
    - Basic aggregates (count, sum, avg, max, min)
    - Statistical functions (stddev, variance, skewness, kurtosis)
    - Collection aggregates (collect_list, collect_set, first, last)
    - Distinct counting (countDistinct)
    - Type-safe operations with proper return types
    - Support for both column references and expressions
    - Proper handling of null values and edge cases

Example:
    >>> from sparkless.sql import SparkSession, functions as F
    >>> spark = SparkSession("test")
    >>> data = [{"dept": "IT", "salary": 50000}, {"dept": "IT", "salary": 60000}]
    >>> df = spark.createDataFrame(data)
    >>> grouped = df.groupBy("dept")
    >>> result = grouped.agg(
    ...     F.count("*").alias("count"),
    ...     F.avg("salary").alias("avg_salary"),
    ...     F.max("salary").alias("max_salary")
    ... )
    >>> result.show()
    DataFrame[1 rows, 4 columns]
    dept count avg_salary max_salary
    IT 2 55000.0 60000
"""
```

--------------------------------

### Mask example

Source: https://sparkless.readthedocs.io/en/latest/api/functions.html

Example of using F.mask to mask sensitive data with custom characters.

```python
>>> df.select(F.mask(F.col("email"), upperChar='U', lowerChar='l', digitChar='d'))
```

--------------------------------

### Run full Robin test suite and failure sample

Source: https://sparkless.readthedocs.io/en/latest/robin_sparkless_ownership_analysis.html

Commands to run the full Robin test suite with parallel execution and to sample failures with long tracebacks.

```bash
# Full Robin run
SPARKLESS_TEST_BACKEND=robin SPARKLESS_BACKEND=robin python -m pytest \
  tests/ --ignore=tests/archive -n 12 --dist loadfile -v --tb=short \
  2>&1 | tee test_results_robin_mode.txt
```

```bash
# Sample with long tracebacks
SPARKLESS_TEST_BACKEND=robin python scripts/run_robin_failure_sample.py --max 50
```
Tokens:
Raw