Apache Spark

https://spark.apache.org/docs/4.1.1
Admin
Apache Spark is a unified analytics engine for large-scale data processing, offering high-level APIs...
Tokens:9,725,648
Snippets:54,797
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Basic DataFrame setup

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.pandas_on_spark.transform_batch.html

Initializes a sample DataFrame for transformation examples.

```python
>>> df = ps.DataFrame([(1, 2), (3, 4), (5, 6)], columns=['A', 'B'])
>>> df
   A  B
0  1  2
1  3  4
2  5  6
```

--------------------------------

### Install Spark with Default Settings

Source: https://spark.apache.org/docs/4.1.1/api/R/reference/install.spark.html

This snippet shows how to install Spark using the default settings for hadoopVersion, mirrorUrl, and localDir. It's a basic usage example.

```R
if (FALSE) {
install.spark()
}
```

--------------------------------

### Initialize DataFrame for regexp_instr examples

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.sql/api/pyspark.sql.functions.regexp_instr.html

Setup the DataFrame used in the subsequent examples.

```python
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("1a 2b 14m", r"\d+(a|b|m)")], ["str", "regexp"])
```

--------------------------------

### Initialize DataFrame for take examples

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.take.html

Setup a sample DataFrame with custom indices to demonstrate positional selection.

```python
>>> df = ps.DataFrame([('falcon', 'bird', 389.0),
...                    ('parrot', 'bird', 24.0),
...                    ('lion', 'mammal', 80.5),
...                    ('monkey', 'mammal', np.nan)],
...                   columns=['name', 'class', 'max_speed'],
...                   index=[0, 2, 3, 1])
>>> df
     name   class  max_speed
0  falcon    bird      389.0
2  parrot    bird       24.0
3    lion  mammal       80.5
1  monkey  mammal        NaN
```

--------------------------------

### Initialize DataFrame for iloc examples

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.iloc.html

Setup a sample DataFrame to demonstrate indexing operations.

```python
>>> mydict = [{'a': 1, 'b': 2, 'c': 3, 'd': 4},
...           {'a': 100, 'b': 200, 'c': 300, 'd': 400},
...           {'a': 1000, 'b': 2000, 'c': 3000, 'd': 4000 }]
>>> df = ps.DataFrame(mydict, columns=['a', 'b', 'c', 'd'])
>>> df
      a     b     c     d
0     1     2     3     4
1   100   200   300   400
2  1000  2000  3000  4000
```

--------------------------------

### Setup Person Table

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-qry-select-sortby.html

Creates and populates the person table used for subsequent sorting examples.

```sql
CREATE TABLE person (zip_code INT, name STRING, age INT);
INSERT INTO person VALUES
    (94588, 'Zen Hui', 50),
    (94588, 'Dan Li', 18),
    (94588, 'Anil K', 27),
    (94588, 'John V', NULL),
    (94511, 'David K', 42),
    (94511, 'Aryan B.', 18),
    (94511, 'Lalit B.', NULL);
```

--------------------------------

### Setup Sample DataFrame

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.scalar.html

Initializes a sample DataFrame for use in subsequent examples. This DataFrame contains employee data including ID, name, salary, and department ID.

```python
data = [
    (1, "Alice", 45000, 101), (2, "Bob", 54000, 101), (3, "Charlie", 29000, 102),
    (4, "David", 61000, 102), (5, "Eve", 48000, 101),
]
employees = spark.createDataFrame(data, ["id", "name", "salary", "department_id"])
```

--------------------------------

### Retrieve Current Timestamp

Source: https://spark.apache.org/docs/4.1.1/api/python/_modules/pyspark/sql/functions/builtin.html

Examples of using current_timestamp, now, and localtimestamp to get the current time at the start of query evaluation.

```python
>>> from pyspark.sql import functions as sf
>>> spark.range(1).select(sf.current_timestamp()).show(truncate=False) # doctest: +SKIP
```

```python
>>> from pyspark.sql import functions as sf
>>> spark.range(1).select(sf.now()).show(truncate=False) # doctest: +SKIP
```

```python
>>> from pyspark.sql import functions as sf
>>> spark.range(1).select(sf.localtimestamp()).show(truncate=False) # doctest: +SKIP
```

--------------------------------

### LIST JAR Usage Examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-resource-mgmt-list-jar.html

Examples showing how to add JARs and list them, including filtering by specific file names.

```sql
ADD JAR /tmp/test.jar;
ADD JAR /tmp/test_2.jar;
LIST JAR;
-- output for LIST JAR
spark://192.168.1.112:62859/jars/test.jar
spark://192.168.1.112:62859/jars/test_2.jar

LIST JAR /tmp/test.jar /some/random.jar /another/random.jar;
-- output
spark://192.168.1.112:62859/jars/test.jar
```

--------------------------------

### ORDER BY Examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-qry-select-orderby.html

Examples demonstrating table creation, data insertion, and various sorting configurations including NULL handling and multi-column sorting.

```sql
CREATE TABLE person (id INT, name STRING, age INT);
INSERT INTO person VALUES
    (100, 'John', 30),
    (200, 'Mary', NULL),
    (300, 'Mike', 80),
    (400, 'Jerry', NULL),
    (500, 'Dan',  50);

-- Sort rows by age. By default rows are sorted in ascending manner with NULL FIRST.
SELECT name, age FROM person ORDER BY age;
+-----+----+
| name| age|
+-----+----+
|Jerry|null|
| Mary|null|
| John|  30|
|  Dan|  50|
| Mike|  80|
+-----+----+

-- Sort rows in ascending manner keeping null values to be last.
SELECT name, age FROM person ORDER BY age NULLS LAST;
+-----+----+
| name| age|
+-----+----+
| John|  30|
|  Dan|  50|
| Mike|  80|
| Mary|null|
|Jerry|null|
+-----+----+

-- Sort rows by age in descending manner, which defaults to NULL LAST.
SELECT name, age FROM person ORDER BY age DESC;
+-----+----+
| name| age|
+-----+----+
| Mike|  80|
|  Dan|  50|
| John|  30|
|Jerry|null|
| Mary|null|
+-----+----+

-- Sort rows in ascending manner keeping null values to be first.
SELECT name, age FROM person ORDER BY age DESC NULLS FIRST;
+-----+----+
| name| age|
+-----+----+
|Jerry|null|
| Mary|null|
| Mike|  80|
|  Dan|  50|
| John|  30|
+-----+----+

-- Sort rows based on more than one column with each column having different
-- sort direction.
SELECT * FROM person ORDER BY name ASC, age DESC;
+---+-----+----+
| id| name| age|
+---+-----+----+
|500|  Dan|  50|
|400|Jerry|null|
|100| John|  30|
|200| Mary|null|
|300| Mike|  80|
+---+-----+----+
```

--------------------------------

### Get Spark Start Time

Source: https://spark.apache.org/docs/4.1.1/api/python/_modules/pyspark/core/context.html

Returns the epoch time when the SparkContext was started.

```python
return self._jsc.startTime()
```

--------------------------------

### sequence(start, stop, step)

Source: https://spark.apache.org/docs/4.1.1/api/sql/index.html

Generates an array of elements from start to stop (inclusive).

```APIDOC
## sequence(start, stop, step)

### Description
Generates an array of elements from start to stop (inclusive), incrementing by step. Supported types are: byte, short, integer, long, date, timestamp.

### Parameters
#### Request Body
- **start** (expression) - Required - The start of the range.
- **stop** (expression) - Required - The end of the range (inclusive).
- **step** (expression) - Optional - The step of the range.
```

--------------------------------

### Setup Sample DataFrames

Source: https://spark.apache.org/docs/4.1.1/api/python/_modules/pyspark/sql/dataframe.html

Initializes customers and orders DataFrames for join operations.

```python
from pyspark.sql import functions as sf
from pyspark.sql import Row
customers_data = [
    Row(customer_id=1, name="Alice"), Row(customer_id=2, name="Bob"),
    Row(customer_id=3, name="Charlie"), Row(customer_id=4, name="Diana")
]
customers = spark.createDataFrame(customers_data)
orders_data = [
    Row(order_id=101, customer_id=1, order_date="2024-01-10",
        items=[Row(product="laptop", quantity=5), Row(product="mouse", quantity=12)]),
    Row(order_id=102, customer_id=1, order_date="2024-02-15",
        items=[Row(product="phone", quantity=2), Row(product="charger", quantity=15)]),
    Row(order_id=105, customer_id=1, order_date="2024-03-20",
        items=[Row(product="tablet", quantity=4)]),
    Row(order_id=103, customer_id=2, order_date="2024-01-12",
        items=[Row(product="tablet", quantity=8)]),
    Row(order_id=104, customer_id=2, order_date="2024-03-05",
        items=[Row(product="laptop", quantity=7)]),
    Row(order_id=106, customer_id=3, order_date="2024-04-05",
        items=[Row(product="monitor", quantity=1)]),
]
orders = spark.createDataFrame(orders_data)
```

--------------------------------

### Get Soundex code

Source: https://spark.apache.org/docs/4.1.1/api/sql/index.html

Returns the Soundex code for a given string.

```sql
> SELECT soundex('Miller');
 M460
```

--------------------------------

### Check if String Starts With Substring

Source: https://spark.apache.org/docs/4.1.1/api/sql/index.html

Returns a boolean indicating if the left string starts with the right string. Returns NULL if either input is NULL. Both inputs must be of STRING or BINARY type.

```sql
> SELECT startswith('Spark SQL', 'Spark');
 true
```

```sql
> SELECT startswith('Spark SQL', 'SQL');
 false
```

```sql
> SELECT startswith('Spark SQL', null);
 NULL
```

```sql
> SELECT startswith(x'537061726b2053514c', x'537061726b');
 true
```

```sql
> SELECT startswith(x'537061726b2053514c', x'53514c');
 false
```

--------------------------------

### Get Start Angle

Source: https://spark.apache.org/docs/4.1.1/api/python/_sources/getting_started/quickstart_ps.ipynb.txt

Accessor function to retrieve the start angle of an arc generator.

```javascript
function jo(t){return t.startAngle}
```

--------------------------------

### Execute SHOW CREATE TABLE examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-show-create-table.html

Demonstrates creating a table and retrieving its definition using standard and Hive SerDe formats.

```sql
CREATE TABLE test (c INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
    STORED AS TEXTFILE
    TBLPROPERTIES ('prop1' = 'value1', 'prop2' = 'value2');

SHOW CREATE TABLE test;

SHOW CREATE TABLE test AS SERDE;
```

--------------------------------

### getStart()

Source: https://spark.apache.org/docs/4.1.1/api/java/index-all.html

Gets the start value from IdentityColumnSpec.

```APIDOC
## getStart()

### Description
Gets the start value from an IdentityColumnSpec object.

### Method
N/A (Method Signature)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### SHOW DATABASES Usage Examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-show-databases.html

Examples demonstrating database creation, listing all databases, filtering by pattern, and using the SCHEMAS keyword.

```sql
-- Create database. Assumes a database named `default` already exists in
-- the system. 
CREATE DATABASE payroll_db;
CREATE DATABASE payments_db;

-- Lists all the databases. 
SHOW DATABASES;
+------------+
|databaseName|
+------------+
|     default|
| payments_db|
|  payroll_db|
+------------+
  
-- Lists databases with name starting with string pattern `pay`
SHOW DATABASES LIKE 'pay*';
+------------+
|databaseName|
+------------+
| payments_db|
|  payroll_db|
+------------+
  
-- Lists all databases. Keywords SCHEMAS and DATABASES are interchangeable. 
SHOW SCHEMAS;
+------------+
|databaseName|
+------------+
|     default|
| payments_db|
|  payroll_db|
+------------+
```

--------------------------------

### ADD JAR Examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-resource-mgmt-add-jar.html

Various examples of adding JAR files using local paths, paths with spaces, and Ivy URIs.

```sql
ADD JAR /tmp/test.jar;
ADD JAR "/path/to/some.jar";
ADD JAR '/some/other.jar';
ADD JAR "/path with space/abc.jar";
ADD JARS "/path with space/def.jar" '/path with space/ghi.jar';
ADD JAR "ivy://group:module:version";
ADD JAR "ivy://group:module:version?transitive=false";
ADD JAR "ivy://group:module:version?transitive=true";
ADD JAR "ivy://group:module:version?exclude=group:module&transitive=true";
```

--------------------------------

### Start PySpark Shell (pip installed)

Source: https://spark.apache.org/docs/4.1.1/quick-start.html

Launch the PySpark interactive shell when PySpark is installed via pip in the current environment.

```bash
pyspark
```

--------------------------------

### Initialize DataFrame for partitioning examples

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.repartition.html

Sets up a sample DataFrame with 'id', 'name', and 'age' columns to demonstrate partitioning behavior.

```python
>>> from pyspark.sql import functions as sf
>>> df = spark.range(0, 64, 1, 9).withColumn(
...     "name", sf.concat(sf.lit("name_"), sf.col("id").cast("string"))
... ).withColumn(
...     "age", sf.col("id") - 32
... )
>>> df.select(
...     sf.spark_partition_id().alias("partition")
... ).distinct().sort("partition").show()
+---------+
|partition|
+---------+
|        0|
|        1|
|        2|
|        3|
|        4|
|        5|
|        6|
|        7|
|        8|
+---------+
```

--------------------------------

### Create DataFrame for Shifting Examples

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.groupby.GroupBy.shift.html

Initializes a PySpark DataFrame used in the subsequent shifting examples. This setup is necessary before applying group-wise operations.

```python
>>> df = ps.DataFrame({
...     'a': [1, 1, 1, 2, 2, 2, 3, 3, 3],
...     'b': [1, 2, 2, 2, 3, 3, 3, 4, 4]}, columns=['a', 'b'])
>>> df
```

--------------------------------

### Get day of week

Source: https://spark.apache.org/docs/4.1.1/api/sql/index.html

Returns the day of the week as an integer (0 = Monday, 6 = Sunday).

```sql
> SELECT weekday('2009-07-30');
 3
```

--------------------------------

### Setup Sample Data for EXISTS Subqueries

Source: https://spark.apache.org/docs/4.1.1/api/python/_modules/pyspark/sql/dataframe.html

Initializes customer and order DataFrames for demonstration purposes.

```python
>>> data_customers = [
...     (101, "Alice", "USA"), (102, "Bob", "Canada"), (103, "Charlie", "USA"),
...     (104, "David", "Australia")
... ]
>>> data_orders = [
...     (1, 101, "2023-01-15", 250), (2, 102, "2023-01-20", 300),
...     (3, 103, "2023-01-25", 400), (4, 101, "2023-02-05", 150)
... ]
>>> customers = spark.createDataFrame(
...     data_customers, ["customer_id", "customer_name", "country"])
>>> orders = spark.createDataFrame(
...     data_orders, ["order_id", "customer_id", "order_date", "total_amount"])
```

--------------------------------

### Get SparkUI URL with sparkR.uiWebUrl()

Source: https://spark.apache.org/docs/4.1.1/api/R/reference/sparkR.uiWebUrl.html

Call sparkR.uiWebUrl() to get the SparkUI URL. Returns NA if disabled or not started. Available since SparkR 2.1.1.

```r
sparkR.uiWebUrl()
```

```r
if (FALSE) {
sparkR.session()
url <- sparkR.uiWebUrl()
}
```

--------------------------------

### SHOW FUNCTIONS Examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-show-functions.html

Various examples demonstrating how to list functions using different filters, namespaces, and database scopes.

```sql
-- List a system function `trim` by searching both user defined and system
-- defined functions.
SHOW FUNCTIONS trim;
+--------+
|function|
+--------+
|    trim|
+--------+

-- List a system function `concat` by searching system defined functions.
SHOW SYSTEM FUNCTIONS concat;
+--------+
|function|
+--------+
|  concat|
+--------+

-- List a qualified function `max` from database `salesdb`. 
SHOW SYSTEM FUNCTIONS FROM salesdb LIKE 'max';
+--------+
|function|
+--------+
|     max|
+--------+

-- List all functions starting with `t`
SHOW FUNCTIONS LIKE 't*';
+-----------------+
|         function|
+-----------------+
|              tan|
|             tanh|
|        timestamp|
|          tinyint|
|           to_csv|
|          to_date|
|          to_json|
|     to_timestamp|
|to_unix_timestamp|
| to_utc_timestamp|
|        transform|
|   transform_keys|
| transform_values|
|        translate|
|             trim|
|            trunc|
|           typeof|
+-----------------+

-- List all functions starting with `yea` or `windo`
SHOW FUNCTIONS LIKE 'yea*|windo*';
+--------+
|function|
+--------+
|  window|
|    year|
+--------+

-- Use normal regex pattern to list function names that has 4 characters
-- with `t` as the starting character.
SHOW FUNCTIONS LIKE 't[a-z][a-z][a-z]';
+--------+
|function|
+--------+
|    tanh|
|    trim|
+--------+
```

--------------------------------

### Get Number of Rows

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/api/pyspark.mllib.linalg.distributed.RowMatrix.html

Examples for retrieving the number of rows in a RowMatrix.

```python
>>> rows = sc.parallelize([[1, 2, 3], [4, 5, 6],
...                        [7, 8, 9], [10, 11, 12]])
```

```python
>>> mat = RowMatrix(rows)
>>> print(mat.numRows())
4
```

```python
>>> mat = RowMatrix(rows, 7, 6)
>>> print(mat.numRows())
7
```

--------------------------------

### Get Number of Columns

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/api/pyspark.mllib.linalg.distributed.RowMatrix.html

Examples for retrieving the number of columns in a RowMatrix.

```python
>>> rows = sc.parallelize([[1, 2, 3], [4, 5, 6],
...                        [7, 8, 9], [10, 11, 12]])
```

```python
>>> mat = RowMatrix(rows)
>>> print(mat.numCols())
3
```

```python
>>> mat = RowMatrix(rows, 7, 6)
>>> print(mat.numCols())
6
```

--------------------------------

### Examples of ADD FILE usage

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-resource-mgmt-add-file.html

Demonstrates adding single files, directories, and multiple files with various path formats.

```sql
ADD FILE /tmp/test;
ADD FILE "/path/to/file/abc.txt";
ADD FILE '/another/test.txt';
ADD FILE "/path with space/abc.txt";
ADD FILE "/path/to/some/directory";
ADD FILES "/path with space/cde.txt" '/path with space/fgh.txt';
```

--------------------------------

### Multiple Partitioning Hints Example

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-qry-select-hints.html

Demonstrates applying multiple partitioning hints to a query. The optimizer picks the leftmost hint when multiple are specified.

```sql
EXPLAIN EXTENDED SELECT /*+ REPARTITION(100), COALESCE(500), REPARTITION_BY_RANGE(3, c) */ * FROM t;
```

--------------------------------

### getAllWithPrefix(String)

Source: https://spark.apache.org/docs/4.1.1/api/java/index-all.html

Gets all parameters from SparkConf that start with a given prefix.

```APIDOC
## getAllWithPrefix(String prefix)

### Description
Gets all parameters from SparkConf that start with the specified prefix.

### Method
N/A (Getter method)

### Endpoint
N/A

### Parameters
- **prefix** (String) - The prefix to filter parameters by.

### Response
- **parameters** (Map<String, String>) - A map of parameters that start with the given prefix.
```

--------------------------------

### RESET Command Examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-conf-mgmt-reset.html

Examples demonstrating how to reset all session configurations or a specific configuration key to its default value.

```sql
-- Reset any runtime configurations specific to the current session which were set via the SET command to their default values.
RESET;

-- If you start your application with --conf spark.foo=bar and set spark.foo=foobar in runtime, the example below will restore it to 'bar'. If spark.foo is not specified during starting, the example below will remove this config from the SQLConf. It will ignore nonexistent keys.
RESET spark.abc;
```

--------------------------------

### ADD ARCHIVE Examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-resource-mgmt-add-archive.html

Examples demonstrating how to add various archive file formats and handle paths containing spaces.

```sql
ADD ARCHIVE /tmp/test.tar.gz;
ADD ARCHIVE "/path/to/some.zip";
ADD ARCHIVE '/some/other.tgz';
ADD ARCHIVE "/path with space/abc.tar";
ADD ARCHIVES "/path with space/def.tgz" '/path with space/ghi.zip';
```

--------------------------------

### Get partition ID

Source: https://spark.apache.org/docs/4.1.1/api/sql/index.html

Returns the current partition ID for the row.

```sql
> SELECT spark_partition_id();
 0
```

--------------------------------

### Extract Substring (substring)

Source: https://spark.apache.org/docs/4.1.1/api/sql/index.html

Returns a portion of a string or byte array starting at a specific position.

```SQL
> SELECT substring('Spark SQL', 5);
 k SQL
> SELECT substring('Spark SQL', -3);
 SQL
> SELECT substring('Spark SQL', 5, 1);
 k
> SELECT substring('Spark SQL' FROM 5);
 k SQL
> SELECT substring('Spark SQL' FROM -3);
 SQL
> SELECT substring('Spark SQL' FROM 5 FOR 1);
 k
> SELECT substring(encode('Spark SQL', 'utf-8'), 5);
 k SQL
```

--------------------------------

### Initialize DataFrame for query examples

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.query.html

Sets up a sample DataFrame with columns A, B, and C C for subsequent query demonstrations.

```python
>>> df = ps.DataFrame({'A': range(1, 6),
...                    'B': range(10, 0, -2),
...                    'C C': range(10, 5, -1)})
>>> df
   A   B  C C
0  1  10   10
1  2   8    9
2  3   6    8
3  4   4    7
4  5   2    6
```

--------------------------------

### Get Current Date

Source: https://spark.apache.org/docs/4.1.1/api/sql

Returns the current date at the start of query evaluation.

```SQL
> SELECT curdate();
 2022-09-06
```

--------------------------------

### Slice array

Source: https://spark.apache.org/docs/4.1.1/api/sql/index.html

Returns a subset of an array starting from a specific index with a defined length. Indices are 1-based.

```sql
> SELECT slice(array(1, 2, 3, 4), 2, 2);
 [2,3]
> SELECT slice(array(1, 2, 3, 4), -2, 2);
 [3,4]
```

--------------------------------

### Hive Format Examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-dml-insert-overwrite-directory.html

Examples demonstrating how to use the Hive format with local directories and specific SerDe configurations.

```sql
INSERT OVERWRITE LOCAL DIRECTORY '/tmp/destination'
    STORED AS orc
    SELECT * FROM test_table;
```

```sql
INSERT OVERWRITE LOCAL DIRECTORY '/tmp/destination'
    ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
    SELECT * FROM test_table;
```

--------------------------------

### GET /initialOffset

Source: https://spark.apache.org/docs/4.1.1/api/scala/org/apache/spark/sql/connector/read/streaming/SupportsAdmissionControl.html

Retrieves the initial offset for a streaming query to start reading from.

```APIDOC
## GET /initialOffset

### Description
Returns the initial offset for a streaming query to start reading from. Note that the streaming data source should not assume that it will start reading from its initial offset if Spark is restarting an existing query.

### Method
GET

### Response
#### Success Response (200)
- **offset** (Offset) - The initial offset.
```

--------------------------------

### Extract Substring

Source: https://spark.apache.org/docs/4.1.1/api/sql/index.html

Returns a portion of a string or byte array starting at a specific position.

```SQL
> SELECT substr('Spark SQL', 5);
 k SQL
> SELECT substr('Spark SQL', -3);
 SQL
> SELECT substr('Spark SQL', 5, 1);
 k
> SELECT substr('Spark SQL' FROM 5);
 k SQL
> SELECT substr('Spark SQL' FROM -3);
 SQL
> SELECT substr('Spark SQL' FROM 5 FOR 1);
 k
> SELECT substr(encode('Spark SQL', 'utf-8'), 5);
 k SQL
```

--------------------------------

### GET SparkContext.uiWebUrl

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/api/pyspark.SparkContext.uiWebUrl.html

Retrieves the URL of the SparkUI instance started by the current SparkContext.

```APIDOC
## GET SparkContext.uiWebUrl

### Description
Returns the URL of the SparkUI instance started by this `SparkContext`. If the web UI is disabled (e.g., `spark.ui.enabled` is set to `False`), it returns `None`.

### Method
GET

### Endpoint
SparkContext.uiWebUrl

### Response
#### Success Response (200)
- **url** (string) - The URL of the SparkUI instance or None if disabled.

#### Response Example
"http://..."
```

--------------------------------

### KMeans Clustering Example (R)

Source: https://spark.apache.org/docs/4.1.1/ml-clustering.html

Demonstrates K-Means clustering in R using Spark. This example shows model fitting, summary, getting fitted results, and making predictions.

```r
# Fit a k-means model with spark.kmeans
t <- as.data.frame(Titanic)
training <- createDataFrame(t)
df_list <- randomSplit(training, c(7,3), 2)
kmeansDF <- df_list[[1]]
kmeansTestDF <- df_list[[2]]
kmeansModel <- spark.kmeans(kmeansDF, ~ Class + Sex + Age + Freq,
                            k = 3)

# Model summary
summary(kmeansModel)

# Get fitted result from the k-means model
head(fitted(kmeansModel))

# Prediction
kmeansPredictions <- predict(kmeansModel, kmeansTestDF)
head(kmeansPredictions)
```

--------------------------------

### Get Streaming Query Timestamp

Source: https://spark.apache.org/docs/4.1.1/api/python/_modules/pyspark/sql/streaming/listener.html

Retrieves the timestamp indicating when the streaming query started.

```python
    @property
    def timestamp(self) -> str:
        """
        The timestamp to start a query.
        """
        return self["timestamp"]
```

--------------------------------

### Initialize DataFrame for lead examples

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.sql/api/pyspark.sql.functions.lead.html

Sets up the initial DataFrame used for demonstrating window function operations.

```python
>>> from pyspark.sql import functions as sf
>>> from pyspark.sql import Window
>>> df = spark.createDataFrame(
...     [("a", 1), ("a", 2), ("a", 3), ("b", 8), ("b", 2)], ["c1", "c2"])
>>> df.show()
+---+---+
| c1| c2|
+---+---+
|  a|  1|
|  a|  2|
|  a|  3|
|  b|  8|
|  b|  2|
+---+---+
```

--------------------------------

### Create and Populate Table

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-describe-table.html

Initializes a partitioned table and inserts a sample record.

```sql
-- Creates a table `customer`. Assumes current database is `salesdb`.
CREATE TABLE customer(
        cust_id INT,
        state VARCHAR(20),
        name STRING COMMENT 'Short name'
    )
    USING parquet
    PARTITIONED BY (state);
    
INSERT INTO customer PARTITION (state = 'AR') VALUES (100, 'Mike');
```

--------------------------------

### Initialize DataFrame for applymap examples

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.applymap.html

Setup a sample DataFrame to demonstrate elementwise operations.

```python
>>> df = ps.DataFrame([[1, 2.12], [3.356, 4.567]])
>>> df
       0      1
0  1.000  2.120
1  3.356  4.567
```

--------------------------------

### Create and Show Columns in Current Database

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-aux-show-columns.html

Demonstrates creating a table and then listing its columns using SHOW COLUMNS IN.

```sql
USE salesdb;
CREATE TABLE customer(
    cust_cd INT,
    name VARCHAR(100),
    cust_addr STRING);

SHOW COLUMNS IN customer;
```

```text
+---------+
| col_name|
+---------+
|  cust_cd|
|     name|
|cust_addr|
+---------+
```

--------------------------------

### gettingResultTime()

Source: https://spark.apache.org/docs/4.1.1/api/java/index-all.html

Returns the time when the task started remotely getting the result. Available in TaskInfo and TaskData.

```APIDOC
## gettingResultTime()

### Description
The time when the task started remotely getting the result.

### Method
N/A (Java method signature)

### Endpoint
N/A (Java method signature)

### Parameters
None explicitly documented for this signature.

### Response
Returns the time in milliseconds.
```

--------------------------------

### Get Job Tags

Source: https://spark.apache.org/docs/4.1.1/api/R/reference/getJobTags.html

Retrieves the tags that are currently set to be assigned to all the jobs started by this thread.

```APIDOC
## Get Job Tags

### Description
Get the tags that are currently set to be assigned to all the jobs started by this thread.

### Method
GET

### Endpoint
/getJobTags

### Parameters
This endpoint does not have any parameters.

### Request Example
```
getJobTags()
```

### Response
#### Success Response (200)
- **tags** (array) - A list of tags assigned to jobs.

#### Response Example
```json
{
  "tags": ["tag1", "tag2"]
}
```

### Note
Available since Spark 3.5.0.
```

--------------------------------

### Initialize DataFrame for selection examples

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.select_dtypes.html

Sets up a sample DataFrame with mixed data types for demonstration.

```python
>>> df = ps.DataFrame({'a': [1, 2] * 3,
...                    'b': [True, False] * 3,
...                    'c': [1.0, 2.0] * 3,
...                    'd': ['a', 'b'] * 3}, columns=['a', 'b', 'c', 'd'])
>>> df
   a      b    c  d
0  1   True  1.0  a
1  2  False  2.0  b
2  1   True  1.0  a
3  2  False  2.0  b
4  1   True  1.0  a
5  2  False  2.0  b
```

--------------------------------

### DROP DATABASE Usage Examples

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-ddl-drop-database.html

Examples demonstrating database creation and subsequent deletion using CASCADE and IF EXISTS clauses.

```sql
-- Create `inventory_db` Database
CREATE DATABASE inventory_db COMMENT 'This database is used to maintain Inventory';

-- Drop the database and it's tables
DROP DATABASE inventory_db CASCADE;

-- Drop the database using IF EXISTS
DROP DATABASE IF EXISTS inventory_db CASCADE;
```

--------------------------------

### Define secondary DataFrame for correlation

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.corrwith.html

Setup of a second DataFrame to be used in subsequent correlation examples.

```python
>>> df2 = ps.DataFrame({
...         "A":[5, 3, 6, 4],
...         "B":[11, 2, 4, 3],
...         "C":[4, 3, 8, 5]})
```

--------------------------------

### Spark SQL CLUSTER BY Example: Data Setup

Source: https://spark.apache.org/docs/4.1.1/sql-ref-syntax-qry-select-clusterby.html

Sets up a sample 'person' table and inserts data for demonstrating the CLUSTER BY clause. It also sets the shuffle partitions to 2 for clearer illustration.

```sql
CREATE TABLE person (name STRING, age INT);
INSERT INTO person VALUES
    ('Zen Hui', 25),
    ('Anil B', 18),
    ('Shone S', 16),
    ('Mike A', 25),
    ('John A', 18),
    ('Jack N', 16);

-- Reduce the number of shuffle partitions to 2 to illustrate the behavior of `CLUSTER BY`.
-- It's easier to see the clustering and sorting behavior with less number of partitions.
SET spark.sql.shuffle.partitions = 2;
```

--------------------------------

### Example of sparkRSQL.init usage

Source: https://spark.apache.org/docs/4.1.1/api/R/reference/sparkRSQL.init-deprecated.html

Demonstrates the initialization of a SQLContext using an existing SparkContext.

```R
if (FALSE) {
sc <- sparkR.init()
sqlContext <- sparkRSQL.init(sc)
}
```

--------------------------------

### Get a local property by key

Source: https://spark.apache.org/docs/4.1.1/api/R/reference/getLocalProperty.html

Use `getLocalProperty` to retrieve a property set in the current thread. This function is available since Spark version 2.3.0. The example demonstrates how to get the value of 'spark.scheduler.pool'.

```R
if (FALSE) {
getLocalProperty("spark.scheduler.pool")
}
```

--------------------------------

### Create DataFrame for Examples

Source: https://spark.apache.org/docs/4.1.1/api/R/reference/column_nonaggregate_functions.html

Initializes the DataFrame used across the documentation examples.

```R
df <- createDataFrame(cbind(model = rownames(mtcars), mtcars))}
```

--------------------------------

### Run Kinesis Data Producer Example

Source: https://spark.apache.org/docs/4.1.1/streaming-kinesis-integration.html

Start a Kinesis data producer to generate sample data for the Spark Streaming example. This pushes lines of random numbers to the specified Kinesis stream.

```bash
./bin/run-example streaming.KinesisWordProducerASL [Kinesis stream name] [endpoint URL] 1000 10
```

--------------------------------

### Get Input File Block Start Offset

Source: https://spark.apache.org/docs/4.1.1/api/python/_modules/pyspark/sql/functions/builtin.html

Retrieves the starting offset of the current input file block. Returns -1 if the offset is not available. This can help in locating data within large files.

```python
from pyspark.sql import functions as sf
df = spark.read.text("python/test_support/sql/ages_newlines.csv", lineSep=",")
df.select(sf.input_file_block_start()).show()
```

--------------------------------

### SparkR Session Examples

Source: https://spark.apache.org/docs/4.1.1/api/R/reference/sparkR.session.html

Examples demonstrating how to initialize a SparkSession with different configurations. This includes default initialization, specifying master and appName, setting Spark properties, including JARs, and adding packages.

```R
sparkR.session()
```

```R
df <- read.json(path)
```

```R
sparkR.session("local[2]", "SparkR", "/home/spark")
```

```R
sparkR.session("yarn", "SparkR", "/home/spark",
               list(spark.executor.memory="4g", spark.submit.deployMode="client"),
               c("one.jar", "two.jar", "three.jar"),
               c("com.databricks:spark-avro_2.12:2.0.1"))
```

```R
sparkR.session(spark.master = "yarn", spark.submit.deployMode = "client",
               spark.executor.memory = "4g")
```

--------------------------------

### Array Slicing with Negative Start Index in PySpark

Source: https://spark.apache.org/docs/4.1.1/api/python/reference/pyspark.sql/api/pyspark.sql.functions.slice.html

This example demonstrates slicing an array using a negative start index, which counts from the end of the array. The length parameter specifies the number of elements to include.

```python
from pyspark.sql import functions as sf
df = spark.createDataFrame([([1, 2, 3],), ([4, 5],)], ['x'])
df.select(sf.slice(df.x, -1, 1)).show()
```

--------------------------------

### Initialize DataFrame for Grouping Examples

Source: https://spark.apache.org/docs/4.1.1/api/python/_modules/pyspark/sql/dataframe.html

Sets up the initial DataFrame used for the groupBy examples.

```python
>>> df = spark.createDataFrame([
...     ("Alice", 2), ("Bob", 2), ("Bob", 2), ("Bob", 5)], schema=["name", "age"])
```
Tokens:
Raw