### Example of installing Spark

Source: https://spark.apache.org/docs/3.5.6/api/R/reference/install.spark.html

An example of how to use the install.spark function to download and install Spark.

```r
if (FALSE) { # \dontrun{
install.spark()
} # }
```

--------------------------------

### Java RFormula Example Setup

Source: https://spark.apache.org/docs/3.5.6/ml-features.html

This Java code snippet shows the initial setup for using RFormula, including schema definition.

```java
import java.util.Arrays;
import java.util.List;

import org.apache.spark.ml.feature.RFormula;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;

import static org.apache.spark.sql.types.DataTypes.*;

StructType schema = createStructType(new StructField[]{
  createStructField("id", IntegerType, false),
  createStructField("country", StringType, false),
  createStructField("hour", IntegerType, false),
  createStructField("clicked", DoubleType, false)
});

```

--------------------------------

### Java Example Setup

Source: https://spark.apache.org/docs/3.5.6/ml-features.html

Sets up the DataFrame and schema for a FeatureHasher example in Java.

```java
import java.util.Arrays;
import java.util.List;

import org.apache.spark.ml.feature.FeatureHasher;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.Metadata;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;

List<Row> data = Arrays.asList(
  RowFactory.create(2.2, true, "1", "foo"),
  RowFactory.create(3.3, false, "2", "bar"),
  RowFactory.create(4.4, false, "3", "baz"),
  RowFactory.create(5.5, false, "4", "foo")
);
StructType schema = new StructType(new StructField[]{
  new StructField("real", DataTypes.DoubleType, false, Metadata.empty()),
  new StructField("bool", DataTypes.BooleanType, false, Metadata.empty()),
  new StructField("stringNum", DataTypes.StringType, false, Metadata.empty()),
  new StructField("string", DataTypes.StringType, false, Metadata.empty())
});

```

--------------------------------

### Starting Spark shell with Protobuf package

Source: https://spark.apache.org/docs/3.5.6/sql-data-sources-protobuf.html

Example of using spark-shell with the --packages option to add the spark-protobuf package.

```bash
./bin/spark-shell --packages org.apache.spark:spark-protobuf_2.12:3.5.6 ...
```

--------------------------------

### Mesos Installation Prefix Example

Source: https://spark.apache.org/docs/3.5.6/running-on-mesos.html

Example of how to specify a custom installation prefix for Mesos if administrative privileges are not available.

```bash
--prefix=/home/me/mesos
```

--------------------------------

### Install Spark

Source: https://spark.apache.org/docs/3.5.6/api/R/articles/sparkr-vignettes.html

Installs Apache Spark if it's not already installed. This is a prerequisite for using SparkR.

```R
install.spark()
```

--------------------------------

### Python SparkSession Example

Source: https://spark.apache.org/docs/3.5.6/sql-getting-started.html

Example of creating a SparkSession in Python.

```python
from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()
```

--------------------------------

### Java SparkSession Example

Source: https://spark.apache.org/docs/3.5.6/sql-getting-started.html

Example of creating a SparkSession in Java.

```java
import org.apache.spark.sql.SparkSession;

SparkSession spark = SparkSession
  .builder()
  .appName("Java Spark SQL basic example")
  .config("spark.some.config.option", "some-value")
  .getOrCreate();
```

--------------------------------

### Example 2: Starting a streaming query

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.StreamingQueryManager.active.html

This example demonstrates starting a streaming query to a memory sink and getting the StreamingQueryManager.

```python
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()
>>> sqm = spark.streams

```

--------------------------------

### Running Example Applications

Source: https://spark.apache.org/docs/3.5.6/quick-start.html

Commands to run built-in Spark example applications for Scala, Java, Python, and R.

```bash
# For Scala and Java, use run-example:
./bin/run-example SparkPi

# For Python examples, use spark-submit directly:
./bin/spark-submit examples/src/main/python/pi.py

# For R examples, use spark-submit directly:
./bin/spark-submit examples/src/main/r/dataframe.R
```

--------------------------------

### Get SparkContext start time

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/api/pyspark.SparkContext.startTime.html

This example shows how to retrieve the epoch time when the SparkContext was started.

```python
>>> _ = sc.startTime

```

--------------------------------

### Python setup.py dependency

Source: https://spark.apache.org/docs/3.5.6/quick-start.html

Example of adding PySpark as a dependency in a Python setup.py file.

```python
install_requires=[
        'pyspark==3.5.6'
    ]
```

--------------------------------

### input_file_block_start Example

Source: https://spark.apache.org/docs/3.5.6/api/sql/index.html

Example of calling input_file_block_start to get the start offset of the current block being read.

```sql
> SELECT input_file_block_start();
 -1

```

--------------------------------

### Configuration Example

Source: https://spark.apache.org/docs/3.5.6/api/java/org/apache/spark/sql/ExperimentalMethods.html

Example of how to configure experimental strategies.

```shell
spark.experimental.extraStrategies += ...
```

--------------------------------

### Launch Spark server with Spark Connect

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_connect.html

To launch Spark with support for Spark Connect sessions, run the start-connect-server.sh script. Ensure environment variables are loaded before running.

```bash
source ~/.profile # Make sure environment variables are loaded.
$HOME/sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:$SPARK_VERSION

```

--------------------------------

### input_file_block_start example

Source: https://spark.apache.org/docs/3.5.6/api/python/_modules/pyspark/sql/functions.html

Demonstrates how to use the input_file_block_start function to get the start offset of the block being read.

```python
df = spark.read.text("python/test_support/sql/ages_newlines.csv", lineSep=",")
df.select(input_file_block_start().alias('r')).first()
```

--------------------------------

### Build the Spark package

Source: https://spark.apache.org/docs/3.5.6/api/python/development/setting_ide.html

After downloading the source code, navigate to the 'spark' directory and build the package using SBT. SBT build is generally much faster than Maven.

```bash
build/sbt package

```

--------------------------------

### Latent Dirichlet Allocation (LDA) - Perplexity

Source: https://spark.apache.org/docs/3.5.6/api/R/articles/sparkr-vignettes.html

Example of getting perplexity from an LDA model.

```r
perplexity <- spark.perplexity(model, corpusDF)
perplexity
```

--------------------------------

### Start Spark server with Spark Connect

Source: https://spark.apache.org/docs/3.5.6/spark-connect-overview.html

Example command to start the Spark server with the Spark Connect package.

```bash
./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.6
```

--------------------------------

### Create DataFrame

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_df.html

Creates a sample DataFrame with specified schema and displays it.

```python
df = spark.createDataFrame([
    ['red', 'banana', 1, 10], ['blue', 'banana', 2, 20], ['red', 'carrot', 3, 30],
    ['blue', 'grape', 4, 40], ['red', 'carrot', 5, 50], ['black', 'carrot', 6, 60],
    ['red', 'banana', 7, 70], ['red', 'grape', 8, 80]], schema=['color', 'fruit', 'v1', 'v2'])
df.show()
```

--------------------------------

### Latent Dirichlet Allocation (LDA) - Posterior Probabilities

Source: https://spark.apache.org/docs/3.5.6/api/R/articles/sparkr-vignettes.html

Example of getting posterior probabilities from an LDA model.

```r
posterior <- spark.posterior(model, corpusDF)
head(posterior)
```

--------------------------------

### Basic example

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.DataStreamWriter.start.html

This example demonstrates the basic usage of starting a streaming query with the memory format.

```python
>>> df = spark.readStream.format("rate").load()
>>> q = df.writeStream.format('memory').queryName('this_query').start()
>>> q.isActive
True
>>> q.name
'this_query'
>>> q.stop()
>>> q.isActive
False
```

--------------------------------

### Typical Scala project directory layout

Source: https://spark.apache.org/docs/3.5.6/quick-start.html

Example of the expected directory structure for a Scala Spark application with sbt.

```shell
# Your directory layout should look like this
$ find .
.
./build.sbt
./src
./src/main
./src/main/scala
./src/main/scala/SimpleApp.scala
```

--------------------------------

### Using Avro Package in Spark Shell

Source: https://spark.apache.org/docs/3.5.6/sql-data-sources-avro.html

Example of starting a Spark shell with the spark-avro package.

```bash
./bin/spark-shell --packages org.apache.spark:spark-avro_2.12:3.5.6 ...
```

--------------------------------

### SparkR substr Method Behavior Change

Source: https://spark.apache.org/docs/3.5.6/sparkr-migration-guide.html

Example demonstrating the change in the 'start' parameter behavior for the substr method in SparkR.

```r
substr(lit('abcdef'), 2, 4)
```

--------------------------------

### Packaging a Scala Spark application with sbt

Source: https://spark.apache.org/docs/3.5.6/quick-start.html

Command to package a Scala Spark application into a JAR file using sbt.

```shell
# Package a jar containing your application
$ sbt package
...
[info] Packaging {..}/{..}/target/scala-2.12/simple-project_2.12-1.0.jar
```

--------------------------------

### Managing Active Queries - Java

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

Java code examples for managing active streaming queries using the StreamingQueryManager, including getting active queries, getting a query by ID, and waiting for any termination.

```java
SparkSession spark = ...

spark.streams().active();    // get the list of currently active streaming queries

spark.streams().get(id);   // get a query object by its unique id

spark.streams().awaitAnyTermination();   // block until any one of them terminates
```

--------------------------------

### Managing Active Queries - Scala

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

Scala code examples for managing active streaming queries using the StreamingQueryManager, including getting active queries, getting a query by ID, and waiting for any termination.

```scala
val spark: SparkSession = ...

spark.streams.active    // get the list of currently active streaming queries

spark.streams.get(id)   // get a query object by its unique id

spark.streams.awaitAnyTermination()   // block until any one of them terminates
```

--------------------------------

### Package and Run Application

Source: https://spark.apache.org/docs/3.5.6/quick-start.html

Commands to package the Java application using Maven and then run it using spark-submit.

```bash
# Package a JAR containing your application
$ mvn package
...
[INFO] Building jar: {..}/{..}/target/simple-project-1.0.jar

# Use spark-submit to run your application
$ YOUR_SPARK_HOME/bin/spark-submit \
  --class "SimpleApp" \
  --master local[4] \
  target/simple-project-1.0.jar
...
Lines with a: 46, lines with b: 23
```

--------------------------------

### Managing Active Queries - Python

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

Python code examples for managing active streaming queries using the StreamingQueryManager, including getting active queries, getting a query by ID, and waiting for any termination.

```python
spark = ...  # spark session

spark.streams.active  # get the list of currently active streaming queries

spark.streams.get(id)  # get a query object by its unique id

spark.streams.awaitAnyTermination()  # block until any one of them terminates
```

--------------------------------

### localtimestamp function example

Source: https://spark.apache.org/docs/3.5.6/api/sql

Shows the usage of the localtimestamp function to get the current timestamp without time zone at the start of query evaluation.

```sql
> SELECT localtimestamp();
 2020-04-25 15:49:11.914
```

--------------------------------

### Get the user-specified name of the query, or null if not specified.

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.StreamingQuery.name.html

This example demonstrates how to start a streaming query with a specified name and then retrieve that name using the .name property.

```python
>>> sdf = spark.readStream.format("rate").load()
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()

>>> sq.name
'this_query'

>>> sq.stop()
```

--------------------------------

### Run Scala example

Source: https://spark.apache.org/docs/3.5.6/streaming-programming-guide.html

This command shows how to run the Scala Spark Streaming example.

```shell
$ ./bin/run-example streaming.NetworkWordCount localhost 9999
```

--------------------------------

### SHOW CREATE TABLE Examples

Source: https://spark.apache.org/docs/3.5.6/sql-ref-syntax-aux-show-create-table.html

Demonstrates the usage of SHOW CREATE TABLE for a basic table and with the AS SERDE option for Hive DDL generation.

```sql
CREATE TABLE test (c INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
    STORED AS TEXTFILE
    TBLPROPERTIES ('prop1' = 'value1', 'prop2' = 'value2');

SHOW CREATE TABLE test;
+----------------------------------------------------+
|                                      createtab_stmt|
+----------------------------------------------------+
|CREATE TABLE `default`.`test` (`c` INT)
 USING text
 TBLPROPERTIES (
   'transient_lastDdlTime' = '1586269021',
   'prop1' = 'value1',
   'prop2' = 'value2')
+----------------------------------------------------+

SHOW CREATE TABLE test AS SERDE;
+------------------------------------------------------------------------------+
|                                                                createtab_stmt|
+------------------------------------------------------------------------------+
|CREATE TABLE `default`.`test`(
  `c` INT)
 ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
 WITH SERDEPROPERTIES (
   'serialization.format' = ',',
   'field.delim' = ',')
 STORED AS
   INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
   OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
 TBLPROPERTIES (
   'prop1' = 'value1',
   'prop2' = 'value2',
   'transient_lastDdlTime' = '1641800515')
+------------------------------------------------------------------------------+
```

--------------------------------

### Run Java example

Source: https://spark.apache.org/docs/3.5.6/streaming-programming-guide.html

This command shows how to run the Java Spark Streaming example.

```shell
$ ./bin/run-example streaming.JavaNetworkWordCount localhost 9999
```

--------------------------------

### get Examples

Source: https://spark.apache.org/docs/3.5.6/sql-ref-functions-builtin.html

Examples demonstrating the usage of the get function.

```sql
SELECT get(array(1, 2, 3), 0);
+----------------------+
|get(array(1, 2, 3), 0)|
+----------------------+
|                     1|
+----------------------+

SELECT get(array(1, 2, 3), 3);
+----------------------+
|get(array(1, 2, 3), 3)|
+----------------------+
|                  NULL|
+----------------------+

SELECT get(array(1, 2, 3), -1);
+-----------------------+
|get(array(1, 2, 3), -1)|
+-----------------------+
|                   NULL|
+-----------------------+
```

--------------------------------

### Managing Streaming Queries - R

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

R code examples for managing a streaming query object, including getting its name, explaining it, stopping it, and accessing progress updates.

```r
query <- write.stream(df, "console")  # get the query object

queryName(query)          # get the name of the auto-generated or user-specified name

explain(query)            # print detailed explanations of the query

stopQuery(query)          # stop the query

awaitTermination(query)   # block until query is terminated, with stop() or with error

lastProgress(query)       # the most recent progress update of this streaming query
```

--------------------------------

### Writing and Reading Parquet

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_ps.html

Demonstrates writing a pandas-on-Spark DataFrame to a Parquet file and reading it back.

```python
psdf.to_parquet('bar.parquet')
ps.read_parquet('bar.parquet').head(10)
```

--------------------------------

### Running Scala/Java Example

Source: https://spark.apache.org/docs/3.5.6/rdd-programming-guide.html

Command to run a Spark example (e.g., SparkPi) using the run-example script.

```bash
./bin/run-example SparkPi
```

--------------------------------

### Get the unique id of this query that persists across restarts from checkpoint data

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.StreamingQuery.id.html

This example demonstrates how to start a streaming query and then retrieve its unique ID, which persists across restarts from checkpoint data.

```python
sdf = spark.readStream.format("rate").load()
sq = sdf.writeStream.format('memory').queryName('this_query').start()
```

```python
sq.id
```

```python
sq.stop()
```

--------------------------------

### Managing Streaming Queries - Java

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

Java code examples for managing a streaming query object, including getting its ID, name, explaining it, stopping it, and accessing progress updates.

```java
StreamingQuery query = df.writeStream().format("console").start();   // get the query object

query.id();          // get the unique identifier of the running query that persists across restarts from checkpoint data

query.runId();       // get the unique id of this run of the query, which will be generated at every start/restart

query.name();        // get the name of the auto-generated or user-specified name

query.explain();   // print detailed explanations of the query

query.stop();      // stop the query

query.awaitTermination();   // block until query is terminated, with stop() or with error

query.exception();       // the exception if the query has been terminated with error

query.recentProgress();  // an array of the most recent progress updates for this query

query.lastProgress();    // the most recent progress update of this streaming query
```

--------------------------------

### Directory Structure

Source: https://spark.apache.org/docs/3.5.6/quick-start.html

The standard Maven directory layout for the project files.

```bash
$ find .
./pom.xml
./src
./src/main
./src/main/java
./src/main/java/SimpleApp.java
```

--------------------------------

### Managing Streaming Queries - Scala

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

Scala code examples for managing a streaming query object, including getting its ID, name, explaining it, stopping it, and accessing progress updates.

```scala
val query = df.writeStream.format("console").start()   // get the query object

query.id          // get the unique identifier of the running query that persists across restarts from checkpoint data

query.runId       // get the unique id of this run of the query, which will be generated at every start/restart

query.name        // get the name of the auto-generated or user-specified name

query.explain()   // print detailed explanations of the query

query.stop()      // stop the query

query.awaitTermination()   // block until query is terminated, with stop() or with error

query.exception       // the exception if the query has been terminated with error

query.recentProgress  // an array of the most recent progress updates for this query

query.lastProgress    // the most recent progress update of this streaming query
```

--------------------------------

### Examples

Source: https://spark.apache.org/docs/3.5.6/api/R/reference/isActive.html

Example of how to use isActive.

```R
if (FALSE)  isActive(sq)  # \dontrun{}
```

--------------------------------

### Plotting a Series

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_ps.html

Demonstrates how to create a pandas-on-Spark Series and plot it.

```python
pser = pd.Series(np.random.randn(1000),
                 index=pd.date_range('1/1/2000', periods=1000))
psser = ps.Series(pser)
psser = psser.cummax()
psser.plot()
```

--------------------------------

### Managing Streaming Queries - Python

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

Python code examples for managing a streaming query object, including getting its ID, name, explaining it, stopping it, and accessing progress updates.

```python
query = df.writeStream.format("console").start()   # get the query object

query.id()          # get the unique identifier of the running query that persists across restarts from checkpoint data

query.runId()       # get the unique id of this run of the query, which will be generated at every start/restart

query.name()        # get the name of the auto-generated or user-specified name

query.explain()   # print detailed explanations of the query

query.stop()      # stop the query

query.awaitTermination()   # block until query is terminated, with stop() or with error

query.exception()       # the exception if the query has been terminated with error

query.recentProgress  # a list of the most recent progress updates for this query

query.lastProgress    # the most recent progress update of this streaming query
```

--------------------------------

### Example 1: Get column types

Source: https://spark.apache.org/docs/3.5.6/api/R/reference/coltypes.html

Example of how to get the column types of a SparkDataFrame.

```R
if (FALSE) { # \dontrun{
irisDF <- createDataFrame(iris)
coltypes(irisDF) # get column types
} # }
```

--------------------------------

### Examples

Source: https://spark.apache.org/docs/3.5.6/api/R/reference/sparkR.init-deprecated.html

Examples demonstrating how to initialize a SparkContext with different configurations.

```R
if (FALSE) { # \dontrun{
sc <- sparkR.init("local[2]", "SparkR", "/home/spark")
sc <- sparkR.init("local[2]", "SparkR", "/home/spark",
                 list(spark.executor.memory="1g"))
sc <- sparkR.init("yarn-client", "SparkR", "/home/spark",
                 list(spark.executor.memory="4g"),
                 list(LD_LIBRARY_PATH="/directory of JVM libraries (libjvm.so) on workers/"),
                 c("one.jar", "two.jar", "three.jar"),
                 c("com.databricks:spark-avro_2.11:2.0.1"))
} # } 
```

--------------------------------

### Examples

Source: https://spark.apache.org/docs/3.5.6/api/R/reference/sparkR.version.html

Example of how to get the Spark version.

```R
if (FALSE) { # \dontrun{
sparkR.session()
version <- sparkR.version()
} # } 
```

--------------------------------

### ADD FILE Examples

Source: https://spark.apache.org/docs/3.5.6/sql-ref-syntax-aux-resource-mgmt-add-file.html

Examples demonstrating how to add files and directories using the ADD FILE command, including paths with spaces.

```sql
ADD FILE /tmp/test;
ADD FILE "/path/to/file/abc.txt";
ADD FILE '/another/test.txt';
ADD FILE "/path with space/abc.txt";
ADD FILE "/path/to/some/directory";
ADD FILES "/path with space/cde.txt" '/path with space/fgh.txt';
```

--------------------------------

### Submit a PySpark Example

Source: https://spark.apache.org/docs/3.5.6/index.html

Submits a sample Python application to Spark.

```bash
./bin/spark-submit examples/src/main/python/pi.py 10
```

--------------------------------

### Example 2: Starting a streaming query and checking its name

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.StreamingQueryManager.get.html

This example shows how to start a streaming query with a specific query name and then access that name.

```python
sq = sdf.writeStream.format('memory').queryName('this_query').start()
sq.name
```

--------------------------------

### Examples

Source: https://spark.apache.org/docs/3.5.6/api/R/reference/sparkRSQL.init-deprecated.html

Example of initializing a SparkContext and then a SQLContext using sparkRSQL.init.

```R
if (FALSE) { # \dontrun{
sc <- sparkR.init()
sqlContext <- sparkRSQL.init(sc)
} # }
```

--------------------------------

### Example of MultiIndex.dtypes

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.pandas/api/pyspark.pandas.MultiIndex.dtypes.html

This example demonstrates how to get the dtypes of a MultiIndex.

```python
>>> psmidx = ps.MultiIndex.from_arrays(
...     [[0, 1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3, 4, 5, 6, 7, 8, 9]],
...     names=("zero", "one"),
... )
>>> psmidx.dtypes
zero    int64
one     int64
dtype: object

```

--------------------------------

### Java Example

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

Demonstrates how to retrieve and print the last progress and status of a streaming query in Java.

```Java
StreamingQuery query = ...

System.out.println(query.lastProgress());
/* Will print something like the following.

{
  "id" : "ce011fdc-8762-4dcb-84eb-a77333e28109",
  "runId" : "88e2ff94-ede0-45a8-b687-6316fbef529a",
  "name" : "MyQuery",
  "timestamp" : "2016-12-14T18:45:24.873Z",
  "numInputRows" : 10,
  "inputRowsPerSecond" : 120.0,
  "processedRowsPerSecond" : 200.0,
  "durationMs" : {
    "triggerExecution" : 3,
    "getOffset" : 2
  },
  "eventTime" : {
    "watermark" : "2016-12-14T18:45:24.873Z"
  },
  "stateOperators" : [ ],
  "sources" : [ {
    "description" : "KafkaSource[Subscribe[topic-0]]",
    "startOffset" : {
      "topic-0" : {
        "2" : 0,
        "4" : 1,
        "1" : 1,
        "3" : 1,
        "0" : 1
      }
    },
    "endOffset" : {
      "topic-0" : {
        "2" : 0,
        "4" : 115,
        "1" : 134,
        "3" : 21,
        "0" : 534
      }
    },
    "numInputRows" : 10,
    "inputRowsPerSecond" : 120.0,
    "processedRowsPerSecond" : 200.0
  } ],
  "sink" : {
    "description" : "MemorySink"
  }
}
*/


System.out.println(query.status());
/*  Will print something like the following.
{
  "message" : "Waiting for data to arrive",
  "isDataAvailable" : false,
  "isTriggerActive" : false
}
*/
```

--------------------------------

### RowMatrix.numCols Example

Source: https://spark.apache.org/docs/3.5.6/api/python/_modules/pyspark/mllib/linalg/distributed.html

Example of getting the number of columns in a RowMatrix.

```python
>>> rows = sc.parallelize([[1, 2, 3], [4, 5, 6],
...                        [7, 8, 9], [10, 11, 12]])

>>> mat = RowMatrix(rows)
>>> print(mat.numCols())
3

>>> mat = RowMatrix(rows, 7, 6)
>>> print(mat.numCols())
6
```

--------------------------------

### Plotting a DataFrame

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_ps.html

Demonstrates how to create a pandas-on-Spark DataFrame and plot it.

```python
pdf = pd.DataFrame(np.random.randn(1000, 4), index=pser.index,
                   columns=['A', 'B', 'C', 'D'])
psdf = ps.from_pandas(pdf)
psdf = psdf.cummax() 
psdf.plot()
```

--------------------------------

### Initializing SparkSession with Spark Packages

Source: https://spark.apache.org/docs/3.5.6/api/R/articles/sparkr-vignettes.html

Example of initializing a SparkSession and including a Spark package for Avro support.

```r
sparkR.session(sparkPackages = "com.databricks:spark-avro_2.12:3.0.0")
```

--------------------------------

### RowMatrix.numRows Example

Source: https://spark.apache.org/docs/3.5.6/api/python/_modules/pyspark/mllib/linalg/distributed.html

Example of getting the number of rows in a RowMatrix.

```python
>>> rows = sc.parallelize([[1, 2, 3], [4, 5, 6],
...                        [7, 8, 9], [10, 11, 12]])

>>> mat = RowMatrix(rows)
>>> print(mat.numRows())
4

>>> mat = RowMatrix(rows, 7, 6)
>>> print(mat.numRows())
7
```

--------------------------------

### Using start parameter

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.str.rfind.html

This example shows how to use the 'start' parameter to specify the starting index for the search.

```python
>>> s.str.rfind('a', start=2)
0   -1
1    2
2    5
dtype: int64
```

--------------------------------

### Creating a Spark DataFrame and converting to pandas-on-Spark

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_ps.html

Demonstrates creating a Spark DataFrame from a pandas DataFrame and then converting the Spark DataFrame back to a pandas-on-Spark DataFrame using the pandas_api() method.

```python
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
sdf = spark.createDataFrame(pdf)
sdf.show()
```

```python
psdf = sdf.pandas_api()
```

```python
psdf
```

--------------------------------

### Writing and Reading CSV

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_ps.html

Demonstrates writing a pandas-on-Spark DataFrame to a CSV file and reading it back.

```python
psdf.to_csv('foo.csv')
ps.read_csv('foo.csv').head(10)
```

--------------------------------

### Using start parameter

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.str.find.html

This example shows how to use the 'start' parameter to specify the starting index for the search.

```python
>>> s.str.find('a', start=2)
0   -1
1    2
2    3
dtype: int64
```

--------------------------------

### current_user Example

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.sql/api/pyspark.sql.functions.current_user.html

Example of using current_user() to get the current database.

```python
spark.range(1).select(current_user()).show()

```

--------------------------------

### LIST JAR Example

Source: https://spark.apache.org/docs/3.5.6/sql-ref-syntax-aux-resource-mgmt-list-jar.html

Example demonstrating how to add JARs and then list them, showing the expected output.

```sql
ADD JAR /tmp/test.jar;
ADD JAR /tmp/test_2.jar;
LIST JAR;
-- output for LIST JAR
spark://192.168.1.112:62859/jars/test.jar
spark://192.168.1.112:62859/jars/test_2.jar
```

--------------------------------

### Examples

Source: https://spark.apache.org/docs/3.5.6/api/R/reference/schema.html

Example of how to get the schema of a SparkDataFrame.

```R
if (FALSE) { # \dontrun{
sparkR.session()
path <- "path/to/file.json"
df <- read.json(path)
dfSchema <- schema(df)
} # }
```

--------------------------------

### Creating a DataFrame for grouping

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_ps.html

Creates a sample Spark DataFrame with columns 'A', 'B', 'C', and 'D' for demonstrating grouping operations.

```python
psdf = ps.DataFrame({'A': ['foo', 'bar', 'foo', 'bar',
                          'foo', 'bar', 'foo', 'foo'],
                    'B': ['one', 'one', 'two', 'three',
                          'two', 'two', 'one', 'three'],
                    'C': np.random.randn(8),
                    'D': np.random.randn(8)})
```

--------------------------------

### SHOW DATABASES Examples

Source: https://spark.apache.org/docs/3.5.6/sql-ref-syntax-aux-show-databases.html

Examples of using the SHOW DATABASES command, including creating databases and filtering results with a LIKE pattern.

```SQL
-- Create database. Assumes a database named `default` already exists in
-- the system. 
CREATE DATABASE payroll_db;
CREATE DATABASE payments_db;

-- Lists all the databases. 
SHOW DATABASES;
+------------+
|databaseName|
+------------+
|     default|
| payments_db|
|  payroll_db|
+------------+
  
-- Lists databases with name starting with string pattern `pay`
SHOW DATABASES LIKE 'pay*';
+------------+
|databaseName|
+------------+
| payments_db|
|  payroll_db|
+------------+
  
-- Lists all databases. Keywords SCHEMAS and DATABASES are interchangeable. 
SHOW SCHEMAS;
+------------+
|databaseName|
+------------+
|     default|
| payments_db|
|  payroll_db|
+------------+


```

--------------------------------

### MapReduceTriplets Example

Source: https://spark.apache.org/docs/3.5.6/graphx-programming-guide.html

An example of using the mapReduceTriplets operator.

```Scala
val graph: Graph[Int, Float] = ...
def msgFun(triplet: Triplet[Int, Float]): Iterator[(Int, String)] = {
  Iterator((triplet.dstId, "Hi"))
}
def reduceFun(a: String, b: String): String = a + " " + b
val result = graph.mapReduceTriplets[String](msgFun, reduceFun)
```

--------------------------------

### Scala Dataset Creation

Source: https://spark.apache.org/docs/3.5.6/quick-start.html

Creating a Dataset from a text file in Scala.

```scala
scala> val textFile = spark.read.textFile("README.md")
textFile: org.apache.spark.sql.Dataset[String] = [value: string]

```

--------------------------------

### Get the day of the week

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.pandas/api/pyspark.pandas.DatetimeIndex.dayofweek.html

This example demonstrates how to get the day of the week from a DatetimeIndex.

```python
>>> idx = ps.date_range('2016-12-31', '2017-01-08', freq='D')
>>> idx.dayofweek
Int64Index([5, 6, 0, 1, 2, 3, 4, 5, 6], dtype='int64')
```

--------------------------------

### Examples

Source: https://spark.apache.org/docs/3.5.6/api/R/reference/printSchema.html

Example of how to use printSchema

```R
if (FALSE) { # \dontrun{
ssparkR.session()
path <- "path/to/file.json"
df <- read.json(path)
printSchema(df)
} # }
```

--------------------------------

### Example: Creating a table and showing properties

Source: https://spark.apache.org/docs/3.5.6/sql-ref-syntax-aux-show-tblproperties.html

This example demonstrates how to create a table with specified properties and then display all user-specified properties for that table.

```sql
-- create a table `customer` in database `salesdb`
USE salesdb;
CREATE TABLE customer(cust_code INT, name VARCHAR(100), cust_addr STRING)
    TBLPROPERTIES ('created.by.user' = 'John', 'created.date' = '01-01-2001');

-- show all the user specified properties for table `customer`
SHOW TBLPROPERTIES customer;
+---------------------+
|                  key|     value|
+---------------------+
|      created.by.user|      John|
|         created.date|01-01-2001|
|transient_lastDdlTime|1567554931|
+---------------------+

```

--------------------------------

### Get the day of the week

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.pandas/api/pyspark.pandas.DatetimeIndex.day_of_week.html

This example demonstrates how to get the day of the week from a DatetimeIndex.

```python
>>> idx = ps.date_range('2016-12-31', '2017-01-08', freq='D')  
>>> idx.dayofweek  
Int64Index([5, 6, 0, 1, 2, 3, 4, 5, 6], dtype='int64')
```

--------------------------------

### Writing and Reading Spark IO (ORC)

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_ps.html

Demonstrates writing a pandas-on-Spark DataFrame to an ORC file using Spark IO and reading it back.

```python
psdf.to_spark_io('zoo.orc', format="orc")
ps.read_spark_io('zoo.orc', format="orc").head(10)
```

--------------------------------

### bitmap_bucket_number example

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.sql/api/pyspark.sql.functions.bitmap_bucket_number.html

Example of using bitmap_bucket_number to get the bucket number for a column.

```python
>>> df = spark.createDataFrame([(123,)], ["a"])
>>> df.select(bitmap_bucket_number(df.a).alias("r")).collect()
[Row(r=1)]
```

--------------------------------

### currentDatabase Example

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.sql/api/pyspark.sql.Catalog.currentDatabase.html

Example of how to get the current default database in a PySpark session.

```python
>>> spark.catalog.currentDatabase()
'default'
```

--------------------------------

### Python Example

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

Demonstrates how to retrieve and print the last progress and status of a streaming query in Python.

```Python
query = ...  # a StreamingQuery
print(query.lastProgress)

'''
Will print something like the following.

{u'stateOperators': [], u'eventTime': {u'watermark': u'2016-12-14T18:45:24.873Z'}, u'name': u'MyQuery', u'timestamp': u'2016-12-14T18:45:24.873Z', u'processedRowsPerSecond': 200.0, u'inputRowsPerSecond': 120.0, u'numInputRows': 10, u'sources': [{u'description': u'KafkaSource[Subscribe[topic-0]]', u'endOffset': {u'topic-0': {u'1': 134, u'0': 534, u'3': 21, u'2': 0, u'4': 115}}, u'processedRowsPerSecond': 200.0, u'inputRowsPerSecond': 120.0, u'numInputRows': 10, u'startOffset': {u'topic-0': {u'1': 1, u'0': 1, u'3': 1, u'2': 0, u'4': 1}}}], u'durationMs': {u'getOffset': 2, u'triggerExecution': 3}, u'runId': u'88e2ff94-ede0-45a8-b687-6316fbef529a', u'id': u'ce011fdc-8762-4dcb-84eb-a77333e28109', u'sink': {u'description': u'MemorySink'}}
'''

print(query.status)
'''
Will print something like the following.

{u'message': u'Waiting for data to arrive', u'isTriggerActive': False, u'isDataAvailable': False}
'''
```

--------------------------------

### S3 Dependency Management Example

Source: https://spark.apache.org/docs/3.5.6/running-on-kubernetes.html

Example of submitting an application with dependencies hosted on S3.

```bash
...
--packages org.apache.hadoop:hadoop-aws:3.2.2
--conf spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path
--conf spark.hadoop.fs.s3a.access.key=...
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.fast.upload=true
--conf spark.hadoop.fs.s3a.secret.key=....
--conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp
file:///full/path/to/app.jar


```

--------------------------------

### percentile_approx example 2

Source: https://spark.apache.org/docs/3.5.6/api/python/_modules/pyspark/sql/functions.html

Example of using percentile_approx to get the median.

```python
>>> df.groupBy("key").agg(
...     percentile_approx("value", 0.5, lit(1000000)).alias("median")
... ).printSchema()
root
 |-- key: long (nullable = true)
 |-- median: double (nullable = true)
```

--------------------------------

### percentile_approx example 1

Source: https://spark.apache.org/docs/3.5.6/api/python/_modules/pyspark/sql/functions.html

Example of using percentile_approx to get quantiles.

```python
>>> key = (col("id") % 3).alias("key")
>>> value = (randn(42) + key * 10).alias("value")
>>> df = spark.range(0, 1000, 1, 1).select(key, value)
>>> df.select(
...     percentile_approx("value", [0.25, 0.5, 0.75], 1000000).alias("quantiles")
... ).printSchema()
root
 |-- quantiles: array (nullable = true)
 |    |-- element: double (containsNull = false)
```

--------------------------------

### Show and Print Schema

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_df.html

Displays the content of the DataFrame and prints its schema.

```python
# All DataFrames above result same.
df.show()
df.printSchema()
```

--------------------------------

### Index Size Example

Source: https://spark.apache.org/docs/3.5.6/api/python/_modules/pyspark/pandas/indexes/base.html

Example demonstrating how to get the size of a pandas-on-Spark Index.

```python
>>> df = ps.DataFrame([(.2, .3), (.0, .6), (.6, .0), (.2, .1)],
...                   columns=['dogs', 'cats'],
...                   index=list('abcd'))
>>> df.index.size
4

>>> df.set_index('dogs', append=True).index.size
4
```

--------------------------------

### Creating a pandas-on-Spark Series

Source: https://spark.apache.org/docs/3.5.6/api/python/getting_started/quickstart_ps.html

Demonstrates creating a pandas-on-Spark Series from a list of values with a default integer index.

```python
import pyspark.pandas as ps
import numpy as np
s = ps.Series([1, 3, 5, np.nan, 6, 8])
```

```python
s
```

--------------------------------

### Run Python example

Source: https://spark.apache.org/docs/3.5.6/streaming-programming-guide.html

This command shows how to run the Python Spark Streaming example.

```shell
$ ./bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost 9999
```

--------------------------------

### BlockMatrix numCols Example

Source: https://spark.apache.org/docs/3.5.6/api/python/_modules/pyspark/mllib/linalg/distributed.html

Example demonstrating how to get the number of columns in a BlockMatrix.

```python
blocks = sc.parallelize([((0, 0), Matrices.dense(3, 2, [1, 2, 3, 4, 5, 6])),
                         ((1, 0), Matrices.dense(3, 2, [7, 8, 9, 10, 11, 12]))])

mat = BlockMatrix(blocks, 3, 2)
print(mat.numCols())
```

```python
blocks = sc.parallelize([((0, 0), Matrices.dense(3, 2, [1, 2, 3, 4, 5, 6])),
                         ((1, 0), Matrices.dense(3, 2, [7, 8, 9, 10, 11, 12]))])

mat = BlockMatrix(blocks, 3, 2, 7, 6)
print(mat.numCols())
```

--------------------------------

### BlockMatrix numRows Example

Source: https://spark.apache.org/docs/3.5.6/api/python/_modules/pyspark/mllib/linalg/distributed.html

Example demonstrating how to get the number of rows in a BlockMatrix.

```python
blocks = sc.parallelize([((0, 0), Matrices.dense(3, 2, [1, 2, 3, 4, 5, 6])),
                         ((1, 0), Matrices.dense(3, 2, [7, 8, 9, 10, 11, 12]))])

mat = BlockMatrix(blocks, 3, 2)
print(mat.numRows())
```

```python
blocks = sc.parallelize([((0, 0), Matrices.dense(3, 2, [1, 2, 3, 4, 5, 6])),
                         ((1, 0), Matrices.dense(3, 2, [7, 8, 9, 10, 11, 12]))])

mat = BlockMatrix(blocks, 3, 2, 7, 6)
print(mat.numRows())
```

--------------------------------

### Examples

Source: https://spark.apache.org/docs/3.5.6/api/R/reference/dropTempView.html

Example of how to use dropTempView.

```R
if (FALSE) { # \dontrun{
sparkR.session()
df <- read.df(path, "parquet")
createOrReplaceTempView(df, "table")
dropTempView("table")
} # } 
```

--------------------------------

### R SparkSession Example

Source: https://spark.apache.org/docs/3.5.6/sql-getting-started.html

Example of creating a SparkSession in R.

```r
sparkR.session(appName = "R Spark SQL basic example", sparkConfig = list(spark.some.config.option = "some-value"))
```

--------------------------------

### HashPartitioner Example

Source: https://spark.apache.org/docs/3.5.6/api/scala/org/apache/spark/HashPartitioner.html

This example shows how to create and use a HashPartitioner.

```scala
val partitioner = new HashPartitioner(10)
val rdd = sc.parallelize(Seq((1, "a"), (2, "b"), (3, "c")))
val partitionedRdd = rdd.partitionBy(partitioner)
partitionedRdd.collect()
```

--------------------------------

### sbt build.sbt configuration

Source: https://spark.apache.org/docs/3.5.6/quick-start.html

sbt build configuration file specifying Spark SQL as a dependency.

```scala
name := "Simple Project"

version := "1.0"

scalaVersion := "2.12.18"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.5.6"
```

--------------------------------

### Scala SparkSession Example

Source: https://spark.apache.org/docs/3.5.6/sql-getting-started.html

Example of creating a SparkSession in Scala.

```scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession
  .builder()
  .appName("Spark SQL basic example")
  .config("spark.some.config.option", "some-value")
  .getOrCreate()
```

--------------------------------

### BisectingKMeans Example

Source: https://spark.apache.org/docs/3.5.6/api/python/reference/api/pyspark.ml.clustering.BisectingKMeans.html

This example demonstrates how to use the BisectingKMeans algorithm, including setting parameters, fitting the model, making predictions, and saving/loading the model.

```python
from pyspark.ml.linalg import Vectors
data = [(Vectors.dense([0.0, 0.0]), 2.0), (Vectors.dense([1.0, 1.0]), 2.0),
        (Vectors.dense([9.0, 8.0]), 2.0), (Vectors.dense([8.0, 9.0]), 2.0)]
df = spark.createDataFrame(data, ["features", "weighCol"])
bkm = BisectingKMeans(k=2, minDivisibleClusterSize=1.0)
bkm.setMaxIter(10)
print(bkm.getMaxIter())
bkm.clear(bkm.maxIter)
bkm.setSeed(1)
bkm.setWeightCol("weighCol")
print(bkm.getSeed())
bkm.clear(bkm.seed)
model = bkm.fit(df)
print(model.getMaxIter())
model.setPredictionCol("newPrediction")
print(model.predict(df.head().features))
centers = model.clusterCenters()
print(len(centers))
print(model.computeCost(df))
print(model.hasSummary)
summary = model.summary
print(summary.k)
print(summary.clusterSizes)
print(summary.trainingCost)
transformed = model.transform(df).select("features", "newPrediction")
rows = transformed.collect()
print(rows[0].newPrediction == rows[1].newPrediction)
print(rows[2].newPrediction == rows[3].newPrediction)
bkm_path = temp_path + "/bkm"
bkm.save(bkm_path)
bkm2 = BisectingKMeans.load(bkm_path)
print(bkm2.getK())
print(bkm2.getDistanceMeasure())
model_path = temp_path + "/bkm_model"
model.save(model_path)
model2 = BisectingKMeansModel.load(model_path)
print(model2.hasSummary)
print(model.clusterCenters()[0] == model2.clusterCenters()[0])
print(model.clusterCenters()[1] == model2.clusterCenters()[1])
print(model.transform(df).take(1) == model2.transform(df).take(1))
```

--------------------------------

### JAAS Login Configuration Example

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-kafka-integration.html

Example of how to configure JAAS login for Kafka integration using JVM options with spark-submit.

```bash
./bin/spark-submit \
    --driver-java-options "-Djava.security.auth.login.config=/path/to/custom_jaas.conf" \
    --conf spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/path/to/custom_jaas.conf \
    ...

```

--------------------------------

### AggregateMessages Example

Source: https://spark.apache.org/docs/3.5.6/graphx-programming-guide.html

Rewriting the mapReduceTriplets example using aggregateMessages.

```Scala
val graph: Graph[Int, Float] = ...
def msgFun(triplet: EdgeContext[Int, Float, String]) {
  triplet.sendToDst("Hi")
}
def reduceFun(a: String, b: String): String = a + " " + b
val result = graph.aggregateMessages[String](msgFun, reduceFun)
```

--------------------------------

### Java foreach example

Source: https://spark.apache.org/docs/3.5.6/structured-streaming-programming-guide.html

Example of using foreach in Java by extending ForeachWriter.

```Java
streamingDatasetOfString.writeStream().foreach(
  new ForeachWriter<String>() {

    @Override public boolean open(long partitionId, long version) {
      // Open connection
    }

    @Override public void process(String record) {
      // Write string to connection
    }

    @Override public void close(Throwable errorOrNull) {
      // Close the connection
    }
  }
).start();
```

--------------------------------

### position function examples

Source: https://spark.apache.org/docs/3.5.6/sql-ref-functions-builtin.html

Examples showing how to use the position function to find the starting position of a substring within a string, with and without a starting position.

```sql
-- position
SELECT position('bar', 'foobarbar');
+---------------------------+
|position(bar, foobarbar, 1)|
+---------------------------+
|                          4|
+---------------------------+

SELECT position('bar', 'foobarbar', 5);
+---------------------------+
|position(bar, foobarbar, 5)|
+---------------------------+
|                          7|
+---------------------------+
```