### Spark 3 with Scala 2.12 Setup (Scala)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Command to start a Spark Scala shell with SparkMeasure for Spark 3 and Scala 2.12.

```bash
spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.28
```

--------------------------------

### Start InfluxDB CLI

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_InfluxDBSink.md

Initiates the InfluxDB command-line interface. Ensure InfluxDB is installed and running.

```bash
/usr/bin/influx
```

--------------------------------

### Build and Run SparkMeasure Example

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Instrument_Scala_code.md

Commands to build the example JAR and submit it to Spark using a packaged version of SparkMeasure.

```bash
# build the example jar
sbt package

bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28 --class ch.cern.testSparkMeasure.testSparkMeasure <path_to_the_example_jar>/testsparkmeasurescala_2.13-0.1.jar

```

--------------------------------

### Build and Run Example JAR

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/testSparkMeasureScala/README.md

Instructions for building the example JAR using sbt and running it with spark-submit, including necessary packages and class.

```bash
# build the example jar
sbt package

bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26 --class ch.cern.testSparkMeasure.testSparkMeasure <path_to_the_example_jar>/testsparkmeasurescala_2.13-0.1.jar
```

--------------------------------

### CLI Example for Task Metrics (Python)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Example of how to run and measure task metrics using the TaskMetrics class in a PySpark shell. Ensure the spark-measure package is installed and included.

```bash
# Python CLI
# pip install pyspark
pip install sparkmeasure
pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
```

```python
from sparkmeasure import TaskMetrics
taskmetrics = TaskMetrics(spark)
taskmetrics.runandmeasure(globals(), 'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()')
```

--------------------------------

### Spark 4 with Scala 2.13 Setup (Scala)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Command to start a Spark Scala shell with SparkMeasure for Spark 4 and Scala 2.13.

```bash
spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
```

--------------------------------

### Example: Launch Spark Shell with PushGatewaySink

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_PrometheusPushgatewaySink.md

This example demonstrates how to launch the Spark shell with the PushGatewaySink listener enabled, specifying the Pushgateway address and the spark-measure package.

```bash
bin/spark-shell \
--conf spark.extraListeners=ch.cern.sparkmeasure.PushGatewaySink \
--conf spark.sparkmeasure.pushgateway=localhost:9091 \
--packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
```

--------------------------------

### Python Example: Stage-Level Metrics to Local File

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_FileSink.md

A Python example that runs pi.py, collects stage-level metrics, and saves them to a local file in JSON format.

```bash
bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28 \
--conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderStageMetrics \
examples/src/main/python/pi.py

# Read the flight recorder output file:
more /tmp/stageMetrics_flightRecorder
```

--------------------------------

### Spark 3 with Scala 2.12 Setup (Python)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Commands to set up a PySpark environment for Spark 3 and Scala 2.12, including installing sparkmeasure.

```bash
pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.28
pip install sparkmeasure
```

--------------------------------

### Install sparkMeasure and PySpark

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Python_shell_and_Jupyter.md

Install the Python wrapper and obtain the JAR from Maven Central. Ensure you use the correct PySpark and Spark version.

```bash
pip install pyspark # Spark 4
pip install sparkmeasure
bin/pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
```

--------------------------------

### Scala Example: Stage-Level Metrics with Custom Output and Stdout

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_FileSink.md

A Scala example that collects stage-level metrics, uses a custom output filename, and prints metrics to stdout.

```bash
bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28 \
--class org.apache.spark.examples.SparkPi \
--conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderStageMetrics \
--conf spark.sparkmeasure.printToStdout=true \
--conf spark.sparkmeasure.outputFilename="/tmp/myoutput_$(date +%s).json" \
examples/jars/spark-examples_2.13-4.0.0.jar 10

# The metrics are printed on stdout and also saved to a file
# Find and read the flight recorder output file:
ls -ltr /tmp/myoutput*.json
```

--------------------------------

### InfluxDB Sink Configuration Example

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_InfluxDBSink.md

This example shows how to configure the InfluxDB sink for SparkMeasure's Flight Recorder mode. Ensure InfluxDB is running and accessible.

```text
1718107791715000000 local-1718107775270 0             0         0            1718107791715  7957128399      312604886                  958                     14359                         0         0                  0                   1000        0              35                      64934      1880                0                    0                         0                     0                  32                    0                          0                      0                            0                         0                     419858111        MyAppName      false                           1       1718107791154
1718107791910000000 local-1718107775270 0             0         0            1718107791910  81533655        4458869                    4                       90                            0         0                  0                   0           0              0                       4006       0                   0                    32                        1880                  32                 0                     0                          0                      0                            32                        1880                  0                MyAppName      false                           3       1718107791799
```

--------------------------------

### Build sparkMeasure from Source

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Python_shell_and_Jupyter.md

Clone the repository, build the JAR using sbt, install the Python wrapper, and run PySpark with the JAR.

```bash
git clone https://github.com/lucacanali/sparkmeasure
cd sparkmeasure
sbt +package
ls -l target/scala-2.13/spark-measure*.jar  # note location of the compiled and packaged jar

# Install the Python wrapper package
cd python
pip install .

# Run as in one of these examples:
bin/pyspark --jars path>/spark-measure_2.13-0.29-SNAPSHOT.jar

#Alternative:
bin/pyspark --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.29-SNAPSHOT.jar
```

--------------------------------

### Start Spark Session with spark-measure Package

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkMeasure_Jupyter_Colab_Example.ipynb

Initializes a SparkSession in local mode and configures it to use the spark-measure package by specifying the Maven coordinates. This setup is necessary for sparkMeasure to attach its listener.

```python
from pyspark.sql import SparkSession

# Start the Spark Session
# This example uses Spark in local mode for simplicity.
# You can modify master to use YARN or K8S if available 

spark = (
  SparkSession
  .builder
  .master("local[*]")
  .appName("Test sparkmeasure instrumentation of Python/PySpark code")
  .config("spark.jars.packages","ch.cern.sparkmeasure:spark-measure_2.13:0.28")
  .getOrCreate() )

```

--------------------------------

### Spark 4 with Scala 2.13 Setup (Python)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Commands to set up a PySpark environment for Spark 4 and Scala 2.13, including installing sparkmeasure.

```bash
pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
pip install sparkmeasure
```

--------------------------------

### Start InfluxDB using Docker (Alternative 2)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_InfluxDBSink.md

Use this command to start a standalone InfluxDB container, version 1.8.10, mapping its port to the host's network.

```bash
# Alternative 2.
# Start InfluxDB stand-alone, for example using a docker image 
docker run --name influx --network=host -d influxdb:1.8.10
```

--------------------------------

### Start Spark with KafkaSink Listener

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_KafkaSink.md

Configure Spark to use the KafkaSink listener and specify Kafka broker and topic details. Ensure the spark-measure package is included.

```bash
# edit my_kafka_server with the Kafka broker server name
bin/spark-shell \
--conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSink \
--conf spark.sparkmeasure.kafkaBroker=localhost:9092 \
--conf spark.sparkmeasure.kafkaTopic=metrics\
--packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Unit-tests.md

Install the necessary Python packages required for running the integration tests. This command reads dependencies from the specified requirements file.

```bash
pip install -r python/requirements.txt
```

--------------------------------

### CLI Example for Task Metrics (Scala)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Example of how to run and measure task metrics using the TaskMetrics class in a Spark Scala shell. Ensure the spark-measure package is included.

```bash
# Scala CLI
spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
```

```scala
val taskMetrics = ch.cern.sparkmeasure.TaskMetrics(spark)
taskMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show())
```

--------------------------------

### Start Spark Session with SparkMeasure Package

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkMeasure_Jupyter_Python_getting_started.ipynb

Initializes a SparkSession in local mode and configures it to use the sparkMeasure package from Maven central. Ensure the Scala version (2.13) matches your Spark installation.

```python
# Start the Spark Session
# This example uses Spark in local mode for simplicity.
# You can modify master to use YARN or K8S if available 
# This example uses sparkMeasure compiled with scala 2.13, downloaded from maven central


from pyspark.sql import SparkSession
spark = (
    SparkSession.builder
    .appName("Test sparkmeasure instrumentation of Python/PySpark code")
    .master("local[*]")
    .config("spark.jars.packages", "ch.cern.sparkmeasure:spark-measure_2.13:0.28")
    .getOrCreate()
)
  

```

--------------------------------

### Start InfluxDB using Docker (Alternative 1)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_InfluxDBSink.md

Use this command to start an InfluxDB container if you plan to use the Spark dashboard v1. It maps the container's port to the host's network.

```bash
# Alternative 1. 
# Use this if you plan to use the Spark dashboard v1 as in
# https://github.com/cerndb/spark-dashboard 
docker run --name influx --network=host -d lucacanali/spark-dashboard:v01
```

--------------------------------

### Start Spark Shell with InfluxDBSink Listener

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_InfluxDBSink.md

This command starts a Spark shell with the InfluxDBSink and InfluxDBSinkExtended listeners enabled, configuring the InfluxDB URL and enabling stage metrics. It also includes the spark-measure package.

```bash
bin/spark-shell \
  --name MyAppName
  --conf spark.sparkmeasure.influxdbURL="http://localhost:8086" \
  --conf spark.extraListeners=ch.cern.sparkmeasure.InfluxDBSink,ch.cern.sparkmeasure.InfluxDBSinkExtended \
  --conf spark.sparkmeasure.influxdbStagemetrics=true
  --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28

// run a Spark job, this will produce metrics  
spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show
```

--------------------------------

### Install SparkMeasure and PySpark

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkMeasure_Jupyter_Python_getting_started.ipynb

Installs the necessary Python packages for PySpark and the sparkMeasure API. This is a prerequisite for using sparkMeasure in a Python environment.

```python
# Dependencies

# 1. Install PySpark/Spark 
# This is optional, if you have already downloaded Spark
# See findspark (3.) in that case
!pip install pyspark

# 2. Install the Python wrapper API for spark-measure
!pip install sparkmeasure

```

--------------------------------

### Create and Activate Python Virtual Environment

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Unit-tests.md

Set up a dedicated Python virtual environment for running sparkMeasure integration tests. Activate the environment to ensure dependencies are installed locally.

```bash
python3 -m venv ~/venv/sparkmeasure
source ~/venv/sparkmeasure/bin/activate
```

--------------------------------

### Install sparkmeasure Package

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkMeasure_Jupyter_Colab_Example.ipynb

Installs the sparkmeasure Python package using pip. This is a prerequisite for using the sparkMeasure API in your Python environment.

```python
# Dependencies

# Install PySpark/Spark 

# !pip install pyspark

# Install the sparkmeasure Python wrapper API

!pip install sparkmeasure
```

--------------------------------

### Measure Stage Metrics and Send to Prometheus

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Prometheus.md

This example demonstrates how to initialize StageMetrics, begin collecting metrics, execute Spark jobs, end collection, and then send the collected metrics to a Prometheus Pushgateway. Ensure the spark-measure package is included.

```scala
bin/spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28

val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
stageMetrics.begin()

...execute one or more Spark jobs...

stageMetrics.end()
stageMetrics.sendReportPrometheus(s"localhost:9091", s"aaa_job", s"label_name", s"label_value")
```

--------------------------------

### Collect and print stage metrics with sparkMeasure

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Scala_shell_and_notebooks.md

Instantiate StageMetrics, run a Spark SQL query, and then retrieve and print the aggregated stage metrics. This example also shows how to print a memory report.

```shell
bin/spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
```

```scala
val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)

// get metrics as a Map
val metrics = stageMetrics.aggregateStageMetrics
```

```scala
> stageMetrics.printMemoryReport
```

--------------------------------

### Attach KafkaSink Listener

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_KafkaSink.md

Attach the KafkaSink listener to a Spark Context using the extra listener infrastructure. This is the recommended way to start the listener.

```bash
--conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSink
```

--------------------------------

### Attach KafkaSinkV2 Listener

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_KafkaSink.md

Start the KafkaSinkV2 listener, which is the recommended version for enhanced metrics and custom labels. This attaches the listener to the Spark Context.

```bash
--conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSinkV2
```

--------------------------------

### Python CLI: Run and Measure Spark SQL Query

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Use this snippet to quickly measure the performance of a Spark SQL query using a one-liner command. Ensure sparkmeasure is installed and the pyspark package is available.

```python
# Python CLI
# pip install pyspark
pip install sparkmeasure
pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28

# Import sparkMeasure
from sparkmeasure import StageMetrics
stagemetrics = StageMetrics(spark)
# Simple one-liner to run a Spark SQL query and measure its performance
stagemetrics.runandmeasure(globals(), 'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()')
```

--------------------------------

### Cluster Mode Example: Task-Level Metrics to HDFS

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_FileSink.md

An example for cluster mode on YARN that collects task-level metrics and writes them to HDFS using the json_to_hadoop format.

```bash
bin/spark-submit --master yarn --deploy-mode cluster \
--packages ch.cern.sparkmeasure:spark-measure_2.13:0.28 \
--conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderTaskMetrics \
--conf spark.sparkmeasure.outputFormat=json_to_hadoop \
--conf spark.sparkmeasure.outputFilename="hdfs://myclustername/user/luca/test/myoutput_$(date +%s).json" \
examples/src/main/python/pi.py
```

--------------------------------

### Run sparkMeasure using packaged version from Maven Central

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Scala_shell_and_notebooks.md

Install sparkMeasure using the --packages option in the spark-shell. Alternatively, specify the JAR path using spark.driver.extraClassPath.

```shell
bin/spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
```

```shell
bin/spark-shell --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.28.jar
```

--------------------------------

### Collect and print stage metrics using begin/end

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Scala_shell_and_notebooks.md

An alternative method to collect stage metrics by manually starting and ending the measurement period. This allows for measuring multiple Spark jobs within a single measurement session.

```scala
val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
stageMetrics.begin()

...execute one or more Spark jobs...

stageMetrics.end()
stageMetrics.printReport()
```

--------------------------------

### Collect and Save Stage Metrics in Scala

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Instrument_Scala_code.md

Example of collecting stage metrics, printing reports, and saving metrics data to files using SparkMeasure in Scala.

```scala
    val spark = SparkSession.
      builder().
      appName("testSparkMeasure").
      getOrCreate()

    val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
    stageMetrics.runAndMeasure {
      spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()
    }

    // print report to standard output
    stageMetrics.printReport()

    // get metric values as a Map
    val metrics = stageMetrics.aggregateStageMetrics()
    assert(metrics("numStages") > 1)

    // Introduced in sparkMeasure v0.21, memory metrics report:
    stageMetrics.printMemoryReport()

    //save session metrics data
    val df = stageMetrics.createStageMetricsDF("PerfStageMetrics")
    stageMetrics.saveData(df.orderBy("jobId", "stageId"), "/tmp/stagemetrics_test1")

    val aggregatedDF = stageMetrics.aggregateStageMetrics("PerfStageMetrics")
    stageMetrics.saveData(aggregatedDF, "/tmp/stagemetrics_report_test2")
```

--------------------------------

### Build SparkMeasure from Source and Run

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Instrument_Scala_code.md

Steps to clone the SparkMeasure repository, build it using sbt, and run it with spark-submit, either by providing the JAR path or setting the driver classpath.

```bash
git clone https://github.com/lucacanali/sparkmeasure
cd sparkmeasure
sbt +package
ls -l target/scala-2.13/spark-measure*.jar  # location of the compiled jar

# Run as in one of these examples:
bin/spark-submit --jars path>/spark-measure_2.13-0.29-SNAPSHOT.jar

# alternative, set classpath for the driver (it is only needed in the driver)
bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.29-SNAPSHOT.jar ...

```

--------------------------------

### Build and Run sparkMeasure from Source

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Instrument_Python_code.md

Steps to clone the sparkMeasure repository, build the JAR, and then run a Python application using the built JAR. This is for development or using the latest versions.

```bash
git clone https://github.com/lucacanali/sparkmeasure
cd sparkmeasure
sbt +package
ls -l target/scala-2.13/spark-measure*.jar  # location of the compiled jar

cd python
pip install .

# Run as in one of these examples:
bin/spark-submit --jars path>/spark-measure_2.13-0.29-SNAPSHOT.jar ...

# alternative, set classpath for the driver (sparkmeasure code runs only in the driver)
bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.29-SNAPSHOT.jar ...
```

--------------------------------

### Build sparkMeasure from source and run

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Scala_shell_and_notebooks.md

Clone the sparkMeasure repository, build the JAR using sbt, and then run spark-shell using the built JAR. The JAR is only needed in the driver.

```shell
git clone https://github.com/lucacanali/sparkmeasure
cd sparkmeasure
sbt +package
ls -l target/scala-2.13/spark-measure*.jar  # location of the compiled jar
```

```shell
bin/spark-shell --jars <path>/spark-measure_2.13-0.29-SNAPSHOT.jar
```

```shell
bin/spark-shell --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.29-SNAPSHOT.jar
```

--------------------------------

### Configure Flight Recorder for Stage-Level Metrics

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_FileSink.md

Add these configurations to spark-submit to record metrics at the stage execution level.

```bash
--packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
--conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderStageMetrics
```

--------------------------------

### Collect Task Metrics in Scala

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Instrument_Scala_code.md

Example of collecting task metrics at a finer granularity using SparkMeasure. This option has additional overhead and should be used only when necessary.

```scala
    val taskMetrics = ch.cern.sparkmeasure.TaskMetrics(spark)
    taskMetrics.runAndMeasure {
      spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()
    }

```

--------------------------------

### Initialize and Collect Task Metrics

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkMeasure_Jupyter_Colab_Example.ipynb

Initializes TaskMetrics, runs a Spark SQL query, and ends the collection. Use this for basic task metric collection.

```python
from sparkmeasure import TaskMetrics
taskmetrics = TaskMetrics(spark)

taskmetrics.begin()
spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()
taskmetrics.end()

taskmetrics.print_report()
```

--------------------------------

### StageMetrics Python Class Example

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Python wrapper for SparkMeasure's StageMetrics Scala class. Use this to collect and process task metrics aggregated by execution stage.

```python
from sparkmeasure import StageMetrics
stagemetrics = StageMetrics(spark)

stagemetrics.runandmeasure(globals(), 'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()')
```

--------------------------------

### Python CLI: Measure Spark SQL Query with begin/end

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

This Python snippet shows how to use the begin() and end() methods to measure the performance of a specific block of Spark code, such as a Spark SQL query. It also demonstrates how to print detailed reports.

```python
# Alternatively, you can use the begin() and end() methods to measure performance
# Start measuring
stagemetrics.begin()

spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()

# Set a stop point for measuring metrics delta values
stagemetrics.end()
# Print the metrics report
stagemetrics.print_report()
stagemetrics.print_memory_report()

# get metrics as a dictionary
metrics = stagemetrics.aggregate_stage_metrics()
```

--------------------------------

### KafkaSinkV2 Configuration Change

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_KafkaSink.md

Demonstrates the configuration change required to migrate from KafkaSink to KafkaSinkV2 by updating the listener class name.

```bash
# Old:
--conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSink 

# New:
--conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSinkV2
```

--------------------------------

### Configure Kafka Topic

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_KafkaSink.md

Specify the Kafka topic where Spark metrics will be sent. The topic will be created if it does not exist.

```bash
--conf spark.sparkmeasure.kafkaTopic=sparkmeasure-stageinfo
```

--------------------------------

### Visualize Average Task Duration and CPU Usage by Host

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkTaskMetricsAnalysisExample.ipynb

Creates a bar chart comparing the average task duration and average CPU time across hosts. This helps in determining if the workload is CPU-bound.

```python
spark.sql("""
select host, avg(duration) avg_duration, avg(executorCpuTime) avg_CPU
from PerfTaskMetrics 
group by host""").toPandas().plot(x='host', kind='bar', figsize=(12, 8))
```

--------------------------------

### TaskMetrics Scala API

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

The TaskMetrics class in Scala allows for detailed collection and reporting of metrics at the task granularity. It provides methods to start and stop data collection, generate reports, and save metrics data.

```APIDOC
## TaskMetrics Scala API

### Description
Collects metrics data at Task granularity and provides aggregation and reporting functions.

### Class Definition
```scala
case class TaskMetrics(sparkSession: SparkSession)
```

### Methods

#### `begin(): Long`
Marks the beginning of data collection.

#### `end(): Long`
Marks the end of data collection.

#### `removeListener(): Unit`
Helper method to remove the listener.

#### `aggregateTaskMetrics(): LinkedHashMap[String, Long]`
Computes basic aggregation on the Task metrics for the metrics report, filtering on time boundaries. Returns a map with metrics names and their aggregated values.

#### `report(): String`
Custom aggregations and post-processing of metrics data.

#### `printReport(): Unit`
Runs the report and prints it.

#### `createTaskMetricsDF(nameTempView: String = "PerfTaskMetrics"): DataFrame`
Legacy transformation of data recorded from the custom Stage listener into a DataFrame and registers it as a view for querying with SQL.

#### `aggregateTaskMetrics(nameTempView: String = "PerfTaskMetrics"): DataFrame`
Legacy metrics aggregation computed using SQL.

#### `reportUsingDataFrame(): String`
Custom aggregations and post-processing of metrics data using Spark DataFrame operations. Use `report()` instead.

#### `runAndMeasure[T](f: => T): T`
Runs and measures the metrics for Spark execution, similar to `spark.time()`.

#### `saveData(df: DataFrame, fileName: String, fileFormat: String = "json", saveMode: String = "default"): Unit`
Saves the collected data. Expects small amounts of data, so collapsing to one partition seems acceptable.

#### `sendReportPrometheus(serverIPnPort: String, metricsJob: String, labelName: String = sparkSession.sparkContext.appName, labelValue: String = sparkSession.sparkContext.applicationId): Unit`
Sends the report to a Prometheus endpoint.
```

--------------------------------

### Scala CLI: Run and Measure Spark SQL Query

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

This Scala snippet demonstrates how to initialize and use StageMetrics in a Scala environment to measure the performance of a Spark SQL query. Ensure the sparkmeasure package is included.

```scala
spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28

val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show())
```

--------------------------------

### PushGateway Client Implementation

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Prometheus.md

Illustrates the basic structure of the PushGateway client class used by sparkMeasure. It includes the constructor and methods for validating metric/label names and posting metrics to Prometheus.

```scala
case class PushGateway(serverIPnPort: String, metricsJob: String)
   * Http Client: send metrics to prometheus pushgateway

Methods:
   * def validateLabel(name: String): String -> validates label name, replace not valid symbols
   * def validateMetric(name: String): String -> validates metric name, replace not valid symbols
   * def post(metrics: String, metricsType: String, labelName: String, labelValue: String): Unit -> post metrics to prometheus.
```

--------------------------------

### Run and Measure Spark SQL with TaskMetrics

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Python_shell_and_Jupyter.md

Use the runandmeasure method with TaskMetrics for a shortcut to execute and measure a Spark SQL query at the task level. The spark object must be initialized.

```python
from sparkmeasure import TaskMetrics
taskmetrics = TaskMetrics(spark)
taskmetrics.runandmeasure(globals(),
'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()')
```

--------------------------------

### Define Metrics File Path and Time Window

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkTaskMetricsAnalysisExample.ipynb

Sets the file path for storing metrics and defines the time window for analysis. Use taskMetrics.beginSnapshot and taskMetrics.endSnapshot for taskMetrics.runAndMeasure, or set to 0 if details are unavailable.

```python
# This is the file path and name where the metrics are stored
metrics_filename = "<path>/myPerfTaskMetrics1"

# This defines the time window for analysis
# when using metrics coming from taskMetrics.runAndMeasure, 
# get the info from: taskMetrics.beginSnapshot and taskMetrics.endSnapshot
# if you don't have the details, set begin_time and end_time to 0
begin_time = 1490554321913
end_time = 1490554663808
```

--------------------------------

### Configure Kafka Topic for KafkaSinkV2

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_KafkaSink.md

Specify the Kafka topic for KafkaSinkV2. This is a required configuration parameter, similar to KafkaSink.

```bash
--conf spark.sparkmeasure.kafkaTopic=sparkmeasure-metrics
```

--------------------------------

### Run and Measure Spark SQL with StageMetrics

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Python_shell_and_Jupyter.md

Use the runandmeasure method for a shortcut to execute and measure a Spark SQL query with StageMetrics. The spark object must be initialized.

```python
from sparkmeasure import StageMetrics
stagemetrics = StageMetrics(spark)

stagemetrics.runandmeasure(globals(),
'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()')

stageMetrics.print_memory_report()
```

--------------------------------

### Initialize Plotting Libraries

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkTaskMetricsAnalysisExample.ipynb

Initializes libraries for plotting, including matplotlib.pyplot and seaborn. Ensures plots are displayed inline within the notebook.

```python
# Initialize libraries used later for plotting

import matplotlib.pyplot as plt
import seaborn as sns; sns.set() # cosmetics
%matplotlib inline
```

--------------------------------

### Add SparkMeasure via --packages

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Preferred method to include SparkMeasure and its dependencies by specifying the package during Spark runtime startup.

```bash
--packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
```

--------------------------------

### Configure PushGateway Sink Parameters

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_PrometheusPushgatewaySink.md

Set the Prometheus Pushgateway server address and optional job name and timeouts using Spark configuration parameters.

```bash
--conf spark.sparkmeasure.pushgateway=SERVER:PORT
```

```bash
--conf spark.sparkmeasure.pushgateway.jobname=JOBNAME
```

```bash
--conf spark.sparkmeasure.pushgateway.http.connection.timeout=TIME_IN_MS
```

```bash
--conf spark.sparkmeasure.pushgateway.http.read.timeout=TIME_IN_MS
```

--------------------------------

### Configure Flight Recorder for Task-Level Metrics

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_FileSink.md

Use this to record metrics at the task execution level. Change FlightRecorderStageMetrics to FlightRecorderTaskMetrics for task-level granularity.

```bash
--packages ch.cern.sparkmeasure:spark-measure_2.13:0.28
--conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderTaskMetrics
```

--------------------------------

### Flight Recorder Configuration Parameters

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_FileSink.md

Parameters to configure the output format, filename, and printing to stdout for flight recorder mode.

```bash
--conf spark.sparkmeasure.outputFormat=<format>
    Note: valid values: json,json_to_hadoop,java the default is "json"
--conf spark.sparkmeasure.outputFilename=<output file>
    Note: default = "/tmp/stageMetrics_flightRecorder"
--conf spark.sparkmeasure.printToStdout=<true|false>
    Note: default is false. Set this to true to print JSON serialized metrics for debug purposes.
```

--------------------------------

### Enable Flight Recorder Print to Stdout

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Set this configuration to true to print JSON serialized metrics to standard output for debugging purposes. The default is false.

```bash
--conf spark.sparkmeasure.printToStdout=<true|false>
```

--------------------------------

### Instantiate TaskMetrics Class

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkTaskMetricsAnalysisExample.ipynb

Initializes the TaskMetrics class for measuring task metrics using custom listeners.

```Scala
val taskMetrics = new ch.cern.sparkmeasure.TaskMetrics(spark)
```

--------------------------------

### KafkaSink Configuration

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Configure KafkaSink to send Spark metrics to Kafka. Specify the Kafka broker endpoint and topic.

```bash
--conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSink
--conf spark.sparkmeasure.kafkaBroker=kafka.your-site.com:9092
--conf spark.sparkmeasure.kafkaTopic=sparkmeasure-stageinfo
```

--------------------------------

### Prepare Data for Heatmap Visualization

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkTaskMetricsAnalysisExample.ipynb

Aggregates the number of concurrent running tasks per host and time interval from the 'ConcurrentRunningTasks' view. The data is then pivoted to create a matrix suitable for heatmap generation.

```Python
# load into Pandas the values of number of concurrent running tasks per host and time sample
# see also the heatmap visualization in the next cell

plot_heatmap_running_tasks_per_host = spark.sql("""
select time, host, count(*) num_running_tasks 
from ConcurrentRunningTasks 
group by time, host
order by time, host
""").toPandas()

pivoted_heatmapPandas = plot_heatmap_running_tasks_per_host.pivot(index='time', columns='host', values='num_running_tasks')
```

--------------------------------

### Attach FlightRecorderTaskMetrics Listener

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Use this configuration to attach the FlightRecorderTaskMetrics listener for task-level metrics collection. Be mindful of potential large data volumes.

```bash
--conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderTaskMetrics
```

--------------------------------

### Enable Stage Metrics in InfluxDB Sink

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_InfluxDBSink.md

Set this configuration to true to populate stage_metrics data when using the InfluxDB Sink.

```bash
--conf spark.sparkmeasure.influxdbStagemetrics=true
```

--------------------------------

### Configure Flight Recorder Output Format

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Set the output format for metrics collected in flight recorder mode. Supported formats include json, json_to_hadoop, and java.

```bash
--conf spark.sparkmeasure.outputFormat=<format>
```

--------------------------------

### Add SparkMeasure via --jars (Local Path)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Alternative method to add SparkMeasure by providing the path to the downloaded JAR file for the driver's classpath.

```bash
--jars /path/to/spark-measure_2.13-0.28.jar
```

--------------------------------

### Visualize Average and Maximum Task Duration by Host

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkTaskMetricsAnalysisExample.ipynb

Generates a bar chart showing the average and maximum task durations for each host. This visualization aids in quickly comparing task execution times across different servers.

```python
spark.sql("""
select host, avg(duration) avg_duration, max(duration) max_duration
from PerfTaskMetrics 
group by host""").toPandas().plot(x='host', kind='bar', figsize=(12, 8))
```

--------------------------------

### List HDFS output files

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_FileSink.md

Lists files matching a pattern in HDFS. Useful for verifying output.

```bash
hdfs dfs -ls <path>/myoutput_*.json
```

--------------------------------

### Configure Kafka Broker Endpoint

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_KafkaSink.md

Set the Kafka broker endpoint URL for the KafkaSink. This is a required configuration parameter.

```bash
--conf spark.sparkmeasure.kafkaBroker=kafka.your-site.com:9092
```

--------------------------------

### Run SparkMeasure using Maven Central Package

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Instrument_Scala_code.md

Instructions for running SparkMeasure with a version packaged on Maven Central using spark-submit.

```bash
bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28

// or just download and use the jar (it is only needed in the driver) as in:
bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.28.jar ...

```

--------------------------------

### Utils: Configuration Parsing

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Functions for parsing various configuration parameters from a SparkConf object, including metrics format, filenames, and output destinations.

```scala
// Handle metrics format parameter
def parseMetricsFormat(conf: SparkConf, logger: Logger, defaultFormat:String) : String
def parseMetricsFilename(conf: SparkConf, logger: Logger, defaultFileName:String) : String 

// Parameter parsing
def parsePrintToStdout(conf: SparkConf, logger: Logger, defaultVal:Boolean) : Boolean
def parseMetricsFilename(conf: SparkConf, logger: Logger, defaultFileName:String) : String 
def parseInfluxDBURL(conf: SparkConf, logger: Logger) : String
def parseInfluxDBCredentials(conf: SparkConf, logger: Logger) : (String,String)
def parseInfluxDBName(conf: SparkConf, logger: Logger) : String
def parseInfluxDBStagemetrics(conf: SparkConf, logger: Logger) : Boolean
def parseKafkaConfig(conf: SparkConf, logger: Logger) : (String,String)
def parsePushGatewayConfig(conf: SparkConf, logger: Logger): (String, String)
```

--------------------------------

### Run and Measure Custom Workload

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkTaskMetricsAnalysisExample.ipynb

Executes a custom SQL workload and measures its performance using TaskMetrics.

```Scala
taskMetrics.runAndMeasure(sql(
"select a.bucket, sum(a.val2) tot from t1 a, t1 b where a.bucket=b.bucket and a.val1+b.val1<1000 group by a.bucket order by a.bucket").show)
```

--------------------------------

### Enable InfluxDB Stage Metrics

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Set this to true to collect stage metrics in addition to task metrics when using InfluxDBSink. The default is false.

```bash
spark.sparkmeasure.influxdbStagemetrics, (boolean, default is false)
```

--------------------------------

### Test Spark Session

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkMeasure_Jupyter_Colab_Example.ipynb

Verifies that the Spark session is working correctly by executing a simple SQL query and displaying the results. This confirms the Spark environment is ready for further operations.

```python
# Test that Spark is working OK
spark.sql("select 1 as id, 'Hello world!' as Greeting").show()
```

--------------------------------

### Utils: Metric Report Data Structures

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Functions to initialize zero-value data structures for computing stage and task metrics reports.

```scala
// Return the data structure use to compute metrics reports
def zeroMetricsStage() : LinkedHashMap[String, Long]
def zeroMetricsTask() : LinkedHashMap[String, Long]
```

--------------------------------

### Run sparkMeasure with Maven Central Package

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Instrument_Python_code.md

Instructions for running a Python application with sparkMeasure using a packaged version from Maven Central. This involves specifying the package during spark-submit.

```bash
bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28 your_python_code.py

// alternative: just download and use the jar (it is only needed in the driver) as in:
bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.28.jar ...
```

--------------------------------

### KafkaSinkV2 Configuration

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Configure KafkaSinkV2 for enhanced Spark metrics with application-level aggregated metrics and custom labels. Ensures backward compatibility.

```bash
--conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSinkV2
--conf spark.sparkmeasure.kafkaBroker=kafka.your-site.com:9092
--conf spark.sparkmeasure.kafkaTopic=spark-metrics
```

--------------------------------

### Utils: Formatting Utilities

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Helper functions for pretty-printing duration and byte sizes in SparkMeasure reports.

```scala
// Boilerplate code for pretty printing
def formatDuration(milliseconds: Long): String 
def formatBytes(bytes: Long): String
def prettyPrintValues(metric: String, value: Long): String
```

--------------------------------

### Attach KafkaSink and KafkaSinkExtended Listeners

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_KafkaSink.md

As an alternative, you can attach both KafkaSink and KafkaSinkExtended listeners simultaneously. This provides both stage-level and task-level metrics.

```bash
--conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSink,ch.cern.sparkmeasure.KafkaSinkExtended
```

--------------------------------

### StageMetrics Class and Basic Usage

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Details on how to instantiate and use the StageMetrics class for collecting and measuring Spark SQL query execution.

```APIDOC
## StageMetrics - Scala

### Description
Collects stage-level metrics with Stage granularity and provides aggregation and reporting functions for the end-user.

### Example Usage
```scala
val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)
```

### Configuration

- **spark.sparkmeasure.stageinfo.verbose** (boolean, default `true`)
  Controls printing the stage info report and collecting executor memory metrics.

- **spark.sparkmeasure.stageinfo.executormetrics** (string, default `"JVMHeapMemory,OnHeapExecutionMemory"`)
  The list of executor metrics that are captured. See [Spark Documentation](https://spark.apache.org/docs/latest/monitoring.html#executor-metrics) for available metrics.
```

--------------------------------

### Run Scala Unit Tests

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Unit-tests.md

Execute the built-in unit tests for the core sparkMeasure Scala codebase using sbt.

```bash
sbt test
```

--------------------------------

### Query 'queries_started' Measurement

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Flight_recorder_mode_InfluxDBSink.md

Retrieves all data points from the 'queries_started' measurement. This query shows details about query execution, including application ID, description, and query ID.

```influxql
select * from queries_started
```

--------------------------------

### Display Available PerfTaskMetrics Schema

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkTaskMetricsAnalysisExample.ipynb

Shows the schema of the PerfTaskMetrics table, including column names, data types, and comments. This is useful for understanding the available metrics for analysis.

```python
spark.sql("desc PerfTaskMetrics").show(50,False)
```

--------------------------------

### Python TaskMetrics Methods

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Provides method signatures for the Python TaskMetrics class, including begin, end, report, and runandmeasure.

```python
# Marks the beginning of data collection
def begin(self):

```

```python
# Marks the end of data collection
def end(self):

```

```python
# Custom aggregations and post-processing of metrics data
def report(self):

```

```python
# Returns a dictionary of the aggregated task metrics data.
def aggregate_taskmetrics(self):

```

```python
# Runs report and prints it
def print_report(self):

```

```python
# Runs the code and measures the metrics for Spark execution
def runandmeasure(self, env, codetorun):

```

```python
# Creates a Spark Dataframe from the collected task metrics data.
def create_taskmetrics_DF(self, viewname="PerfTaskMetrics"):

```

```python
# Creates a Spark Dataframe from the aggregated task metrics data.
def aggregate_taskmetrics_DF(self, viewname="PerfTaskMetrics"):

```

```python
# Saves the collected task metrics data to a file.
def save_data(self, df, filepathandname, fileformat):

```

```python
# Removes the listener from the SparkContext
def remove_listener(self):

```

--------------------------------

### Add SparkMeasure via --jars (URL)

Source: https://github.com/lucacanali/sparkmeasure/blob/master/README.md

Alternative method to add SparkMeasure by providing a URL to the JAR file for the driver's classpath.

```bash
--jars https://github.com/LucaCanali/SparkMeasure/releases/download/v0.28/spark-measure_2.13-0.28.jar
```

--------------------------------

### Spark-shell Configuration for Workload

Source: https://github.com/lucacanali/sparkmeasure/blob/master/examples/SparkTaskMetricsAnalysisExample.ipynb

Configures the spark-shell environment with specified executors, cores, and memory, including the sparkMeasure JAR.

```Shell
spark-shell --num-executors 14 --executor-cores 4 --driver-memory 2g  --executor-memory 2g --jars sparkMeasure/target/scala-2.11/spark-measure_2.11-0.1-SNAPSHOT.jar

val testNumRows = 1e7.toLong
sql(s"select id from range($testNumRows)").createOrReplaceTempView("t0")
sql("select id, floor(200*rand()) bucket, floor(1000*rand()) val1, floor(10*rand()) val2 from t0").cache().createOrReplaceTempView("t1")
sql("select count(*) from t1").show
```

--------------------------------

### Configure Flight Recorder Output Filename

Source: https://github.com/lucacanali/sparkmeasure/blob/master/docs/Reference_SparkMeasure_API_and_Configs.md

Specify the output filename for metrics collected in flight recorder mode. The default is /tmp/stageMetrics_flightRecorder.

```bash
--conf spark.sparkmeasure.outputFilename=<output file>
```