SparkMonitor

https://github.com/swan-cern/sparkmonitor
Admin
SparkMonitor is a Jupyter extension for monitoring Apache Spark jobs launched from notebooks. It...
Tokens:1,970
Snippets:25
Trust Score:8.9
License:Apache-2.0
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install SparkMonitor for JupyterLab

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Install SparkMonitor along with PySpark and JupyterLab using pip.

```bash
pip install sparkmonitor pyspark jupyterlab
```

--------------------------------

### Install SparkMonitor in Editable Mode

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Install the SparkMonitor package in editable mode for local development. This command is executed in the terminal.

```bash
# Install the package in editable mode
pip install -e .
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Before installing SparkMonitor, create and activate a Python virtual environment to manage dependencies.

```bash
python -m venv venv
source venv/bin/activate
```

--------------------------------

### Inspect Installed Package Location

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Use this Python snippet to find the installation path of the sparkmonitor package. This is useful for locating the listener JAR manually.

```python
import sparkmonitor

print(sparkmonitor.__path__)
```

--------------------------------

### Build Frontend Assets

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Build the frontend assets for SparkMonitor. The specific action can be found in the package.json file. This command is executed in the terminal.

```bash
# Build the frontend (see package.json for available scripts)
yarn run build:<action>
```

--------------------------------

### Enable SparkMonitor IPython Kernel Extension

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Enable the SparkMonitor extension for the IPython kernel. This configuration is profile-specific and needs to be done only once.

```bash
ipython profile create
echo "c.InteractiveShellApp.extensions.append('sparkmonitor.kernelextension')" >> "$(ipython profile locate default)/ipython_kernel_config.py"
```

--------------------------------

### Build Spark 3 Listener JARs

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Build the Spark 3 listener JARs for Scala 2.12 and 2.13. Navigate to the correct directory and run the sbt command. This command is executed in the terminal.

```bash
cd scalalistener_spark3 # Spark 3 / Scala 2.12 and 2.13
sbt +package
```

--------------------------------

### Watch Frontend Files

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Continuously watch frontend files for changes during development. This command is executed in the terminal.

```bash
# Watch frontend files for changes
yarn run watch
```

--------------------------------

### Build Spark 4 Listener JAR

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Build the Spark 4 listener JAR for Scala 2.13. Navigate to the correct directory and run the sbt command. This command is executed in the terminal.

```bash
cd ../scalalistener_spark4 # Spark 4 / Scala 2.13
sbt package
```

--------------------------------

### Manually Specify Listener JAR Path

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Use this method when the exact path to the SparkMonitor listener JAR is known. Ensure the path is correct for your environment.

```python
from pyspark.sql import SparkSession

spark = (
    SparkSession.builder.config(
        "spark.extraListeners",
        "sparkmonitor.listener.JupyterSparkMonitorListener",
    )
    .config(
        "spark.driver.extraClassPath",
        # Put the path to the matching SparkMonitor listener JAR here.
        "venv/lib/python3.13/site-packages/sparkmonitor/listener_spark4_2.13.jar",
    )
    .getOrCreate()
)
```

--------------------------------

### Develop JupyterLab Extension

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

Link the JupyterLab extension into your local Jupyter environment for development. This command is executed in the terminal.

```bash
# Link the JupyterLab extension into your local Jupyter environment
jupyter labextension develop --overwrite .
```

--------------------------------

### Dynamically Resolve Listener JAR Path

Source: https://github.com/swan-cern/sparkmonitor/blob/master/README.md

This approach automatically detects the correct listener JAR path based on Spark and Scala versions. It checks SPARK_HOME and the pyspark package layout.

```python
import os
from pathlib import Path

import pyspark
import sparkmonitor
from pyspark.sql import SparkSession


def iter_spark_jar_dirs() -> list[Path]:
    candidates = []

    spark_home = os.environ.get("SPARK_HOME")
    if spark_home:
        candidates.append(Path(spark_home) / "jars")

    candidates.append(Path(pyspark.__file__).resolve().parent / "jars")
    return [path for path in candidates if path.exists()]


def resolve_listener_jar(sparkmonitor_dir: Path) -> Path:
    for jars_dir in iter_spark_jar_dirs():
        for jar in jars_dir.glob("spark-core_*.jar"):
            # spark-core_2.13-3.5.8.jar => scala=2.13, spark_major=3
            scala_ver, spark_ver = jar.name.split("_")[1].split("-")[:2]
            spark_major = spark_ver.split(".")[0]
            if spark_major == "3" and scala_ver == "2.12":
                return sparkmonitor_dir / "listener_spark3_2.12.jar"
            if spark_major == "3" and scala_ver == "2.13":
                return sparkmonitor_dir / "listener_spark3_2.13.jar"
            if spark_major == "4" and scala_ver == "2.13":
                return sparkmonitor_dir / "listener_spark4_2.13.jar"

    raise RuntimeError(
        "Could not detect Spark/Scala version from SPARK_HOME or the pyspark installation"
    )


sparkmonitor_dir = Path(sparkmonitor.__file__).resolve().parent
listener_jar = resolve_listener_jar(sparkmonitor_dir)

spark = (
    SparkSession.builder.config(
        "spark.extraListeners",
        "sparkmonitor.listener.JupyterSparkMonitorListener",
    )
    .config("spark.driver.extraClassPath", str(listener_jar))
    .getOrCreate()
)
```
Tokens:
Raw