### Start development server and proxy

Source: https://github.com/dataflint/spark/blob/main/CONTRIBUTING.md

Starts the development server and proxy for live frontend development.

```bash
cd spark-ui
npm run start
npm run proxy
```

--------------------------------

### Start with Docker Compose

Source: https://github.com/dataflint/spark/blob/main/docker/README.md

Sets the SPARK_HISTORY_DIR environment variable to the host event logs directory and starts the service with docker-compose up -d.

```bash
# Set your event logs directory
export SPARK_HISTORY_DIR=/path/to/spark-events

# Start the service
docker-compose up -d
```

--------------------------------

### Install Java 17 with Homebrew

Source: https://github.com/dataflint/spark/blob/main/spark-plugin/pyspark-testing/README.md

Commands to install Java 17 and set JAVA_HOME on macOS with Homebrew.

```bash
# macOS with Homebrew
brew install openjdk@17
export JAVA_HOME=/opt/homebrew/opt/openjdk@17
export PATH=$JAVA_HOME/bin:$PATH
```

--------------------------------

### Install UI dependencies

Source: https://github.com/dataflint/spark/blob/main/CONTRIBUTING.md

Installs the UI dependencies for the spark-ui project.

```bash
cd spark-ui
npm install
```

--------------------------------

### Run dataflint_pyspark_example.py with run-with-spark.sh

Source: https://github.com/dataflint/spark/blob/main/spark-plugin/pyspark-testing/README.md

Commands to run the simple example script with different Spark versions. Requires Java 17+ for Spark 4.0.2.

```bash
cd utils
./run-with-spark.sh 3.3.4 ../pyspark-testing/dataflint_pyspark_example.py
./run-with-spark.sh 3.5.1 ../pyspark-testing/dataflint_pyspark_example.py
./run-with-spark.sh 4.0.2 ../pyspark-testing/dataflint_pyspark_example.py  # Requires Java 17+
```

--------------------------------

### Install Local CORS Proxy (LCP)

Source: https://github.com/dataflint/spark/blob/main/CONTRIBUTING.md

Optionally installs the Local CORS Proxy (LCP) for local development.

```bash
brew install lcp
```

--------------------------------

### Build with custom Spark and Scala versions

Source: https://github.com/dataflint/spark/blob/main/docker/README.md

Shows how to build the Docker image with different Spark and Scala versions using build arguments. Includes examples for Spark 3.5.3, 3.4.1 with Scala 2.13, and Spark 4.0.0 (automatically uses Scala 2.13).

```bash
# Spark 3.5.3 from Maven
docker build -t dataflint-hs:3.5.3 --build-arg SPARK_VERSION=3.5.3 .

# Spark 3.4.1 with Scala 2.13 from Maven
docker build -t dataflint-hs:3.4.1 \
  --build-arg SPARK_VERSION=3.4.1 \
  --build-arg SCALA_VERSION=2.13 .

# Spark 4.0.0 from Maven (automatically uses Scala 2.13)
docker build -t dataflint-hs:4.0.0 --build-arg SPARK_VERSION=4.0.0 .

# Local JAR for Spark 3.x
./build-jars.sh
docker build -t dataflint-hs:local --build-arg USE_LOCAL_JAR=true .

# Local JAR for Spark 4.x
./build-jars.sh
docker build -t dataflint-hs:4.0.0-local \
  --build-arg USE_LOCAL_JAR=true \
  --build-arg SPARK_VERSION=4.0.0 .
```

--------------------------------

### Build plugin JARs with sbt

Source: https://github.com/dataflint/spark/blob/main/spark-plugin/pyspark-testing/README.md

Commands to build the required plugin JARs for Spark 3.x and Spark 4.x before running the examples.

```bash
# For Spark 3.x
sbt pluginspark3/assembly

# For Spark 4.x
sbt pluginspark4/assembly
```

--------------------------------

### Install DataFlint OSS via spark-submit for Spark 3.X

Source: https://github.com/dataflint/spark/blob/main/README.md

Install DataFlint OSS with no code change as a spark ivy package for Spark 3.X by adding these two lines to your spark-submit command.

```bash
spark-submit
--packages io.dataflint:spark_2.12:0.9.9 \
--conf spark.plugins=io.dataflint.spark.SparkDataflintPlugin \
...
```

--------------------------------

### Example output from Spark UI

Source: https://github.com/dataflint/spark/blob/main/spark-plugin/pyspark-testing/README.md

Shows the expected output in the Spark UI when DataFlint instrumentation is working, including the duration metric.

```text
DataFlintMapInPandasExec
  duration total (min, med, max (stageId: taskId))
  2.3 s (581 ms, 581 ms, 582 ms (stage 2.0: task 19))
```

--------------------------------

### Build the UI for the plugin

Source: https://github.com/dataflint/spark/blob/main/CONTRIBUTING.md

Builds the UI for the plugin.

```bash
cd spark-ui
npm run deploy
```

--------------------------------

### Install DataFlint OSS via spark-submit for Spark 4.X

Source: https://github.com/dataflint/spark/blob/main/README.md

Install DataFlint OSS with no code change as a spark ivy package for Spark 4.X by adding these two lines to your spark-submit command.

```bash
spark-submit
--packages io.dataflint:dataflint-spark4_2.13:0.9.9 \
--conf spark.plugins=io.dataflint.spark.SparkDataflintPlugin \
...
```

--------------------------------

### Build and run with local JARs for development

Source: https://github.com/dataflint/spark/blob/main/docker/README.md

Builds JARs locally, then builds the Docker image with the local JAR and runs it. Requires Node.js 20+, Java 8+, and sbt.

```bash
# Step 1: Build the JARs locally
./build-jars.sh

# Step 2: Build Docker image with local JAR
docker build -t dataflint-history-server --build-arg USE_LOCAL_JAR=true .

# Step 3: Run
docker run -d \
  -p 18080:18080 \
  -v /path/to/spark-events:/spark-history:ro \
  dataflint-history-server
```