### Starting Hadoop Daemons - Batch

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/HadoopWindowsUserSetup.md

Navigates to the Hadoop sbin directory and executes the start-all.cmd script to launch all Hadoop daemons (NameNode, DataNode, ResourceManager, NodeManager).

```Batch
cd sbin
start-all.cmd
```

--------------------------------

### Configuring Hadoop HDFS Site (hdfs-site.xml) - XML

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/HadoopWindowsUserSetup.md

This configuration for hdfs-site.xml defines the replication factor for HDFS (set to 1 for a single-node setup) and specifies the local file system directories for storing NameNode metadata and DataNode blocks.

```XML
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///C:/hadoop/data/dfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///C:/hadoop/data/dfs/datanode</value>
</property>
```

--------------------------------

### Configuring Hadoop YARN Site (yarn-site.xml) - XML

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/HadoopWindowsUserSetup.md

This configuration for yarn-site.xml sets up the auxiliary services for the YARN NodeManager. 'mapreduce_shuffle' is required for running MapReduce applications on YARN.

```XML
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>Yarn Node Manager Aux Service</description>
</property>
<configuration>
```

--------------------------------

### Configuring Hadoop Core Site (core-site.xml) - XML

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/HadoopWindowsUserSetup.md

This snippet shows the configuration for the Hadoop core-site.xml file. It sets the default file system URI to hdfs://localhost:9000, which is the address for the NameNode.

```XML
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>
```

--------------------------------

### Configuring Hadoop MapReduce Site (mapred-site.xml) - XML

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/HadoopWindowsUserSetup.md

This snippet configures the mapred-site.xml file to specify the framework used for MapReduce jobs. Setting the value to 'yarn' indicates that MapReduce will run on top of YARN.

```XML
<configuration>
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
</configuration>
```

--------------------------------

### Example Output of Docker Spark-Submit

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/spark-cluster/README.md

Shows the expected console output after successfully submitting a Spark application using the docker run command. It includes logs from the RestSubmissionClient detailing the submission process, status updates, and the final JSON response from the server.

```bash
Running Spark using the REST application submission protocol.
2018-09-23 15:17:52 INFO  RestSubmissionClient:54 - Submitting a request to launch an application in spark://spark-master:6066.
2018-09-23 15:17:53 INFO  RestSubmissionClient:54 - Submission successfully created as driver-20180923151753-0000. Polling submission state...
2018-09-23 15:17:53 INFO  RestSubmissionClient:54 - Submitting a request for the status of submission driver-20180923151753-0000 in spark://spark-master:6066.
2018-09-23 15:17:53 INFO  RestSubmissionClient:54 - State of driver driver-20180923151753-0000 is now RUNNING.
2018-09-23 15:17:53 INFO  RestSubmissionClient:54 - Driver is running on worker worker-20180923151711-10.5.0.4-45381 at 10.5.0.4:45381.
2018-09-23 15:17:53 INFO  RestSubmissionClient:54 - Server responded with CreateSubmissionResponse:
{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20180923151753-0000",
  "serverSparkVersion" : "3.5.0",
  "submissionId" : "driver-20180923151753-0000",
  "success" : true
}

```

--------------------------------

### Running Spark Cluster with Docker Compose (Shell)

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/spark-cluster/README.md

Starts the Spark standalone cluster defined in the `docker-compose.yml` file. The `--scale spark-worker=3` option specifies that three worker containers should be launched alongside the master.

```sh
docker-compose up --scale spark-worker=3
```

--------------------------------

### Building Spark Docker Images (Shell)

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/spark-cluster/README.md

Executes the `build-images.sh` script to create the necessary Docker images for the Spark cluster components (base, master, worker, submit). This script must be made executable before running.

```sh
chmod +x build-images.sh
./build-images.sh
```

--------------------------------

### Copying Spark Application Files to Host Volumes (Bash)

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/spark-cluster/README.md

Copies the application JAR, configuration directory, and input data file from local host paths to the host directories (`/mnt/spark-apps`, `/mnt/spark-files`) that are mounted as volumes in the Docker containers. This makes the files accessible to the Spark master and workers.

```bash
cp /home/workspace/crimes-app/build/libs/crimes-app.jar /mnt/spark-apps
cp -r /home/workspace/crimes-app/config /mnt/spark-apps
cp /home/Crimes_-_2001_to_present.csv /mnt/spark-files
```

--------------------------------

### Submitting Spark Application using Docker Spark-Submit

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/spark-cluster/README.md

Explains how to use a spark-submit Docker image to submit a Spark application JAR to a Spark cluster running in Docker. It sets environment variables for the JAR location, main class, and extra Spark submit arguments, then runs the docker container, connecting it to the cluster network and mounting the application directory.

```bash
#Creating some variables to make the docker run command more readable
#App jar environment used by the spark-submit image
SPARK_APPLICATION_JAR_LOCATION="/opt/spark-apps/crimes-app.jar"
#App main class environment used by the spark-submit image
SPARK_APPLICATION_MAIN_CLASS="org.mvb.applications.CrimesApp"
#Extra submit args used by the spark-submit image
SPARK_SUBMIT_ARGS="--conf spark.executor.extraJavaOptions='-Dconfig-path=/opt/spark-apps/dev/config.conf'"

#We have to use the same network as the spark cluster(internally the image resolves spark master as spark://spark-master:7077)
docker run --network docker-spark-cluster_spark-network \
-v /mnt/spark-apps:/opt/spark-apps \
--env SPARK_APPLICATION_JAR_LOCATION=$SPARK_APPLICATION_JAR_LOCATION \
--env SPARK_APPLICATION_MAIN_CLASS=$SPARK_APPLICATION_MAIN_CLASS \
spark-submit:3.5.0

```

--------------------------------

### Validating Application Files on Spark Worker Containers (Shell)

Source: https://github.com/rockthejvm/spark-performance-tuning/blob/master/spark-cluster/README.md

Uses the `docker exec` command to run `ls -l` inside each Spark worker container (`spark-worker-1`, `spark-worker-2`, `spark-worker-3`). This verifies that the application JAR and data files were successfully copied to the mounted volumes (`/opt/spark-apps`, `/opt/spark-data`) within the containers.

```sh
docker exec -ti spark-worker-1 ls -l /opt/spark-apps
docker exec -ti spark-worker-1 ls -l /opt/spark-data

docker exec -ti spark-worker-2 ls -l /opt/spark-apps
docker exec -ti spark-worker-2 ls -l /opt/spark-data

docker exec -ti spark-worker-3 ls -l /opt/spark-apps
docker exec -ti spark-worker-3 ls -l /opt/spark-data
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.