Splunk Lantern

https://lantern.splunk.com
Admin
Splunk Lantern is a comprehensive guidance platform providing step-by-step instructions and best...
Tokens:288,108
Snippets:2,071
Trust Score:10
Update:5 days ago
Show doc for...
Tokens:
Raw
### Start local documentation web server

Source: https://lantern.splunk.com/Splunk_and_Cisco_Use_Cases/Deploying_and_managing_your_Splunk_POD_environment

Starts a new process to serve install-specific documentation. The example starts the web server on port 8080 (the default). After starting, access it via any worker IP on port 8080.

```bash
${INSTALLED_PATH}/kubernetes-install-standalone –web [–web.port 8080]
```

--------------------------------

### Example command.path file content

Source: https://lantern.splunk.com/Manage_Performance_and_Health/Node.js_deprecation_FAQ

Example content for a command.path file that specifies the location of a wrapper script for custom search commands.

```plaintext
$SPLUNK_HOME/etc/apps/<app_name>/linux_x86_64/bin/command.sh
```

--------------------------------

### Install Certbot with pip3

Source: https://lantern.splunk.com/Manage_Performance_and_Health/Renewing_a_certificate_on_a_new_Splunk_Enterprise_installation

Install Certbot if it is not already installed on the server.

```bash
pip3 install certbot
```

--------------------------------

### Shell script for starting Nmon with FIFO and version handling

Source: https://lantern.splunk.com/Manage_Performance_and_Health/Troubleshooting_slow_Splunk_platform_restarts_when_running_under_systemd_on_Linux

This snippet is a shell script that starts Nmon, handling FIFO configurations and Nmon version checks. It sets NMON_START and NMON_SNAP variables, checks if Nmon version is 16 or higher to enable disk group statistics, and manages exceptions for SLES 11. It starts Nmon, stores the PID, and handles errors or fallback if disk group file is missing.

```shell
     # fifo_started variable is exported by the function start_fifo_reader
     case $fifo_started in
     "fifo1")
         case $nmon_external_generation in
         1)
             # nmon_external
             create_nmon_external
             NMON_START="${APP_VAR}/bin/nmon_external_cmd/nmon_external_start_fifo1.sh"
             export NMON_START
             NMON_SNAP="${APP_VAR}/bin/nmon_external_cmd/nmon_external_snap_fifo1.sh"
             export NMON_SNAP
         ;;
         esac
         nmon_command=${nmon_command_fifo1} ;;
     "fifo2")
         case $nmon_external_generation in
         1)
             # nmon_external
             create_nmon_external
             NMON_START="${APP_VAR}/bin/nmon_external_cmd/nmon_external_start_fifo2.sh"
             export NMON_START
             NMON_SNAP="${APP_VAR}/bin/nmon_external_cmd/nmon_external_snap_fifo2.sh"
             export NMON_SNAP
         ;;
         esac
         nmon_command=${nmon_command_fifo2} ;;
     esac

            # Retrieve the nmon Linux version
            # Nmon 16x or superior is required to run disk group statistics

        NMON_VERSION=`$NMON -h | sed -n 's/.*[v|V]ersion[^0-9]*\([0-9][0-9]*\).*$/\1/p' | head -1`

        # Assume we can fail
        case $NMON_VERSION in
        "")
            # Set a default to 14 in case of identification failure
            NMON_VERSION="14" ;;
        esac

        if [ $NMON_VERSION -ge "16" ]; then

            # Activation of Linux disks extended stats generate a message in stdout
            # We don't want this as we need to retrieve the pid from nmon output
            # However, we also want to analyse the return code, so we can't filter out in only one operation

            # Manage exceptions - these systems will not generate data but will start and output an lsblk message
            linux_distrib="${linux_vendor}_${linux_mainversion}"
            disk_group_option_exception="sles_11"
            echo $disk_group_option_exception | grep -o $linux_distrib >/dev/null
            if [ $? -eq 0 ]; then
                     nmon_command=`echo ${nmon_command} | sed "s/-g ${Linux_disk_dg_group} -D//g"`
            fi

            echo "`log_date`, ${HOST} INFO: starting nmon : ${nmon_command} in ${NMON_EXTERNAL_DIR}"
            ${nmon_command} > ${APP_VAR}/nmon_output.txt
            #echo "${nmon_command}" >> /tmp/nmon_command.txt

            if [ $? -ne 0 ]; then
                echo "`log_date`, ${HOST} ERROR, nmon binary returned a non 0 code while trying to start, please verify error traces in splunkd log (missing shared libraries?)"
            fi

            # Store the PID file (very last line of nmon output)
            if [ -f ${APP_VAR}/nmon_output.txt ]; then
                awk 'END{print}' ${APP_VAR}/nmon_output.txt > ${PIDFILE}
            fi

            # old nmon versions might not be compatible with disks extended stats, or the group file does not exist
            # In such a case, echo a WARN, remove the option and last chance start
            if grep 'opening disk group file' ${APP_VAR}/nmon_output.txt >/dev/null; then

                echo "`log_date`, ${HOST} WARN, nmon disks extended statistics cannot be collected, either this nmon version is not compatible or the disk group file does not exist, see ${APP_VAR}/nmon_output.txt"

                nmon_command=`echo ${nmon_command} | sed "s/-g ${Linux_disk_dg_group} -D//g"`
                echo "`log_date`, ${HOST} INFO: starting nmon : ${nmon_command} in ${NMON_EXTERNAL_DIR}"
                ${nmon_command} &> ${PIDFILE}

                if [ $? -ne 0 ]; then
                    echo "`log_date`, ${HOST} ERROR, nmon binary returned a non 0 code while trying to start, please verify error traces in splunkd log (missing shared libraries?)"
                fi

            fi

        else

            # This version is not compatible with the auto group disk
            nmon_command=`echo ${nmon_command} | sed "s/-g ${Linux_disk_dg_group} -D//g"`
            echo "`log_date`, ${HOST} INFO: starting nmon : ${nmon_command} in ${NMON_EXTERNAL_DIR}"
            ${nmon_command} > ${PIDFILE}

            if [ $? -ne 0 ]; then
                echo "`log_date`, ${HOST} ERROR, nmon binary returned a non 0 code while trying to start, please verify error traces in splunkd log (missing shared libraries?)"
            fi

        fi

        ;;

        SunOS )

       # global nmon_external
       NMON_EXTERNAL_DIR="${APP_VAR}/var/nmon_repository/${fifo_started}"
       export NMON_EXTERNAL_DIR
       NMON_EXTERNAL_FIFO="${APP_VAR}/var/nmon_repository/${fifo_started}/nmon.fifo"
       export NMON_EXTERNAL_FIFO
       TIMESTAMP=0
       export TIMESTAMP
       NMON_ONE_IN=1
       export NMON_ONE_IN
       unset NMON_END
```

--------------------------------

### AIX nmon startup script with FIFO handling

Source: https://lantern.splunk.com/Manage_Performance_and_Health/Troubleshooting_slow_Splunk_platform_restarts_when_running_under_systemd_on_Linux

This script handles starting nmon on AIX. It sets environment variables, manages two FIFO configurations (fifo1 and fifo2), starts nmon, checks for errors, handles the -y option incompatibility, and stores the PID file.

```shell
case $UNAME in

        AIX )

     # on AIX, prevent error messages linked to /usr/opt/freeware/bin/rpm
     unset LIBPATH

     # global nmon_external
     NMON_EXTERNAL_DIR="${APP_VAR}/var/nmon_repository/${fifo_started}"
     export NMON_EXTERNAL_DIR
     NMON_EXTERNAL_FIFO="${APP_VAR}/var/nmon_repository/${fifo_started}/nmon.fifo"
     export NMON_EXTERNAL_FIFO
     TIMESTAMP=0
     export TIMESTAMP
     NMON_ONE_IN=1
     export NMON_ONE_IN
     unset NMON_END

     # fifo_started variable is exported by the function start_fifo_reader
     case $fifo_started in
     "fifo1")
         case $nmon_external_generation in
         1)
             # nmon_external
             create_nmon_external
             NMON_START="${APP_VAR}/bin/nmon_external_cmd/nmon_external_start_fifo1.sh"
             export NMON_START
             NMON_SNAP="${APP_VAR}/bin/nmon_external_cmd/nmon_external_snap_fifo1.sh"
             export NMON_SNAP
         ;;
         esac

         echo "`log_date`, ${HOST} INFO: starting nmon : ${nmon_command_fifo1} in ${NMON_EXTERNAL_DIR}"
         ${nmon_command_fifo1} 2>&1 > ${APP_VAR}/nmon_output.txt

         if [ $? -ne 0 ]; then
             echo "`log_date`, ${HOST} ERROR, nmon binary returned a non 0 code while trying to start, please verify error traces in splunkd log"
         fi

         # old topas-nmon version might not be compatible with the -y option, let's manage this
         cat ${APP_VAR}/nmon_output.txt | grep -i 'invalid option[^y]*y' >/dev/null
         if [ $? -eq 0 ]; then
             # option -y is not compatible and not mandatory
             echo "`log_date`, ${HOST}, ERROR, This system is running a topas-nmon version that does not support the -y option, you might need to consider an AIX upgrade: `cat ${APP_VAR}/nmon_output.txt`"
             nmon_command_fifo1=`echo ${nmon_command_fifo1} | sed 's/\-yoverwrite=1//g'`
             ${nmon_command_fifo1} 2>&1 > ${APP_VAR}/nmon_output.txt
         fi

         # Store the PID file (very last line of nmon output)
         if [ -f ${APP_VAR}/nmon_output.txt ]; then
             tail -1 ${APP_VAR}/nmon_output.txt > ${PIDFILE}
         fi

     ;;

     "fifo2")
         case $nmon_external_generation in
         1)
             # nmon_external
             create_nmon_external
             NMON_START="${APP_VAR}/bin/nmon_external_cmd/nmon_external_start_fifo2.sh"
             export NMON_START
             NMON_SNAP="${APP_VAR}/bin/nmon_external_cmd/nmon_external_snap_fifo2.sh"
             export NMON_SNAP
         ;;
         esac

         echo "`log_date`, ${HOST} INFO: starting nmon : ${nmon_command_fifo2} in ${NMON_EXTERNAL_DIR}"
         ${nmon_command_fifo2} 2>&1 > ${APP_VAR}/nmon_output.txt

         if [ $? -ne 0 ]; then
             echo "`log_date`, ${HOST} ERROR, nmon binary returned a non 0 code while trying to start, please verify error traces in splunkd log"
         fi

         # old topas-nmon version might not be compatible with the -y option, let's manage this
         cat ${APP_VAR}/nmon_output.txt | grep -i 'invalid option[^y]*y' >/dev/null
         if [ $? -eq 0 ]; then
             # option -y is not compatible and not mandatory
             echo "`log_date`, ${HOST}, ERROR, This system is running a topas-nmon version that does not support the -y option, you might need to consider an AIX upgrade: `cat ${APP_VAR}/nmon_output.txt`"
             nmon_command_fifo2=`echo ${nmon_command_fifo2} | sed 's/\-yoverwrite=1//g'`
             ${nmon_command_fifo2} 2>&1 > ${APP_VAR}/nmon_output.txt
         fi

         # Store the PID file (very last line of nmon output)
         if [ -f ${APP_VAR}/nmon_output.txt ]; then
             tail -1 ${APP_VAR}/nmon_output.txt > ${PIDFILE}
         fi

     ;;

     esac

        ;;

        Linux )

     # global nmon_external
     NMON_EXTERNAL_DIR="${APP_VAR}/var/nmon_repository/${fifo_started}"
     export NMON_EXTERNAL_DIR
     NMON_EXTERNAL_FIFO="${APP_VAR}/var/nmon_repository/${fifo_started}/nmon.fifo"
     export NMON_EXTERNAL_FIFO
     TIMESTAMP=0
     export TIMESTAMP
     NMON_ONE_IN=1
     export NMON_ONE_IN
     unset NMON_END


```

--------------------------------

### Start nmon with FIFO and external generation configuration

Source: https://lantern.splunk.com/Manage_Performance_and_Health/Troubleshooting_slow_Splunk_platform_restarts_when_running_under_systemd_on_Linux

Configures nmon startup based on the fifo_started variable, setting NMON_START, NMON_SNAP, and NMONOUTPUTFILE for fifo1 or fifo2. Also sets NMONNOSAFILE, and optionally enables UARG and VxVM statistics based on Solaris_UARG and Solaris_VxVM. Finally starts nmon in the background.

```bash
       # fifo_started variable is exported by the function start_fifo_reader
       case $fifo_started in
       "fifo1")
           case $nmon_external_generation in
           1)
               # nmon_external
               create_nmon_external
               NMON_START="${APP_VAR}/bin/nmon_external_cmd/nmon_external_start_fifo1.sh"
               export NMON_START
               NMON_SNAP="${APP_VAR}/bin/nmon_external_cmd/nmon_external_snap_fifo1.sh"
               export NMON_SNAP
           ;;
           esac
           NMONOUTPUTFILE="${APP_VAR}/var/nmon_repository/${fifo_started}/nmon.fifo"
           export NMONOUTPUTFILE
           ;;
       "fifo2")
           case $nmon_external_generation in
           1)
               # nmon_external
               create_nmon_external
               NMON_START="${APP_VAR}/bin/nmon_external_cmd/nmon_external_start_fifo2.sh"
               export NMON_START
               NMON_SNAP="${APP_VAR}/bin/nmon_external_cmd/nmon_external_snap_fifo2.sh"
               export NMON_SNAP
           ;;
           esac
           NMONOUTPUTFILE="${APP_VAR}/var/nmon_repository/${fifo_started}/nmon.fifo"
           export NMONOUTPUTFILE
           ;;
       esac

                NMONNOSAFILE=1 # Do not generate useless sa files
                export NMONNOSAFILE

                # Manage UARG activation, default is on (1)
                NMONUARG_VALUE=${Solaris_UARG}
                if [ ! -z ${NMONUARG_VALUE} ]; then

                        if [ ${NMONUARG_VALUE} -eq 1 ]; then
                        NMONUARG=1
                        export NMONUARG
                        fi

                fi

                # Manage VxVM volume statistics activation, default is off (0)
                NMONVXVM_VALUE=${Solaris_VxVM}
                if [ ! -z ${NMONVXVM_VALUE} ]; then

                        if [ ${NMONVXVM_VALUE} -eq 1 ]; then
                        NMONVXVM=1
                        export NMONVXVM
                        fi

                fi

        echo "`log_date`, ${HOST} INFO: starting nmon : ${nmon_command} in ${NMON_REPOSITORY}"
                ${nmon_command} >/dev/null 2>&1 &
        ;;

esac

}
```

--------------------------------

### Run the installer with -deploy argument

Source: https://lantern.splunk.com/Splunk_and_Cisco_Use_Cases/Deploying_and_managing_your_Splunk_POD_environment

Initiates the deployment of Kubernetes, object store, and Splunk platform. On first run, you will be prompted to accept the Terms and Conditions (one-time action).

```bash
${path_to_install}/kubernetes-installer-standalone -static.cluster cluster-config.yaml -deploy
```

--------------------------------

### Deploy apps or upgrade Splunk POD

Source: https://lantern.splunk.com/Splunk_and_Cisco_Use_Cases/Deploying_and_managing_your_Splunk_POD_environment

Re-run the deploy command after adding app paths to cluster-config.yaml or after downloading a new installer binary. The installer handles app upload/deployment and component upgrades.

```bash
./kubernetes-installer-standalone -static.cluster cluster-config.yaml -deploy
```

--------------------------------

### nmon configuration file loading loop

Source: https://lantern.splunk.com/Manage_Performance_and_Health/Troubleshooting_slow_Splunk_platform_restarts_when_running_under_systemd_on_Linux

Loop that sources nmon configuration files from default, local, and /etc/nmon.conf, reformatting if needed.

```bash
# source default / local / per server nmon.conf
for nmon_conf_file in $APP/default/nmon.conf $APP/local/nmon.conf /etc/nmon.conf; do
    if [ -f $nmon_conf_file ]; then
        # Verify and reformat if required
        grep '\[nmon\]' $nmon_conf_file >/dev/null
        if [ $? -eq 0 ]; then
            reformat_default_nmon_conf $nmon_conf_file
            . $nmon_conf_file
        else
            . $nmon_conf_file
        fi
    fi
done
```
Tokens:
Raw