### Installing OpenVINO for Legacy Models

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/README.md

Installs the OpenVINO toolkit, which is required for running specific legacy MiDaS models optimized for Intel CPUs. This is an optional dependency.

```shell
pip install openvino
```

--------------------------------

### Install TensorFlow

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/ios/README.md

Installs the TensorFlow Python package using pip. Ensure python3 is set as the default Python version before installation. This is a prerequisite for model conversion.

```shell
echo 'export PATH=/usr/local/opt/python/libexec/bin:$PATH' >> ~/.zshenv
 echo 'alias python=python3' >> ~/.zshenv
 echo 'alias pip=pip3' >> ~/.zshenv
 pip install tensorflow
```

--------------------------------

### Install TensorFlowLiteSwift via Cocoapods

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/ios/README.md

Installs necessary build tools and Cocoapods. It then integrates the TensorFlowLiteSwift library into the project by running 'pod install' in the project's root directory. This step is crucial for using the TensorFlow Lite library in the iOS application.

```shell
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install mc rbenv ruby-build
sudo gem install cocoapods
```

```ruby
pod install
```

--------------------------------

### Install ROS Melodic for Ubuntu 17.10/18.04

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/README.md

This script installs ROS Melodic and its dependencies for Ubuntu 17.10 and 18.04. It's a prerequisite for running the MiDaS ROS1 node on compatible systems.

```bash
wget https://raw.githubusercontent.com/isl-org/MiDaS/master/ros/additions/install_ros_melodic_ubuntu_17_18.sh
./install_ros_melodic_ubuntu_17_18.sh
```

--------------------------------

### Install Dependencies with Pip

Source: https://github.com/rsgao/lfdiff/blob/master/README.md

Installs necessary Python packages for LFdiff, including libraries for image processing, deep learning, and data handling. Ensure CUDA 11.3 is available.

```bash
pip install opencv-python scikit-image h5py matplotlib einops xlwt tensorboard diffusers==0.20.0 imutils==0.5.4 timm==0.6.12 transformers accelerate
```

--------------------------------

### Final Installation Configuration in ROS Catkin

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/midas_cpp/CMakeLists.txt

A consolidated installation command for the main target, specifying archive, library, and runtime destinations. This is a common pattern for defining the final installation artifacts of a ROS package.

```cmake
install(TARGETS ${PROJECT_NAME}
  ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)
```

--------------------------------

### Install ROS Noetic for Ubuntu 20.04

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/README.md

This script installs ROS Noetic and its dependencies for Ubuntu 20.04. It's a prerequisite for running the MiDaS ROS1 node on Ubuntu 20.04.

```bash
wget https://raw.githubusercontent.com/isl-org/MiDaS/master/ros/additions/install_ros_noetic_ubuntu_20.sh
./install_ros_noetic_ubuntu_20.sh
```

--------------------------------

### Install LibTorch 1.7 with CUDA 11.0 on Jetson (ARM)

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/README.md

Installs LibTorch version 1.7.0 with CUDA 11.0 support for ARM-based Jetson devices. This involves downloading a pre-compiled wheel, installing system dependencies, and then installing PyTorch. Python 3 and pip3 are required.

```bash
wget https://nvidia.box.com/shared/static/wa34qwrwtk9njtyarwt5nvo6imenfy26.whl -O torch-1.7.0-cp36-cp36m-linux_aarch64.whl
sudo apt-get install python3-pip libopenblas-base libopenmpi-dev 
pip3 install Cython
pip3 install torch-1.7.0-cp36-cp36m-linux_aarch64.whl
```

--------------------------------

### Environment Setup with Conda

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/README.md

Sets up the necessary Python environment for running MiDaS models using Conda. This involves creating a new environment from a YAML file and activating it.

```shell
conda env create -f environment.yaml
conda activate midas-py310
```

--------------------------------

### Download and Install MiDaS ROS Package

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/README.md

Clones the MiDaS ROS repository, copies its contents into a ROS workspace's src directory, makes necessary scripts executable, and then builds the workspace. It also runs a download script for additional assets.

```bash
source ~/.bashrc
cd ~/
mkdir catkin_ws
cd catkin_ws
git clone https://github.com/isl-org/MiDaS
mkdir src
cp -r MiDaS/ros/* src

chmod +x src/additions/*.sh
chmod +x src/*.sh
chmod +x src/midas_cpp/scripts/*.py
cp src/additions/do_catkin_make.sh ./do_catkin_make.sh
./do_catkin_make.sh
./src/additions/downloads.sh
```

--------------------------------

### Install PyTorch and TensorFlow for Model Training and Conversion

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/README.md

Installs necessary PyTorch and TensorFlow versions along with specific torchvision and tensorflow-addons. It also clones the onnx-tensorflow repository for model conversion utilities.

```shell
pip install torch==1.6.0 torchvision==0.7.0
pip install tf-nightly-gpu==2.5.0.dev20201031 tensorflow-addons==0.11.2 numpy==1.18.0
git clone --depth 1 --branch v1.6.0 https://github.com/onnx/onnx-tensorflow
```

--------------------------------

### Install Dependencies for ONNX-Runtime Inference

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/tf/README.md

Installs required libraries for running inference using ONNX models with ONNX-Runtime. This includes OpenCV, ONNX, and ONNX-Runtime, with specific version requirements.

```shell
pip install --upgrade pip
pip install opencv-python
pip install onnx==1.7.0
pip install onnxruntime==1.5.2
```

--------------------------------

### Install Dependencies for ONNX Model Conversion

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/tf/README.md

Sets up the necessary environment for converting a PyTorch model to an ONNX model. This involves installing PyTorch, TensorFlow, ONNX, and the onnx-tensorflow package with a specific commit hash.

```shell
# install OpenCV
pip install --upgrade pip
pip install opencv-python

# install PyTorch TorchVision
pip install -I torch==1.7.0 torchvision==0.8.0

# install TensorFlow
pip install -I grpcio tensorflow==2.3.0 tensorflow-addons==0.11.2 numpy==1.18.0

# install ONNX
pip install onnx==1.7.0

# install ONNX-TensorFlow
git clone https://github.com/onnx/onnx-tensorflow.git
cd onnx-tensorflow 
git checkout 095b51b88e35c4001d70f15f80f31014b592b81e 
pip install -e .
```

--------------------------------

### Install Other Files in ROS Catkin Package

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/midas_cpp/CMakeLists.txt

Installs miscellaneous files, such as launch files or bag files, to the package's share directory. This is a flexible way to include any non-code assets required by your ROS package.

```cmake
install(FILES
  # myfile1
  # myfile2
  DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
)
```

--------------------------------

### Install Libraries in ROS Catkin Package

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/midas_cpp/CMakeLists.txt

Installs target libraries to archive, library, and runtime destinations. This ensures that your package's shared and static libraries are correctly placed for use by other packages or nodes.

```cmake
install(TARGETS ${PROJECT_NAME}
  ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  RUNTIME DESTINATION ${CATKIN_GLOBAL_BIN_DESTINATION}
)
```

--------------------------------

### CMake Project Setup and Dependency Finding (CMake)

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/midas_cpp/CMakeLists.txt

Initializes the CMake project, specifies the C++ standard, and finds required ROS catkin packages, OpenCV, and PyTorch. It also adjusts include and link directories for PyTorch based on Python version.

```cmake
cmake_minimum_required(VERSION 3.0.2)
project(midas_cpp)

# add_compile_options(-std=c++11)

find_package(catkin REQUIRED COMPONENTS
  cv_bridge
  image_transport
  roscpp
  rospy
  sensor_msgs
  std_msgs
)

# find_package(Boost REQUIRED COMPONENTS system)

list(APPEND CMAKE_PREFIX_PATH "~/libtorch")
list(APPEND CMAKE_PREFIX_PATH "/usr/local/lib/python3.6/dist-packages/torch/lib")
list(APPEND CMAKE_PREFIX_PATH "/usr/local/lib/python2.7/dist-packages/torch/lib")

if(NOT EXISTS "~/libtorch")
    if (EXISTS "/usr/local/lib/python3.6/dist-packages/torch")     
        include_directories(/usr/local/include)
        include_directories(/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include)
        include_directories(/usr/local/lib/python3.6/dist-packages/torch/include)

        link_directories(/usr/local/lib)
        link_directories(/usr/local/lib/python3.6/dist-packages/torch/lib)

        set(CMAKE_PREFIX_PATH /usr/local/lib/python3.6/dist-packages/torch)
        set(Boost_USE_MULTITHREADED ON)
        set(Torch_DIR /usr/local/lib/python3.6/dist-packages/torch)
            
    elseif (EXISTS "/usr/local/lib/python2.7/dist-packages/torch")
    
        include_directories(/usr/local/include)
        include_directories(/usr/local/lib/python2.7/dist-packages/torch/include/torch/csrc/api/include)
        include_directories(/usr/local/lib/python2.7/dist-packages/torch/include)

        link_directories(/usr/local/lib)
        link_directories(/usr/local/lib/python2.7/dist-packages/torch/lib)

        set(CMAKE_PREFIX_PATH /usr/local/lib/python2.7/dist-packages/torch)
        set(Boost_USE_MULTITHREADED ON)
        set(Torch_DIR /usr/local/lib/python2.7/dist-packages/torch)
    endif()
endif()



find_package(Torch REQUIRED)
find_package(OpenCV REQUIRED)
include_directories( ${OpenCV_INCLUDE_DIRS} )

add_executable(midas_cpp src/main.cpp)
target_link_libraries(midas_cpp "${TORCH_LIBRARIES}" "${OpenCV_LIBS} ${catkin_LIBRARIES}")
set_property(TARGET midas_cpp PROPERTY CXX_STANDARD 14)


###################################
## catkin specific configuration ##
###################################
catkin_package(
#  INCLUDE_DIRS include
#  LIBRARIES midas_cpp
#  CATKIN_DEPENDS cv_bridge image_transport roscpp sensor_msgs std_msgs
#  DEPENDS system_lib
)

###########
## Build ##
###########

include_directories(
# include
  ${catkin_INCLUDE_DIRS}
)

# add_library(${PROJECT_NAME}
#   src/${PROJECT_NAME}/midas_cpp.cpp
# )

# add_dependencies(${PROJECT_NAME} ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})

# add_executable(${PROJECT_NAME}_node src/midas_cpp_node.cpp)

# set_target_properties(${PROJECT_NAME}_node PROPERTIES OUTPUT_NAME node PREFIX "")

# add_dependencies(${PROJECT_NAME}_node ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})

# target_link_libraries(${PROJECT_NAME}_node
#   ${catkin_LIBRARIES}
# )

#############
## Install ##
#############

# catkin_install_python(PROGRAMS
#   scripts/my_python_script
#   DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
# )

```

--------------------------------

### Install Executable in ROS Catkin Package

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/midas_cpp/CMakeLists.txt

Installs the specified target executable to the runtime destination within the ROS package. This is crucial for making your ROS nodes discoverable and runnable.

```cmake
install(TARGETS ${PROJECT_NAME}_node
  RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)
```

--------------------------------

### Install Dependencies for TensorFlow Inference

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/tf/README.md

Installs necessary Python packages for running inference with TensorFlow models, including OpenCV and TensorFlow itself. Specific versions are pinned for compatibility.

```shell
pip install --upgrade pip
pip install opencv-python
pip install -I grpcio tensorflow==2.3.0 tensorflow-addons==0.11.2 numpy==1.18.0
```

--------------------------------

### Build Docker Image for MiDaS

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/README.md

Builds the Docker image for MiDaS using the provided Dockerfile. Ensure Docker and NVIDIA Docker runtime are installed. The tag 'midas' is applied to the image.

```shell
docker build -t midas .
```

--------------------------------

### Install C++ Header Files in ROS Catkin Package

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/midas_cpp/CMakeLists.txt

Installs C++ header files from the include directory into the package's include destination. It includes pattern matching to select specific files and excludes version control directories.

```cmake
install(DIRECTORY include/${PROJECT_NAME}/
  DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION}
  FILES_MATCHING PATTERN "*.h"
  PATTERN ".svn" EXCLUDE
)
```

--------------------------------

### Install LibTorch 1.7 with CUDA 11.0 on Linux (x86_64)

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/README.md

Downloads and unzips LibTorch version 1.7.0 with CUDA 11.0 support for x86_64 Linux systems. This provides the necessary C++ library for PyTorch inference.

```bash
cd ~/
wget https://download.pytorch.org/libtorch/cu110/libtorch-cxx11-abi-shared-with-deps-1.7.0%2Bcu110.zip
unzip libtorch-cxx11-abi-shared-with-deps-1.7.0+cu110.zip
```

--------------------------------

### Load TensorFlow Lite Model and Create Interpreter

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

Loads a TensorFlow Lite model file into a MappedByteBuffer and instantiates an Interpreter. It configures the interpreter with options such as hardware delegates (NNAPI, GPU) and the number of threads. It also loads the class labels.

```java
protected Classifier(Activity activity, Device device, int numThreads) throws
    IOException {
  tfliteModel = FileUtil.loadMappedFile(activity, getModelPath());
  switch (device) {
    case NNAPI:
      nnApiDelegate = new NnApiDelegate();
      tfliteOptions.addDelegate(nnApiDelegate);
      break;
    case GPU:
      gpuDelegate = new GpuDelegate();
      tfliteOptions.addDelegate(gpuDelegate);
      break;
    case CPU:
      break;
  }
  tfliteOptions.setNumThreads(numThreads);
  tflite = new Interpreter(tfliteModel, tfliteOptions);
  labels = FileUtil.loadLabels(activity, getLabelPath());
...
```

--------------------------------

### Capture User Preferences in CameraActivity (Java)

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

This snippet from CameraActivity.java shows how to capture user preferences for the classification model, device, and number of threads from UI elements. It parses the selected model and device from spinners and the thread count from a text view.

```java
model = Model.valueOf(modelSpinner.getSelectedItem().toString().toUpperCase());
device = Device.valueOf(deviceSpinner.getSelectedItem().toString());
numThreads = Integer.parseInt(threadsTextView.getText().toString().trim());
```

--------------------------------

### Map Labels to Probabilities - Java

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

This Java code snippet demonstrates how to map labels to their corresponding probabilities using TensorLabel and a probability processor. It takes a map of labels and a processed output buffer to generate a map of string labels to float probabilities.

```java
/** Gets the label to probability map. */
Map<String, Float> labeledProbability =
    new TensorLabel(labels,
        probabilityProcessor.process(outputProbabilityBuffer))
        .getMapWithFloatValue();
```

--------------------------------

### Load Model and Create ImageClassifier with Task Library (Java)

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

This code demonstrates loading a TensorFlow Lite model and creating an ImageClassifier instance using the Task Library. It configures inference options like maximum results and score threshold. This approach simplifies inference with a few lines of code.

```java
// Create the ImageClassifier instance.
ImageClassifierOptions options =
    ImageClassifierOptions.builder().setMaxResults(MAX_RESULTS).build();
imageClassifier = ImageClassifier.createFromFileAndOptions(activity,
    getModelPath(), options);
```

--------------------------------

### Run MiDaS Inference with Docker

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/README.md

Executes MiDaS inference within a Docker container. It utilizes all available NVIDIA GPUs, mounts local input, output, and weights directories into the container, and runs the inference process. Ensure Docker and NVIDIA Docker runtime are installed.

```shell
docker run --rm --gpus all -v $PWD/input:/opt/MiDaS/input -v $PWD/output:/opt/MiDaS/output -v $PWD/weights:/opt/MiDaS/weights midas
```

--------------------------------

### Define Normalization Parameters for Quantized Model Output

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

Specifies the mean and standard deviation for post-processing the output probabilities of a quantized model. These parameters are used to de-quantize the predictions, which are typically linear transformations.

```java
private static final float PROBABILITY_MEAN = 0.0f;
private static final float PROBABILITY_STD = 255.0f;
```

--------------------------------

### Define Normalization Parameters for Float Model

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

Defines the mean and standard deviation constants used for normalizing image data in a float model. These values are crucial for the pre-processing pipeline to ensure consistent input ranges for the model.

```java
private static final float IMAGE_MEAN = 127.5f;
private static final float IMAGE_STD = 127.5f;
```

--------------------------------

### Sort Top Probabilities using PriorityQueue - Java

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

This Java method, getTopKProbability, sorts a map of label probabilities to extract the top k results. It utilizes a PriorityQueue to efficiently find and return the most confident recognitions, bounded by MAX_RESULTS. The confidence scores are used to prioritize the results in descending order.

```java
/** Gets the top-k results. */
private static List<Recognition> getTopKProbability(
    Map<String, Float> labelProb) {
  // Find the best classifications.
  PriorityQueue<Recognition> pq =
      new PriorityQueue<>(
          MAX_RESULTS,
          new Comparator<Recognition>() {
            @Override
            public int compare(Recognition lhs, Recognition rhs) {
              // Intentionally reversed to put high confidence at the head of
              // the queue.
              return Float.compare(rhs.getConfidence(), lhs.getConfidence());
            }
          });

  for (Map.Entry<String, Float> entry : labelProb.entrySet()) {
    pq.add(new Recognition("" + entry.getKey(), entry.getKey(),
               entry.getValue(), null));
  }

  final ArrayList<Recognition> recognitions = new ArrayList<>();
  int recognitionsSize = Math.min(pq.size(), MAX_RESULTS);
  for (int i = 0; i < recognitionsSize; ++i) {
    recognitions.add(pq.poll());
  }
  return recognitions;
}
```

--------------------------------

### Allocate Output TensorBuffer for Model Predictions

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

This Java code snippet demonstrates how to initialize an output TensorBuffer for a TensorFlow Lite model. It retrieves the shape and data type of the output tensor from the model file and creates a TensorBuffer with fixed dimensions.

```java
/** Output probability TensorBuffer. */
private final TensorBuffer outputProbabilityBuffer;

//... 
// Get the array size for the output buffer from the TensorFlow Lite model file
int probabilityTensorIndex = 0;
int[] probabilityShape =
    tflite.getOutputTensor(probabilityTensorIndex).shape(); // {1, 1001}
DataType probabilityDataType =
    tflite.getOutputTensor(probabilityTensorIndex).dataType();

// Creates the output tensor and its processor.
outputProbabilityBuffer =
    TensorBuffer.createFixedSize(probabilityShape, probabilityDataType);

// Creates the post processor for the output probability.
probabilityProcessor =
    new TensorProcessor.Builder().add(getPostprocessNormalizeOp()).build();
```

--------------------------------

### Convert TensorFlow Model to TFLite

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/ios/README.md

Converts a TensorFlow saved model to the TensorFlow Lite format. This Python script utilizes tf.lite.TFLiteConverter to perform the conversion. The resulting TFLite model can then be deployed on mobile devices.

```python
saved_model_export_dir = "./saved_model"
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_export_dir)    
tflite_model = converter.convert()
open(model_tflite_name, "wb").write("model.tflite")
```

--------------------------------

### Define Normalization Parameters for Quantized Model

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

Defines the mean and standard deviation constants for image normalization in a quantized model. In this specific implementation, normalization is not required, indicated by mean 0.0f and std 1.0f.

```java
private static final float IMAGE_MEAN = 0.0f;
private static final float IMAGE_STD = 1.0f;
```

--------------------------------

### Run Inference with TensorFlow Lite Model

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

This code snippet executes the inference process using a loaded TensorFlow Lite model. It feeds the pre-processed input image buffer into the model and populates the output probability buffer with the model's predictions.

```java
tflite.run(inputImageBuffer.getBuffer(),
    outputProbabilityBuffer.getBuffer().rewind());
```

--------------------------------

### Pre-process Bitmap Image with TensorFlow Lite

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

This Java method loads a bitmap image, applies various pre-processing steps including resizing, cropping, rotation, and normalization, preparing it for a TensorFlow Lite model. It handles image dimensions and orientation for accurate processing.

```java
/** Loads input image, and applys preprocessing. */
private TensorImage loadImage(final Bitmap bitmap, int sensorOrientation) {
  // Loads bitmap into a TensorImage.
  image.load(bitmap);

  // Creates processor for the TensorImage.
  int cropSize = Math.min(bitmap.getWidth(), bitmap.getHeight());
  int numRoration = sensorOrientation / 90;
  ImageProcessor imageProcessor =
      new ImageProcessor.Builder()
          .add(new ResizeWithCropOrPadOp(cropSize, cropSize))
          .add(new ResizeOp(imageSizeX, imageSizeY, ResizeMethod.BILINEAR))
          .add(new Rot90Op(numRoration))
          .add(getPreprocessNormalizeOp())
          .build();
  return imageProcessor.process(inputImageBuffer);
}
```

--------------------------------

### Recreate Classifier with Model, Device, and Threads in Java

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

The `recreateClassifier()` function in `ClassifierActivity.java` handles the instantiation and reconfiguration of the `Classifier`. It closes the existing classifier if one exists, checks for compatibility (e.g., GPU not supporting quantized models), and then creates a new `Classifier` instance based on the provided model, device, and number of threads. It includes error handling for `IOException` during creation.

```java
private void recreateClassifier(Model model, Device device, int numThreads) {
  if (classifier != null) {
    LOGGER.d("Closing classifier.");
    classifier.close();
    classifier = null;
  }
  if (device == Device.GPU && model == Model.QUANTIZED) {
    LOGGER.d("Not creating classifier: GPU doesn't support quantized models.");
    runOnUiThread(
        () -> {
          Toast.makeText(this, "GPU does not yet supported quantized models.",
              Toast.LENGTH_LONG)
              .show();
        });
    return;
  }
  try {
    LOGGER.d(
        "Creating classifier (model=%s, device=%s, numThreads=%d)", model,
        device, numThreads);
    classifier = Classifier.create(this, model, device, numThreads);
  } catch (IOException e) {
    LOGGER.e(e, "Failed to create classifier.");
  }
}
```

--------------------------------

### Define Normalization Parameters for Float Model Output

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

Defines the mean and standard deviation for post-processing the output probabilities of a float model. For float models, de-quantization is not strictly required but is included for API uniformity, using a mean of 0.0f and std of 1.0f.

```java
private static final float PROBABILITY_MEAN = 0.0f;
private static final float PROBABILITY_STD = 1.0f;
```

--------------------------------

### Preprocess Input Image for Classification

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

Preprocesses an input bitmap for image classification by rotating it to the up-right angle and cropping it to a square center region. This ensures the input image matches the model's expected dimensions (e.g., 224x224). The output is a list of classifications.

```java
TensorImage inputImage = TensorImage.fromBitmap(bitmap);
int width = bitmap.getWidth();
int height = bitmap.getHeight();
int cropSize = min(width, height);
ImageProcessingOptions imageOptions = 
    ImageProcessingOptions.builder()
        .setOrientation(getOrientation(sensorOrientation))
        // Set the ROI to the center of the image.
        .setRoi(
            new Rect(
                /*left=*/ (width - cropSize) / 2,
                /*top=*/ (height - cropSize) / 2,
                /*right=*/ (width + cropSize) / 2,
                /*bottom=*/ (height + cropSize) / 2))
        .build();

List<Classifications> results = imageClassifier.classify(inputImage,
    imageOptions);
```

--------------------------------

### Process Camera Image and Display Results in Java

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/mobile/android/EXPLORE_THE_CODE.md

The `processImage()` function in `ClassifierActivity.java` is responsible for processing camera frames, running image classification on a background thread, and displaying the results on the UI thread. It takes the RGB frame, performs recognition using the `classifier`, and updates the UI with inference details and detected objects.

```java
@Override
protected void processImage() {
  rgbFrameBitmap.setPixels(getRgbBytes(), 0, previewWidth, 0, 0, previewWidth,
      previewHeight);
  final int imageSizeX = classifier.getImageSizeX();
  final int imageSizeY = classifier.getImageSizeY();

  runInBackground(
      new Runnable() {
        @Override
        public void run() {
          if (classifier != null) {
            final long startTime = SystemClock.uptimeMillis();
            final List<Classifier.Recognition> results = 
                classifier.recognizeImage(rgbFrameBitmap, sensorOrientation);
            lastProcessingTimeMs = SystemClock.uptimeMillis() - startTime;
            LOGGER.v("Detect: %s", results);

            runOnUiThread(
                new Runnable() {
                  @Override
                  public void run() {
                    showResultsInBottomSheet(results);
                    showFrameInfo(previewWidth + "x" + previewHeight);
                    showCropInfo(imageSizeX + "x" + imageSizeY);
                    showCameraResolution(imageSizeX + "x" + imageSizeY);
                    showRotationInfo(String.valueOf(sensorOrientation));
                    showInference(lastProcessingTimeMs + "ms");
                  }
                });
          }
          readyForNextImage();
        }
      });
}
```

--------------------------------

### Train LFdiff Model with Accelerate

Source: https://github.com/rsgao/lfdiff/blob/master/README.md

Launches the training process for LFdiff using the 'accelerate' library for multi-GPU support. Requires setting 'state' to 'train' and configuring paths in 'MainCondition.py'.

```bash
accelerate launch --multi_gpu MainCondition.py
```

--------------------------------

### Run MiDaS Test Nodes

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/README.md

Launches talker and listener nodes to test the MiDaS ROS node. This requires a video file named 'test.mp4' in the `~/catkin_ws/src/` directory and should be run in a separate terminal after the MiDaS node is active.

```bash
cd ~/catkin_ws/src && ./run_talker_listener_test.sh
```

--------------------------------

### Run MiDaS Node

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/README.md

Executes the MiDaS ROS node using a provided launch script. This script assumes the ROS environment is sourced and the MiDaS package has been built within the catkin workspace.

```bash
~/catkin_ws/src/launch_midas_cpp.sh
```

--------------------------------

### Create Symlink for OpenCV

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/README.md

Creates a symbolic link from `/usr/include/opencv4` to `/usr/include/opencv`. This is often required by ROS packages that expect OpenCV to be located at `/usr/include/opencv`.

```bash
sudo ln -s /usr/include/opencv4 /usr/include/opencv
```

--------------------------------

### Perform LFdiff Inference

Source: https://github.com/rsgao/lfdiff/blob/master/README.md

Runs the inference process for LF synthesis. Users need to set 'state' to 'eval' in 'MainCondition.py' and prepare test data according to the specified formats.

```python
python MainCondition.py
```

--------------------------------

### Running MiDaS Inference with Camera Input

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/README.md

Performs depth estimation using live camera feed as input. This mode displays the input RGB image and the output depth map side-by-side for real-time comparison.

```shell
python run.py --model_type <model_type> --side
```

--------------------------------

### Running MiDaS Inference

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/README.md

Executes the MiDaS model for depth estimation from input images. Users specify the model type, input directory, and output directory. Various model types are supported.

```shell
python run.py --model_type <model_type> --input_path input --output_path output
```

--------------------------------

### Adding Next-ViT Submodule

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/README.md

Integrates the Next-ViT model as a Git submodule into the project's external directory. This is an optional step for specific model usage.

```shell
git submodule add https://github.com/isl-org/Next-ViT midas/external/next_vit
```

--------------------------------

### Custom Post-Build Command for File Copy

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/midas_cpp/CMakeLists.txt

Adds a custom command that executes after the target is built, copying the built executable to the source directory. This is often used for development workflows or specific integration needs.

```cmake
add_custom_command(
        TARGET midas_cpp POST_BUILD
        COMMAND ${CMAKE_COMMAND} -E copy
        ${CMAKE_CURRENT_BINARY_DIR}/midas_cpp
        ${CMAKE_SOURCE_DIR}/midas_cpp
)
```

--------------------------------

### Generate RGB-D Training Data

Source: https://github.com/rsgao/lfdiff/blob/master/README.md

Preprocesses data to create RGB-D format for training. This script handles cropping input images (single or central views) into 32x32 patches.

```python
./DataProcessing/Generate_Data_for_Training_rgbd.py
```

--------------------------------

### Run MidaS for Monocular Disparity Estimation

Source: https://github.com/rsgao/lfdiff/blob/master/README.md

Executes the MidaS script to generate initial disparity maps from input images. Requires pretrained MidaS weights and places input images in '/MidaS/input', saving results to '/MidaS/output'.

```python
python MidaS/run.py
```

--------------------------------

### Run ONNX Model Inference

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/tf/README.md

Performs depth estimation inference using an ONNX model (`.onnx` file) via ONNX-Runtime. Input images are read from 'tf/input', and depth maps are saved to 'tf/output'. Allows selection of a small model variant.

```python
python tf/run_onnx.py

```

```python
python tf/run_onnx.py --model_weights model-small.onnx --model_type small
```

--------------------------------

### Add Python Nosetests in ROS Catkin

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/midas_cpp/CMakeLists.txt

Configures the package to run Python tests using nosetests. This simplifies the execution of your Python-based test suites within the ROS build environment.

```cmake
catkin_add_nosetests(test)
```

--------------------------------

### Convert PyTorch Model to ONNX

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/tf/README.md

Converts a downloaded PyTorch model (`.pt` file) into an ONNX format (`.onnx` file) using the provided Python script. The resulting ONNX model is saved in the '/tf/' directory.

```python
python tf/make_onnx_model.py
```

--------------------------------

### Add C++ gtest Test Target in ROS Catkin

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/midas_cpp/CMakeLists.txt

Adds a gtest-based C++ test target and links the necessary libraries. This allows you to define and run unit tests for your C++ ROS nodes.

```cmake
catkin_add_gtest(${PROJECT_NAME}-test test/test_midas_cpp.cpp)
if(TARGET ${PROJECT_NAME}-test)
  target_link_libraries(${PROJECT_NAME}-test ${PROJECT_NAME})
endif()
```

--------------------------------

### Run TensorFlow Model Inference

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/tf/README.md

Executes a TensorFlow model (`.pb` file) for depth estimation. It takes input images from the 'tf/input' directory and saves the resulting inverse depth maps to the 'tf/output' directory. Supports a small model variant.

```python
python tf/run_pb.py

```

```python
python tf/run_pb.py --model_weights model-small.pb --model_type small
```

--------------------------------

### Generate H5 Data for Single Image Inference

Source: https://github.com/rsgao/lfdiff/blob/master/README.md

Converts single RGB images and rescaled disparities into H5 format required for inference when synthesizing LFs from single images. Results are saved to 'test_image/test_h5/scene_name'.

```python
python DataProcessing/generated_h5_data.py
```

--------------------------------

### Convert Python Scripts to Python 2

Source: https://github.com/rsgao/lfdiff/blob/master/MidaS/ros/README.md

Modifies Python scripts within the MiDaS ROS package to use `python2` instead of `python3`. This is useful if the ROS environment or project specifically requires Python 2 compatibility.

```bash
sed -i 's/python3/python2/' ~/catkin_ws/src/midas_cpp/scripts/*.py
```

--------------------------------

### Aggregate Overlapping Views for Full LF

Source: https://github.com/rsgao/lfdiff/blob/master/README.md

Combines overlapping views to reconstruct the full light field after initial synthesis. This script is used to obtain the complete set of views.

```python
python overlap_aggregate.py
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.