NeoWorld

https://github.com/zyp123494/neoworld
Admin
NeoWorld provides the official code for the neural simulation of explorable virtual worlds, achieved...
Tokens:23,845
Snippets:326
Trust Score:5.9
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Project Submodules and Requirements

Source: https://github.com/zyp123494/neoworld/blob/main/README.md

Install necessary submodules for depth-difference Gaussian rasterization and simple kNN, followed by the main project requirements. Also, install the RepViT submodule and download a spaCy model.

```bash
pip install submodules/depth-diff-gaussian-rasterization-min-features/
pip install submodules/simple-knn/
pip install -r requirements.txt
cd ./RepViT/sam && pip install -e . && cd ../..
python -m spacy download en_core_web_sm
```

--------------------------------

### Install Dependencies

Source: https://github.com/zyp123494/neoworld/blob/main/mpm_solver_warp/README.md

Installs the necessary Python packages listed in the requirements.txt file.

```shell
pip install -r requirements.txt
```

--------------------------------

### Example: Test RepViT-M1.1 on ADE20K

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/segmentation/README.md

Example command to test the RepViT-M1.1 model on the ADE20K dataset using 8 GPUs.

```bash
./tools/dist_test.sh configs/sem_fpn/fpn_repvit_m1_1_ade20k_40k.py path/to/repvit_m1_1_ade20k.pth 8 --eval mIoU
```

--------------------------------

### Install Other Files

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Installs other files such as launch and bag files to the package share destination.

```cmake
install(FILES
  # myfile1
  # myfile2
  DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
)
```

--------------------------------

### Install TensorFlow

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/ios/README.md

Install the TensorFlow library using pip. Ensure Python 3 is set as the default.

```shell
pip install tensorflow

```

--------------------------------

### Install RepViT-SAM

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/sam/README.md

Installs the RepViT-SAM package using pip. It also downloads a pretrained checkpoint to the weights directory.

```bash
pip install -e .
# download pretrained checkpoint
mkdir weights && cd weights
wget https://github.com/THU-MIG/RepViT/releases/download/v1.0/repvit_sam.pt
```

--------------------------------

### Install OpenVINO for Legacy Models

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/README.md

Install the OpenVINO library, which is required for running the legacy MiDaS models optimized for Intel CPUs.

```shell
pip install openvino
```

--------------------------------

### Example: Test RepViT-M1.1 on COCO 2017

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/detection/README.md

Example command to test the RepViT-M1.1 model on the COCO 2017 dataset using an 8-GPU machine. Includes evaluation for bounding box and segmentation.

```bash
./dist_test.sh configs/mask_rcnn_repvit_m1_1_fpn_1x_coco.py path/to/repvit_m1_1_coco.pth 8 --eval bbox segm
```

--------------------------------

### Install Executable Target

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Installs the executable target to the runtime destination. ${CATKIN_PACKAGE_BIN_DESTINATION} is typically used for executables.

```cmake
install(TARGETS ${PROJECT_NAME}_node
  RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)
```

--------------------------------

### Install Dependencies with Pip

Source: https://github.com/zyp123494/neoworld/blob/main/marigold_module/README.md

Create a Python virtual environment named 'marigold' and install dependencies from 'requirements.txt' using pip. Activate the environment before running scripts.

```bash
python -m venv venv/marigold
source venv/marigold/bin/activate
pip install -r requirements.txt
```

--------------------------------

### Install MMSegmentation and MMCV

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/segmentation/README.md

Install mmcv-full and MMSegmentation using MIM. Ensure you use compatible versions.

```bash
pip install -U openmim
mim install mmcv-full==1.7.1
mim install mmseg==0.30.0
```

--------------------------------

### Install MMDetection and MMCV

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/detection/README.md

Install the necessary libraries for object detection using MIM. Ensure compatibility with specified versions.

```bash
pip install -U openmim
mim install mmcv-full==1.7.1
mim install mmdet==2.28.2
```

--------------------------------

### Install PyTorch and Torchvision

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/README.md

Installs specific versions of PyTorch and Torchvision required for model training and conversion.

```bash
pip install torch==1.6.0 torchvision==0.7.0
```

--------------------------------

### CoreML Export Example

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/sam/README.md

Instructions for exporting the RepViT-SAM model to CoreML format. Refer to the specified Jupyter notebook for detailed steps.

```python
Please refer to [coreml_example.ipynb](./notebooks/coreml_example.ipynb)
```

--------------------------------

### Install Library Targets

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Installs library targets, including archives, libraries, and runtime components. Use appropriate CATKIN package destination variables.

```cmake
install(TARGETS ${PROJECT_NAME}
  ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  RUNTIME DESTINATION ${CATKIN_GLOBAL_BIN_DESTINATION}
)
```

--------------------------------

### Install Python Scripts

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Installs Python scripts to the specified destination. Use ${CATKIN_PACKAGE_BIN_DESTINATION} for the binary directory.

```cmake
catkin_install_python(PROGRAMS
  scripts/my_python_script
  DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)
```

--------------------------------

### Install ROS Melodic Dependencies

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/README.md

Installs necessary ROS Melodic dependencies for Ubuntu 17.10/18.04.

```bash
wget https://raw.githubusercontent.com/isl-org/MiDaS/master/ros/additions/install_ros_melodic_ubuntu_17_18.sh
./install_ros_melodic_ubuntu_17_18.sh
```

--------------------------------

### Set Up Conda Environment

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/README.md

Create and activate a conda environment for RepViT. It is recommended to use a Python version of 3.8 and install dependencies from requirements.txt.

```bash
conda create -n repvit python=3.8
pip install -r requirements.txt
```

--------------------------------

### Install TensorFlow Nightly and Addons

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/README.md

Installs the nightly build of TensorFlow GPU and TensorFlow Addons, along with NumPy, for model development.

```bash
pip install tf-nightly-gpu==2.5.0.dev20201031 tensorflow-addons==0.11.2 numpy==1.18.0
```

--------------------------------

### Install ROS Noetic Dependencies

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/README.md

Installs necessary ROS Noetic dependencies for Ubuntu 20.04.

```bash
wget https://raw.githubusercontent.com/isl-org/MiDaS/master/ros/additions/install_ros_noetic_ubuntu_20.sh
./install_ros_noetic_ubuntu_20.sh
```

--------------------------------

### Install C++ Header Files

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Installs C++ header files from the include directory. Excludes .svn directories and matches only .h files.

```cmake
install(DIRECTORY include/${PROJECT_NAME}/
  DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION}
  FILES_MATCHING PATTERN "*.h"
  PATTERN ".svn" EXCLUDE
)
```

--------------------------------

### Install Homebrew, rbenv, ruby-build, and Cocoapods

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/ios/README.md

Install necessary tools for managing Ruby versions and dependencies, including Cocoapods for iOS project management.

```shell
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

```

```shell
brew install mc rbenv ruby-build

```

```shell
sudo gem install cocoapods

```

--------------------------------

### Install TensorFlow Dependencies

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/tf/README.md

Installs necessary packages for TensorFlow inference, including OpenCV and TensorFlow itself.

```shell
pip install --upgrade pip
pip install opencv-python

pip install -I grpcio tensorflow==2.3.0 tensorflow-addons==0.11.2 numpy==1.18.0
```

--------------------------------

### Install LibTorch 1.7 for Jetson (ARM)

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/README.md

Installs LibTorch version 1.7.0 with CUDA 11.0 for Jetson devices.

```bash
wget https://nvidia.box.com/shared/static/wa34qwrwtk9njtyarwt5nvo6imenfy26.whl -O torch-1.7.0-cp36-cp36m-linux_aarch64.whl
sudo apt-get install python3-pip libopenblas-base libopenmpi-dev 
pip3 install Cython
pip3 install numpy torch-1.7.0-cp36-cp36m-linux_aarch64.whl
```

--------------------------------

### Run Sand Column Collapse Example

Source: https://github.com/zyp123494/neoworld/blob/main/mpm_solver_warp/README.md

Executes the simulation for a sand column collapse using the 'run_sand.py' script.

```shell
python run_sand.py
```

--------------------------------

### Install ONNX Model Conversion Dependencies

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/tf/README.md

Sets up the environment for converting PyTorch models to ONNX format. This includes PyTorch, TensorFlow, ONNX, and ONNX-TensorFlow.

```shell
pip install --upgrade pip
pip install opencv-python

pip install -I torch==1.7.0 torchvision==0.8.0

pip install -I grpcio tensorflow==2.3.0 tensorflow-addons==0.11.2 numpy==1.18.0

pip install onnx==1.7.0

git clone https://github.com/onnx/onnx-tensorflow.git
cd onnx-tensorflow 
git checkout 095b51b88e35c4001d70f15f80f31014b592b81e 
pip install -e .
```

--------------------------------

### Install ONNX Runtime Dependencies

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/tf/README.md

Installs packages required for running ONNX models with ONNX Runtime, including OpenCV and ONNX.

```shell
pip install --upgrade pip
pip install opencv-python

pip install onnx==1.7.0

pip install onnxruntime==1.5.2
```

--------------------------------

### Install PyTorch with CUDA Support

Source: https://github.com/zyp123494/neoworld/blob/main/README.md

Install specific versions of PyTorch, torchvision, and torchaudio compatible with CUDA 12.1. Ensure the correct index URL is used for PyTorch wheels.

```bash
pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu121
```

--------------------------------

### Install Dependencies with Mamba

Source: https://github.com/zyp123494/neoworld/blob/main/marigold_module/README.md

Create and activate a Conda environment named 'marigold' and install dependencies from 'environment.yaml' using Mamba. Ensure Miniforge is activated first.

```bash
mamba env create -n marigold --file environment.yaml
conda activate marigold
```

--------------------------------

### Build MiDaS Docker Image

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/README.md

Build the Docker image for MiDaS. Ensure Docker and NVIDIA Docker runtime are installed.

```shell
docker build -t midas .
```

--------------------------------

### Install LibTorch 1.7 for Linux x86_64

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/README.md

Downloads and unzips LibTorch version 1.7.0 with CUDA 11.0 for Linux x86_64 systems.

```bash
cd ~/
wget https://download.pytorch.org/libtorch/cu110/libtorch-cxx11-abi-shared-with-deps-1.7.0%2Bcu110.zip
unzip libtorch-cxx11-abi-shared-with-deps-1.7.0+cu110.zip
```

--------------------------------

### Download and Install MiDaS ROS Package

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/README.md

Clones the MiDaS repository, copies ROS-specific files, sets execute permissions, and builds the workspace.

```bash
source ~/.bashrc
cd ~/
mkdir catkin_ws
cd catkin_ws
git clone https://github.com/isl-org/MiDaS
mkdir src
cp -r MiDaS/ros/* src

chmod +x src/additions/*.sh
chmod +x src/*.sh
chmod +x src/midas_cpp/scripts/*.py
cp src/additions/do_catkin_make.sh ./do_catkin_make.sh
./do_catkin_make.sh
./src/additions/downloads.sh
```

--------------------------------

### Declare C++ Library Example

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

An example of how to declare a C++ library. This is typically used for shared or static libraries within the ROS package.

```cmake
add_library(${PROJECT_NAME}
  src/${PROJECT_NAME}/midas_cpp.cpp
)
```

--------------------------------

### Install PyTorch3D and Dependencies

Source: https://github.com/zyp123494/neoworld/blob/main/README.md

Install PyTorch3D and its required dependencies (fvcore, iopath) using Conda and pip. This is necessary for 3D operations within the project.

```bash
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
```

--------------------------------

### Set Up Conda Environment for MiDaS

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/README.md

Create and activate the Conda environment required for MiDaS. Ensure you have Conda installed and the environment.yaml file present.

```shell
conda env create -f environment.yaml
conda activate midas-py310
```

--------------------------------

### Install Targets for Midas C++

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Specifies the installation destinations for the Midas C++ executable, libraries, and archives within the ROS package structure.

```cmake
install(TARGETS ${PROJECT_NAME}
  ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)
```

--------------------------------

### Run Marigold Docker Container

Source: https://github.com/zyp123494/neoworld/blob/main/marigold_module/README.md

Execute the Marigold demo locally using Docker. Ensure you have a GPU and nvidia-docker2 installed. This command maps port 7860 for access.

```bash
docker run -it -p 7860:7860 --platform=linux/amd64 --gpus all registry.hf.space/toshas-marigold:latest python app.py
```

--------------------------------

### Add Executable Dependencies Example

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Specifies CMake target dependencies for an executable, similar to library dependencies. This ensures that necessary targets are built prior to the executable.

```cmake
add_dependencies(${PROJECT_NAME}_node ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})
```

--------------------------------

### Set Default Python Version and Aliases

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/ios/README.md

Configure your shell environment to use Python 3 and pip3 by default. This is a prerequisite for installing TensorFlow.

```shell
echo 'export PATH=/usr/local/opt/python/libexec/bin:$PATH' >> ~/.zshenv

```

```shell
echo 'alias python=python3' >> ~/.zshenv

```

```shell
echo 'alias pip=pip3' >> ~/.zshenv

```

--------------------------------

### Declare C++ Executable Example

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Defines a C++ executable target. In a Catkin build, all packages are built within a single CMake context, so using a recommended prefix helps avoid name collisions.

```cmake
add_executable(${PROJECT_NAME}_node src/midas_cpp_node.cpp)
```

--------------------------------

### Add CMake Target Dependencies Example

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Adds dependencies for a CMake target, ensuring that certain targets are built before others. This is common for targets that depend on generated code or other libraries.

```cmake
add_dependencies(${PROJECT_NAME} ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})
```

--------------------------------

### Configure Torch Include and Link Directories (Python 3.6)

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Configures include and link directories for Torch when installed via Python 3.6's dist-packages. This block is executed if the specified Torch path exists.

```cmake
if(NOT EXISTS "~/libtorch")
    if (EXISTS "/usr/local/lib/python3.6/dist-packages/torch")     
        include_directories(/usr/local/include) 
        include_directories(/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include)
        include_directories(/usr/local/lib/python3.6/dist-packages/torch/include)

        link_directories(/usr/local/lib)
        link_directories(/usr/local/lib/python3.6/dist-packages/torch/lib)

        set(CMAKE_PREFIX_PATH /usr/local/lib/python3.6/dist-packages/torch)
        set(Boost_USE_MULTITHREADED ON)
        set(Torch_DIR /usr/local/lib/python3.6/dist-packages/torch)
            
    elseif (EXISTS "/usr/local/lib/python2.7/dist-packages/torch")
    
        include_directories(/usr/local/include) 
        include_directories(/usr/local/lib/python2.7/dist-packages/torch/include/torch/csrc/api/include)
        include_directories(/usr/local/lib/python2.7/dist-packages/torch/include)

        link_directories(/usr/local/lib)
        link_directories(/usr/local/lib/python2.7/dist-packages/torch/lib)

        set(CMAKE_PREFIX_PATH /usr/local/lib/python2.7/dist-packages/torch)
        set(Boost_USE_MULTITHREADED ON)
        set(Torch_DIR /usr/local/lib/python2.7/dist-packages/torch)
    endif()
endif()
```

--------------------------------

### Run the Interactive Demo on the Server

Source: https://github.com/zyp123494/neoworld/blob/main/README.md

Execute the main demo script on the server using a bash script. This command initiates the interactive visualization server component.

```bash
# On user_id@server_name
bash run_demo.sh
```

--------------------------------

### Run RepViT-SAM Demo

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/sam/README.md

Executes the Hugging Face demo application for RepViT-SAM. The demo is accessible online via a provided link.

```python
python app/app.py
```

--------------------------------

### Download Sample Data

Source: https://github.com/zyp123494/neoworld/blob/main/marigold_module/README.md

Use this script to download sample images for testing if you do not have your own.

```bash
bash script/download_sample_data.sh
```

--------------------------------

### Download Pre-trained Models

Source: https://github.com/zyp123494/neoworld/blob/main/README.md

Download the RepViT model and the fine-tuned Amodal3R checkpoint. Place these files in the root directory of the project as required for certain functionalities.

```bash
wget https://github.com/THU-MIG/RepViT/releases/download/v1.0/repvit_sam.pt
# Download our fine-tuned [Amodal3R checkpoint](https://drive.google.com/file/d/1xDwU7dOReeGLUf8WmsOubOFXZJT6VH4U/view?usp=sharing) and place it in the root directory.
```

--------------------------------

### Run Basic Generation Locally

Source: https://github.com/zyp123494/neoworld/blob/main/README.md

Configure the rotation path in the base configuration file and then run the main generation script using a bash command.

```bash
# Set `rotation_path` in `config/base-config.yaml`
# Run `bash run.sh` for basic generation
```

--------------------------------

### Run MiDaS Node

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/README.md

Launches the MiDaS ROS node for depth estimation.

```bash
~/catkin_ws/src/launch_midas_cpp.sh
```

--------------------------------

### Load Model and Create Interpreter

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/android/EXPLORE_THE_CODE.md

Loads the model file and instantiates an Interpreter, configuring it with device-specific options and delegates (NNAPI, GPU, CPU).

```java
protected Classifier(Activity activity, Device device, int numThreads) throws
    IOException {
  tfliteModel = FileUtil.loadMappedFile(activity, getModelPath());
  switch (device) {
    case NNAPI:
      nnApiDelegate = new NnApiDelegate();
      tfliteOptions.addDelegate(nnApiDelegate);
      break;
    case GPU:
      gpuDelegate = new GpuDelegate();
      tfliteOptions.addDelegate(gpuDelegate);
      break;
    case CPU:
      break;
  }
  tfliteOptions.setNumThreads(numThreads);
  tflite = new Interpreter(tfliteModel, tfliteOptions);
  labels = FileUtil.loadLabels(activity, getLabelPath());
...
```

--------------------------------

### Multi-GPU Training Script

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/detection/README.md

Launch distributed data parallel training using the provided script. Specify the configuration file and the number of GPUs.

```bash
./dist_train.sh configs/mask_rcnn_repvit_m1_1_fpn_1x_coco.py 8
```

--------------------------------

### Get Camera Input Preferences

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/android/EXPLORE_THE_CODE.md

Extracts user preferences for model, device, and number of threads from UI elements in CameraActivity.java.

```java
model = Model.valueOf(modelSpinner.getSelectedItem().toString().toUpperCase());
device = Device.valueOf(deviceSpinner.getSelectedItem().toString());
numThreads = Integer.parseInt(threadsTextView.getText().toString().trim());
```

--------------------------------

### Set Minimum CMake Version

Source: https://github.com/zyp123494/neoworld/blob/main/submodules/depth-diff-gaussian-rasterization-min-features/CMakeLists.txt

Specifies the minimum required version of CMake for the project. Ensure your CMake installation meets this requirement.

```cmake
cmake_minimum_required(VERSION 3.20)
```

--------------------------------

### Test MiDaS Node with Talker/Listener

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/README.md

Runs a test sequence involving capturing video, running the MiDaS node, and then executing talker and listener nodes for 30 seconds.

```bash
cd ~/catkin_ws/src && ./run_talker_listener_test.sh
```

--------------------------------

### Install TensorFlowLiteSwift via Cocoapods

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/ios/README.md

Integrate the TensorFlowLiteSwift library into your iOS project using Cocoapods. Ensure the correct version is specified in your Podfile.

```ruby
pod install

```

--------------------------------

### Download Checkpoint Weights Locally

Source: https://github.com/zyp123494/neoworld/blob/main/marigold_module/README.md

Use this script to download the Marigold checkpoint weights to a local directory.

```bash
bash script/download_weights.sh
```

--------------------------------

### Multi-GPU Testing Script

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/segmentation/README.md

Use this script for multi-GPU testing. Specify the configuration file, checkpoint path, and the number of GPUs.

```bash
./tools/dist_test.sh config_file path/to/checkpoint #GPUs --eval mIoU
```

--------------------------------

### Get Top K Probabilities

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/android/EXPLORE_THE_CODE.md

Sorts and retrieves the top K recognition results from a map of labels to probabilities. Uses a PriorityQueue for efficient sorting.

```java
/** Gets the top-k results. */
private static List<Recognition> getTopKProbability(
    Map<String, Float> labelProb) {
  // Find the best classifications.
  PriorityQueue<Recognition> pq =
      new PriorityQueue<>(
          MAX_RESULTS,
          new Comparator<Recognition>() {
            @Override
            public int compare(Recognition lhs, Recognition rhs) {
              // Intentionally reversed to put high confidence at the head of
              // the queue.
              return Float.compare(rhs.getConfidence(), lhs.getConfidence());
            }
          });

  for (Map.Entry<String, Float> entry : labelProb.entrySet()) {
    pq.add(new Recognition("" + entry.getKey(), entry.getKey(),
               entry.getValue(), null));
  }

  final ArrayList<Recognition> recognitions = new ArrayList<>();
  int recognitionsSize = Math.min(pq.size(), MAX_RESULTS);
  for (int i = 0; i < recognitionsSize; ++i) {
    recognitions.add(pq.poll());
  }
  return recognitions;
}
```

--------------------------------

### Run MiDaS with Camera Input

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/README.md

Perform inference using live camera feed. Omit input and output paths and specify the model type. The --side argument displays input and output side-by-side.

```shell
python run.py --model_type <model_type> --side
```

--------------------------------

### Clone Repository and Set Up Conda Environment

Source: https://github.com/zyp123494/neoworld/blob/main/README.md

Clone the NeoWorld repository and create a dedicated Conda environment with Python 3.10 for project dependencies.

```bash
git clone https://github.com/zyp123494/NeoWorld.git && cd NeoWorld
conda create -n neoworld python=3.10
conda activate neoworld
```

--------------------------------

### Basic HTML Structure for Viewer Controls

Source: https://github.com/zyp123494/neoworld/blob/main/splat/index_stream.html

This snippet shows the HTML elements used for user interaction, such as input boxes for prompts and buttons for actions like sending, simulating, and animating. It also includes elements for displaying information and progress.

```html
<div id="prompt-box"></div>
<button id="send-button"></button>
<div id="sim-prompt-box"></div>
<button id="sim-button"></button>
<div id="ani-prompt-box"></div>
<button id="ani-button"></button>
<div id="info"></div>
<div id="progress"></div>
<div id="quality"></div>
<div id="server-state"></div>
<div id="caminfo"></div>
```

--------------------------------

### Rename C++ Executable Example

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Renames a C++ executable target to a shorter name, removing the package prefix. This simplifies the command used to run the executable, e.g., 'rosrun someones_pkg node' instead of 'rosrun someones_pkg someones_pkg_node'.

```cmake
set_target_properties(${PROJECT_NAME}_node PROPERTIES OUTPUT_NAME node PREFIX "")
```

--------------------------------

### Load Model and Create ImageClassifier

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/android/EXPLORE_THE_CODE.md

Instantiates an ImageClassifier using a model file and custom options. Ensure the model has metadata and a corresponding label file. Configure inference parameters like max results and score threshold.

```java
ImageClassifierOptions options = 
    ImageClassifierOptions.builder().setMaxResults(MAX_RESULTS).build();
imageClassifier = ImageClassifier.createFromFileAndOptions(activity,
    getModelPath(), options);
```

--------------------------------

### Run Inference with Input Images

Source: https://github.com/zyp123494/neoworld/blob/main/marigold_module/README.md

Execute the inference script by specifying input and output directories. Results will be saved in the output directory.

```python
python run.py \
    --input_rgb_dir input/in-the-wild_example \
    --output_dir output/in-the-wild_example
```

--------------------------------

### Create OpenCV Symlink

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/README.md

Creates a symbolic link for OpenCV to ensure compatibility.

```bash
sudo ln -s /usr/include/opencv4 /usr/include/opencv
```

--------------------------------

### Run Simulation or Animation Tasks Locally

Source: https://github.com/zyp123494/neoworld/blob/main/README.md

Execute simulation or animation tasks locally by running the corresponding bash scripts after setting the rotation path in the configuration file.

```bash
# Set `rotation_path` in `config/base-config.yaml`
# Run `bash run_simulation.sh` or `bash run_animation.sh` for simulation/animation tasks
```

--------------------------------

### Set Up SSH Tunnel for Local Visualization

Source: https://github.com/zyp123494/neoworld/blob/main/README.md

Establish an SSH tunnel to forward the server's port 7777 to your local machine. This is required for the interactive visualization demo using splat.

```shell
# On your local machine
ssh -L 7777:localhost:7777 server_name
```

--------------------------------

### Export Model to Core ML

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/README.md

Use this command to export a RepViT model to the Core ML format for use on Apple devices. Specify the model name and the path to the checkpoint file.

```python
python export_coreml.py --model repvit_m0_9 --ckpt pretrain/repvit_m0_9_distill_300e.pth
```

--------------------------------

### Define Executable and Link Libraries

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Defines the main executable 'midas_cpp' from 'src/main.cpp' and links it with Torch, OpenCV, and catkin libraries. Sets the C++ standard to C++14.

```cmake
add_executable(midas_cpp src/main.cpp)
target_link_libraries(midas_cpp "${TORCH_LIBRARIES}" "${OpenCV_LIBS} ${catkin_LIBRARIES}")
set_property(TARGET midas_cpp PROPERTY CXX_STANDARD 14)
```

--------------------------------

### Include Directories Configuration

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Specifies additional locations for header files. The package's own include directories should be listed before other locations.

```cmake
include_directories(
# include
  ${catkin_INCLUDE_DIRS}
)
```

--------------------------------

### Run Inference with Local Checkpoint

Source: https://github.com/zyp123494/neoworld/blob/main/marigold_module/README.md

Run inference specifying a local path to the checkpoint weights. Ensure input and output directories are also provided.

```python
python run.py \
    --checkpoint checkpoint/Marigold_v1_merged_2 \
    --input_rgb_dir input/in-the-wild_example\
    --output_dir output/in-the-wild_example
```

--------------------------------

### Pre-process Bitmap Image

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/android/EXPLORE_THE_CODE.md

Loads a bitmap, applies preprocessing steps including resizing, cropping, and normalization using an ImageProcessor.

```java
/** Loads input image, and applys preprocessing. */
private TensorImage loadImage(final Bitmap bitmap, int sensorOrientation) {
  // Loads bitmap into a TensorImage.
  image.load(bitmap);

  // Creates processor for the TensorImage.
  int cropSize = Math.min(bitmap.getWidth(), bitmap.getHeight());
  int numRoration = sensorOrientation / 90;
  ImageProcessor imageProcessor =
      new ImageProcessor.Builder()
          .add(new ResizeWithCropOrPadOp(cropSize, cropSize))
          .add(new ResizeOp(imageSizeX, imageSizeY, ResizeMethod.BILINEAR))
          .add(new Rot90Op(numRoration))
          .add(getPreprocessNormalizeOp())
          .build();
  return imageProcessor.process(inputImageBuffer);
}
```

--------------------------------

### Convert PyTorch Model to ONNX

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/tf/README.md

Converts a downloaded PyTorch model file (.pt) into an ONNX model file (.onnx). The output is saved in the /tf/ folder.

```shell
python tf/make_onnx_model.py
```

--------------------------------

### Clone Marigold Repository

Source: https://github.com/zyp123494/neoworld/blob/main/marigold_module/README.md

Clone the Marigold repository using git and navigate into the project directory.

```bash
git clone https://github.com/prs-eth/Marigold.git
cd Marigold
```

--------------------------------

### Distributed Data Parallel Training Script

Source: https://github.com/zyp123494/neoworld/blob/main/RepViT/segmentation/README.md

Use this script for PyTorch distributed data parallel training. Specify the configuration file and the number of GPUs.

```bash
./tools/dist_train.sh configs/sem_fpn/fpn_repvit_m1_1_ade20k_40k.py 8
```

--------------------------------

### Find Torch and OpenCV Libraries

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Finds the Torch and OpenCV libraries, and includes their respective include directories.

```cmake
find_package(Torch REQUIRED)
find_package(OpenCV REQUIRED)
include_directories( ${OpenCV_INCLUDE_DIRS} )
```

--------------------------------

### Link Libraries for Node

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

Links the necessary libraries to the node target. Ensure ${catkin_LIBRARIES} is defined in your environment.

```cmake
target_link_libraries(${PROJECT_NAME}_node
  ${catkin_LIBRARIES}
)
```

--------------------------------

### Map Labels to Probabilities

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/mobile/android/EXPLORE_THE_CODE.md

Processes the output buffer to create a map of labels to their corresponding probabilities.

```java
/** Gets the label to probability map. */
Map<String, Float> labeledProbability =
    new TensorLabel(labels,
        probabilityProcessor.process(outputProbabilityBuffer))
        .getMapWithFloatValue();
```

--------------------------------

### Specify Link Libraries

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/ros/midas_cpp/CMakeLists.txt

A comment indicating where to specify the libraries that a library or executable target should link against.

```cmake
## Specify libraries to link a library or executable target against
```

--------------------------------

### Run ONNX Model Inference

Source: https://github.com/zyp123494/neoworld/blob/main/midas_module/tf/README.md

Performs inference using an ONNX model. Use command-line arguments to specify different model weights or types.

```shell
python tf/run_onnx.py
```

```shell
python tf/run_onnx.py --model_weights model-small.onnx --model_type small
```

--------------------------------

### Set Hugging Face Cache Directory

Source: https://github.com/zyp123494/neoworld/blob/main/marigold_module/README.md

Override the default Hugging Face cache location by setting the HF_HOME environment variable.

```bash
export HF_HOME=new/path
```
Tokens:
Raw