TAPIP3D

https://github.com/zbw001/tapip3d
Admin
TAPIP3D is a method for long-term feed-forward 3D point tracking in monocular RGB and RGB-D video...
Tokens:357,208
Snippets:2,097
Trust Score:7.8
License:Apache-2.0
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install OpenCV and Open3D

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/README.md

Install OpenCV and Open3D for running the examples. Depending on the OS, these may need to be built from source.

```bash
pip install opencv-python open3d
```

--------------------------------

### Install from source

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/README.md

Install LieTorch from source. The build may take up to 10 minutes. Run the test script after installation.

```bash
git clone --recursive https://github.com/princeton-vl/lietorch.git
python setup.py install
./run_tests.sh
```

--------------------------------

### Comma-initialization example and output

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/doc/TutorialMatrixClass.dox

Example and output for comma-initialization syntax. The example is included from an external file, and the output is shown separately.

```cpp
Tutorial_commainit_01.cpp
```

```text
Tutorial_commainit_01.out
```

--------------------------------

### Resizing example and output

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/doc/TutorialMatrixClass.dox

Example and output for resizing dynamic-size matrices using the resize() method. The example is included from an external file, and the output is shown separately.

```cpp
tut_matrix_resize.cpp
```

```text
tut_matrix_resize.out
```

--------------------------------

### Assignment and resizing example and output

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/doc/TutorialMatrixClass.dox

Example and output for assignment with automatic resizing. The example is included from an external file, and the output is shown separately.

```cpp
tut_matrix_assignment_resizing.cpp
```

```text
tut_matrix_assignment_resizing.out
```

--------------------------------

### Fixed-size resize example and output

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/doc/TutorialMatrixClass.dox

Example and output for resizing fixed-size matrices. The example is included from an external file, and the output is shown separately.

```cpp
tut_matrix_resize_fixed_size.cpp
```

```text
tut_matrix_resize_fixed_size.out
```

--------------------------------

### Eigen example: TopicStorageOrders_example.cpp and its output

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/StorageOrders.dox

The example code (TopicStorageOrders_example.cpp) and its output (TopicStorageOrders_example.out) illustrate the two storage orders using the PlainObjectBase::data() function, which returns a pointer to the first entry of the matrix. The example initializes a column-major matrix and assigns it to a row-major matrix, showing how Eigen reorders entries automatically.

```cpp
\include TopicStorageOrders_example.cpp
```

```text
\verbinclude TopicStorageOrders_example.out
```

--------------------------------

### Output of first Eigen example

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/QuickStartGuide.dox

Output produced by the first example program (QuickStart_example.cpp). It shows the 2x2 matrix m after the assignments, with the bottom-right entry computed as m(1,0) + m(0,1).

```text
3 -1
2.5 1.5

```

--------------------------------

### Install notebook dependencies

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/tartanair_tools/TartanAir_Sample.ipynb

Install the required Python packages for the notebook: numpy, azure-storage-blob, and opencv-python.

```bash
pip install numpy
```

```bash
pip install azure-storage-blob
```

```bash
pip install opencv-python
```

--------------------------------

### Simple Eigen MatrixXd program

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/QuickStartGuide.dox

A minimal Eigen program that creates a 2x2 MatrixXd, sets its entries, and prints the matrix. It is the first example in the Getting Started guide and is used to demonstrate basic matrix creation and output.

```cpp
#include <iostream>
#include <Eigen/Dense>
 
using Eigen::MatrixXd;
 
int main()
{
  MatrixXd m(2,2);
  m(0,0) = 3;
  m(1,0) = 2.5;
  m(0,1) = -1;
  m(1,1) = m(1,0) + m(0,1);
  std::cout << m << std::endl;
}

```

--------------------------------

### Install via pip

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/README.md

Install LieTorch directly via pip from the GitHub repository.

```bash
pip install git+https://github.com/princeton-vl/lietorch.git
```

--------------------------------

### Install xformers for UniDepth

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/README.md

Downloads and installs xformers 0.0.22.post7 from a prebuilt file for Python 3.10, CUDA 11.8, and PyTorch 2.0.1. This is needed for the UniDepth model.

```bash
wget https://anaconda.org/xformers/xformers/0.0.22.post7/download/linux-64/xformers-0.0.22.post7-py310_cu11.8.0_pyt2.0.1.tar.bz2
```

```bash
conda install xformers-0.0.22.post7-py310_cu11.8.0_pyt2.0.1.tar.bz2
```

--------------------------------

### Install dependencies

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/MoGe/README.md

Install the required Python packages. MoGe is compatible with most versions of the requirements.

```bash
pip install -r requirements.txt
```

--------------------------------

### Coefficient accessors example and output

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/doc/TutorialMatrixClass.dox

Example and output for coefficient accessors using the overloaded parenthesis operators. The example is included from an external file, and the output is shown separately.

```cpp
tut_matrix_coefficient_accessors.cpp
```

```text
tut_matrix_coefficient_accessors.out
```

--------------------------------

### Output of second Eigen example

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/QuickStartGuide.dox

Output produced by the second example program (QuickStart_example2_dynamic.cpp). It shows the scaled random matrix m and the result of multiplying m by the vector v.

```text
m =
  94.0188  89.844 43.5223
  89.018   97.4804 88.6024
  49.3967  88.5646 82.6404

m * v =
  404.274
  512.237
  441.273

```

--------------------------------

### EIGEN_INITIALIZE_MATRICES_BY_ZERO: 1x1 constructor example

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/PreprocessorDirectives.dox

Shows the unary constructor of a 1x1 fixed-size matrix being interpreted as an initialization constructor, where the argument is the coefficient value, not the size. This example creates a vector with coefficients [2,1].

```cpp
Vector2d v(2,1);
```

--------------------------------

### Simple vector addition example

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/InsideEigenExample.dox

This is the main example program that the page analyzes. It creates three dynamic-size float vectors and adds two of them, storing the result in the third.

```cpp
#include<Eigen/Core>

int main()
{
  int size = 50;
  // VectorXf is a vector of floats, with dynamic size.
  Eigen::VectorXf u(size), v(size), w(size);
  u = v + w;
}

```

--------------------------------

### Example program: Eigen::VectorXf addition

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/doc/InsideEigenExample.dox

The main example program that demonstrates Eigen's compilation process. It creates three dynamic-size vectors of floats and adds two of them. The page analyzes how Eigen compiles this code with SSE2 vectorization enabled.

```cpp
#include<Eigen/Core>

int main()
{
  int size = 50;
  // VectorXf is a vector of floats, with dynamic size.
  Eigen::VectorXf u(size), v(size), w(size);
  u = v + w;
}
```

--------------------------------

### Coefficient accessors example

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/TutorialMatrixClass.dox

Example of using the parenthesis operator to access and mutate matrix coefficients. The row index is passed first for matrices, and a single index for vectors. Indexing starts at 0.

```cpp
#include <iostream>
#include <Eigen/Dense>

using namespace Eigen;

int main()
{
  MatrixXd m(2,2);
  m(0,0) = 3;
  m(1,0) = 2.5;
  m(0,1) = -1;
  m(1,1) = m(1,0) + m(0,1);
  
  std::cout << "Here is the matrix m:\n" << m << std::endl;
  VectorXd v(2);
  v(0) = 4;
  v(1) = v(0) - 1;
  std::cout << "Here is the vector v:\n" << v << std::endl;
}
```

```text
Here is the matrix m:
  3  -1
2.5 1.5
Here is the vector v:
4
3
```

--------------------------------

### Run MoGe inference with scripts/infer.py

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/MoGe/README.md

Two example invocations of scripts/infer.py. The first saves maps, GLB, and PLY outputs; the second displays the result in a window and requires pyglet < 2.0 (e.g., pip install pyglet==1.5.29).

```bash
python scripts/infer.py --input IMAGES_FOLDER_OR_IMAGE_PATH --output OUTPUT_FOLDER --maps --glb --ply
```

```bash
python scripts/infer.py --input IMAGES_FOLDER_OR_IMAGE_PATH --output OUTPUT_FOLDER --show
```

--------------------------------

### Run web demo

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/MoGe/README.md

Run the web demo using Gradio. Add --share for public sharing.

```bash
python scripts/app.py   # --share for Gradio public sharing
```

--------------------------------

### Comma-initializer example

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/TutorialMatrixClass.dox

Shows how to set matrix and vector coefficients using the comma-initializer syntax.

```cpp
#include <iostream>
#include <Eigen/Dense>

using namespace Eigen;

int main()
{
  Matrix3f m;
  m << 1, 2, 3,
       4, 5, 6,
       7, 8, 9;
  std::cout << m;
}
```

```text
1 2 3
4 5 6
7 8 9
```

--------------------------------

### Run demo.py for SE3/Sim3 registration

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/examples/registration/readme.md

Run the demo for SE3 or Sim3 registration. Requires Open3D and a downloaded model checkpoint (se3.pth or sim3.pth).

```bash
python demo.py --transformation=SE3 --ckpt=se3.pth
python demo.py --transformation=Sim3 --ckpt=sim3.pth
```

--------------------------------

### Run demo.py for RGB-D SLAM

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/examples/rgbdslam/readme.md

Requires a GPU with at least 8GB memory. Download a TUM-RGBD sequence first. The --frame_rate flag controls subsampling rate; --go enables global optimization. With RTX-3090 and no visualization, --frame_rate <= 8.0 gives real-time performance.

```bash
python demo.py --ckpt=lietorch_rgbdslam.pth --datapath=<sequence path> --frame_rate=8.0 --go --viz
```

--------------------------------

### Transpose and conjugate example (tut_arithmetic_transpose_conjugate.cpp)

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/TutorialMatrixArithmetic.dox

Shows how to obtain the transpose, conjugate, and adjoint (conjugate transpose) of a matrix using transpose(), conjugate(), and adjoint(). For real matrices, conjugate() is a no-op and adjoint() equals transpose().

```cpp
#include <iostream>
#include <Eigen/Dense>

using namespace Eigen;

int main()
{
  MatrixXcf a = MatrixXcf::Random(2,2);
  std::cout << "Here is the matrix a\n" << a << std::endl;

  std::cout << "Here is the matrix a^T\n" << a.transpose() << std::endl;

  std::cout << "Here is the conjugate of a\n" << a.conjugate() << std::endl;

  std::cout << "Here is the matrix a^*\n" << a.adjoint() << std::endl;
}
```

```cpp
Here is the matrix a
 (-0.211234,0.680375)  (0.59688,0.823295)
 (0.566198,-0.604897) (-0.329554,0.536459)
Here is the matrix a^T
 (-0.211234,0.680375)  (0.566198,-0.604897)
 (0.59688,0.823295) (-0.329554,0.536459)
Here is the conjugate of a
 (-0.211234,-0.680375)  (0.59688,-0.823295)
 (0.566198,0.604897) (-0.329554,-0.536459)
Here is the matrix a^*
 (-0.211234,-0.680375)  (0.566198,0.604897)
 (0.59688,-0.823295) (-0.329554,-0.536459)
```

--------------------------------

### RowMajor output for extract_patches example

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/unsupported/Eigen/CXX11/src/Tensor/README.md

Output of the extract_patches example when the data layout is RowMajor. The set of patches is the same as in ColMajor but indexed differently.

```text
patch index: 0
0 1
4 5
patch index: 1
1 2
5 6
patch index: 2
2 3
6 7
patch index: 3
4 5
8 9
patch index: 4
5 6
9 10
patch index: 5
6 7
10 11
```

--------------------------------

### ColMajor output for extract_patches example

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/unsupported/Eigen/CXX11/src/Tensor/README.md

Output of the extract_patches example when the data layout is ColMajor. The set of patches is the same as in RowMajor but indexed differently.

```text
patch index: 0
0 1
4 5
patch index: 1
4 5
8 9
patch index: 2
1 2
5 6
patch index: 3
5 6
9 10
patch index: 4
2 3
6 7
patch index: 5
6 7
10 11
```

--------------------------------

### Run inference with known depths and camera parameters (dexycb.npz)

Source: https://github.com/zbw001/tapip3d/blob/main/README.md

Runs inference on an .npz file containing rgb, depths, intrinsics, and extrinsics, operating in an aligned global frame to generate world-coordinate trajectories. The example file must be downloaded and placed in demo_inputs/.

```bash
python inference.py --input_path demo_inputs/dexycb.npz --checkpoint checkpoints/tapip3d_final.pth --resolution_factor 2
```

--------------------------------

### Get total element count with size()

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/unsupported/Eigen/CXX11/src/Tensor/README.md

Shows how to get the total number of elements in a Tensor using size(). The output is printed to cout.

```cpp
Eigen::Tensor<float, 2> a(3, 4);
cout << "Size: " << a.size();
=> Size: 12
```

--------------------------------

### Naive implementation with temporary

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/InsideEigenExample.dox

This shows what a naive C++ library would do, introducing a temporary and causing two loops.

```cpp
  VectorXf tmp = v + w;
  VectorXf u = tmp;

```

--------------------------------

### List Easy trajectories for first environment

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/tartanair_tools/TartanAir_Sample.ipynb

List the 'Easy' trajectories for the first environment (abandonedfactory). The output shows 10 trajectories.

```python
diff_level = 'Easy'
env_ind = 0
trajlist = get_trajectory_list(envlist[env_ind], easy_hard = diff_level)
print('Find {} trajectories in {}'.format(len(trajlist), envlist[env_ind]+diff_level))
print(trajlist)
```

```text
Output:
Find 10 trajectories in abandonedfactory/Easy
['abandonedfactory/Easy/P000/', 'abandonedfactory/Easy/P001/', 'abandonedfactory/Easy/P002/', 'abandonedfactory/Easy/P004/', 'abandonedfactory/Easy/P005/', 'abandonedfactory/Easy/P006/', 'abandonedfactory/Easy/P008/', 'abandonedfactory/Easy/P009/', 'abandonedfactory/Easy/P010/', 'abandonedfactory/Easy/P011/']
```

--------------------------------

### Dot product and cross product

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/TutorialMatrixArithmetic.dox

Demonstrates the dot() and cross() methods for vectors. The example and its output are provided.

```cpp
\include tut_arithmetic_dot_cross.cpp
```

```text
\verbinclude tut_arithmetic_dot_cross.out
```

--------------------------------

### Install lapacke on macOS with MacPorts

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/UsingBlasLapackBackends.dox

Required for Mac users to use the LAPACK version shipped with the Accelerate framework. After installation, link with -framework Accelerate and /opt/local/lib/lapack/liblapacke.dylib.

```bash
sudo port install lapack
```

--------------------------------

### Clone and set up conda environment

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/README.md

Clone the repository with submodules and set up a conda environment with dependencies. Requires CUDA >= 10.1 and PyTorch >= 1.8.

```bash
git clone --recursive https://github.com/princeton-vl/lietorch.git
cd lietorch

conda create -n lie_env
conda activate lie_env
conda install scipy pyyaml pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
```

--------------------------------

### Get Tensor dimensions with dimensions()

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/unsupported/Eigen/CXX11/src/Tensor/README.md

Shows how to get the dimensions of a Tensor using dimensions() and access the size and individual dimension values. The output is printed to cout.

```cpp
Eigen::Tensor<float, 2> a(3, 4);
const Eigen::Tensor<float, 2>::Dimensions& d = a.dimensions();
cout << "Dim size: " << d.size << ", dim 0: " << d[0]
     << ", dim 1: " << d[1];
=> Dim size: 2, dim 0: 3, dim 1: 4
```

--------------------------------

### N-D Scaling initialization

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/TutorialGeometry.dox

Shows the various ways to construct an N-D scaling transformation.

```cpp
Scaling(sx, sy)
Scaling(sx, sy, sz)
Scaling(s)
Scaling(vecN)
```

--------------------------------

### Compile camera tracking extensions

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/README.md

Compiles the extensions for the camera tracking module. Run from the project root.

```bash
cd base; python setup.py install
```

--------------------------------

### Transpose aliasing example

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/eigen/doc/TopicAliasing.dox

Example from TutorialMatrixArithmetic showing aliasing when transposing a matrix in-place. The output shows the issue, and by default Eigen uses a run-time assertion to detect it.

```cpp
#include <iostream>
#include <Eigen/Dense>

using namespace Eigen;

int main()
{
  Matrix2i a; a << 1, 2, 3, 4;
  std::cout << "Here is the matrix a:\n" << a << std::endl;

  a = a.transpose(); // !!! do NOT do this !!!
  std::cout << "and the result of the aliasing effect:\n" << a << std::endl;
}
```

```text
Here is the matrix a:
1 2
3 4
and the result of the aliasing effect:
1 2
2 4
```

--------------------------------

### Initialization

Source: https://github.com/zbw001/tapip3d/blob/main/third_party/megasam/base/thirdparty/lietorch/eigen/doc/AsciiQuickReference.txt

Initialize matrices and vectors with comma-separated values, fill with a constant, or use static factory methods.

```APIDOC
## Initialization

### Description
Initialize matrices and vectors using comma-initializers, fill with a constant, or create identity, zero, ones, random, and linearly spaced matrices/vectors.

### Methods
- `A << ...` - Comma-initializer to set matrix elements row by row.
- `A.fill(value)` - Fill all elements of `A` with the given value.
- `MatrixXd::Identity(rows, cols)` - Create an identity matrix of given size.
- `C.setIdentity(rows, cols)` - Resize and set `C` to identity.
- `MatrixXd::Zero(rows, cols)` - Create a zero matrix.
- `C.setZero(rows, cols)` - Resize and set `C` to zero.
- `MatrixXd::Ones(rows, cols)` - Create a matrix of ones.
- `C.setOnes(rows, cols)` - Resize and set `C` to ones.
- `MatrixXd::Random(rows, cols)` - Create a matrix with uniform random numbers in (-1, 1).
- `C.setRandom(rows, cols)` - Resize and set `C` to random values.
- `VectorXd::LinSpaced(size, low, high)` - Create a vector with `size` linearly spaced values from `low` to `high`.
- `v.setLinSpaced(size, low, high)` - Resize and set `v` to linearly spaced values.
- `VectorXi::LinSpaced(size, low, high)` - Create an integer vector with linearly spaced values.

### Example
```cpp
A << 1, 2, 3,
     4, 5, 6,
     7, 8, 9;
B << A, A, A;     // B is three horizontally stacked A's.
A.fill(10);       // Fill A with all 10's.

MatrixXd::Identity(rows,cols);
C.setIdentity(rows,cols);
MatrixXd::Zero(rows,cols);
C.setZero(rows,cols);
MatrixXd::Ones(rows,cols);
C.setOnes(rows,cols);
MatrixXd::Random(rows,cols);
C.setRandom(rows,cols);
VectorXd::LinSpaced(size,low,high);
v.setLinSpaced(size,low,high);
VectorXi::LinSpaced(((hi-low)/step)+1, low, low+step*(size-1));
```
```
Tokens:
Raw