### Set up Python 3.10 environment and install fmpose3d Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/README.md Creates a conda environment named fmpose_3d with Python 3.10, activates it, and installs the fmpose3d package. The installation and demos are tested with Python 3.10. ```bash conda create -n fmpose_3d python=3.10 conda activate fmpose_3d pip install fmpose3d ``` -------------------------------- ### Install optional DeepLabCut dependency for animal pipeline Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/README.md Installs the optional DeepLabCut dependency required for the animal pipeline. Use this only if you need animal pose estimation. ```bash pip install "fmpose3d[animals]" ``` -------------------------------- ### Animal pose estimation with FMPose3DInference.for_animals() Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Uses the convenience method to load the animal config and predicts 3D poses for animal images. Output shape is (1, 26, 3). Requires the optional DeepLabCut dependency, installed via pip install "fmpose3d[animals]". ```python from fmpose3d import FMPose3DInference # The api has a convenience method for loading directly with the animal config api = FMPose3DInference.for_animals() result = api.predict("dog.jpg") print(result.poses_3d.shape) # (1, 26, 3) ``` -------------------------------- ### setup_runtime() Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Manually initializes all runtime components (2D estimator, 3D model, weights). Called automatically on first use of other methods. ```APIDOC ## setup_runtime ### Description Initializes the 2D estimator, 3D model, and loads weights. This is called automatically on first use of `predict`, `prepare_2d`, or `pose_3d`, but can be invoked manually. ### Parameters None ### Returns None ### Example ```python inference.setup_runtime() ``` ``` -------------------------------- ### Run in-the-wild visualization script for humans Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/README.md Runs the visualization script for single-frame 3D human pose estimation on images in the demo/images folder. Pre-trained weights are downloaded automatically from Hugging Face on first run, or you can place your own weights in ./pre_trained_models and set model_weights_path in the script. Predictions are saved to demo/predictions. ```bash sh vis_in_the_wild.sh ``` -------------------------------- ### Train FMPose3D on Human3.6M Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/README.md Run the training script for Human3.6M. Training logs, checkpoints, and related files are saved in the './checkpoint' folder. ```bash sh ./scripts/FMPose3D_train.sh ``` -------------------------------- ### Dataset directory structure Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/README.md Place the downloaded preprocessed dataset files in the dataset/ folder of the project root. The required files are data_3d_h36m.npz, data_2d_h36m_gt.npz, and data_2d_h36m_cpn_ft_h36m_dbb.npz. ```text / ├── dataset/ │ ├── data_3d_h36m.npz │ ├── data_2d_h36m_gt.npz │ └── data_2d_h36m_cpn_ft_h36m_dbb.npz ``` -------------------------------- ### Run inference with pretrained model Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/animals/README.md Download the pretrained model and place it in `./pre_trained_models`, then run the test script from the `animals` directory. ```bash cd animals # the current path is: ./animals bash ./scripts/test_animal3d.sh ``` -------------------------------- ### HRNetEstimator.setup_runtime Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Loads the YOLO and HRNet models for the human 2D estimator. ```APIDOC ## HRNetEstimator.setup_runtime ### Description Loads the YOLO and HRNet models for the human 2D estimator. ### Method setup_runtime ### Parameters None ### Returns None ``` -------------------------------- ### Run visualization script for in-the-wild animal images Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/animals/README.md Place test images in `demo/images` and run `sh vis_animals.sh` to perform 3D animal pose estimation. Predictions are saved to `demo/predictions`. Pre-trained checkpoints are auto-downloaded from Hugging Face on first run. ```bash sh vis_animals.sh ``` -------------------------------- ### Train FMPose3D on Animal3D and Control_Animal3D Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/animals/README.md Run the training script from the `animals` directory. Training logs, checkpoints, and related files are saved in the `./checkpoint` folder. ```bash cd animals bash ./scripts/train_animal3d.sh ``` -------------------------------- ### SuperAnimalEstimator.setup_runtime Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md No-op; DeepLabCut loads lazily. ```APIDOC ## SuperAnimalEstimator.setup_runtime ### Description No-op; DeepLabCut loads lazily. ### Method setup_runtime ### Parameters None ### Returns None ``` -------------------------------- ### Download official MPI-INF-3DHP test set Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/3dhp_test/README.md Run the test-set downloader from the official package after editing source/conf.ig as instructed. ```bash cd /path/to/mpi_inf_3dhp/source bash get_testset.sh ``` -------------------------------- ### Run inference on Human3.6M Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/README.md Run the inference script for Human3.6M. Pre-trained weights are fetched automatically from Hugging Face on first run; you can use local weights by setting model_weights_path in the shell script. ```bash sh ./scripts/FMPose3D_test.sh ``` -------------------------------- ### FMPose3DInference Constructor Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Creates an instance of the main inference class, managing the full pipeline from input ingestion to 3D pose lifting. ```APIDOC ## Constructor ### Description Initializes the FMPose3DInference object with optional configuration, model weights, device, and overrides for the 2D estimator and postprocessor. ### Parameters - **model_cfg** (FMPose3DConfig | None) - Optional - Model architecture settings. Defaults to human (17 H36M joints). - **inference_cfg** (InferenceConfig | None) - Optional - Inference settings (sample steps, test augmentation, etc.). - **model_weights_path** (str | Path | None) - Optional - Path to a `.pth` checkpoint. `None` triggers automatic download from HuggingFace. - **device** (str | torch.device | None) - Optional - Compute device. `None` auto-selects CUDA if available. - **estimator_2d** (HRNetEstimator | SuperAnimalEstimator | None) - Optional - Override the 2D pose estimator (auto-selected by default). - **postprocessor** (HumanPostProcessor | AnimalPostProcessor | None) - Optional - Override the post-processor (auto-selected by default). ### Example ```python FMPose3DInference( model_cfg: FMPose3DConfig | None = None, inference_cfg: InferenceConfig | None = None, model_weights_path: str | Path | None = None, device: str | torch.device | None = None, *, estimator_2d: HRNetEstimator | SuperAnimalEstimator | None = None, postprocessor: HumanPostProcessor | AnimalPostProcessor | None = None, ) ``` ``` -------------------------------- ### Dataset folder structure for Animal3D and Control_Animal3D Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/animals/README.md Place downloaded Animal3D and control_animal3D datasets in the `dataset/` folder with the specified structure. The `images` and JSON files are required for training and testing. ```text / ├── dataset/ │ ├── animal3d │ ├── images │ ├── train.json │ ├── test.json │ ├── control_animal3d │ ├── images │ ├── train.json │ ├── test.json ``` -------------------------------- ### prepare_2d(source, progress) → Pose2DResult Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Runs only the 2D pose estimation step, returning keypoints, scores, image size, and a validity mask. ```APIDOC ## prepare_2d ### Description Performs 2D pose estimation on the input source, returning a `Pose2DResult` with keypoints, scores, image size, and a validity mask. ### Parameters - **source** (Source) - Required - Same flexible input as `predict()`. - **progress** (ProgressCallback | None) - Optional - Optional progress callback. ### Returns - **Pose2DResult** - Contains `keypoints`, `scores`, `image_size`, and `valid_frames_mask`. Also exposes `status` and `status_message` properties. ### Example ```python result_2d = inference.prepare_2d(source, progress=callback) ``` ``` -------------------------------- ### Human pose estimation (two-step) with FMPose3DInference Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Initializes the API with a local model weights path and calls the 2D and 3D inference steps separately. The 2D step returns keypoints and image size, which are passed to the 3D step. ```python from fmpose3d import FMPose3DInference api = FMPose3DInference(model_weights_path="weights.pth") # The 2D and 3D inference step can be called separately result_2d = api.prepare_2d("photo.jpg") result_3d = api.pose_3d(result_2d.keypoints, result_2d.image_size) ``` -------------------------------- ### pose_3d(keypoints_2d, image_size, *, camera_rotation, seed, progress) → Pose3DResult Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Lifts pre-computed 2D keypoints to 3D using the flow-matching model. ```APIDOC ## pose_3d ### Description Converts 2D keypoints to 3D poses using the flow-matching model. Accepts keypoints for one or multiple persons; if multiple, the first person is used. ### Parameters - **keypoints_2d** (ndarray) - Required - Shape `(num_persons, num_frames, J, 2)` or `(num_frames, J, 2)`. First person is used if 4D. - **image_size** (tuple[int, int]) - Required - `(height, width)` of the source frames. - **camera_rotation** (ndarray | None) - Optional - Camera-to-world quaternion (human only). - **seed** (int | None) - Optional - Seed for reproducible results. - **progress** (ProgressCallback | None) - Optional - Per-frame progress callback. ### Returns - **Pose3DResult** - The 3D pose estimation result. ### Example ```python result_3d = inference.pose_3d(keypoints_2d, image_size, camera_rotation=quat, seed=42, progress=callback) ``` ``` -------------------------------- ### FMPose3DInference.prepare_2d Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Prepares 2D pose estimation from the provided source. Accepts the same source types as predict. ```APIDOC ## FMPose3DInference.prepare_2d ### Description Prepares 2D pose estimation from the provided source. Accepts the same source types as predict. ### Method prepare_2d ### Parameters #### Source - **source** (Source) - Required - A path to an image file or directory, a numpy array of shape (H, W, C) or (N, H, W, C), or a list of file paths or arrays. ### Returns - **Pose2DResult** - The 2D pose estimation result. ``` -------------------------------- ### Human pose estimation (end-to-end) with FMPose3DInference Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Initializes the API with a config (defaults to human model) and predicts 3D poses from an image path or array. Weights are auto-downloaded from HuggingFace. Output shapes are (1, 17, 3) for both camera and world coordinates. ```python from fmpose3d import FMPose3DInference, FMPose3DConfig # Create a config (optional) config = FMPose3DConfig(model_type="fmpose3d_humans") # or "fmpose3d_animals" # Initialize the API api = FMPose3DInference(config) # weights auto-downloaded # Predict from source (path, or an image array) result = api.predict("photo.jpg") print(result.poses_3d.shape) # (1, 17, 3) print(result.poses_3d_world.shape) # (1, 17, 3) ``` -------------------------------- ### FMPose3DInference.for_animals(...) — Class Method Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Convenience constructor for the animal pipeline, setting the model type to 'fmpose3d_animals' and loading the appropriate config. ```APIDOC ## Class Method: for_animals ### Description Creates an FMPose3DInference instance configured for animal pose estimation, using a 26-joint Animal3D skeleton and disabling flip augmentation by default. ### Parameters - **model_weights_path** (str | None) - Optional - Path to a `.pth` checkpoint. `None` triggers automatic download. - **device** (str | torch.device | None) - Optional - Compute device. `None` auto-selects CUDA if available. - **inference_cfg** (InferenceConfig | None) - Optional - Inference settings. ### Returns - **FMPose3DInference** - Configured instance for animal pose estimation. ### Example ```python @classmethod def for_animals( cls, model_weights_path: str | None = None, *, device: str | torch.device | None = None, inference_cfg: InferenceConfig | None = None, ) -> FMPose3DInference ``` ``` -------------------------------- ### Verify annot_data.mat presence for all subjects Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/3dhp_test/README.md Checks that each subject directory contains annot_data.mat and prints ok or missing. ```bash for subject in TS1 TS2 TS3 TS4 TS5 TS6; do test -f "${MPI_INF_3DHP_TEST_ROOT}/${subject}/annot_data.mat" \ && echo "${subject}: ok" \ || echo "${subject}: missing annot_data.mat" done ``` -------------------------------- ### Use local weights for 3DHP test Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/3dhp_test/README.md Set model_weights_path to a local checkpoint, such as the human pretrained weights provided on Google Drive. ```bash model_weights_path="${SCRIPT_DIR}/pretrained/fmpose3d_h36m/FMpose3D_pretrained_weights.pth" ``` -------------------------------- ### Set model type and weights for 3DHP test Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/3dhp_test/README.md Default configuration for the 3DHP test uses the packaged human FMPose3D lifting model and leaves weights empty for automatic download from Hugging Face Hub. ```bash model_type="fmpose3d_humans" model_weights_path="" model_path="" ``` -------------------------------- ### Generate data_test_3dhp.npz Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/3dhp_test/README.md Run from the repository root to generate the processed npz file from the official test set. ```bash python 3dhp_test/dataset/prepare_3dhp_test_npz.py \ --test-root "${MPI_INF_3DHP_TEST_ROOT}" \ --output 3dhp_test/dataset/data_test_3dhp.npz ``` -------------------------------- ### Source type alias for FMPose3DInference.predict and prepare_2d Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Defines the accepted source types for the inference API: a path, an ndarray, or a sequence of these. Use this alias to type-check inputs to predict and prepare_2d. ```python Source = Union[str, Path, np.ndarray, Sequence[Union[str, Path, np.ndarray]]] ``` -------------------------------- ### FMPose3DInference.predict Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Runs 2D and 3D pose estimation on the provided source. Accepts image paths, directories, numpy arrays, or lists of these. ```APIDOC ## FMPose3DInference.predict ### Description Runs 2D and 3D pose estimation on the provided source. Accepts image paths, directories, numpy arrays, or lists of these. ### Method predict ### Parameters #### Source - **source** (Source) - Required - A path to an image file or directory, a numpy array of shape (H, W, C) or (N, H, W, C), or a list of file paths or arrays. ### Returns - **Pose3DResult** - The 3D pose estimation result. ``` -------------------------------- ### Use local model definition for 3DHP test Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/3dhp_test/README.md Set model_path to a Python file that defines Model, for using a local model definition instead of the packaged registry model. ```bash model_path="${SCRIPT_DIR}/pretrained/fmpose3d_h36m/model_GAMLP.py" ``` -------------------------------- ### Expected processed 3DHP test file Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/3dhp_test/README.md infer_3dhp.py expects the processed MPI-INF-3DHP test file at this path. The .npz file is not committed and must be generated. ```bash 3dhp_test/dataset/data_test_3dhp.npz ``` -------------------------------- ### Schema of generated npz file Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/3dhp_test/README.md The generated npz contains per-subject data with 2D, 3D, and valid-frame annotations. ```text data = { "TS1": { "data_2d": annot2, "data_3d": univ_annot3, "valid": valid_frame, }, ... "TS6": ... } ``` -------------------------------- ### HRNetEstimator.predict Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Returns H36M-format 2D keypoints from BGR frames plus a frame-level validity mask. ```APIDOC ## HRNetEstimator.predict ### Description Returns H36M-format 2D keypoints from BGR frames plus a frame-level validity mask. ### Method predict ### Parameters #### frames - **frames** (ndarray) - Required - BGR frames of shape (N, H, W, C). ### Returns - **keypoints** (ndarray) - H36M-format 2D keypoints. - **scores** (ndarray) - Per-joint confidence scores. - **valid_frames_mask** (ndarray) - Frame-level validity mask. ``` -------------------------------- ### SuperAnimalEstimator.predict Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md Returns Animal3D-format 2D keypoints plus a frame-level validity mask. ```APIDOC ## SuperAnimalEstimator.predict ### Description Returns Animal3D-format 2D keypoints plus a frame-level validity mask. ### Method predict ### Parameters #### frames - **frames** (ndarray) - Required - BGR frames of shape (N, H, W, C). ### Returns - **keypoints** (ndarray) - Animal3D-format 2D keypoints. - **scores** (ndarray) - Per-joint confidence scores. - **valid_frames_mask** (ndarray) - Frame-level validity mask. ``` -------------------------------- ### predict(source, *, camera_rotation, seed, progress) → Pose3DResult Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/fmpose3d/inference_api/README.md End-to-end prediction: performs 2D pose estimation followed by 3D lifting in a single call. ```APIDOC ## predict ### Description Runs the full inference pipeline: 2D estimation and 3D lifting. Raises `ValueError` if 2D estimation is unusable for lifting (status is `EMPTY` or `INVALID`). For partial detections, invalid frames are masked to `NaN` in the 3D results. ### Parameters - **source** (Source) - Required - Image path, directory, numpy array `(H,W,C)` or `(N,H,W,C)`, or list thereof. Video files are not supported. - **camera_rotation** (ndarray | None) - Optional - Length-4 quaternion for camera-to-world rotation. Defaults to official demo rotation. `None` skips the transform. Ignored for animals. - **seed** (int | None) - Optional - Seed for reproducible sampling. - **progress** (ProgressCallback | None) - Optional - Callback `(current_step, total_steps) -> None`. ### Returns - **Pose3DResult** - The 3D pose estimation result. ### Example ```python result = inference.predict(source, camera_rotation=quat, seed=42, progress=callback) ``` ``` -------------------------------- ### Expected test-set directory structure Source: https://github.com/adaptivemotorcontrollab/fmpose3d/blob/main/3dhp_test/README.md After extraction, the test-set root should contain TS1 through TS6, each with annot_data.mat. ```text /path/to/mpi_inf_3dhp/ mpi_inf_3dhp_test_set/ TS1/ annot_data.mat ... TS2/ annot_data.mat ... ... TS6/ annot_data.mat ... ```