MedPerf

https://docs.medperf.org/
Admin
MedPerf is an open-source framework for benchmarking medical ML models, using federated evaluation...
Tokens:12,015
Snippets:233
Trust Score:7.1
Update:4 months ago
Show doc for...
Tokens:
Raw
### Start MedPerf WebUI Local Server

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

Run this command in your codespaces terminal to start the local web UI server. After execution, a URL will be provided to access the UI in your browser.

```bash
medperf_webui

```

--------------------------------

### Initiate Login via Command Line

Source: https://docs.medperf.org/concepts/auth

Use this command to start the login process from the command line interface. You will be prompted to enter your email address.

```bash
medperf auth login

```

--------------------------------

### Start MedPerf Web UI

Source: https://docs.medperf.org/concepts/confidential_computing

Run this command in your terminal to start the local web user interface for MedPerf.

```bash
medperf_webui
```

--------------------------------

### Install MedPerf from Source

Source: https://docs.medperf.org/getting_started/installation

Install MedPerf using pip in editable mode from the local source code. This command should be run after cloning the repository and navigating into its directory.

```bash
pip install -e ./cli
```

--------------------------------

### Run Local MedPerf Server

Source: https://docs.medperf.org/getting_started/setup

Set up and run a local MedPerf server for tutorial purposes. This involves copying an environment file and executing a setup script. The server runs on localhost and can be stopped with CTRL+C.

```bash
cd server
cp .env.local.local-auth.sqlite .env
sh setup-dev-server.sh
```

--------------------------------

### Check Singularity Version

Source: https://docs.medperf.org/getting_started/installation

Verify if Singularity is installed on your system.

```bash
singularity --version
```

--------------------------------

### Check Docker Version

Source: https://docs.medperf.org/getting_started/installation

Verify if Docker is installed on your system. Ensure you can run Docker as a non-root user if applicable.

```bash
docker --version
```

--------------------------------

### Verify MedPerf Installation

Source: https://docs.medperf.org/getting_started/installation

Check if MedPerf has been installed correctly by running the version command. This confirms that the MedPerf CLI is accessible.

```bash
medperf --version
```

--------------------------------

### Install MedPerf Server Requirements

Source: https://docs.medperf.org/getting_started/setup

Install the necessary Python packages for the MedPerf server and its testing utilities. Ensure you are in the MedPerf root directory before running these commands.

```bash
pip install -r server/requirements.txt
pip install -r server/test-requirements.txt
```

--------------------------------

### Check Python and Pip Versions

Source: https://docs.medperf.org/getting_started/installation

Verify if Python and pip are installed on your system. Use 'python3' and 'pip3' if your machine is configured differently.

```bash
python --version
pip --version
```

```bash
python3 --version
pip3 --version
```

--------------------------------

### Import Dataset (Development)

Source: https://docs.medperf.org/concepts/import_export_data

Import a dataset that is still in development on a new machine. This command will place the raw data in the specified path. Ensure MedPerf is installed on the new machine and the .gz file is accessible.

```bash
medperf dataset import --data_uid YOUR_DATASET_ID --input PATH_TO_THE_GZ_FILE --raw_dataset_path PATH_TO_PLACE_YOUR_RAW_DATA
```

--------------------------------

### Clone MedPerf Repository

Source: https://docs.medperf.org/getting_started/installation

Clone the MedPerf repository from GitHub and navigate into the cloned directory. This is necessary to install MedPerf from source.

```bash
git clone https://github.com/mlcommons/medperf.git
cd medperf
```

--------------------------------

### Import Dataset (Operational)

Source: https://docs.medperf.org/concepts/import_export_data

Import an operational dataset on a new machine. This command only copies the prepared data; raw data is not transferred or required by MedPerf. Ensure MedPerf is installed on the new machine and the .gz file is accessible.

```bash
medperf dataset import --data_uid YOUR_DATASET_ID --input PATH_TO_THE_GZ_FILE
```

--------------------------------

### Activate Conda Virtual Environment

Source: https://docs.medperf.org/getting_started/installation

Activate the 'medperf-env' virtual environment created previously. This ensures MedPerf is installed and run within the isolated environment.

```bash
conda activate medperf-env
```

--------------------------------

### Download Tutorial Files

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Downloads all necessary files to smoothly follow the benchmark tutorial. Ensure you are in the MedPerf root folder before running.

```bash
sh tutorials_scripts/setup_benchmark_tutorial.sh
```

--------------------------------

### Prepare Local MedPerf Server

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Initializes a local MedPerf server with a fresh database and creates necessary entities for the tutorial. Ensure you are in the MedPerf root folder before running.

```bash
cd server
sh reset_db.sh
python seed.py --demo benchmark
cd ..
```

--------------------------------

### Download Tutorial Files

Source: https://docs.medperf.org/getting_started/model_owner_demo

Downloads all necessary files to smoothly follow the model owner tutorial. Ensure you are in MedPerf's root folder before running. This creates a 'medperf_tutorial' workspace.

```bash
sh tutorials_scripts/setup_model_tutorial.sh

```

--------------------------------

### Initialize Local MedPerf Server

Source: https://docs.medperf.org/getting_started/data_owner_demo

Initializes a local MedPerf server with a fresh database and populates it with demo data. Ensure you are in the MedPerf's root folder before running.

```bash
cd server
sh reset_db.sh
python seed.py --demo data
cd ..

```

--------------------------------

### List available models

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Use this command to list the server UIDs for reference models.

```bash
medperf model ls
```

--------------------------------

### List Available Benchmarks

Source: https://docs.medperf.org/getting_started/data_owner_demo

Execute this command to view a list of all available benchmarks. This helps in identifying the correct benchmark ID for your dataset registration.

```bash
medperf benchmark ls

```

--------------------------------

### Run a Benchmark

Source: https://docs.medperf.org/getting_started/data_owner_demo

Execute a benchmark with a specified ID and data UID. This command initiates the benchmark process.

```bash
medperf benchmark run --benchmark 1 --data_uid 1

```

--------------------------------

### List available containers

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Use this command to list the server UIDs for data preparation and metrics containers.

```bash
medperf container ls
```

--------------------------------

### Submit a benchmark

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Submit a benchmark with its description, demo URL, container UIDs, reference model UID, and evaluator UID. The `--operational` flag indicates it's ready for operational use.

```bash
medperf benchmark submit \
   --name tutorial_bmk \
   --description "MedPerf demo bmk" \
   --demo-url "https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo_data.tar.gz" \
   --data-preparation-container 1 \
   --reference-model 1 \
   --evaluator-container 3 \
   --operational
```

--------------------------------

### Register Training Container Configuration

Source: https://docs.medperf.org/getting_started/webui_training

Provide the path to the training container configuration file when registering the training container.

```yaml
medperf_tutorial/fl_container/container_config.yaml
```

--------------------------------

### Submit Model Container with Additional Files (Direct Link)

Source: https://docs.medperf.org/concepts/hosting_files

When submitting a model container, provide the URL of hosted additional files using the --additional-file argument. Ensure files have anonymous public read access.

```bash
--additional-file <the-direct-download-link-to-the-additional-files-tarball>
```

--------------------------------

### List My Submitted Containers

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Check the server UID for each submitted container by listing your own containers.

```bash
medperf container ls --mine
```

--------------------------------

### List My Datasets

Source: https://docs.medperf.org/getting_started/data_owner_demo

Lists all datasets registered by the current user. Use this to find the server UID of your dataset.

```bash
medperf dataset ls --mine

```

--------------------------------

### Demo Dataset URL for Benchmark Registration

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

This is the URL for the demo dataset required when registering a new benchmark.

```text
https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo_data.tar.gz
```

--------------------------------

### Reset Local MedPerf Database and Seed Demo Data

Source: https://docs.medperf.org/getting_started/model_owner_demo

Initializes a local MedPerf server with a fresh database and populates it with demo data for the model owner tutorial. Ensure you are in the MedPerf's root folder before running.

```bash
cd server
sh reset_db.sh
python seed.py --demo model
cd ..

```

--------------------------------

### Submit Model Container with Additional Files (Synapse ID)

Source: https://docs.medperf.org/concepts/hosting_files

When submitting a model container, provide the Synapse ID of hosted additional files with the 'synapse:' prefix. Authentication with Synapse credentials is required.

```bash
--additional-file synapse:<the-synapse-id-of-the-file>
```

--------------------------------

### Register Data Preparation Container Parameters

Source: https://docs.medperf.org/getting_started/webui_training

Specify the path to the data preparation container parameters file during registration.

```yaml
medperf_tutorial/data_preparator/workspace/parameters.yaml
```

--------------------------------

### Provide Training Container Additional Files URL

Source: https://docs.medperf.org/getting_started/webui_training

Specify the URL for the hosted tarball containing additional files, such as initial model weights, for the training container.

```text
https://storage.googleapis.com/medperf-storage/init_weights_flower.tar.gz
```

--------------------------------

### Compress Demo Dataset

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Package the demo data and paths.yaml file into a compressed tarball archive. This command should be run from within the medperf_tutorial directory.

```bash
cd medperf_tutorial
tar -czf demo_data.tar.gz demo_data paths.yaml
cd ..

```

--------------------------------

### List submitted benchmarks

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Check the server UID of your submitted benchmark.

```bash
medperf benchmark ls --mine
```

--------------------------------

### Register Data Preparation Container Configuration

Source: https://docs.medperf.org/getting_started/webui_training

Specify the path to the data preparation container configuration file when registering a new container.

```yaml
medperf_tutorial/data_preparator/container_config.yaml
```

--------------------------------

### Submit Data Preparator Container

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Use this command to submit a data preparator container. Ensure you have the correct paths to the container configuration and parameters files.

```bash
medperf container submit \
    --name my-prep \
    --container-config-file "medperf_tutorial/data_preparator/container_config.yaml" \
    --parameters-file "medperf_tutorial/data_preparator/workspace/parameters.yaml" \
    --operational
```

--------------------------------

### Login to Local MedPerf Server

Source: https://docs.medperf.org/getting_started/model_owner_demo

Logs into the local MedPerf server using a dummy authentication system for tutorial purposes. This command is used when communicating with the local MedPerf server for the tutorial.

```bash
medperf auth login -e testmo@example.com

```

--------------------------------

### Request Model Association with Benchmark

Source: https://docs.medperf.org/getting_started/model_owner_demo

Use this command to initiate an association request for your model with a specific benchmark. It ensures model compatibility and prompts for confirmation.

```bash
medperf model associate --benchmark 1 --model_uid 2

```

--------------------------------

### Prepare Dataset

Source: https://docs.medperf.org/getting_started/data_owner_demo

Prepares a dataset for use with MedPerf. This command calculates statistics defined by the benchmark owner.

```bash
medperf dataset prepare --data_uid 1

```

--------------------------------

### Submit Reference Model Container

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Submit a reference model container, including its configuration, parameters, and hosted additional files like model weights.

```bash
medperf model submit \
--name my-refmodel \
--container-config-file "medperf_tutorial/model_custom_cnn/container_config.yaml" \
--parameters-file "medperf_tutorial/model_custom_cnn/workspace/parameters.yaml" \
--additional-file "https://storage.googleapis.com/medperf-storage/chestxray_tutorial/cnn_weights.tar.gz" \
--operational
```

--------------------------------

### Login to Local MedPerf Server

Source: https://docs.medperf.org/getting_started/data_owner_demo

Logs into the local MedPerf server using a dummy email address for tutorial purposes. This command is used when communicating with the real MedPerf server as well.

```bash
medperf auth login -e testdo@example.com

```

--------------------------------

### Register Dataset Information

Source: https://docs.medperf.org/getting_started/data_owner_demo

Run this command from MedPerf's root folder to register your dataset. Ensure you replace placeholder values with your dataset's specific details and the correct benchmark ID. The data itself remains on your local machine.

```bash
medperf dataset submit \
  --name "mytestdata" \
  --description "A tutorial dataset" \
  --location "My machine" \
  --data_path "medperf_tutorial/sample_raw_data/images" \
  --labels_path "medperf_tutorial/sample_raw_data/labels" \
  --benchmark 1

```

--------------------------------

### Show Local Results

Source: https://docs.medperf.org/getting_started/data_owner_demo

Display the results of a specific benchmark execution using its result UID. This is useful for local inspection before submission.

```bash
medperf result show_local_results 2

```

--------------------------------

### Submit a Result

Source: https://docs.medperf.org/getting_started/data_owner_demo

Submit a benchmark result identified by its UID. This action sends the result for further processing and confirmation.

```bash
medperf result submit --result 2

```

--------------------------------

### Define Data and Labels Paths

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Create a paths.yaml file to specify the location of your data records and labels. This file is used by the Data Preparator container.

```yaml
data_path: demo_data/images
labels_path: demo_data/labels

```

--------------------------------

### Request Dataset Association with Benchmark

Source: https://docs.medperf.org/getting_started/data_owner_demo

Requests to associate a dataset with a specific benchmark. This command first runs the benchmark's reference model on your dataset to ensure compatibility.

```bash
medperf dataset associate --benchmark_uid 1 --data_uid 1

```

--------------------------------

### Container Configuration File Path for Model Registration

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

The path to the container configuration YAML file needed during model registration.

```text
medperf_tutorial/model_mobilenetv2/container_config.yaml
```

--------------------------------

### Check Authentication Status via Command Line

Source: https://docs.medperf.org/concepts/auth

Run this command to verify the current login status of the MedPerf client. The client remembers your login state per profile.

```bash
medperf auth status

```

--------------------------------

### List My Submitted Models

Source: https://docs.medperf.org/getting_started/model_owner_demo

After submitting a model, you can check its assigned server UID by listing your submitted models.

```bash
medperf model ls --mine
```

--------------------------------

### Submit a Model to MedPerf

Source: https://docs.medperf.org/getting_started/model_owner_demo

Use this command to submit your model to MedPerf. Ensure you have hosted the container configuration, parameters, and additional files, and provide their respective paths or URLs.

```bash
medperf model submit \
   --name my-model \
   --container-config-file "medperf_tutorial/model_mobilenetv2/container_config.yaml" \
   --parameters-file "medperf_tutorial/model_mobilenetv2/workspace/parameters.yaml" \
   --additional-file "https://storage.googleapis.com/medperf-storage/chestxray_tutorial/mobilenetv2_weights.tar.gz" \
   --operational
```

--------------------------------

### Export Dataset

Source: https://docs.medperf.org/concepts/import_export_data

Use this command to export your dataset into a .gz file. This file can then be transferred to another machine for import. Ensure you replace YOUR_DATASET_ID and OUTPUT_FOLDER with your specific values.

```bash
medperf dataset export --data_uid YOUR_DATASET_ID --output OUTPUT_FOLDER
```

```bash
medperf dataset export --data_uid 71 --output ./my_exported_dataset
```

--------------------------------

### Parameters File Path for Model Registration

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

The path to the parameters YAML file required when registering a model.

```text
medperf_tutorial/model_mobilenetv2/workspace/parameters.yaml
```

--------------------------------

### Simulate Data Association Approval

Source: https://docs.medperf.org/getting_started/data_owner_demo

Simulates the approval of a data association request by the benchmark committee. This is typically used for tutorial purposes.

```bash
sh tutorials_scripts/simulate_data_association_approval.sh

```

--------------------------------

### Register Reference Model Container Parameters

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

Specifies the path to the parameters file for the Reference Model container.

```yaml
medperf_tutorial/model_custom_cnn/workspace/parameters.yaml
```

--------------------------------

### Register Metrics Container Parameters

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

Specifies the path to the parameters file for the Metrics container.

```yaml
medperf_tutorial/metrics/workspace/parameters.yaml
```

--------------------------------

### Register Reference Model Additional Files

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

Provides the URL to the hosted tarball file containing additional files, such as model weights, for the Reference Model container.

```url
https://storage.googleapis.com/medperf-storage/chestxray_tutorial/cnn_weights.tar.gz
```

--------------------------------

### Register Reference Model Container Configuration

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

Specifies the path to the container configuration file for the Reference Model.

```yaml
medperf_tutorial/model_custom_cnn/container_config.yaml
```

--------------------------------

### Login to Google Cloud Application Default Credentials

Source: https://docs.medperf.org/concepts/confidential_computing

Authenticate your application with Google Cloud using application default credentials. Follow the on-screen prompts to complete the login process.

```bash
gcloud auth application-default login
```

--------------------------------

### Register Metrics Container Configuration

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

Specifies the path to the container configuration file for the Metrics container.

```yaml
medperf_tutorial/metrics/container_config.yaml
```

--------------------------------

### Configure MedPerf Client Platform to Singularity

Source: https://docs.medperf.org/getting_started/setup

Modify the 'platform' parameter of the currently activated MedPerf client profile to use Singularity as the container runner. The default profile is configured for Docker.

```bash
medperf profile set --platform singularity
```

--------------------------------

### View Active MedPerf Profile Configuration

Source: https://docs.medperf.org/getting_started/setup

Display the configuration parameters of the currently active MedPerf client profile. This is useful for reviewing settings like server endpoints.

```bash
medperf profile view
```

--------------------------------

### Compressing Additional Files for Container

Source: https://docs.medperf.org/concepts/container_components

Use this command to compress additional files into a tar.gz archive. The archive should be hosted and its URL provided to the MedPerf submission command.

```bash
tar -czf additional_files.tar.gz -C <path_to_the_additional_files_folder> .
```

--------------------------------

### Set Google Cloud Project ID

Source: https://docs.medperf.org/concepts/confidential_computing

Configure the Google Cloud CLI to use your specific project ID. Replace PROJECT_ID with the actual project ID obtained from your cloud administrator.

```bash
gcloud config set project PROJECT_ID
```

--------------------------------

### Cleanup Downloaded Files

Source: https://docs.medperf.org/getting_started/model_owner_demo

Remove the downloaded files workspace to clean up the MedPerf tutorial environment. Ensure you are in the MedPerf's root directory before running.

```bash
rm -fr medperf_tutorial

```

--------------------------------

### Submit Metrics Container

Source: https://docs.medperf.org/getting_started/benchmark_owner_demo

Submit a metrics container to the MedPerf server. This command requires paths to the container configuration and parameters files.

```bash
medperf container submit \
--name my-metrics \
--container-config-file "medperf_tutorial/metrics/container_config.yaml" \
--parameters-file "medperf_tutorial/metrics/workspace/parameters.yaml" \
--operational
```

--------------------------------

### Mark Dataset as Operational

Source: https://docs.medperf.org/getting_started/data_owner_demo

Marks a dataset as operational, making it available for association with benchmarks. Once operational, a dataset cannot be reverted to the development stage.

```bash
medperf dataset set_operational --data_uid 1

```

--------------------------------

### Perform Logout via Command Line

Source: https://docs.medperf.org/concepts/auth

Execute this command to log out from the MedPerf client using the command line interface.

```bash
medperf auth logout

```

--------------------------------

### Activate Local MedPerf Profile

Source: https://docs.medperf.org/getting_started/setup

Activate the 'local' profile for the MedPerf client to communicate with the local MedPerf server. This profile is preconfigured for local server communication.

```bash
medperf profile activate local
```

--------------------------------

### Reset Local MedPerf Server Database

Source: https://docs.medperf.org/getting_started/model_owner_demo

Clean up the local MedPerf server database. Navigate to the 'server' directory and execute the reset script. Ensure you are in the MedPerf's root directory first.

```bash
cd server
sh reset_db.sh

```

--------------------------------

### List Active MedPerf Profiles

Source: https://docs.medperf.org/getting_started/setup

Check the currently active profile for the MedPerf client. This command helps verify which configuration is currently in use.

```bash
medperf profile ls
```

--------------------------------

### Model Weights Tarball URL for Model Registration

Source: https://docs.medperf.org/getting_started/webui_full_tutorial

The URL for the hosted tarball containing model weights, necessary for model registration.

```text
https://storage.googleapis.com/medperf-storage/chestxray_tutorial/mobilenetv2_weights.tar.gz
```

--------------------------------

### Create Conda Virtual Environment

Source: https://docs.medperf.org/getting_started/installation

Create a new Conda virtual environment named 'medperf-env' with Python 3.11. This is an optional but recommended step for isolating MedPerf.

```bash
conda create -n medperf-env python=3.11
```

--------------------------------

### Cleanup Test Storage

Source: https://docs.medperf.org/getting_started/model_owner_demo

Remove the test storage directory located in the user's home directory. This command cleans up local test artifacts.

```bash
rm -fr ~/.medperf/localhost_8000

```
Tokens:
Raw