### Start MedPerf WebUI Local Server Source: https://docs.medperf.org/getting_started/webui_full_tutorial Run this command in your codespaces terminal to start the local web UI server. After execution, a URL will be provided to access the UI in your browser. ```bash medperf_webui ``` -------------------------------- ### Initiate Login via Command Line Source: https://docs.medperf.org/concepts/auth Use this command to start the login process from the command line interface. You will be prompted to enter your email address. ```bash medperf auth login ``` -------------------------------- ### Start MedPerf Web UI Source: https://docs.medperf.org/concepts/confidential_computing Run this command in your terminal to start the local web user interface for MedPerf. ```bash medperf_webui ``` -------------------------------- ### Install MedPerf from Source Source: https://docs.medperf.org/getting_started/installation Install MedPerf using pip in editable mode from the local source code. This command should be run after cloning the repository and navigating into its directory. ```bash pip install -e ./cli ``` -------------------------------- ### Run Local MedPerf Server Source: https://docs.medperf.org/getting_started/setup Set up and run a local MedPerf server for tutorial purposes. This involves copying an environment file and executing a setup script. The server runs on localhost and can be stopped with CTRL+C. ```bash cd server cp .env.local.local-auth.sqlite .env sh setup-dev-server.sh ``` -------------------------------- ### Check Singularity Version Source: https://docs.medperf.org/getting_started/installation Verify if Singularity is installed on your system. ```bash singularity --version ``` -------------------------------- ### Check Docker Version Source: https://docs.medperf.org/getting_started/installation Verify if Docker is installed on your system. Ensure you can run Docker as a non-root user if applicable. ```bash docker --version ``` -------------------------------- ### Verify MedPerf Installation Source: https://docs.medperf.org/getting_started/installation Check if MedPerf has been installed correctly by running the version command. This confirms that the MedPerf CLI is accessible. ```bash medperf --version ``` -------------------------------- ### Install MedPerf Server Requirements Source: https://docs.medperf.org/getting_started/setup Install the necessary Python packages for the MedPerf server and its testing utilities. Ensure you are in the MedPerf root directory before running these commands. ```bash pip install -r server/requirements.txt pip install -r server/test-requirements.txt ``` -------------------------------- ### Check Python and Pip Versions Source: https://docs.medperf.org/getting_started/installation Verify if Python and pip are installed on your system. Use 'python3' and 'pip3' if your machine is configured differently. ```bash python --version pip --version ``` ```bash python3 --version pip3 --version ``` -------------------------------- ### Import Dataset (Development) Source: https://docs.medperf.org/concepts/import_export_data Import a dataset that is still in development on a new machine. This command will place the raw data in the specified path. Ensure MedPerf is installed on the new machine and the .gz file is accessible. ```bash medperf dataset import --data_uid YOUR_DATASET_ID --input PATH_TO_THE_GZ_FILE --raw_dataset_path PATH_TO_PLACE_YOUR_RAW_DATA ``` -------------------------------- ### Clone MedPerf Repository Source: https://docs.medperf.org/getting_started/installation Clone the MedPerf repository from GitHub and navigate into the cloned directory. This is necessary to install MedPerf from source. ```bash git clone https://github.com/mlcommons/medperf.git cd medperf ``` -------------------------------- ### Import Dataset (Operational) Source: https://docs.medperf.org/concepts/import_export_data Import an operational dataset on a new machine. This command only copies the prepared data; raw data is not transferred or required by MedPerf. Ensure MedPerf is installed on the new machine and the .gz file is accessible. ```bash medperf dataset import --data_uid YOUR_DATASET_ID --input PATH_TO_THE_GZ_FILE ``` -------------------------------- ### Activate Conda Virtual Environment Source: https://docs.medperf.org/getting_started/installation Activate the 'medperf-env' virtual environment created previously. This ensures MedPerf is installed and run within the isolated environment. ```bash conda activate medperf-env ``` -------------------------------- ### Download Tutorial Files Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Downloads all necessary files to smoothly follow the benchmark tutorial. Ensure you are in the MedPerf root folder before running. ```bash sh tutorials_scripts/setup_benchmark_tutorial.sh ``` -------------------------------- ### Prepare Local MedPerf Server Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Initializes a local MedPerf server with a fresh database and creates necessary entities for the tutorial. Ensure you are in the MedPerf root folder before running. ```bash cd server sh reset_db.sh python seed.py --demo benchmark cd .. ``` -------------------------------- ### Download Tutorial Files Source: https://docs.medperf.org/getting_started/model_owner_demo Downloads all necessary files to smoothly follow the model owner tutorial. Ensure you are in MedPerf's root folder before running. This creates a 'medperf_tutorial' workspace. ```bash sh tutorials_scripts/setup_model_tutorial.sh ``` -------------------------------- ### Initialize Local MedPerf Server Source: https://docs.medperf.org/getting_started/data_owner_demo Initializes a local MedPerf server with a fresh database and populates it with demo data. Ensure you are in the MedPerf's root folder before running. ```bash cd server sh reset_db.sh python seed.py --demo data cd .. ``` -------------------------------- ### List available models Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Use this command to list the server UIDs for reference models. ```bash medperf model ls ``` -------------------------------- ### List Available Benchmarks Source: https://docs.medperf.org/getting_started/data_owner_demo Execute this command to view a list of all available benchmarks. This helps in identifying the correct benchmark ID for your dataset registration. ```bash medperf benchmark ls ``` -------------------------------- ### Run a Benchmark Source: https://docs.medperf.org/getting_started/data_owner_demo Execute a benchmark with a specified ID and data UID. This command initiates the benchmark process. ```bash medperf benchmark run --benchmark 1 --data_uid 1 ``` -------------------------------- ### List available containers Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Use this command to list the server UIDs for data preparation and metrics containers. ```bash medperf container ls ``` -------------------------------- ### Submit a benchmark Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Submit a benchmark with its description, demo URL, container UIDs, reference model UID, and evaluator UID. The `--operational` flag indicates it's ready for operational use. ```bash medperf benchmark submit \ --name tutorial_bmk \ --description "MedPerf demo bmk" \ --demo-url "https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo_data.tar.gz" \ --data-preparation-container 1 \ --reference-model 1 \ --evaluator-container 3 \ --operational ``` -------------------------------- ### Register Training Container Configuration Source: https://docs.medperf.org/getting_started/webui_training Provide the path to the training container configuration file when registering the training container. ```yaml medperf_tutorial/fl_container/container_config.yaml ``` -------------------------------- ### Submit Model Container with Additional Files (Direct Link) Source: https://docs.medperf.org/concepts/hosting_files When submitting a model container, provide the URL of hosted additional files using the --additional-file argument. Ensure files have anonymous public read access. ```bash --additional-file ``` -------------------------------- ### List My Submitted Containers Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Check the server UID for each submitted container by listing your own containers. ```bash medperf container ls --mine ``` -------------------------------- ### List My Datasets Source: https://docs.medperf.org/getting_started/data_owner_demo Lists all datasets registered by the current user. Use this to find the server UID of your dataset. ```bash medperf dataset ls --mine ``` -------------------------------- ### Demo Dataset URL for Benchmark Registration Source: https://docs.medperf.org/getting_started/webui_full_tutorial This is the URL for the demo dataset required when registering a new benchmark. ```text https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo_data.tar.gz ``` -------------------------------- ### Reset Local MedPerf Database and Seed Demo Data Source: https://docs.medperf.org/getting_started/model_owner_demo Initializes a local MedPerf server with a fresh database and populates it with demo data for the model owner tutorial. Ensure you are in the MedPerf's root folder before running. ```bash cd server sh reset_db.sh python seed.py --demo model cd .. ``` -------------------------------- ### Submit Model Container with Additional Files (Synapse ID) Source: https://docs.medperf.org/concepts/hosting_files When submitting a model container, provide the Synapse ID of hosted additional files with the 'synapse:' prefix. Authentication with Synapse credentials is required. ```bash --additional-file synapse: ``` -------------------------------- ### Register Data Preparation Container Parameters Source: https://docs.medperf.org/getting_started/webui_training Specify the path to the data preparation container parameters file during registration. ```yaml medperf_tutorial/data_preparator/workspace/parameters.yaml ``` -------------------------------- ### Provide Training Container Additional Files URL Source: https://docs.medperf.org/getting_started/webui_training Specify the URL for the hosted tarball containing additional files, such as initial model weights, for the training container. ```text https://storage.googleapis.com/medperf-storage/init_weights_flower.tar.gz ``` -------------------------------- ### Compress Demo Dataset Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Package the demo data and paths.yaml file into a compressed tarball archive. This command should be run from within the medperf_tutorial directory. ```bash cd medperf_tutorial tar -czf demo_data.tar.gz demo_data paths.yaml cd .. ``` -------------------------------- ### List submitted benchmarks Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Check the server UID of your submitted benchmark. ```bash medperf benchmark ls --mine ``` -------------------------------- ### Register Data Preparation Container Configuration Source: https://docs.medperf.org/getting_started/webui_training Specify the path to the data preparation container configuration file when registering a new container. ```yaml medperf_tutorial/data_preparator/container_config.yaml ``` -------------------------------- ### Submit Data Preparator Container Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Use this command to submit a data preparator container. Ensure you have the correct paths to the container configuration and parameters files. ```bash medperf container submit \ --name my-prep \ --container-config-file "medperf_tutorial/data_preparator/container_config.yaml" \ --parameters-file "medperf_tutorial/data_preparator/workspace/parameters.yaml" \ --operational ``` -------------------------------- ### Login to Local MedPerf Server Source: https://docs.medperf.org/getting_started/model_owner_demo Logs into the local MedPerf server using a dummy authentication system for tutorial purposes. This command is used when communicating with the local MedPerf server for the tutorial. ```bash medperf auth login -e testmo@example.com ``` -------------------------------- ### Request Model Association with Benchmark Source: https://docs.medperf.org/getting_started/model_owner_demo Use this command to initiate an association request for your model with a specific benchmark. It ensures model compatibility and prompts for confirmation. ```bash medperf model associate --benchmark 1 --model_uid 2 ``` -------------------------------- ### Prepare Dataset Source: https://docs.medperf.org/getting_started/data_owner_demo Prepares a dataset for use with MedPerf. This command calculates statistics defined by the benchmark owner. ```bash medperf dataset prepare --data_uid 1 ``` -------------------------------- ### Submit Reference Model Container Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Submit a reference model container, including its configuration, parameters, and hosted additional files like model weights. ```bash medperf model submit \ --name my-refmodel \ --container-config-file "medperf_tutorial/model_custom_cnn/container_config.yaml" \ --parameters-file "medperf_tutorial/model_custom_cnn/workspace/parameters.yaml" \ --additional-file "https://storage.googleapis.com/medperf-storage/chestxray_tutorial/cnn_weights.tar.gz" \ --operational ``` -------------------------------- ### Login to Local MedPerf Server Source: https://docs.medperf.org/getting_started/data_owner_demo Logs into the local MedPerf server using a dummy email address for tutorial purposes. This command is used when communicating with the real MedPerf server as well. ```bash medperf auth login -e testdo@example.com ``` -------------------------------- ### Register Dataset Information Source: https://docs.medperf.org/getting_started/data_owner_demo Run this command from MedPerf's root folder to register your dataset. Ensure you replace placeholder values with your dataset's specific details and the correct benchmark ID. The data itself remains on your local machine. ```bash medperf dataset submit \ --name "mytestdata" \ --description "A tutorial dataset" \ --location "My machine" \ --data_path "medperf_tutorial/sample_raw_data/images" \ --labels_path "medperf_tutorial/sample_raw_data/labels" \ --benchmark 1 ``` -------------------------------- ### Show Local Results Source: https://docs.medperf.org/getting_started/data_owner_demo Display the results of a specific benchmark execution using its result UID. This is useful for local inspection before submission. ```bash medperf result show_local_results 2 ``` -------------------------------- ### Submit a Result Source: https://docs.medperf.org/getting_started/data_owner_demo Submit a benchmark result identified by its UID. This action sends the result for further processing and confirmation. ```bash medperf result submit --result 2 ``` -------------------------------- ### Define Data and Labels Paths Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Create a paths.yaml file to specify the location of your data records and labels. This file is used by the Data Preparator container. ```yaml data_path: demo_data/images labels_path: demo_data/labels ``` -------------------------------- ### Request Dataset Association with Benchmark Source: https://docs.medperf.org/getting_started/data_owner_demo Requests to associate a dataset with a specific benchmark. This command first runs the benchmark's reference model on your dataset to ensure compatibility. ```bash medperf dataset associate --benchmark_uid 1 --data_uid 1 ``` -------------------------------- ### Container Configuration File Path for Model Registration Source: https://docs.medperf.org/getting_started/webui_full_tutorial The path to the container configuration YAML file needed during model registration. ```text medperf_tutorial/model_mobilenetv2/container_config.yaml ``` -------------------------------- ### Check Authentication Status via Command Line Source: https://docs.medperf.org/concepts/auth Run this command to verify the current login status of the MedPerf client. The client remembers your login state per profile. ```bash medperf auth status ``` -------------------------------- ### List My Submitted Models Source: https://docs.medperf.org/getting_started/model_owner_demo After submitting a model, you can check its assigned server UID by listing your submitted models. ```bash medperf model ls --mine ``` -------------------------------- ### Submit a Model to MedPerf Source: https://docs.medperf.org/getting_started/model_owner_demo Use this command to submit your model to MedPerf. Ensure you have hosted the container configuration, parameters, and additional files, and provide their respective paths or URLs. ```bash medperf model submit \ --name my-model \ --container-config-file "medperf_tutorial/model_mobilenetv2/container_config.yaml" \ --parameters-file "medperf_tutorial/model_mobilenetv2/workspace/parameters.yaml" \ --additional-file "https://storage.googleapis.com/medperf-storage/chestxray_tutorial/mobilenetv2_weights.tar.gz" \ --operational ``` -------------------------------- ### Export Dataset Source: https://docs.medperf.org/concepts/import_export_data Use this command to export your dataset into a .gz file. This file can then be transferred to another machine for import. Ensure you replace YOUR_DATASET_ID and OUTPUT_FOLDER with your specific values. ```bash medperf dataset export --data_uid YOUR_DATASET_ID --output OUTPUT_FOLDER ``` ```bash medperf dataset export --data_uid 71 --output ./my_exported_dataset ``` -------------------------------- ### Parameters File Path for Model Registration Source: https://docs.medperf.org/getting_started/webui_full_tutorial The path to the parameters YAML file required when registering a model. ```text medperf_tutorial/model_mobilenetv2/workspace/parameters.yaml ``` -------------------------------- ### Simulate Data Association Approval Source: https://docs.medperf.org/getting_started/data_owner_demo Simulates the approval of a data association request by the benchmark committee. This is typically used for tutorial purposes. ```bash sh tutorials_scripts/simulate_data_association_approval.sh ``` -------------------------------- ### Register Reference Model Container Parameters Source: https://docs.medperf.org/getting_started/webui_full_tutorial Specifies the path to the parameters file for the Reference Model container. ```yaml medperf_tutorial/model_custom_cnn/workspace/parameters.yaml ``` -------------------------------- ### Register Metrics Container Parameters Source: https://docs.medperf.org/getting_started/webui_full_tutorial Specifies the path to the parameters file for the Metrics container. ```yaml medperf_tutorial/metrics/workspace/parameters.yaml ``` -------------------------------- ### Register Reference Model Additional Files Source: https://docs.medperf.org/getting_started/webui_full_tutorial Provides the URL to the hosted tarball file containing additional files, such as model weights, for the Reference Model container. ```url https://storage.googleapis.com/medperf-storage/chestxray_tutorial/cnn_weights.tar.gz ``` -------------------------------- ### Register Reference Model Container Configuration Source: https://docs.medperf.org/getting_started/webui_full_tutorial Specifies the path to the container configuration file for the Reference Model. ```yaml medperf_tutorial/model_custom_cnn/container_config.yaml ``` -------------------------------- ### Login to Google Cloud Application Default Credentials Source: https://docs.medperf.org/concepts/confidential_computing Authenticate your application with Google Cloud using application default credentials. Follow the on-screen prompts to complete the login process. ```bash gcloud auth application-default login ``` -------------------------------- ### Register Metrics Container Configuration Source: https://docs.medperf.org/getting_started/webui_full_tutorial Specifies the path to the container configuration file for the Metrics container. ```yaml medperf_tutorial/metrics/container_config.yaml ``` -------------------------------- ### Configure MedPerf Client Platform to Singularity Source: https://docs.medperf.org/getting_started/setup Modify the 'platform' parameter of the currently activated MedPerf client profile to use Singularity as the container runner. The default profile is configured for Docker. ```bash medperf profile set --platform singularity ``` -------------------------------- ### View Active MedPerf Profile Configuration Source: https://docs.medperf.org/getting_started/setup Display the configuration parameters of the currently active MedPerf client profile. This is useful for reviewing settings like server endpoints. ```bash medperf profile view ``` -------------------------------- ### Compressing Additional Files for Container Source: https://docs.medperf.org/concepts/container_components Use this command to compress additional files into a tar.gz archive. The archive should be hosted and its URL provided to the MedPerf submission command. ```bash tar -czf additional_files.tar.gz -C . ``` -------------------------------- ### Set Google Cloud Project ID Source: https://docs.medperf.org/concepts/confidential_computing Configure the Google Cloud CLI to use your specific project ID. Replace PROJECT_ID with the actual project ID obtained from your cloud administrator. ```bash gcloud config set project PROJECT_ID ``` -------------------------------- ### Cleanup Downloaded Files Source: https://docs.medperf.org/getting_started/model_owner_demo Remove the downloaded files workspace to clean up the MedPerf tutorial environment. Ensure you are in the MedPerf's root directory before running. ```bash rm -fr medperf_tutorial ``` -------------------------------- ### Submit Metrics Container Source: https://docs.medperf.org/getting_started/benchmark_owner_demo Submit a metrics container to the MedPerf server. This command requires paths to the container configuration and parameters files. ```bash medperf container submit \ --name my-metrics \ --container-config-file "medperf_tutorial/metrics/container_config.yaml" \ --parameters-file "medperf_tutorial/metrics/workspace/parameters.yaml" \ --operational ``` -------------------------------- ### Mark Dataset as Operational Source: https://docs.medperf.org/getting_started/data_owner_demo Marks a dataset as operational, making it available for association with benchmarks. Once operational, a dataset cannot be reverted to the development stage. ```bash medperf dataset set_operational --data_uid 1 ``` -------------------------------- ### Perform Logout via Command Line Source: https://docs.medperf.org/concepts/auth Execute this command to log out from the MedPerf client using the command line interface. ```bash medperf auth logout ``` -------------------------------- ### Activate Local MedPerf Profile Source: https://docs.medperf.org/getting_started/setup Activate the 'local' profile for the MedPerf client to communicate with the local MedPerf server. This profile is preconfigured for local server communication. ```bash medperf profile activate local ``` -------------------------------- ### Reset Local MedPerf Server Database Source: https://docs.medperf.org/getting_started/model_owner_demo Clean up the local MedPerf server database. Navigate to the 'server' directory and execute the reset script. Ensure you are in the MedPerf's root directory first. ```bash cd server sh reset_db.sh ``` -------------------------------- ### List Active MedPerf Profiles Source: https://docs.medperf.org/getting_started/setup Check the currently active profile for the MedPerf client. This command helps verify which configuration is currently in use. ```bash medperf profile ls ``` -------------------------------- ### Model Weights Tarball URL for Model Registration Source: https://docs.medperf.org/getting_started/webui_full_tutorial The URL for the hosted tarball containing model weights, necessary for model registration. ```text https://storage.googleapis.com/medperf-storage/chestxray_tutorial/mobilenetv2_weights.tar.gz ``` -------------------------------- ### Create Conda Virtual Environment Source: https://docs.medperf.org/getting_started/installation Create a new Conda virtual environment named 'medperf-env' with Python 3.11. This is an optional but recommended step for isolating MedPerf. ```bash conda create -n medperf-env python=3.11 ``` -------------------------------- ### Cleanup Test Storage Source: https://docs.medperf.org/getting_started/model_owner_demo Remove the test storage directory located in the user's home directory. This command cleans up local test artifacts. ```bash rm -fr ~/.medperf/localhost_8000 ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.