### Progress Indicator Example

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This output shows a progress bar, typically from a library like `tqdm`, indicating that an operation is running. The '0%' and '?' iterations/second suggest the process has started but is not yet significantly advanced or has completed very quickly.

```text
Result:
  0%|          | 0/3 [00:00<?, ?it/s]
```

--------------------------------

### Progress Indicator Example (Extended)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This output shows a progress bar, indicating that an operation is running. The '0%' and '?' iterations/second suggest the process has started but is not yet significantly advanced or has completed very quickly.

```text
Result:
  0%|          | 0/9 [00:00<?, ?it/s]
```

--------------------------------

### Experiment Setup and Execution in Python

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

This Python code defines lists of datasets and models, along with their corresponding colors, to be used in experiments. It then calls a function `run_idrid_exp` to execute these experiments with specified parameters.

```python
name_X_list = [
    ('RETFound-Green', 
     'IRID_Train_X_ours.npy',
     'IRID_Test_X_ours.npy'),
    ('DinoV2', 
     'IRID_Train_X_dino392.npy',
     'IRID_Test_X_dino392.npy'),
]

models = ['RETFound-Green', 'DinoV2']
colors = ['#8ac926', '#4d759a']

run_idrid_exp(name_X_list, models, colors,
              figsize=(8.5, 4), dashed_line_offsets=(-0.25, 1.5), xtick_loc_scaling=0.5, title_size='medium')
```

--------------------------------

### Model Performance Output Example (DERETFound)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This output presents the performance metrics (AUC scores) for the 'DERETFound' model across various retinal image tasks. The metrics are formatted as a tab-separated string, followed by total and mean processing times.

```text
Output:
        DERETFound:macula:0.8551	optic_disc:0.8873	vessels:0.7924	diabetic_retinopathy:0.9379	amd:0.9155	macular_edema:0.9382	quality:0.8962	insuline:0.8671	diabetes:0.8624
Total time: 19.0831s / mean time: 2.1203s
```

--------------------------------

### Model Performance Output Example

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This output presents the performance metrics (AUC scores) for a specific model ('RETFound-MEH') across various retinal image tasks. The metrics are formatted as a tab-separated string.

```text
Output:
        RETFound-MEH:macula:0.8442	optic_disc:0.8487	vessels:0.7990	diabetic_retinopathy:0.9205	amd:0.9079	macular_edema:0.9201	quality:0.8784	insuline:0.7959	diabetes:0.8575
Total time: 19.4282s / mean time: 2.1587s
```

--------------------------------

### Model Performance Output Example (RETFound-Green)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This output presents the performance metrics (AUC scores) for the 'RETFound-Green' model across various retinal image tasks. The metrics are formatted as a tab-separated string, followed by total and mean processing times.

```text
Output:
      RETFound-Green:macula:0.8922	optic_disc:0.8561	vessels:0.8296	diabetic_retinopathy:0.9624	amd:0.9482	macular_edema:0.9751	quality:0.8855	insuline:0.8306	diabetes:0.8503
Total time: 5.8532s / mean time: 0.6504s
```

--------------------------------

### Data Loading and Setup (Python)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/IDRiD_2dProjections.ipynb

Loads training and testing datasets using pandas and sets up configurations for a diabetic retinopathy grading task. Defines target columns, model names, and color mappings for visualization.

```python
import pandas as pd

dftrain = pd.read_csv('/mnt/e/IDRiD_B_DiseaseGrading/B. Disease Grading/2. Groundtruths/a. IDRiD_Disease Grading_Training Labels.csv')
dftest = pd.read_csv('/mnt/e/IDRiD_B_DiseaseGrading/B. Disease Grading/2. Groundtruths/b. IDRiD_Disease Grading_Testing Labels.csv')

target_col = 'Retinopathy grade'
dataset = 'Diabetic Retinopathy (IDRiD)'
target_cols = list(range(5)) + [-1]

name_X_list = [
    ('RETFound-MEH', 
     '/home/justin/RETFound_MAE/IRID_Train_RETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Test_RETFound_Feature_latent.npy'),
    ('DERETFound', 
     '/home/justin/RETFound_MAE/IRID_Train_DERETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Test_DERETFound_Feature_latent.npy'),
    ('RETFound-Green', 
     'IRID_Train_X_ours.npy',
     'IRID_Test_X_ours.npy'),
]


target_to_colour = {
    'Normal': '#5bc0de',   # Light blue (neutral/safe)
    'Grade 1': '#66c266',  # Medium-light green
    'Grade 2': '#ffcc00',  # Golden yellow
    'Grade 3': '#ff6600',  # Orange-red
    'Grade 4': '#990000',  # Dark red
    'No/Mild': None,
}


target_int_to_name = {_: f"Grade {_}" for _ in target_cols}
target_int_to_name[0] = 'Normal'
target_int_to_name[-1] = 'No/Mild'


models = ['RETFound-MEH', 'DERETFound', 'RETFound-Green']
colors = ['#ff595e', '#ffca3a', '#8ac926']  # Colors for models

```

--------------------------------

### Data Structure Definitions for BRSET Experiment

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET_DR.ipynb

Defines the `name_X_list`, `models`, and `colors` variables used in the `run_brset_exp` function. These variables configure the experiment setup.

```python
name_X_list = [
    ('RETFound-MEH', '/home/justin/RETFound_MAE/BRSET_RETFound_Feature_latent.npy'),
    ('DERETFound', '/home/justin/RETFound_MAE/BRSET_DERETFound_Feature_latent.npy'),
    ('RETFound-Green@224', 'X_ours_brset_224.npy'),
]
models = ['RETFound-MEH', 'DERETFound', 'RETFound-Green@224']
colors = ['#ff595e', '#ffca3a', '#B8FF3D']
```

--------------------------------

### Model Training and Visualization Setup (Python)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/ROP_2dProjections.ipynb

This snippet sets up the environment for model training and visualization, including importing necessary libraries like PCA, LogisticRegression, and KNeighborsClassifier. It initializes dataframes and dictionaries to store results and bootstrap AUCs, and prepares for iterating through different feature sets.

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from tqdm.notebook import tqdm

fig, ax = plt.subplots(1, 3, figsize=(15, 4), dpi=120)

target_cols = ['Normal', 'laser_scars', 'ROP', 'Stage_1_ROP', 'Stage_2_ROP', 'Stage_3_ROP']

projection_method_name = 'Principal Component Analysis (PCA)'
dim_name = 'Principal Component'
projector = PCA(n_components=2)

results_df = pd.DataFrame()
bootstrap_results_dict = {}

```

--------------------------------

### Configure and Run IDRID Experiments (Python)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Sets up model configurations, including names, data paths, and colors, then initiates the IDRID experiment run. This snippet defines the parameters for training and testing data for multiple models like RETFound-MEH, DERETFound, and RETFound-Green@224.

```python
name_X_list = [
    ('RETFound-MEH',
     '/home/justin/RETFound_MAE/IRID_Train_RETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Test_RETFound_Feature_latent.npy'),
    ('DERETFound',
     '/home/justin/RETFound_MAE/IRID_Train_DERETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Test_DERETFound_Feature_latent.npy'),
    ('RETFound-Green@224',
     'IRID_Train_X_ours224.npy',
     'IRID_Test_X_ours224.npy'),
]
models = ['RETFound-MEH', 'DERETFound', 'RETFound-Green@224']
colors = ['#ff595e', '#ffca3a', '#B8FF3D']

run_idrid_exp(name_X_list, models, colors)
```

--------------------------------

### Python Function for Experiment Visualization

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET_DR.ipynb

This Python code snippet calls the 'run_brset_exp' function to visualize experiment results. It takes lists of model names, colors, and configuration options for plotting.

```python
run_brset_exp(name_X_list, models, colors, diab_only=False,
              figsize=(4.5, 4), dashed_line_offsets=(-0.25, 1.5), xtick_loc_scaling=0.5, title_size='medium')
```

--------------------------------

### Define Model Configurations (Python)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Sets up lists for model names, feature file paths, and corresponding colors for visualization. This configuration is used to iterate through different models and their associated data.

```python
name_X_list = [
    ('RETFound-MEH', 
     '/home/justin/RETFound_MAE/IRID_Train_RETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Test_RETFound_Feature_latent.npy'),
    ('DERETFound', 
     '/home/justin/RETFound_MAE/IRID_Train_DERETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Test_DERETFound_Feature_latent.npy'),
    ('RETFound-Green', 
     'IRID_Train_X_ours.npy',
     'IRID_Test_X_ours.npy'),
]
models = ['RETFound-MEH', 'DERETFound', 'RETFound-Green']
colors = ['#ff595e', '#ffca3a', '#8ac926']
```

--------------------------------

### Statistical Comparison of Wins and Ties (Output)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Summarizes the number of statistically significant wins and ties for first place between the models being compared. This provides a high-level overview of the competitive performance of each model.

```output
Output:
Statistically significant wins:  {'RETFound-MEH': 2, 'DERETFound': 3, 'RETFound-Green@224': 2}
Ties for first place:  {'RETFound-MEH': 2, 'DERETFound': 2, 'RETFound-Green@224': 4}
```

```output
Output:
Statistically significant wins:  {'RETFound-Green@224': 9, 'DinoV2@224': 2}
Ties for first place:  {'RETFound-Green@224': 0, 'DinoV2@224': 0}
```

--------------------------------

### Execute BRSET Experiment

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This function call likely initiates a specific experiment or set of experiments related to 'BRSET', using a list of names, models, and colors provided as arguments. The output indicates progress, possibly through a series of tasks or iterations.

```python
run_brset_exp(name_X_list, models, colors)
```

--------------------------------

### Populate and Display Results DataFrame

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This snippet populates a pandas DataFrame with performance results and then displays the rounded DataFrame. It's used to consolidate and present the final performance metrics.

```python
results_df.loc[name] = [name] + median_aucs

display(results_df.round(4))
```

--------------------------------

### Run BRSET Experiment for RETFound-Green and DinoV2

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This Python code snippet configures and runs a BRSET experiment comparing RETFound-Green and DinoV2 models. It defines the datasets, model names, and corresponding colors for visualization. The `run_brset_exp` function likely handles the experiment execution and result generation.

```python
name_X_list = [
    ('RETFound-Green', 'X_ours_brset.npy'),
    ('DinoV2', 'X_ViTSDinov2_brset.npy'),
]

models = ['RETFound-Green', 'DinoV2']
colors = ['#8ac926', '#4d759a']

run_brset_exp(name_X_list, models, colors,
              figsize=(8.5, 4), dashed_line_offsets=(-0.25, 1.5), xtick_loc_scaling=0.5)
```

--------------------------------

### Run BRSET Experiment

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET_DR.ipynb

Executes the `run_brset_exp` function with specified name lists, models, and colors. This function is used to run experiments and generate results.

```python
run_brset_exp(name_X_list, models, colors)
```

```python
run_brset_exp(name_X_list, models, colors, diab_only=True)
```

```python
run_brset_exp(name_X_list, models, colors)
```

```python
run_brset_exp(name_X_list, models, colors, diab_only=True)
```

--------------------------------

### Load Training and Testing Data (Pandas)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Loads training and testing datasets from CSV files using the pandas library. These datasets are typically used for supervised learning tasks.

```python
dftrain = pd.read_csv('/mnt/e/IDRiD_B_DiseaseGrading/B. Disease Grading/2. Groundtruths/a. IDRiD_Disease Grading_Training Labels.csv')
dftest = pd.read_csv('/mnt/e/IDRiD_B_DiseaseGrading/B. Disease Grading/2. Groundtruths/b. IDRiD_Disease Grading_Testing Labels.csv')
```

--------------------------------

### Load RETFound-Green Model with PyTorch and Timm

Source: https://github.com/justinengelmann/retfound_green/blob/main/README.md

This snippet demonstrates how to load the RETFound-Green model using the timm library. It requires PyTorch and timm to be installed, and the model weights to be downloaded. The model is initialized and set to evaluation mode.

```python
import timm

rfg = timm.create_model('vit_small_patch14_reg4_dinov2',
                        img_size=(392, 392), num_classes=0,
                        checkpoint_path='retfoundgreen_statedict.pth').eval()
rfg.global_pool = 'avg'
```

--------------------------------

### Run BRSET Experiment for RETFound-Green@224 and DinoV2@224

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This Python code initiates a BRSET experiment comparing RETFound-Green@224 and DinoV2@224. It defines the input data paths, model names, and associated colors for visualization purposes. The `run_brset_exp` function is utilized to execute the experiment with specific figure size and offset configurations.

```python
name_X_list = [
    ('RETFound-Green@224', 'X_ours_brset_224.npy'),
    ('DinoV2@224', 'X_original_dinov2_224_brset.npy'),
]


models = ['RETFound-Green@224', 'DinoV2@224']
colors = ['#B8FF3D', '#6EA6D1']

run_brset_exp(name_X_list, models, colors,
              figsize=(8.5, 4), dashed_line_offsets=(-0.25, 1.5), xtick_loc_scaling=0.5)
```

--------------------------------

### Run Experiment with RETFound-Green and DinoV2 (All Data)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET_DR.ipynb

This Python code executes an experiment using the `run_brset_exp` function, comparing 'RETFound-Green' against 'DinoV2' across all data. It utilizes predefined lists for model names, data files, and colors, along with parameters for plot customization.

```python
name_X_list = [
    ('RETFound-Green', 'X_ours_brset.npy'),
    ('DinoV2', 'X_ViTSDinov2_brset.npy'),
]

models = ['RETFound-Green', 'DinoV2']
colors = ['#8ac926', '#4d759a']

run_brset_exp(name_X_list, models, colors, diab_only=False,
              figsize=(4.5, 4), dashed_line_offsets=(-0.25, 1.1), xtick_loc_scaling=0.5, title_size='medium')
```

--------------------------------

### Configure and Run IDRID Experiments with Customization (Python)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Configures and runs IDRID experiments, specifically comparing RETFound-Green@224 with DinoV2@224. This snippet includes parameters for plot customization such as figure size, dashed line offsets, and title size.

```python
name_X_list = [
    ('RETFound-Green@224',
     'IRID_Train_X_ours224.npy',
     'IRID_Test_X_ours224.npy'),
    ('DinoV2@224',
     'IRID_Train_X_dino224.npy',
     'IRID_Test_X_dino224.npy'),
]

models = ['RETFound-Green@224', 'DinoV2@224']
colors = ['#B8FF3D', '#6EA6D1']

run_idrid_exp(name_X_list, models, colors,
              figsize=(8.5, 4), dashed_line_offsets=(-0.25, 1.5), xtick_loc_scaling=0.5, title_size='medium')
```

--------------------------------

### Run BRSET Experiment for RETFound-MEH, DERETFound, and RETFound-Green@224

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This Python code sets up and executes a BRSET experiment to compare three models: RETFound-MEH, DERETFound, and RETFound-Green@224. It specifies the data files, model names, and distinct colors for each model's representation in the experiment results. The `run_brset_exp` function is called to perform the comparison.

```python
name_X_list = [
    ('RETFound-MEH', '/home/justin/RETFound_MAE/BRSET_RETFound_Feature_latent.npy'),
    ('DERETFound', '/home/justin/RETFound_MAE/BRSET_DERETFound_Feature_latent.npy'),
    ('RETFound-Green@224', 'X_ours_brset_224.npy'),
]
models = ['RETFound-MEH', 'DERETFound', 'RETFound-Green@224']
colors = ['#ff595e', '#ffca3a', '#B8FF3D']

run_brset_exp(name_X_list, models, colors)
```

--------------------------------

### Consolidated Model Performance Table (Output)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Displays a summary table comparing the performance metrics of RETFound-MEH, DERETFound, and RETFound-Green@224 across various grading categories. This table provides a clear side-by-side comparison of the models' effectiveness.

```output
Result:
                                  Name  DR Grade 0  DR Grade 1  DR Grade 2  \
RETFound-MEH              RETFound-MEH      0.8622      0.5361      0.7631   
DERETFound                  DERETFound      0.8661      0.5709      0.6566   
RETFound-Green@224  RETFound-Green@224      0.8696      0.7500      0.6392   

                    DR Grade 3  DR Grade 4  Mac. Edema Grade 0  \
RETFound-MEH            0.7982      0.8313              0.8834   
DERETFound              0.8168      0.9047              0.8781   
RETFound-Green@224      0.7770      0.7857              0.9044   

                    Mac. Edema Grade 1  Mac. Edema Grade 2  No/Mild vs rest  \
RETFound-MEH                    0.7117              0.8888           0.9240   
DERETFound                      0.7583              0.8706           0.9046   
RETFound-Green@224              0.7032              0.8687           0.9245   

                     AnyDR  AnyDME  
RETFound-MEH        0.8687  0.8922  
DERETFound          0.8721  0.8737  
RETFound-Green@224  0.8708  0.8879  
```

--------------------------------

### Consolidated Performance Table for RETFound-Green@224 and DinoV2@224 (Output)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Presents a comparative table of performance metrics for RETFound-Green@224 and DinoV2@224 models. It summarizes scores across various grading categories, facilitating a direct comparison of their effectiveness.

```output
Result:
                                  Name  DR Grade 0  DR Grade 1  DR Grade 2  \
RETFound-Green@224  RETFound-Green@224      0.8696      0.7500      0.6392   
DinoV2@224                  DinoV2@224      0.8277      0.7104      0.5799   

                    DR Grade 3  DR Grade 4  Mac. Edema Grade 0  \
RETFound-Green@224      0.7770      0.7857              0.9044   
DinoV2@224              0.7663      0.8907              0.8755   

                    Mac. Edema Grade 1  Mac. Edema Grade 2  No/Mild vs rest  \
RETFound-Green@224              0.7032              0.8687           0.9245   
DinoV2@224                      0.6361              0.8737           0.8911   

                     AnyDR  AnyDME  
RETFound-Green@224  0.8708  0.8879  
DinoV2@224          0.8058  0.8791  
```

--------------------------------

### Configure Model and Feature File Paths

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/DR_BRSET_IDRiD_Transportability.ipynb

Defines lists of model names, their corresponding feature file paths for different datasets (BRSET, IDRiD train, IDRiD test), and associated colors for plotting. This setup is crucial for iterating through various experimental configurations.

```python
name_X_list = [
    ('RETFound-MEH', 
     '/home/justin/RETFound_MAE/BRSET_RETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Train_RETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Test_RETFound_Feature_latent.npy'),
    ('DERETFound', 
     '/home/justin/RETFound_MAE/BRSET_DERETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Train_DERETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/IRID_Test_DERETFound_Feature_latent.npy'),
    ('RETFound-Green', 
     'X_ours_brset.npy',
     'IRID_Train_X_ours.npy',
     'IRID_Test_X_ours.npy'),
]

models = ['RETFound-MEH', 'DERETFound', 'RETFound-Green']
colors = ['#ff595e', '#ffca3a', '#8ac926']
```

--------------------------------

### Run Retinopathy Experiment with Python

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET_DR.ipynb

Orchestrates an experiment for retinopathy detection using various models and feature sets. It loads data, trains logistic regression classifiers, calculates AUCs for different targets, and measures performance metrics.

```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
import numpy as np
import pandas as pd
from tqdm.notebook import tqdm
import time

# Assume get_bootstrap_aucs is defined elsewhere
# Assume dftrain, dftest, train_mask are defined globally or passed as arguments

def run_brset_exp(name_X_list, models, colors, diab_only=False,
                  figsize=(5.5, 4), dashed_line_offsets=(-2.75, 0.25), xtick_loc_scaling=1, title_size='large'):
    targets = ['DR Grade', 'No/Mild vs rest', 'AnyDR', ]

    results_df = pd.DataFrame(
        columns=['Name'] + [f'DR Grade {_}' for _ in range(5)] 
                + ['No/Mild vs rest', 'AnyDR', ])

    name_bootstrap_aucs_dict = {}
    name_times_dict = {}
    for name, Xfname in tqdm(name_X_list):
        np.random.seed(42)

        if diab_only:
            diab_mask_train = dftrain['diabetes'].apply(lambda x: x=='yes').values
            diab_mask_test = dftest['diabetes'].apply(lambda x: x=='yes').values
        else:
            # all True
            diab_mask_train = dftrain['diabetes'].apply(lambda x: x is not None).values
            diab_mask_test = dftest['diabetes'].apply(lambda x: x is not None).values
        
        name_times_dict[name] = []
        median_aucs = []

        X = np.load(Xfname)
        X_train = X[train_mask]
        X_test = X[~train_mask]

        perf_str = f'{name:>20}:'
        for target_col in tqdm(targets, leave=True):

            if target_col.startswith('DR Grade'):
                y_train = dftrain['Retinopathy grade'].values
                y_test = dftest['Retinopathy grade'].values

                start = time.time()
                clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=20_000, random_state=42))
                clf.fit(X_train[diab_mask_train], y_train[diab_mask_train])
                preds = clf.predict_proba(X_test[diab_mask_test])
                stop = time.time()
                delta = stop - start
                name_times_dict[name].append(delta)

                for grade in range(5):
                    curr_median_auc, curr_bootstrap_aucs = get_bootstrap_aucs(y_test[diab_mask_test] == grade, preds[:, grade])
                    name_bootstrap_aucs_dict[f'{name}-DR Grade {grade}'] = curr_bootstrap_aucs
                    median_aucs.append(curr_median_auc)

            elif target_col.startswith('No/Mild vs rest'):
                y_train = dftrain['Retinopathy grade'].apply(lambda x: x > 1).values
                y_test = dftest['Retinopathy grade'].apply(lambda x: x > 1).values

                start = time.time()
                clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=20_000, random_state=42))
                clf.fit(X_train[diab_mask_train], y_train[diab_mask_train])
```

--------------------------------

### Display Detailed Model Performance Metrics (Output)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Presents the evaluation metrics for different models, including DR Grade, Macular Edema, No/Mild vs rest, AnyDR, and AnyDME. The output is formatted to show individual model scores and total processing time.

```output
Output:
        RETFound-MEH:DR Grade:0.8622	Mac. Edema:0.5361	No/Mild vs rest:0.7631	AnyDR:0.7982	AnyDME:0.8313
Total time: 0.3792s / mean time: 0.0758s
```

```output
Output:
          DERETFound:DR Grade:0.8661	Mac. Edema:0.5709	No/Mild vs rest:0.6566	AnyDR:0.8168	AnyDME:0.9047
Total time: 0.3810s / mean time: 0.0762s
```

```output
Output:
  RETFound-Green@224:DR Grade:0.8696	Mac. Edema:0.7500	No/Mild vs rest:0.6392	AnyDR:0.7770	AnyDME:0.7857
Total time: 0.4451s / mean time: 0.0890s
```

--------------------------------

### Custom PyTorch Dataset and DataLoader Setup (Python)

Source: https://context7.com/justinengelmann/retfound_green/llms.txt

Provides code for setting up custom PyTorch datasets and dataloaders for retinal image analysis. It demonstrates creating an `ImgDataset` from a directory or a list of image paths and configuring an optimized dataloader for efficient batch processing during training or inference.

```python
from Training_and_eval_code.data import ImgDataset, get_dataloader, get_default_transforms, get_default_training_transforms
from pathlib import Path

# Create dataset from directory (recursively finds all .jpg files)
img_directory = 'path/to/retinal_images/'
inference_transforms = get_default_transforms(res=392)
dataset = ImgDataset(img_directory, transform=inference_transforms)

print(f"Found {len(dataset)} images")

# Create optimized dataloader for batch processing
dataloader = get_dataloader(
    dataset,
    batch_size=32,
    num_workers=4,
    shuffle=False,
    pin_memory=True,
    prefetch_factor=2,
    persistent_workers=True
)

# Extract features for entire dataset
all_features = []
with torch.inference_mode():
    for batch in dataloader:
        batch = batch.to(device)
        features = rfg(batch)  # Shape: [batch_size, 384]
        all_features.append(features.cpu())

all_features = torch.cat(all_features, dim=0)
print(f"Extracted features for {len(all_features)} images")

# Alternative: Create dataset from explicit list of image paths
img_paths = list(Path('dataset_folder').rglob('*.jpg'))
train_transforms = get_default_training_transforms(res=392)
train_dataset = ImgDataset(img_paths, transform=train_transforms)
train_loader = get_dataloader(train_dataset, batch_size=128, shuffle=True)
```

--------------------------------

### Run Experiment with RETFound-Green@224 and DinoV2@224 (Diabetic Only)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET_DR.ipynb

This Python code initiates an experiment using `run_brset_exp` to compare 'RETFound-Green@224' and 'DinoV2@224' on diabetic-only data. It defines the input data, model names, and color schemes, alongside parameters for figure size and plot aesthetics.

```python
name_X_list = [
    ('RETFound-Green@224', 'X_ours_brset_224.npy'),
    ('DinoV2@224', 'X_original_dinov2_224_brset.npy'),
]


models = ['RETFound-Green@224', 'DinoV2@224']
colors = ['#B8FF3D', '#6EA6D1']

run_brset_exp(name_X_list, models, colors, diab_only=True,
              figsize=(4.5, 4), dashed_line_offsets=(-0.25, 1.5), xtick_loc_scaling=0.5, title_size='medium')
```

--------------------------------

### Data Loading and Configuration (Python)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/ROP_2dProjections.ipynb

This snippet loads training and testing datasets using pandas and defines configurations for feature extraction, target columns, and model-specific colors. It sets up data paths and mappings required for subsequent analysis and visualization.

```python
import pandas as pd
import numpy as np

dftrain = pd.read_csv('/mnt/c/Users/Justin/Desktop/retina_imaging_datasets/ROP_dataset/dftrain.csv')
dftest = pd.read_csv('/mnt/c/Users/Justin/Desktop/retina_imaging_datasets/ROP_dataset/dftest.csv')

target_col = 'class'
dataset = 'Retinopathy of Prematurity (ROP)'
name_X_list = [
    ('RETFound-MEH', 
     '/home/justin/RETFound_MAE/ROP_Train_RETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/ROP_Test_RETFound_Feature_latent.npy'),
    ('DERETFound', 
     '/home/justin/RETFound_MAE/ROP_Train_DERETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/ROP_Test_DERETFound_Feature_latent.npy'),
    ('RETFound-Green', 
     'ROP_Train_X_ours.npy',
     'ROP_Test_X_ours.npy'),
]

target_to_colour = {
    'Normal': 'gold',
    'laser_scars': 'crimson',
    'Stage_1_ROP': 'cornflowerblue',
    'Stage_2_ROP': 'royalblue',
    'Stage_3_ROP': 'navy',
}
models = ['RETFound-MEH', 'DERETFound', 'RETFound-Green']
colors = ['#ff595e', '#ffca3a', '#8ac926']  # Colors for models

```

--------------------------------

### Calculate and Format P-values

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

Calculates p-values from AUC scores and formats them into a human-readable string. It uses the Wilcoxon signed-rank test for comparison. Dependencies include numpy and scipy.stats.wilcoxon.

```python
import numpy as np
from scipy.stats import wilcoxon

def get_p_values(all_aucs):
    all_aucs = np.vstack(all_aucs)
    median_aucs = np.median(all_aucs, -1)
    second_best_idx, best_idx = median_aucs.argsort()[-2:]
    p_val = wilcoxon(all_aucs[best_idx], all_aucs[second_best_idx])[1]
    return p_val, (best_idx, second_best_idx)

def pval_to_str(p_val):
    if p_val < 0.0001:
        p_val = 'p<0.0001'
        p_val = '***'
    else:
        p_val = f'p={p_val:.4f}'
    return p_val
```

--------------------------------

### Format and Print Performance String

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This code formats performance metrics (AUC scores) into a tab-separated string and prints it. It also prints the total and mean processing times for a given model.

```python
perf_str += '\t'.join([f'{c}:{v:.4f}' for c, v in zip(targets, median_aucs)])
print(perf_str)
print(f'Total time: {sum(name_times_dict[name]):.4f}s / mean time: {np.mean(name_times_dict[name]):.4f}s')
```

--------------------------------

### Calculate and Store Bootstrap AUCs

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This snippet calculates bootstrap AUCs for model predictions and stores them along with median AUCs. It's used to assess the variability and reliability of model performance.

```python
curr_median_auc, curr_bootstrap_aucs = get_bootstrap_aucs(y_test, preds)
name_bootstrap_aucs_dict[f'{name}-{target_col}'] = curr_bootstrap_aucs
median_aucs.append(curr_median_auc)
```

--------------------------------

### Load and Prepare Datasets

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/DR_BRESET_IDRiD_2dProjections.ipynb

Loads training and testing datasets using pandas and sets up target column and dataset configurations. It also defines lists of model names and corresponding file paths for feature data.

```python
dftrain = pd.read_csv('/mnt/e/IDRiD_B_DiseaseGrading/B. Disease Grading/2. Groundtruths/a. IDRiD_Disease Grading_Training Labels.csv')
dftest = pd.read_csv('/mnt/e/IDRiD_B_DiseaseGrading/B. Disease Grading/2. Groundtruths/b. IDRiD_Disease Grading_Testing Labels.csv')

target_col = 'Retinopathy grade'
dataset = 'Diabetic Retinopathy (IDRiD → BRSET) - Points correspond to unseen BRSET dataset'
target_cols = list(range(5)) + [-1]

name_X_list = [
    ('RETFound-MEH', 
     '/home/justin/RETFound_MAE/IRID_Train_RETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/BRSET_RETFound_Feature_latent.npy'),
    ('DERETFound', 
     '/home/justin/RETFound_MAE/IRID_Train_DERETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/BRSET_DERETFound_Feature_latent.npy'),
    ('RETFound-Green', 
     'IRID_Train_X_ours.npy',
     'X_ours_brset.npy'),
]
dfbr = pd.read_csv('/mnt/c/Users/Justin/Desktop/labels.csv')
dfbr['Retinopathy grade'] = dfbr['DR_ICDR']
br_diab_mask = dfbr.diabetes=='yes'
dfbr = dfbr[br_diab_mask].copy().reset_index()

target_to_colour = {
    'Normal': '#5bc0de',   # Light blue (neutral/safe)
    'Grade 1': '#66c266',  # Medium-light green
    'Grade 2': '#ffcc00',  # Golden yellow
    'Grade 3': '#ff6600',  # Orange-red
    'Grade 4': '#990000',  # Dark red
    'No/Mild': None,
}


target_int_to_name = {_: f"Grade {_}" for _ in target_cols}
target_int_to_name[0] = 'Normal'
target_int_to_name[-1] = 'No/Mild'


models = ['RETFound-MEH', 'DERETFound', 'RETFound-Green']
colors = ['#ff595e', '#ffca3a', '#8ac926']  # Colors for models

```

--------------------------------

### Import Core ML and Data Science Libraries (Python)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET_DR.ipynb

Imports essential Python libraries for data analysis and machine learning. This includes pandas for data manipulation, numpy for numerical operations, scikit-learn for ML algorithms and preprocessing, and joblib for saving models. These libraries are fundamental for building and evaluating machine learning models.

```python
import pandas as pd
import numpy as np

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score

import joblib
```

--------------------------------

### Convert P-value to String Representation (Python)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Formats a given p-value into a human-readable string, indicating significance levels like 'p<0.0001' or '***'. This is useful for presenting statistical test results.

```python
def pval_to_str(p_val):
    if p_val < 0.0001:
        p_val = 'p<0.0001'
        p_val = '***'
    else:
        p_val = f'p={p_val:.4f}'
    return p_val
```

--------------------------------

### Calculate Bootstrap AUCs

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

Calculates Area Under the ROC Curve (AUC) using bootstrapping. It generates multiple AUC scores by resampling with replacement from the true labels and predicted probabilities. Dependencies include numpy and sklearn.metrics.roc_auc_score.

```python
import numpy as np
from sklearn.metrics import roc_auc_score

def get_bootstrap_aucs(y_true, y_pred, bootstrap_iters=100):
    rng = np.random.RandomState(seed=42)
    idxs = list(range(y_pred.shape[0]))

    bootstrap_aucs = []
    for _ in range(bootstrap_iters):
        curr_idxs = rng.choice(idxs, size=len(idxs), replace=True)
        bootstrap_aucs.append(roc_auc_score(y_true[curr_idxs], y_pred[curr_idxs],
                                            multi_class='ovr', average='macro'))

    return np.median(bootstrap_aucs), bootstrap_aucs
```

--------------------------------

### Calculate Bootstrap AUC Scores (Python)

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_IDRiD.ipynb

Computes bootstrap estimates for Area Under the Receiver Operating Characteristic Curve (AUC) scores. This function helps in assessing the uncertainty and robustness of model performance.

```python
def get_bootstrap_aucs(y_true, y_pred, bootstrap_iters=100):
    rng = np.random.RandomState(seed=42)
    idxs = list(range(y_pred.shape[0]))

    bootstrap_aucs = []
    for _ in range(bootstrap_iters):
        curr_idxs = rng.choice(idxs, size=len(idxs), replace=True)
        bootstrap_aucs.append(roc_auc_score(y_true[curr_idxs], y_pred[curr_idxs],
                                            multi_class='ovr', average='macro'))

    return np.median(bootstrap_aucs), bootstrap_aucs
```

--------------------------------

### Results DataFrame Snapshot

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET.ipynb

This is a partial view of a pandas DataFrame summarizing model performance metrics (AUC scores) for different tasks. It displays rows for 'RETFound-MEH', 'DERETFound', and 'RETFound-Green', showing their scores for 'macula', 'optic_disc', and 'vessels'.

```python
Result:
                          Name  macula  optic_disc  vessels  \
RETFound-MEH      RETFound-MEH  0.8442      0.8487   0.7990   
DERETFound          DERETFound  0.8551      0.8873   0.7924   
RETFound-Green  RETFound-Green  0.8922      0.8561   0.8296   

                diabetic_retinopathy     amd  macular_edema  quality  \
```

--------------------------------

### Darken Hex Color Code in Python

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET_DR.ipynb

Adjusts the brightness of a given hexadecimal color code by a specified factor. This function is useful for creating color variations in visualizations, ensuring the factor is within a valid range.

```python
def darken_hex(hex_color, factor=0.5):
    # Ensure the factor is between 0 and 1
    factor = max(0, min(factor, 1))

    # Remove the hash symbol if present
    hex_color = hex_color.lstrip('#')

    # Convert hex to RGB
    rgb = tuple(int(hex_color[i:i + 2], 16) for i in (0, 2, 4))

    # Darken the RGB values by multiplying by the factor
    darkened_rgb = tuple(int(c * factor) for c in rgb)

    # Convert the RGB values back to hex
    darkened_hex = '#{:02x}{:02x}{:02x}'.format(*darkened_rgb)

    return darkened_hex
```

--------------------------------

### Run ROP Experiment with RETFound-Green@224 and DinoV2@224 Models

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_ROP.ipynb

This Python code initializes an experiment comparing 'RETFound-Green@224' and 'DinoV2@224' models. It defines the model names, corresponding data files, and specific colors for plotting. The `run_rop_exp` function is then called to execute the experiment with these settings.

```python
name_X_list = [
    ('RETFound-Green@224', 
     'ROP_Train_X_ours224.npy',
     'ROP_Test_X_ours224.npy'),
    ('DinoV2@224', 
     'ROP_Train_X_dino224.npy',
     'ROP_Test_X_dino224.npy'),
]

models = ['RETFound-Green@224', 'DinoV2@224']
colors = ['#B8FF3D', '#6EA6D1']

run_rop_exp(name_X_list, models, colors,

```

--------------------------------

### Retrieve and Format P-values with Python

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_BRSET_DR.ipynb

Processes arrays of AUC scores to compute a p-value using the Wilcoxon signed-rank test and formats it into a human-readable string. It identifies the best and second-best performing models based on median AUC.

```python
from scipy.stats import wilcoxon
import numpy as np

def get_p_values(all_aucs):
    all_aucs = np.vstack(all_aucs)
    median_aucs = np.median(all_aucs, -1)
    second_best_idx, best_idx = median_aucs.argsort()[-2:]
    p_val = wilcoxon(all_aucs[best_idx], all_aucs[second_best_idx])[1]
    return p_val, (best_idx, second_best_idx)

def pval_to_str(p_val):
    if p_val < 0.0001:
        p_val = 'p<0.0001'
        p_val = '***'
    else:
        p_val = f'p={p_val:.4f}'
    return p_val
```

--------------------------------

### Run ROP Experiment with RETFound-Green and DinoV2 Models

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_ROP.ipynb

This code configures and runs an experiment comparing the 'RETFound-Green' and 'DinoV2' models. It defines lists for model names, associated data files, and visualization colors. The `run_rop_exp` function is called to execute the experiment with specified parameters like figure size and line offsets.

```python
name_X_list = [
    ('RETFound-Green', 
     'ROP_Train_X_ours.npy',
     'ROP_Test_X_ours.npy'),
    ('DinoV2', 
     'ROP_Train_X_dino392.npy',
     'ROP_Test_X_dino392.npy'),
]

models = ['RETFound-Green', 'DinoV2']
colors = ['#8ac926', '#4d759a']

run_rop_exp(name_X_list, models, colors,
            figsize=(8.5, 4), dashed_line_offsets=(-0.25, 1.5), xtick_loc_scaling=0.5, title_size='medium')
```

--------------------------------

### Run ROP Experiment with RETFound-MEH, DERETFound, and RETFound-Green@224 Models

Source: https://github.com/justinengelmann/retfound_green/blob/main/Training_and_eval_code/evaluation/Eval_ROP.ipynb

This Python script sets up and executes a comparative experiment for three models: 'RETFound-MEH', 'DERETFound', and 'RETFound-Green@224'. It specifies the training and testing data file paths and assigns distinct colors for visualization. The `run_rop_exp` function is invoked to process these configurations.

```python
name_X_list = [
    ('RETFound-MEH', 
     '/home/justin/RETFound_MAE/ROP_Train_RETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/ROP_Test_RETFound_Feature_latent.npy'),
    ('DERETFound', 
     '/home/justin/RETFound_MAE/ROP_Train_DERETFound_Feature_latent.npy',
     '/home/justin/RETFound_MAE/ROP_Test_DERETFound_Feature_latent.npy'),
    ('RETFound-Green@224', 
     'ROP_Train_X_ours224.npy',
     'ROP_Test_X_ours224.npy'),
]
models = ['RETFound-MEH', 'DERETFound', 'RETFound-Green@224']
colors = ['#ff595e', '#ffca3a', '#B8FF3D']

run_rop_exp(name_X_list, models, colors)
```