### Training Command

Source: https://github.com/antonotnawang/naviairway/blob/main/README.md

Command to start the training process.

```text
Run ```train.py``` or ```train_semi_supervised_learning.py``` to start training. You can change the hyperparameters. Model parameters are saved in ```checkpoint```.
```

--------------------------------

### Create Train/Test Split Dictionary

Source: https://github.com/antonotnawang/naviairway/blob/main/Get_dataset_info.ipynb

Creates a dictionary to hold training and testing data splits based on predefined train and test names. It iterates through loaded dataset dictionaries and assigns cases to either the 'train' or 'test' set.

```python
train_test_set_dict_EXACT09_LIDC_IDRI_128={}
train_test_set_dict_EXACT09_LIDC_IDRI_128["train"]={}
train_test_set_dict_EXACT09_LIDC_IDRI_128["test"]={}

for case in data_dict_EXACT09_128.keys():
    if (case.split("_")[0]+"_"+case.split("_")[1]) in train_names:
        train_test_set_dict_EXACT09_LIDC_IDRI_128["train"][case] = data_dict_EXACT09_128[case]
    elif (case.split("_")[0]+"_"+case.split("_")[1]) in test_names:
        train_test_set_dict_EXACT09_LIDC_IDRI_128["test"][case] = data_dict_EXACT09_128[case]

for case in data_dict_LIDC_IDRI_128.keys():
    if (case.split("_")[0]+"_"+case.split("_")[1]+"_"+case.split("_")[2]) in train_names:
        train_test_set_dict_EXACT09_LIDC_IDRI_128["train"][case] = data_dict_LIDC_IDRI_128[case]
    elif (case.split("_")[0]+"_"+case.split("_")[1]+"_"+case.split("_")[2]) in test_names:
        train_test_set_dict_EXACT09_LIDC_IDRI_128["test"][case] = data_dict_LIDC_IDRI_128[case]

train_test_set_dict_EXACT09_LIDC_IDRI_128["train_names"]=train_names
train_test_set_dict_EXACT09_LIDC_IDRI_128["test_names"]=test_names
```

--------------------------------

### Load Dataset Dictionaries

Source: https://github.com/antonotnawang/naviairway/blob/main/Get_dataset_info.ipynb

Loads dataset information dictionaries for EXACT09 and LIDC_IDRI datasets, likely containing information about image crops.

```python
data_dict_EXACT09_128=load_obj("dataset_info/dataset_info_EXACT09_crops_128")
data_dict_LIDC_IDRI_128=load_obj("dataset_info/dataset_info_LIDC_IDRI_crops_128")
```

--------------------------------

### Get pre-cropped EXACT09 dataset info

Source: https://github.com/antonotnawang/naviairway/blob/main/Get_dataset_info.ipynb

This script processes the pre-cropped EXACT09 dataset, performing similar operations as the LIDC-IDRI script: extracting paths, calculating airway pixel counts, and determining boundary and inner airway pixels using EDT. The results are saved to a pickle file.

```python
Precrop_dataset_for_train_path = "/data/Airway/Precrop_dataset_for_EXACT09"
Precrop_dataset_for_train_raw_path = Precrop_dataset_for_train_path+"/image"
Precrop_dataset_for_train_label_path = Precrop_dataset_for_train_path+"/label"

raw_case_name_list = os.listdir(Precrop_dataset_for_train_raw_path)
label_case_name_list = os.listdir(Precrop_dataset_for_train_label_path)

assert raw_case_name_list == label_case_name_list

data_dict = dict()

for idx, name in enumerate(raw_case_name_list):
    print("process "+str(name)+" | "+str(idx/len(raw_case_name_list)), end="\r")
    data_dict[name.split(".")[0]]={}
    data_dict[name.split(".")[0]]["image"]=Precrop_dataset_for_train_raw_path+"/"+name
    data_dict[name.split(".")[0]]["label"]=Precrop_dataset_for_train_label_path+"/"+name
    label_temp = np.load(Precrop_dataset_for_train_label_path+"/"+name)
    data_dict[name.split(".")[0]]["airway_pixel_num"]=np.sum(label_temp)
    
    label_temp=np.array(label_temp, dtype=np.uint32, order='F')
    label_temp_edt=edt.edt(
        label_temp,
        black_border=True, order='F',
        parallel=1)
    
    data_dict[name.split(".")[0]]["airway_pixel_num_boundary"] = len(np.where(label_temp_edt==1)[0])
    data_dict[name.split(".")[0]]["airway_pixel_num_inner"] = len(np.where(label_temp_edt>1)[0])

save_obj(data_dict, "dataset_info_EXACT09_crops_128")
```

--------------------------------

### Utility functions for saving and loading Python objects

Source: https://github.com/antonotnawang/naviairway/blob/main/Get_dataset_info.ipynb

These functions use the pickle module to serialize and deserialize Python objects, allowing them to be saved to and loaded from files.

```python
import os
import numpy as np
import pickle
import edt

def save_obj(obj, name ):
    with open(name + '.pkl', 'wb') as f:
        pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)

def load_obj(name ):
    with open(name + '.pkl', 'rb') as f:
        return pickle.load(f)
```

--------------------------------

### Option 1: Get 3D image from DCM images

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Converts DICOM images from a specified directory into a 3D NIfTI (.nii.gz) file.

```python
intput_img_path = "test_data/DCM_imgs" # these DCM images come from a dataset which was not used for training
raw_img_path = "results/raw_img.nii.gz"
get_and_save_3d_img_for_one_case(img_path = intput_img_path,
                                 output_file_path = raw_img_path)
```

--------------------------------

### Data Preprocessing for CASE13 and CASE14

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Examples of how to use the `reverse_img_3d_np` function to preprocess data for specific cases (CASE13 and CASE14) by concatenating reversed image segments.

```python
#mark: for CASE13, raw_img = np.concatenate((reverse_img_3d_np(raw_img[:93,:,:]), reverse_img_3d_np(raw_img[93:193,:,:]), reverse_img_3d_np(raw_img[193:,:,:])), axis=0)
#mark: for CASE14, raw_img = np.concatenate((reverse_img_3d_np(raw_img[:82,:,:]), reverse_img_3d_np(raw_img[82:181,:,:]),
#                          reverse_img_3d_np(raw_img[181:282,:,:]), reverse_img_3d_np(raw_img[282:,:,:])), axis=0)
```

--------------------------------

### Get pre-cropped LIDC-IDRI dataset info

Source: https://github.com/antonotnawang/naviairway/blob/main/Get_dataset_info.ipynb

This script processes the pre-cropped LIDC-IDRI dataset, extracting image and label paths, calculating the total number of airway pixels, and determining the number of boundary and inner airway pixels using the Euclidean Distance Transform (EDT). The processed information is then saved to a pickle file.

```python
Precrop_dataset_for_train_path = "/data/Airway/Precrop_dataset_for_LIDC-IDRI"
Precrop_dataset_for_train_raw_path = Precrop_dataset_for_train_path+"/image"
Precrop_dataset_for_train_label_path = Precrop_dataset_for_train_path+"/label"

raw_case_name_list = os.listdir(Precrop_dataset_for_train_raw_path)
label_case_name_list = os.listdir(Precrop_dataset_for_train_label_path)

assert raw_case_name_list == label_case_name_list

data_dict = dict()

for idx, name in enumerate(raw_case_name_list):
    print("process "+str(name)+" | "+str(idx/len(raw_case_name_list)), end="\r")
    data_dict[name.split(".")[0]]={}
    data_dict[name.split(".")[0]]["image"]=Precrop_dataset_for_train_raw_path+"/"+name
    data_dict[name.split(".")[0]]["label"]=Precrop_dataset_for_train_label_path+"/"+name
    label_temp = np.load(Precrop_dataset_for_train_label_path+"/"+name)
    data_dict[name.split(".")[0]]["airway_pixel_num"]=np.sum(label_temp)
    
    label_temp=np.array(label_temp, dtype=np.uint32, order='F')
    label_temp_edt=edt.edt(
        label_temp,
        black_border=True, order='F',
        parallel=1)
    
    data_dict[name.split(".")[0]]["airway_pixel_num_boundary"] = len(np.where(label_temp_edt==1)[0])
    data_dict[name.split(".")[0]]["airway_pixel_num_inner"] = len(np.where(label_temp_edt>1)[0])

save_obj(data_dict, "dataset_info_LIDC_IDRI_crops_128")
```

--------------------------------

### Get 3D Image for a Case

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

Assembles a 3D image volume from a list of image slices for a single case.

```python
def get_3d_img_for_one_case(img_path_list, img_format="dcm"):
    img_3d=[]
    for idx, img_path in enumerate(img_path_list):
        print("progress: "+str(idx/len(img_path_list))+"; "+str(img_path), end="\r")
        img_slice, frame_num, _, _ = loadFile(img_path)
        assert frame_num==1
        img_3d.append(img_slice)
    img_3d=np.array(img_3d)
    return img_3d.reshape(img_3d.shape[0], img_3d.shape[2], img_3d.shape[3])
```

--------------------------------

### Get the case names of LIDC-IDRI and EXACT09

Source: https://github.com/antonotnawang/naviairway/blob/main/Get_dataset_info.ipynb

This script retrieves and organizes case names from both the LIDC-IDRI and EXACT09 datasets. It lists image and label files, sorts them, and then creates unique lists of case names prefixed with their respective dataset identifiers ('EXACT09_' or 'LIDC_IDRI_'). Finally, it concatenates these lists and demonstrates how to split them into training and testing sets.

```python
EXACT_img_niigz_path = "/data/Airway/EXACT09_3D/train"
EXACT_label_niigz_path = "/data/Airway/EXACT09_3D/train_label"
LIDC_IDRI_img_niigz_path = "/data/Airway/LIDC-IDRI_3D/annotated_data/image"
LIDC_IDRI_label_niigz_path = "/data/Airway/LIDC-IDRI_3D/annotated_data/label"

EXACT_names = os.listdir(EXACT_img_niigz_path)
EXACT_names.sort()
EXACT_label_names = os.listdir(EXACT_label_niigz_path)
EXACT_label_names.sort()
print(EXACT_names, EXACT_label_names)

LIDC_names = os.listdir(LIDC_IDRI_img_niigz_path)
LIDC_names.sort()
LIDC_IDRI_label_names = os.listdir(LIDC_IDRI_label_niigz_path)
LIDC_IDRI_label_names.sort()
print(LIDC_names, LIDC_IDRI_label_names)

EXACT09_names = []
for EXACT09_name in EXACT_names:
    EXACT09_names.append("EXACT09_"+EXACT09_name.split(".")[0])
EXACT09_names = np.array(EXACT09_names)
EXACT09_names = np.unique(EXACT09_names)

LIDC_IDRI_names = []
for LIDC_IDRI_name in LIDC_names:
    LIDC_IDRI_names.append("LIDC_IDRI_"+LIDC_IDRI_name.split(".")[0])
LIDC_IDRI_names = np.array(LIDC_IDRI_names)
LIDC_IDRI_names = np.unique(LIDC_IDRI_names)
```

```python
names = np.concatenate((EXACT09_names, LIDC_IDRI_names))

# you can split train/test by yourself
# just show an example

test_names = ['LIDC_IDRI_0698', 'LIDC_IDRI_0710', 'LIDC_IDRI_0810',
        'LIDC_IDRI_0376', 'EXACT09_CASE13', 'LIDC_IDRI_1004',
        'EXACT09_CASE08', 'EXACT09_CASE01', 'EXACT09_CASE05',
        'LIDC_IDRI_0744']
print("test name: "+str(test_names))
train_names = []
for name in names:
    if name not in test_names:
        train_names.append(name)
train_names=np.array(train_names)
print("train names: "+str(train_names))
```

--------------------------------

### Get 3D Image for One Case Function

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Processes a list of image paths for a single case, loads each slice, and stacks them into a single 3D image array.

```python
def get_3d_img_for_one_case(img_path_list):
    img_3d=[]
    for idx, img_path in enumerate(img_path_list):
        print("progress: "+str(idx/len(img_path_list))+"; "+str(img_path), end="\r")
        img_slice, frame_num, _, _ = loadFile(img_path)
        assert frame_num==1
        img_3d.append(img_slice)
    img_3d=np.array(img_3d)
    return img_3d.reshape(img_3d.shape[0], img_3d.shape[2], img_3d.shape[3])
```

--------------------------------

### Prepare Dataset for Iterative Training

Source: https://github.com/antonotnawang/naviairway/blob/main/Get_dataset_info.ipynb

Prepares an extended dataset for an iterative training strategy. It duplicates existing training data based on airway pixel counts to prioritize training on either high or low generation airways.

```python
data_dict_org=train_test_set_dict_EXACT09_LIDC_IDRI_128['train']

import copy
data_dict_extended = copy.deepcopy(data_dict_org)

is_more_big = True # higher freq on airways of low gen (thicker airways)
copy_times_I = 10

for idx, case in enumerate(data_dict_org.keys()):
    if data_dict_org[case]["airway_pixel_num"]>0:
        if is_more_big:
            copy_times_II = np.ceil(data_dict_org[case]["airway_pixel_num_inner"]/data_dict_org[case]["airway_pixel_num_boundary"])
        else:
            if data_dict_org[case]["airway_pixel_num_inner"]==0:
                copy_times_II = np.ceil(data_dict_org[case]["airway_pixel_num_boundary"])
            else:
                copy_times_II = np.ceil(data_dict_org[case]["airway_pixel_num_boundary"]/data_dict_org[case]["airway_pixel_num_inner"])

        for i in range(int(copy_times_I*copy_times_II)):
            data_dict_extended[case+"_copy_"+str(i+1)]=data_dict_org[case]
if is_more_big:
    save_obj(data_dict_extended, "dataset_info/train_dataset_info_EXACT09_LIDC_IDRI_crops_128_extended_"+"more_low_gen_"+str(copy_times_I))
else:
    save_obj(data_dict_extended, "dataset_info/train_dataset_info_EXACT09_LIDC_IDRI_crops_128_extended_"+"more_high_gen_"+str(copy_times_I))
```

--------------------------------

### Access Training Data

Source: https://github.com/antonotnawang/naviairway/blob/main/Get_dataset_info.ipynb

Accesses the training data portion of the created train/test set dictionary. The comment indicates it contains information about image crops for training.

```python
train_test_set_dict_EXACT09_LIDC_IDRI_128["train"]
# it has all info of images crops for training
```

--------------------------------

### Dataset Preparation Steps

Source: https://github.com/antonotnawang/naviairway/blob/main/README.md

Outlines the steps required to prepare the datasets for training.

```text
- Download the two datasets: EXACT09 and LIDC-IDRI.

- Run ```dataset_preprocess_EXACT09.ipynb``` and ```dataset_preprocess_LIDC-IDRI.ipynb``` to preprocess the images.

- Run ```Pre_crop_images.ipynb``` to pre-crop the images to be samll cubes.

- Run ```Get_dataset_info.ipynb``` to generate dataset info which are pkl files (our iterative training strategy (training with focus on airways of low and high generations iteratively) is achieved by it).
```

--------------------------------

### Display Initial Image Order (Test Cases)

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Prints the case name and the first 7 image filenames for testing cases to show their initial order.

```python
for name in test_case_dict.keys():
    print(name, [test_case_dict[name][i].split("/")[-1] for i in range(7)], end="\n")
```

--------------------------------

### Option 2: Load a pre-existing 3D .nii.gz image

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Loads a 3D NIfTI image directly from a file path.

```python
raw_img_path = "test_data/test_image.nii.gz"
```

```python
raw_img = load_one_CT_img(raw_img_path)
```

--------------------------------

### Download Link and Password

Source: https://github.com/antonotnawang/naviairway/blob/main/README.md

Provides a link to download the demonstration data and the password for extraction.

```text
password: ```2333```
```

--------------------------------

### Get DataFrame of Centerline

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Helper function to convert connection dictionary into a pandas DataFrame for centerline visualization.

```python
# # show the airway centerline
def get_df_of_centerline(connection_dict):
    d = {}
    d["x"] = []
    d["y"] = []
    d["z"] = []
    d["val"] = []
    d["text"] = []
    for item in connection_dict.keys():
        print(item, end="\r")
        d["x"].append(connection_dict[item]['loc'][0])
        d["y"].append(connection_dict[item]['loc'][1])
        d["z"].append(connection_dict[item]['loc'][2])
        d["val"].append(connection_dict[item]['generation'])
        d["text"].append(str(item)+": "+str({"before":connection_dict[item]["before"], "next":connection_dict[item]["next"$})))
    df = pd.DataFrame(data=d)
    return df

# # show the airway centerline
def get_df_of_line_of_centerline(connection_dict):
    d = {}
    for label in connection_dict.keys():
        if connection_dict[label]["before"][0]==0:
            start_label = label
            break
    def get_next_point(connection_dict, current_label, d, idx):
        while (idx in d.keys()):
            idx+=1
        
        d[idx]={}
        if "x" not in d[idx].keys():
            d[idx]["x"]=[]
        if "y" not in d[idx].keys():
            d[idx]["y"]=[]
        if "z" not in d[idx].keys():
            d[idx]["z"]=[]
        if "val" not in d[idx].keys():
            d[idx]["val"]=[]
        
        before_label = connection_dict[current_label]["before"][0]
        if before_label not in connection_dict.keys():
            before_label = current_label
        d[idx]["x"].append(connection_dict[before_label]["loc"][0])
        d[idx]["y"].append(connection_dict[before_label]["loc"][1])
        d[idx]["z"].append(connection_dict[before_label]["loc"][2])
        d[idx]["val"].append(connection_dict[before_label]["generation"])
        
        d[idx]["x"].append(connection_dict[current_label]["loc"][0])
        d[idx]["y"].append(connection_dict[current_label]["loc"][1])
        d[idx]["z"].append(connection_dict[current_label]["loc"][2])
        d[idx]["val"].append(connection_dict[current_label]["generation"])
        
        if connection_dict[current_label]["number_of_next"]==0:
            return
        else:
            for next_label in connection_dict[current_label]["next"]:
                get_next_point(connection_dict, next_label, d, idx+1)
    
    get_next_point(connection_dict, start_label, d,0)
    return d
```

--------------------------------

### Create Output Directories

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Creates necessary directories for saving the processed 3D images (train, test, and train_label).

```python
output_file_path = "TEMP" #"/data/Airway/EXACT09_3D"

if not os.path.exists(output_file_path):
    os.mkdir(output_file_path)

if not os.path.exists(output_file_path+"/train"):
    os.mkdir(output_file_path+"/train")

if not os.path.exists(output_file_path+"/test"):
    os.mkdir(output_file_path+"/test")

if not os.path.exists(output_file_path+"/train_label"):
    os.mkdir(output_file_path+"/train_label")
```

--------------------------------

### Display Initial Image Order

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Prints the case name and the first 7 image filenames for training cases to show their initial order.

```python
for name in train_case_dict.keys():
    print(name, [train_case_dict[name][i].split("/")[-1] for i in range(7)], end="\n")
```

--------------------------------

### Preprocess and Save a Single 3D Image

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Demonstrates how to process one case ('CASE20' from training data) and save the resulting 3D image as a NIfTI file.

```python
# preprocess and save one case
img_3d = get_3d_img_for_one_case(train_case_dict["CASE20"])
sitk.WriteImage(sitk.GetImageFromArray(img_3d), output_file_path+"/train/CASE20.nii.gz")
```

--------------------------------

### Crop Parameters

Source: https://github.com/antonotnawang/naviairway/blob/main/Pre_crop_images.ipynb

Defines the size of the cubic crop and the stride for moving the crop window.

```python
crop_cube_size=(256, 256, 256)
stride=(128,128,128)
```

--------------------------------

### Create Case Dictionaries

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Organizes image file paths into dictionaries, mapping case names to lists of their corresponding image file paths. It sorts the case names and image names alphabetically.

```python
train_case_dict = dict()
test_case_dict = dict()

train_case_names=os.listdir(raw_train_file_path)
train_case_names.sort()

test_case_names=os.listdir(raw_test_file_path)
test_case_names.sort()

for case_name in train_case_names:
    imgs=os.listdir(raw_train_file_path+"/"+case_name)
    imgs.sort()
    img_path_list = []
    for img in imgs:
        img_path_list.append(raw_train_file_path+"/"+case_name+"/"+img)
    train_case_dict[case_name]=img_path_list

for case_name in test_case_names:
    imgs=os.listdir(raw_test_file_path+"/"+case_name)
    imgs.sort()
    img_path_list = []
    for img in imgs:
        img_path_list.append(raw_test_file_path+"/"+case_name+"/"+img)
    test_case_dict[case_name]=img_path_list
```

--------------------------------

### Preprocess and Save All 3D Images

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Iterates through all training and testing cases, converts them into 3D NIfTI images, and saves them to the respective output directories.

```python
# preprocess and save cases
for case in train_case_dict.keys():
    print(case, end="\n")
    img_3d = get_3d_img_for_one_case(train_case_dict[case])
    sitk.WriteImage(sitk.GetImageFromArray(img_3d),
                    output_file_path+"/train/"+case+'.nii.gz')
    

for case in test_case_dict.keys():
    print(case, end="\n")
    img_3d = get_3d_img_for_one_case(test_case_dict[case])
    sitk.WriteImage(sitk.GetImageFromArray(img_3d),
                    output_file_path+"/test/"+case+'.nii.gz')
```

--------------------------------

### Define Raw File Paths

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Specifies the root directories for the raw training and testing data.

```python
raw_train_file_path = "/data/Airway/EXACT09/Training"
raw_test_file_path = "/data/Airway/EXACT09/Testing"
```

--------------------------------

### Pre-crop EXACT09 Dataset

Source: https://github.com/antonotnawang/naviairway/blob/main/Pre_crop_images.ipynb

This code snippet demonstrates pre-cropping images and labels from the EXACT09 dataset and saving them as .npy files.

```python
crop_cube_size=(128,128,128)
stride=(64,64,64)

# -----INPUT-----
output_file_path = "Precrop_dataset_for_EXACT09"

if not os.path.exists(output_file_path+"/image/"):
    os.makedirs(output_file_path+"/image/")

if not os.path.exists(output_file_path+"/label/"):
    os.makedirs(output_file_path+"/label/")

raw_data_dict = EXACT09_data_dict
# -----END-----

for i, case in enumerate(raw_data_dict.keys()):
    raw_img = io.imread(raw_data_dict[case]["image"], plugin='simpleitk')
    label_img = io.imread(raw_data_dict[case]["label"], plugin='simpleitk')
    
    raw_img_crop_list = crop_one_3d_img(raw_img, crop_cube_size=crop_cube_size, stride=stride)
    label_img_crop_list = crop_one_3d_img(label_img, crop_cube_size=crop_cube_size, stride=stride)
    
    assert len(raw_img_crop_list)==len(label_img_crop_list)
    
    for idx in range(len(raw_img_crop_list)):
        print("progress: "+str(idx)+"th crop | "+str(i)+"th 3d img: "+str(case), end="\r")
        
        #sitk.WriteImage(sitk.GetImageFromArray(raw_img_crop_list[idx]), output_file_path+"/image/"+case+"_"+str(idx)+".nii.gz")
        #sitk.WriteImage(sitk.GetImageFromArray(label_img_crop_list[idx]), output_file_path+"/label/"+case+"_"+str(idx)+".nii.gz")
        
        np.save(output_file_path+"/image/"+case+"_"+str(idx)+".npy", raw_img_crop_list[idx])
        np.save(output_file_path+"/label/"+case+"_"+str(idx)+".npy", label_img_crop_list[idx])
```

--------------------------------

### Import Libraries

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Imports necessary libraries for data manipulation, image processing, and file handling.

```python
import os
import numpy as np
import SimpleITK as sitk
from PIL import Image
import pydicom
import cv2
import nibabel as nib
```

--------------------------------

### Load Semi-Supervised Model Checkpoint

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Initializes another SegAirwayModel for semi-supervised learning and loads its state dictionary from a different checkpoint file.

```python
model_semi_supervise_learning=SegAirwayModel(in_channels=1, out_channels=2)
model_semi_supervise_learning.to(device)
load_path = "checkpoint/checkpoint_semi_supervise_learning.pkl"
checkpoint = torch.load(load_path)
model_semi_supervise_learning.load_state_dict(checkpoint['model_state_dict'])
```

--------------------------------

### Find Image Files

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

Recursively searches through directories to find image files for each case and stores their paths in a dictionary.

```python
LIDC_IDRI_raw_img_dict = {}
img_names = os.listdir(LIDC_IDRI_raw_path)
img_names.sort()
img_names

path_to_a_case = ""

def find_imgs(input_path):
    global path_to_a_case
    items = os.listdir(input_path)
    items.sort()
    #print("There are "+str(items)+" in "+str(input_path))
    All_file_flag = True
    for item in items:
        if os.path.isdir(input_path+"/"+item):
            All_file_flag = False
            break
    if All_file_flag and len(items)>10:
        #print("we get "+str(input_path))
        path_to_a_case = input_path
    else:
        for item in items:
            if os.path.isdir(input_path+"/"+item):
                #print("open filefloder: "+str(input_path+"/"+item))
                find_imgs(input_path+"/"+item)
    
for idx, img_name in enumerate(img_names):
    print(idx/len(img_names), end="\r")
    find_imgs(LIDC_IDRI_raw_path+"/"+img_name)
    slice_names = os.listdir(path_to_a_case)
    slice_names.sort()
    LIDC_IDRI_raw_img_dict[img_name]=[]
    for slice_name in slice_names:
        if slice_name.split(".")[1]=="dcm":
            LIDC_IDRI_raw_img_dict[img_name].append(path_to_a_case+"/"+slice_name)
```

--------------------------------

### Build raw_data_dict for LIDC-IDRI dataset

Source: https://github.com/antonotnawang/naviairway/blob/main/Pre_crop_images.ipynb

This code snippet initializes a dictionary to store raw data paths for the LIDC-IDRI dataset, including image and label paths.

```python
import os
import numpy as np
import skimage.io as io
import SimpleITK as sitk


# build the raw_data_dict for train

raw_data_dict = dict()

# LIDC-IDRI data
LIDC_IDRI_file_path = "/data/Airway/LIDC-IDRI_3D/annotated_data"
LIDC_IDRI_raw_path = LIDC_IDRI_file_path+"/image"
LIDC_IDRI_label_path = LIDC_IDRI_file_path+"/label"

LIDC_IDRI_raw_names = os.listdir(LIDC_IDRI_raw_path)
LIDC_IDRI_raw_names.sort()

LIDC_IDRI_label_names = os.listdir(LIDC_IDRI_label_path)
LIDC_IDRI_label_names.sort()

case_names = []

for case in LIDC_IDRI_raw_names:
    temp = case.split(".")[0]
    #print(temp)
    case_names.append(temp)
    raw_data_dict["LIDC_IDRI_"+temp]={}
    raw_data_dict["LIDC_IDRI_"+temp]["image"]=LIDC_IDRI_raw_path+"/"+case

for case in LIDC_IDRI_label_names:
    temp = case.split(".")[0]
    #print(temp)
    if temp in case_names:
        raw_data_dict["LIDC_IDRI_"+temp]["label"]=LIDC_IDRI_label_path+"/"+case

LIDC_IDRI_data_dict = raw_data_dict
```

--------------------------------

### Load Image File Function

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

A helper function to load an image file using SimpleITK and return its array, frame count, width, and height.

```python
def loadFile(filename):
    ds = sitk.ReadImage(filename)
    img_array = sitk.GetArrayFromImage(ds)
    frame_num, width, height = img_array.shape
    return img_array, frame_num, width, height
```

--------------------------------

### Apply Image Name Resorting

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Applies the `resort_names` function to specific cases in both training and testing dictionaries, using 'I' as the indicator for sorting.

```python
resort_names(test_case_dict, "CASE37", "I")
resort_names(test_case_dict, "CASE36", "I")
resort_names(test_case_dict, "CASE38", "I")
resort_names(train_case_dict, "CASE16", "I")
resort_names(train_case_dict, "CASE17", "I")
resort_names(train_case_dict, "CASE18", "I")
```

--------------------------------

### Load Model Checkpoint

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Initializes a SegAirwayModel and loads its state dictionary from a specified checkpoint file.

```python
model=SegAirwayModel(in_channels=1, out_channels=2)
model.to(device)
load_path = "checkpoint/checkpoint.pkl"
checkpoint = torch.load(load_path)
model.load_state_dict(checkpoint['model_state_dict'])
```

--------------------------------

### Set Output Path

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

Defines the directory where the processed NIfTI images will be saved.

```python
# set output path

output_image_path = "LIDC-IDRI"
```

--------------------------------

### Inspect DICOM Headers

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Iterates through the test cases, reads the first DICOM file of each, and prints specific metadata (Pixel Spacing).

```python
for case in test_case_dict.keys():
    print(case, end="\n")
    dicom_file = pydicom.dcmread(test_case_dict[case][0])
    keys = list(dicom_file.keys())
    for idx, key in enumerate(keys):
        if str(key) == "(0028, 0030)":
            i = idx
            break
    print(dicom_file[list(dicom_file.keys())[i]])
    print("----------")
```

--------------------------------

### EXACT09 Dataset File Structure

Source: https://github.com/antonotnawang/naviairway/blob/main/README.md

Illustrates the expected file structure for the EXACT09 dataset.

```text
/data/Airway/EXACT09
    /Training
        /CASE01
            /1093782
            /1093783
            ...
        /CASE02
        ...
    /Testing
        /CASE21
        ...
```

--------------------------------

### EXACT09 Data Processing

Source: https://github.com/antonotnawang/naviairway/blob/main/Pre_crop_images.ipynb

This code snippet processes the EXACT09 dataset by iterating through raw and label directories, creating a dictionary `raw_data_dict` that stores the file paths for images and labels, keyed by a combined dataset and case name (e.g., 'EXACT09_case_name').

```python
raw_data_dict = dict()

EXACT09_file_path = "/data/Airway/EXACT09_3D"
EXACT09_train_raw_path = EXACT09_file_path+"/train"
EXACT09_train_label_path = EXACT09_file_path+"/train_label"

EXACT09_raw_names = os.listdir(EXACT09_train_raw_path)
EXACT09_raw_names.sort()

EXACT09_label_names = os.listdir(EXACT09_train_label_path)
EXACT09_label_names.sort()

case_names = []

for case in EXACT09_raw_names:
    temp = case.split(".")[0]
    case_names.append(temp)
    raw_data_dict["EXACT09_"+temp]={}
    raw_data_dict["EXACT09_"+temp]["image"]=EXACT09_train_raw_path+"/"+case

for case in EXACT09_label_names:
    temp = case.split("_")[0]
    if temp in case_names:
        raw_data_dict["EXACT09_"+temp]["label"]=EXACT09_train_label_path+"/"+case

EXACT09_data_dict = raw_data_dict
```

--------------------------------

### Import Libraries

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Imports all necessary libraries for the NaviAirway pipeline, including numerical computation, deep learning, image processing, and visualization tools.

```python
import numpy as np
import torch
import copy
import pandas as pd
import SimpleITK as sitk
from PIL import Image
import pydicom
import cv2
import nibabel as nib
import os
import skimage.io as io

import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.express as px

from func.model_arch import SegAirwayModel
from func.model_run import get_image_and_label, get_crop_of_image_and_label_within_the_range_of_airway_foreground, \
semantic_segment_crop_and_cat, dice_accuracy
from func.post_process import post_process, add_broken_parts_to_the_result, find_end_point_of_the_airway_centerline, \
get_super_vox, Cluster_super_vox, delete_fragments, get_outlayer_of_a_3d_shape, get_crop_by_pixel_val, fill_inner_hole
from func.detect_tree import tree_detection
from func.ulti import save_obj, load_obj, get_and_save_3d_img_for_one_case,load_one_CT_img, \
get_df_of_centerline, get_df_of_line_of_centerline
```

--------------------------------

### Load Raw Image Paths

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

This snippet iterates through directories to find and store paths to DICOM image files.

```python
LIDC_IDRI_all_raw_img_dict = {}
for case in os.listdir(LIDC_IDRI_raw_path):
    for name_1 in os.listdir(LIDC_IDRI_raw_path+"/"+case):
        for name_2 in os.listdir(LIDC_IDRI_raw_path+"/"+case+"/"+name_1):
            img_names = os.listdir(LIDC_IDRI_raw_path+"/"+case+"/"+name_1+"/"+name_2)
            img_names.sort()
            if len(img_names)>10:
                LIDC_IDRI_all_raw_img_dict[case.split("-")[2]] = []
                for slice_name in img_names:
                    if slice_name.split(".")[1]=="dcm":
                        LIDC_IDRI_all_raw_img_dict[case.split("-")[2]].append(LIDC_IDRI_raw_path+"/"+case+"/"+name_1+"/"+name_2+"/"+slice_name)
```

--------------------------------

### Process and Save Images

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

Iterates through each case, loads the 3D image, and saves it as a NIfTI file (.nii.gz) in the specified output directory.

```python
if not os.path.exists(output_image_path):
    os.mkdir(output_image_path)
    
for case in LIDC_IDRI_raw_img_dict.keys():
    img_3d = get_3d_img_for_one_case(LIDC_IDRI_raw_img_dict[case])
    sitk.WriteImage(sitk.GetImageFromArray(img_3d),
                    output_image_path+"/"+case+".nii.gz")
```

--------------------------------

### Run Segmentation (Supervised Model)

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Performs semantic segmentation on the raw image using the supervised model. The output is thresholded to create a binary segmentation mask.

```python
seg_result = semantic_segment_crop_and_cat(raw_img, model, device,
                                           crop_cube_size=[32, 128, 128], stride=[16, 64, 64],
                                           windowMin=-1000, windowMax=600)
seg_onehot = np.array(seg_result>threshold, dtype=np.int)
```

--------------------------------

### Import Libraries

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

Imports necessary Python libraries for image processing and file handling.

```python
import numpy as np
import os
import SimpleITK as sitk
from PIL import Image
import pydicom
import cv2
import nibabel as nib
import pydicom
```

--------------------------------

### Load Image Slice

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

Loads a single image slice from a file using SimpleITK and returns the image array along with its dimensions.

```python
def loadFile(filename):
    ds = sitk.ReadImage(filename)
    #pydicom.dcmread(filename)
    img_array = sitk.GetArrayFromImage(ds)
    frame_num, width, height = img_array.shape
    #print("frame_num, width, height: "+str((frame_num, width, height)))
    return img_array, frame_num, width, height
```

--------------------------------

### Print Case Names

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

Prints the names of all cases found in the dataset.

```python
print("Show the case names: "+str(LIDC_IDRI_raw_img_dict.keys()))
```

--------------------------------

### Map Annotations to Images

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

This snippet associates image directories with their corresponding annotation files.

```python
LIDC_IDRI_annotated = {}
LIDC_IDRI_annotation_path = LIDC_IDRI_anno_path

for case in os.listdir(LIDC_IDRI_raw_path):
    for name_1 in os.listdir(LIDC_IDRI_raw_path+"/"+case):
        for name_2 in os.listdir(LIDC_IDRI_raw_path+"/"+case+"/"+name_1):
            if name_2 in anno_names:
                print(LIDC_IDRI_raw_path+"/"+case+"/"+name_1+"/"+name_2)
                img_names = os.listdir(LIDC_IDRI_raw_path+"/"+case+"/"+name_1+"/"+name_2)
                img_names.sort()
                LIDC_IDRI_annotated[case.split("-")[2]] = {}
                LIDC_IDRI_annotated[case.split("-")[2]]["image"] = []
                LIDC_IDRI_annotated[case.split("-")[2]]["label"] = LIDC_IDRI_annotation_path+"/"+name_2+".nii.gz"
                for slice_name in img_names:
                    if slice_name.split(".")[1]=="dcm":
                        LIDC_IDRI_annotated[case.split("-")[2]]["image"].append(LIDC_IDRI_raw_path+"/"+case+"/"+name_1+"/"+name_2+"/"+slice_name)
```

--------------------------------

### Run Segmentation (Semi-Supervised Model)

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Performs semantic segmentation on the raw image using the semi-supervised model. The output is thresholded to create a binary segmentation mask.

```python
threshold = 0.5
seg_result_semi_supervise_learning = semantic_segment_crop_and_cat(raw_img, model_semi_supervise_learning, device,
                                                                   crop_cube_size=[32, 128, 128], stride=[16, 64, 64],
                                                                   windowMin=-1000, windowMax=600)
seg_onehot_semi_supervise_learning = np.array(seg_result_semi_supervise_learning>threshold, dtype=np.int)
```

--------------------------------

### Pre-crop LIDC-IDRI Dataset

Source: https://github.com/antonotnawang/naviairway/blob/main/Pre_crop_images.ipynb

This code snippet shows how to pre-crop images and labels from the LIDC-IDRI dataset and save them as .npy files.

```python
output_file_path = "Precrop_dataset_for_LIDC-IDRI"

if not os.path.exists(output_file_path+"/image/"):
    os.makedirs(output_file_path+"/image/")

if not os.path.exists(output_file_path+"/label/"):
    os.makedirs(output_file_path+"/label/")

raw_data_dict = LIDC_IDRI_data_dict

for i, case in enumerate(raw_data_dict.keys()):
    raw_img = io.imread(raw_data_dict[case]["image"], plugin='simpleitk')
    label_img = io.imread(raw_data_dict[case]["label"], plugin='simpleitk')
    
    raw_img_crop_list = crop_one_3d_img(raw_img, crop_cube_size=crop_cube_size, stride=stride)
    label_img_crop_list = crop_one_3d_img(label_img, crop_cube_size=crop_cube_size, stride=stride)
    
    assert len(raw_img_crop_list)==len(label_img_crop_list)
    
    for idx in range(len(raw_img_crop_list)):
        print("progress: "+str(idx)+"th crop | "+str(i)+"th 3d img: "+str(case), end="\r")
        
        #sitk.WriteImage(sitk.GetImageFromArray(raw_img_crop_list[idx]), output_file_path+"/image/"+case+"_"+str(idx)+".nii.gz")
        #sitk.WriteImage(sitk.GetImageFromArray(label_img_crop_list[idx]), output_file_path+"/label/"+case+"_"+str(idx)+".nii.gz")
        
        np.save(output_file_path+"/image/"+case+"_"+str(idx)+".npy", raw_img_crop_list[idx])
        np.save(output_file_path+"/label/"+case+"_"+str(idx)+".npy", label_img_crop_list[idx])
```

--------------------------------

### LIDC-IDRI Dataset File Structure

Source: https://github.com/antonotnawang/naviairway/blob/main/README.md

Illustrates the expected file structure for the LIDC-IDRI dataset.

```text
/data/Airway/LIDC-IDRI
    /LIDC-IDRI-0001
        /1.3.6.1.4.1.14519.5.2.1.6279.6001.298806137288633453246975630178
            /1.3.6.1.4.1.14519.5.2.1.6279.6001.179049373636438705059720603192
                /1-001.dcm
                /1-002.dcm
                ...
    /LIDC-IDRI-0002
    ...
```

--------------------------------

### Inference Command

Source: https://github.com/antonotnawang/naviairway/blob/main/README.md

Command to run the inference pipeline.

```text
Run ```NaviAirway_pipeline.ipynb```.
```

--------------------------------

### Training Data Label Extraction

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Code to extract and save labels for training data from MHD files and convert them to NIfTI format.

```python
# get labels of the training data
label_file_path = "/data/Airway/EXACT09_annotation"

for case in train_case_dict.keys():
    print(case)
    img_label_arr, _, _, _ = loadFile(label_file_path+'/'+case+'.mhd')
    sitk.WriteImage(sitk.GetImageFromArray(img_label_arr),
                    output_file_path+"/train_label/"+case+'_label.nii.gz')
```

--------------------------------

### Produce 3D OBJ Mesh

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Converts the segmentation masks into 3D OBJ mesh files.

```python
from func.points_to_mesh import produce_3d_obj

produce_3d_obj(seg_processed, output_file_path+"/segmentation")

produce_3d_obj(seg_processed_II, output_file_path+"/segmentation_add_broken_parts")
```

--------------------------------

### Visualize Raw Image with Segmentation

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Displays the raw image with overlaid segmentation contours for a specific slice.

```python
N=200
plt.figure(figsize=(20,20))
plt.title("raw image with label after post process I")
plt.imshow(raw_img[N,:,:], cmap='gray')
plt.contour(seg_processed[N,:,:], colors='r', linestyles='-')
```

--------------------------------

### Print Data Dictionaries

Source: https://github.com/antonotnawang/naviairway/blob/main/Pre_crop_images.ipynb

This code snippet prints the contents of two dictionaries: `LIDC_IDRI_data_dict` and `EXACT09_data_dict`. This is likely used to inspect the processed data structures.

```python
print(LIDC_IDRI_data_dict)
print(EXACT09_data_dict)
```

--------------------------------

### Set Device

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Configures the computation device to be used, prioritizing GPU (CUDA) if available, otherwise falling back to CPU.

```python
device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
```

--------------------------------

### Define LIDC-IDRI Raw Path

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_LIDC-IDRI.ipynb

Specifies the root directory for the LIDC-IDRI raw image data.

```python
# the path to LIDC-IDRI raw images

LIDC_IDRI_raw_path = "/data/Airway/LIDC-IDRI"
```

--------------------------------

### 2D Visualization: Top View of Model Output

Source: https://github.com/antonotnawang/naviairway/blob/main/NaviAirway_pipeline.ipynb

Displays a top-down view of the model's segmentation output using a grayscale colormap.

```python
plt.figure(figsize=(20,20))
plt.title("model output (top view)")
plt.imshow(np.sum(seg_result_semi_supervise_learning, axis=1), cmap='gray')
```

--------------------------------

### Resort Image Names Function

Source: https://github.com/antonotnawang/naviairway/blob/main/dataset_preprocess_EXACT09.ipynb

Defines a function to resort image names within a specific case based on a numerical indicator in their filenames.

```python
def resort_names(case_dict, chosen_name, indicator):
    new_path_idx_list = []
    for path in case_dict[chosen_name]:
        name = path.split("/")[-1]
        new_path_idx_list.append(int(name.split(indicator)[1]))
    new_path_idx_list = np.array(new_path_idx_list)
    locs = np.argsort(new_path_idx_list)

    new_path_list = []
    for i in locs:
        new_path_list.append(case_dict[chosen_name][i])

    case_dict[chosen_name]=new_path_list
```