### Installation: Clone Repository and Run Example Source: https://github.com/jeremykawahara/derm7pt/blob/master/README.md These commands show how to clone the derm7pt repository and execute the minimal example script. Ensure you change the directory path to match your data folder. ```bash git clone https://github.com/jeremykawahara/derm7pt.git cd derm7pt python minimal_example.py '/local-scratch/jer/data/argenziano/release_v0' ``` -------------------------------- ### Import Libraries and Setup Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/web_images.ipynb Imports necessary libraries like matplotlib, pandas, and json, and sets up the environment for autoreload and inline plotting. It also configures pandas display options and adjusts the system path. ```python from __future__ import print_function, division %load_ext autoreload %autoreload 2 %matplotlib inline import matplotlib.pyplot as plt import pandas as pd pd.set_option('display.max_columns', 500) import json import sys, os from shutil import copyfile sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..'))) ``` -------------------------------- ### Check Pandas Version Source: https://github.com/jeremykawahara/derm7pt/blob/master/version_check.ipynb Verify the Pandas library version. Ensure Pandas is installed and imported. ```python import pandas pandas.__version__ ``` -------------------------------- ### Get Validation Image Paths Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Demonstrates how to retrieve validation image paths. These are typically used for hyper-parameter tuning. ```python # Note, you can get images for validation by: valid_derm_paths = derm_data_group.get_img_paths(data_type='valid', img_type='derm') print(len(valid_derm_paths)) # We don't use them within this demo, but they could/should be used when choosing hyper-parameters. ``` -------------------------------- ### Check Python Version Source: https://github.com/jeremykawahara/derm7pt/blob/master/version_check.ipynb Use this snippet to verify the installed Python version. It requires the `sys` module. ```python # Python version. import sys sys.version ``` -------------------------------- ### Check NumPy Version Source: https://github.com/jeremykawahara/derm7pt/blob/master/version_check.ipynb Verify the installed NumPy version. Ensure NumPy is installed and imported as `np`. ```python import numpy as np np.__version__ ``` -------------------------------- ### Load IPython Extensions and Libraries Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Loads necessary IPython extensions and Python libraries for data analysis and visualization. Ensure these libraries are installed. ```python %load_ext autoreload %autoreload 2 %matplotlib inline import matplotlib.pyplot as plt plt.style.use('ggplot') import pandas as pd pd.set_option('display.max_columns', 500) import sys, os from sklearn.linear_model import LogisticRegression sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..'))) ``` -------------------------------- ### Minimal Example: Load and Preprocess Dermatology Data Source: https://github.com/jeremykawahara/derm7pt/blob/master/README.md This snippet demonstrates how to load the Seven-Point Checklist Dermatology Dataset using the Derm7PtDatasetGroupInfrequent class. It preprocesses the data by grouping infrequent class labels and assigning numeric values. ```python import sys, os import pandas as pd sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..'))) # To import derm7pt from derm7pt.dataset import Derm7PtDatasetGroupInfrequent # Change this line to your data directory. dir_release = '/local-scratch/jer/data/argenziano/release_v0' # Dataset after grouping infrequent labels. derm_data = Derm7PtDatasetGroupInfrequent( dir_images=os.path.join(dir_release, 'images'), metadata_df=pd.read_csv(os.path.join(dir_release, 'meta/meta.csv')), train_indexes=list(pd.read_csv(os.path.join(dir_release, 'meta/train_indexes.csv'))['indexes']), valid_indexes=list(pd.read_csv(os.path.join(dir_release, 'meta/valid_indexes.csv'))['indexes']), test_indexes=list(pd.read_csv(os.path.join(dir_release, 'meta/test_indexes.csv'))['indexes'])) # Outputs to screen the preprocessed dataset in a Pandas format. derm_data.df ``` -------------------------------- ### Check Scikit-learn Version Source: https://github.com/jeremykawahara/derm7pt/blob/master/version_check.ipynb Check the installed version of Scikit-learn. Import the `sklearn` module to access its version. ```python import sklearn sklearn.__version__ ``` -------------------------------- ### Get Training Image Paths and Labels Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Retrieves training image paths for dermatology and clinic, along with corresponding labels. The labels dictionary contains abbreviations for diagnosis and seven criteria. ```python # Get the dermatology and clinic training images and corresponding labels. train_derm_paths = derm_data_group.get_img_paths(data_type='train', img_type='derm') train_clinic_paths = derm_data_group.get_img_paths(data_type='train', img_type='clinic') train_labels = derm_data_group.get_labels(data_type='train', one_hot=False) # The 8 abbreviations that indicate the different types of categories # i.e., 1 diagnosis + 7 critiera. print(train_labels.keys()) ``` -------------------------------- ### Check Matplotlib Version Source: https://github.com/jeremykawahara/derm7pt/blob/master/version_check.ipynb Confirm the Matplotlib version. This requires the `matplotlib` library to be installed. ```python import matplotlib matplotlib.__version__ ``` -------------------------------- ### Get Testing Image Paths and Labels Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Retrieves testing image paths for dermatology and clinic, along with corresponding labels. This is used for evaluating the trained model. ```python # Get the dermatology and clinic test images and corresponding labels. test_derm_paths = derm_data_group.get_img_paths(data_type='test', img_type='derm') test_clinic_paths = derm_data_group.get_img_paths(data_type='test', img_type='clinic') test_labels = derm_data_group.get_labels(data_type='test', one_hot=False) ``` -------------------------------- ### Define Data Directories Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Sets up the directory paths for the dataset's release meta-data and images. Make sure to change 'dir_release' to your actual data directory. ```python # CHANGE THIS LINE TO YOUR DATA DIRECTORY. dir_release = '/local-scratch/jer/data/argenziano/release_v0' dir_meta = os.path.join(dir_release, 'meta') dir_images = os.path.join(dir_release, 'images') ``` -------------------------------- ### Initialize Derm7Pt Datasets Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Creates instances of Derm7PtDataset and Derm7PtDatasetGroupInfrequent. The first handles the full dataset, while the second groups infrequent labels. Both require image directory, metadata, and data split indices. ```python # The full dataset before any grouping of the labels. derm_data = Derm7PtDataset(dir_images=dir_images, metadata_df=meta_df.copy(), # Copy as is modified. train_indexes=train_indexes, valid_indexes=valid_indexes, test_indexes=test_indexes) # The dataset after grouping infrequent labels. derm_data_group = Derm7PtDatasetGroupInfrequent(dir_images=dir_images, metadata_df=meta_df.copy(), # Copy as is modified. train_indexes=train_indexes, valid_indexes=valid_indexes, test_indexes=test_indexes) ``` -------------------------------- ### Load Dataset Metadata and Indices Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Reads CSV files to load the main metadata, and the indices for training, validation, and testing sets. These are crucial for initializing the dataset objects. ```python meta_df = pd.read_csv(os.path.join(dir_meta, 'meta.csv')) train_indexes = list(pd.read_csv(os.path.join(dir_meta, 'train_indexes.csv'))['indexes']) valid_indexes = list(pd.read_csv(os.path.join(dir_meta, 'valid_indexes.csv'))['indexes']) test_indexes = list(pd.read_csv(os.path.join(dir_meta, 'test_indexes.csv'))['indexes']) ``` -------------------------------- ### Load and Inspect Derm7Pt Dataset Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/minimal_example.ipynb Use this snippet to load the Derm7Pt dataset with infrequent labels grouped. Ensure the `dir_release` variable points to your data directory. This code preprocesses the dataset and displays its initial rows. ```python import sys, os import pandas as pd sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..'))) # To import derm7pt from derm7pt.dataset import Derm7PtDatasetGroupInfrequent # Change this line to your data directory. dir_release = '/local-scratch/jer/data/argenziano/release_v0' # Dataset after grouping infrequent labels. derm_data = Derm7PtDatasetGroupInfrequent( dir_images=os.path.join(dir_release, 'images'), metadata_df=pd.read_csv(os.path.join(dir_release, 'meta/meta.csv')), train_indexes=list(pd.read_csv(os.path.join(dir_release, 'meta/train_indexes.csv'))['indexes']), valid_indexes=list(pd.read_csv(os.path.join(dir_release, 'meta/valid_indexes.csv'))['indexes']), test_indexes=list(pd.read_csv(os.path.join(dir_release, 'meta/test_indexes.csv'))['indexes'])) # Outputs to screen the first 5 rows of the preprocessed dataset in a Pandas format. derm_data.df.head(n=5) ``` -------------------------------- ### Load Pretrained MobileNet Model Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Loads a pre-trained MobileNet model for feature extraction. Ensure the input shape is compatible with the model. ```python # Choose a pretrained model to extract features from. # Can use any model. Just have to make sure the input_shape is appropriate. from keras.applications.mobilenet import MobileNet, preprocess_input input_shape = (224, 224, 3) model = MobileNet(include_top=False, input_shape=input_shape) ``` -------------------------------- ### Check Pillow Version Source: https://github.com/jeremykawahara/derm7pt/blob/master/version_check.ipynb Confirm the Pillow (PIL fork) version. Import the `PIL` module to check its version. ```python import PIL PIL.__version__ ``` -------------------------------- ### HTML Structure Definitions Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/web_images.ipynb Defines the HTML header and footer templates used for creating web pages. These include Bootstrap CSS for styling and container divs for layout. ```python html_header = '''
''' html_footer = '''
''' ``` -------------------------------- ### Select Category for Linear Model Training Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Selects a specific category (abbreviation) for training the linear model and retrieves the corresponding label names. ```python # Select one of the categories. abbrev = 'DIAG' # These are the labels associated with the given abbreviation. label_names = derm_data_group.get_label_by_abbrev(abbrev).abbrevs.values ``` -------------------------------- ### Display Dataset Statistics Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Calculates and displays statistics for the initialized Derm7PtDataset, including the total number of cases and the counts for training, validation, and testing sets. ```python derm_data.dataset_stats() ``` -------------------------------- ### Define Crop Amount Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Sets the amount to crop from the image boundaries. This is used to remove black borders before feature extraction. ```python # Apply cropping around the boundary of the images to get rid of the black boundary. crop_amount= 25 ``` -------------------------------- ### Generate Clinic HTML Page Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/web_images.ipynb Generates an HTML page for clinical images. It reads metadata, constructs image paths, generates HTML using `html_image_src`, and writes the output to 'clinic.html'. ```python # Clinic HTML page. clinic_paths = [os.path.join(dir_images, img_path) for img_path in meta_df.clinic] html_clinic_image = html_image_src(clinic_paths) html_clinic_out = html_header + html_clinic_image + html_footer with open(os.path.join(dir_write_html, "clinic.html"), "w") as text_file: text_file.write(html_clinic_out) ``` -------------------------------- ### Train Logistic Regression and Predict on Clinical Images Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Trains a logistic regression model on clinical image features and makes predictions. Performance may be worse than derm images due to standardization. ```python # Linear classifier on clinic images. reg = LogisticRegression(C=0.01, class_weight='balanced').fit(train_clinic_f, train_labels[abbrev]) test_preds = reg.predict(test_clinic_f) # Performance is worse for clinical images as they are less standardized than derm images. plot_confusion(y_true=test_labels[abbrev], y_pred=test_preds, labels=label_names, figsize=(6,4)) plt.title(abbrev + ' - clinical images'); ``` -------------------------------- ### Import Derm7Pt Specific Modules Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Imports specific modules from the derm7pt library for dataset handling and visualization. These modules are essential for using the dataset classes and plotting functions. ```python from derm7pt.dataset import Derm7PtDataset, Derm7PtDatasetGroupInfrequent from derm7pt.vis import plot_confusion from derm7pt.kerasutils import deep_features ``` -------------------------------- ### View Processed Metadata Head Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Displays the first few rows of the dataset's metadata after it has been converted to categorical labels. This is useful for inspecting the processed data structure. ```python # Converted the meta-data to categorical (*_numeric) labels. derm_data_group.df.head() ``` -------------------------------- ### Generate Derm HTML Page Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/web_images.ipynb Generates an HTML page for dermoscopic images. It reads metadata, constructs image paths, generates HTML using `html_image_src`, and writes the output to 'derm.html'. ```python from derm7pt.utils import html_image_src dir_write_html = '/local-scratch/jer/data/argenziano/release_v0' dir_images = "images" metadata_csv= os.path.join(dir_write_html, 'meta/meta.csv') meta_df = pd.read_csv(metadata_csv) # Derm HTML page. derm_paths = [os.path.join(dir_images, derm_path) for derm_path in meta_df.derm] html_derm_image = html_image_src(derm_paths) html_derm_out = html_header + html_derm_image + html_footer with open(os.path.join(dir_write_html, "derm.html"), "w") as text_file: text_file.write(html_derm_out) ``` -------------------------------- ### Train Logistic Regression and Predict on Derm Images Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Trains a logistic regression model on dermoscopic image features and makes predictions. Use this for classifying dermoscopic images. ```python # Linear classifier for derm images. reg = LogisticRegression(C=0.01, class_weight='balanced').fit(train_derm_f, train_labels[abbrev]) # Make predictions. test_preds = reg.predict(test_derm_f) # Display a confusion matrix. plot_confusion(y_true=test_labels[abbrev], y_pred=test_preds, labels=label_names, figsize=(6,4)) plt.title(abbrev + ' - dermoscopic images'); ``` -------------------------------- ### Train Logistic Regression for Specific Category (Dots and Globules) Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Trains a logistic regression model on derm images for a specific category, 'Dots and Globules'. Use this to analyze performance for a particular image sub-type. ```python # Example with category "Dots and Globules" using derm images. abbrev = 'DaG' reg = LogisticRegression(C=0.01, class_weight='balanced').fit(train_derm_f, train_labels[abbrev]) test_preds = reg.predict(test_derm_f) label_names = derm_data_group.get_label_by_abbrev(abbrev).abbrevs.values plot_confusion(y_true=test_labels[abbrev], y_pred=test_preds, labels=label_names, figsize=(6,4)) plt.title(abbrev + ' - dermoscopic images'); ``` -------------------------------- ### Display Derm and Clinical Images for a Case Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Displays the derm and clinical images side-by-side for a specified row index, along with the corresponding data information. Ensure the `derm_data_group` object and `plt` are imported and available. ```python # Show the derm and clinical image for a single case. row_index = 3 derm = derm_data_group.derm_image(row_index=row_index) clinic = derm_data_group.clinic_image(row_index=row_index) plt.figure(figsize=(8,4)) plt.subplot(1,2,1) plt.imshow(derm) plt.grid(False) plt.subplot(1,2,2) plt.imshow(clinic) plt.grid(False) # Show the corresponding info. print(derm_data_group.df.iloc[row_index]) ``` -------------------------------- ### Plot Label Distribution (Derm Data) Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Generates a histogram showing the distribution of granular labels in the Derm7pt dataset. Use this to understand the frequency of each specific diagnostic tag before any grouping. ```python # Distribution of the labels before any grouping (most granular labels). derm_data.plot_tags_hist(figsize=(18,8), fontsize=12) ``` -------------------------------- ### Extract Deep Features from Testing Dermatology Images Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Extracts deep features from testing dermatology images. Similar to training data, this can be time-consuming. ```python # Extract features from testing images (might take a few minutes) test_derm_f = deep_features(img_paths=test_derm_paths, model=model, func_preprocess_input=preprocess_input, target_size=input_shape, crop_amount=crop_amount) ``` -------------------------------- ### Check Keras Version Source: https://github.com/jeremykawahara/derm7pt/blob/master/version_check.ipynb Determine the Keras version. Note that Keras might use a specific backend like TensorFlow, as indicated by its output. ```python import keras keras.__version__ ``` -------------------------------- ### Extract Deep Features from Training Dermatology Images Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Extracts deep features from training dermatology images using a specified pre-trained model and preprocessing function. This process may take time, especially without a GPU. ```python # Extract features from training images (might take a few minutes... ideally you have a GPU.) train_derm_f = deep_features(img_paths=train_derm_paths, model=model, func_preprocess_input=preprocess_input, target_size=input_shape, crop_amount=crop_amount) ``` -------------------------------- ### Extract Deep Features from Testing Clinical Images Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Extracts deep features from testing clinical images. Similar to training data, this can be time-consuming. ```python test_clinic_f = deep_features(img_paths=test_clinic_paths, model=model, func_preprocess_input=preprocess_input, target_size=input_shape, crop_amount=crop_amount) print(train_clinic_f.shape) print(test_clinic_f.shape) ``` -------------------------------- ### Plot Label Distribution (Grouped Derm Data) Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Generates a histogram showing the distribution of grouped labels in the Derm7pt dataset. This is useful after clinically grouping less frequent granular labels to simplify the distribution. ```python # Distribution of the labels after the grouping. # We group some clinicially labels as many granular labels occur very infrequently. derm_data_group.plot_tags_hist(figsize=(18,8), fontsize=12) ``` -------------------------------- ### Extract Deep Features from Training Clinical Images Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Extracts deep features from training clinical images. This process may require a significant amount of time. ```python # May take a bit of time. train_clinic_f = deep_features(img_paths=train_clinic_paths, model=model, func_preprocess_input=preprocess_input, target_size=input_shape, crop_amount=crop_amount) ``` -------------------------------- ### Display Feature Extraction Shape Source: https://github.com/jeremykawahara/derm7pt/blob/master/notebooks/example.ipynb Prints the shape of the extracted features for both training and testing dermatology images. The shape indicates (number_of_images, number_of_features). ```python # Number_of_images x number_of_features print(train_derm_f.shape) # 413 images. print(test_derm_f.shape) # 395 images. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.