### Install Project Dependencies

Source: https://github.com/veccam/vectorbrain/blob/main/README.md

Install all required Python packages listed in the requirements.txt file.

```sh
pip install -r requirements.txt
```

--------------------------------

### Start Jupyter Notebook Server

Source: https://github.com/veccam/vectorbrain/blob/main/README.md

Launch the Jupyter Notebook server to access and run the provided notebooks.

```sh
jupyter notebook
```

--------------------------------

### Import Libraries

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Imports necessary Python libraries for data manipulation, file operations, and utility functions. Ensure these libraries are installed before running the script.

```python
import pandas as pd
import os
import random
from utils import specimen_train_test_split
from tqdm import tqdm
import shutil
```

--------------------------------

### Navigate to Project Directory

Source: https://github.com/veccam/vectorbrain/blob/main/README.md

Change the current directory to the cloned project folder.

```sh
cd Mosquito-Dataset-Preparation
```

--------------------------------

### Define Image Directories and Create Them

Source: https://github.com/veccam/vectorbrain/blob/main/download_data.ipynb

Sets up directory paths for storing images and ensures these directories exist. This is a prerequisite for saving downloaded and processed images.

```python
IMAGES_DIR = "./images"
BEFORE_PADDING_DIR = f"{IMAGES_DIR}/before_padding_cropped/"
AFTER_PADDING_DIR = f"{IMAGES_DIR}/after_padding_cropped/"
FULL_IMAGES_DIR = f"{IMAGES_DIR}/full_images/"

# Create directories if they don't exist
for directory in [IMAGES_DIR, BEFORE_PADDING_DIR, AFTER_PADDING_DIR, FULL_IMAGES_DIR]:
    os.makedirs(directory, exist_ok=True)
```

--------------------------------

### Define and Create Directories

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Sets up essential directories for storing CSV files, images, and organized data. It checks for the existence of the image store and creates data folders if they are missing.

```python
import os

# Define constants
CURR_DIR = os.getcwd()
CSV_DIR = os.path.join(CURR_DIR, 'csv')
IMAGE_STORE = os.path.join(CURR_DIR, 'images/after_padding_cropped/')
DATA_FOLDER = os.path.join(CURR_DIR, 'data')
ORGANIZED_DATA_FOLDER = os.path.join(DATA_FOLDER, 'organized_data')
TRAIN_FOLDER = os.path.join(ORGANIZED_DATA_FOLDER, 'train')
TEST_FOLDER = os.path.join(ORGANIZED_DATA_FOLDER, 'test')

COLUMNS_OF_INTEREST = ['Image_name', 'specimen', 'species', 'source_dataset']

# Check for existence of required directories and create them if necessary
assert os.path.exists(IMAGE_STORE), "IMAGE_STORE does not exist. Run download_data.ipynb first"
if not os.path.exists(DATA_FOLDER):
    os.mkdir(DATA_FOLDER)
if not os.path.exists(ORGANIZED_DATA_FOLDER):
    os.mkdir(ORGANIZED_DATA_FOLDER)
if not os.path.exists(TRAIN_FOLDER):
    os.mkdir(TRAIN_FOLDER)
if not os.path.exists(TEST_FOLDER):
    os.mkdir(TEST_FOLDER)
```

--------------------------------

### Database Credentials Configuration

Source: https://github.com/veccam/vectorbrain/blob/main/README.md

Create a .env file in the root directory to store database access credentials. Replace placeholders with actual values.

```makefile
dp_export_username=your_username
dp_export_password=your_password
dp_export_url=your_url
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://github.com/veccam/vectorbrain/blob/main/README.md

Create a Python virtual environment named 'venv' and activate it for project dependencies.

```sh
python -m venv venv
source venv/bin/activate
```

--------------------------------

### Clone Repository

Source: https://github.com/veccam/vectorbrain/blob/main/README.md

Clone the dataset preparation repository from GitHub.

```sh
git clone https://github.com/your_username/Mosquito-Dataset-Preparation.git
```

--------------------------------

### Create Dataset Folders

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Iterates through a dictionary of dataframes, assigns a source dataset key, and creates a corresponding folder in the main data directory if it doesn't exist. Prints the number of rows for each dataset.

```python
for k in df_map.keys():
    df_map[k]['source_dataset'] = k
    # create folder k in DATA_FOLDER if it doesn't exist
    if not os.path.exists(os.path.join(DATA_FOLDER, k)):
        os.mkdir(os.path.join(DATA_FOLDER, k))
    print(f"{k} has {len(df_map[k])} rows")
```

--------------------------------

### Copy Images to Train/Test Folders by Species

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Copies images from their source locations to destination folders, organized by species. It creates species-specific subfolders (e.g., 'C_species') within the main train or test folder. Requires `DATA_FOLDER`, `TRAIN_FOLDER`, `TEST_FOLDER`, `os`, `shutil`, and `tqdm` to be imported.

```python
# copy the images to the train and test folders according to the train and test dataframes and create folders for each species as C_{species}
def copy_images_to_train_test_folders(df, dest_folder):
    for idx, row in tqdm(df.iterrows(), total=len(df), desc=f"Copying images to {dest_folder.split('/')[-1]} folder"):
        image_name = row['Image_name']
        species = row['species']
        image_path = os.path.join(DATA_FOLDER, row['source_dataset'], image_name)
        if not os.path.exists(image_path):
            print(f"{image_path} doesn't exist")
            continue
        destination_path = os.path.join(dest_folder, f"C_{species}")
        # create the destination path if it doesn't exist
        if not os.path.exists(destination_path):
            os.makedirs(destination_path)
        # copy the image to the destination path
        shutil.copy(image_path, destination_path)

copy_images_to_train_test_folders(train_df, TRAIN_FOLDER)
copy_images_to_train_test_folders(test_df, TEST_FOLDER)
```

--------------------------------

### Prepare M4 Dataset

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Prepares the M4 dataset by copying it, renaming columns, mapping species IDs, filtering for valid species, adding a source dataset identifier, and asserting column presence. It then adds the processed M4 data to the cleaned dataframe map and creates a corresponding directory.

```python
# make a copy of the df_M4 DataFrame
df_M4 = df_M4.copy()

# rename 'value' and 'mosquito_id' columns
df_M4.rename(columns={'value': 'Image_name', 'mosquito_id': 'specimen'}, inplace=True)

# map 'morph_id_anopheles_species' column values to 'gambiae' and 'stephensi'
species_map = {'gambiae': 1, 'stephensi': 2}
df_M4['species'] = df_M4['morph_id_anopheles_species'].map(species_map)

# select only the rows with non-null species values
df_M4 = df_M4[df_M4['species'].notna()].copy()

# add 'source_dataset' column with value 'M4'
df_M4['source_dataset'] = 'M4'

# print the count of unique species values
print(df_M4['species'].value_counts())

for col in COLUMNS_OF_INTEREST:
    assert col in df_M4.columns, f"{col} not in df_M4's columns"

# add the M4 DataFrame to the cleaned_df_map
cleaned_df_map['M4'] = df_M4[COLUMNS_OF_INTEREST].copy()

# create folder M4 in DATA_FOLDER if it doesn't exist
if not os.path.exists(os.path.join(DATA_FOLDER, 'M4')):
    os.mkdir(os.path.join(DATA_FOLDER, 'M4'))
```

--------------------------------

### Delete and Recreate Train/Test Folders

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Removes existing content from specified train and test directories and recreates them. Requires `TRAIN_FOLDER`, `TEST_FOLDER`, and `os`, `shutil` modules to be imported.

```python
# delete everything in the train and test folders
for folder in [TRAIN_FOLDER, TEST_FOLDER]:
    if os.path.exists(folder):
        shutil.rmtree(folder)
    os.mkdir(folder)
```

--------------------------------

### Download Database Export

Source: https://github.com/veccam/vectorbrain/blob/main/download_data.ipynb

Downloads a database export as a CSV file. Raises an exception if the download fails, indicating potential credential issues.

```python
success, df = download_data_export('cleaned_export.csv')
if not success:
    raise Exception("No data downloaded, check your credentials in .env file")
```

--------------------------------

### Import Libraries

Source: https://github.com/veccam/vectorbrain/blob/main/download_data.ipynb

Imports necessary libraries for data manipulation, file operations, image processing, and multithreading.

```python
import pandas as pd
import requests
import io
import random
import os
from utils import download_image, download_data_export, save_image, pad_image
from PIL import Image
from tqdm import tqdm
from threading import Thread, Event
from queue import Queue
```

--------------------------------

### Download and Process Images

Source: https://github.com/veccam/vectorbrain/blob/main/download_data.ipynb

Initiates the image download and processing pipeline using the filtered DataFrame. The 'zoomed=True' argument indicates that zoomed images should be processed.

```python
download_images_from_df(df, zoomed=True)
```

--------------------------------

### Move Valid Images to Source Folders

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Defines a function that iterates through a dataframe, checks for the existence of each image, copies available images to their respective source dataset folders, and returns a dataframe containing only rows with available images. This function uses `tqdm` for progress visualization.

```python
def identify_and_move_valid_images(df):
    df["Image_Available"] = True
    for idx, row in tqdm(df.iterrows(), total=len(df)):
        image_name = row['Image_name']
        source_dataset = row['source_dataset']
        species = row['species']
        specimen = row['specimen']
        image_path = os.path.join(IMAGE_STORE, image_name)
        # check if the image exists
        if not os.path.exists(image_path):
            df.at[idx, 'Image_Available'] = False
            continue
        destination_path = os.path.join(DATA_FOLDER, source_dataset)
        # create the destination path if it doesn't exist
        if not os.path.exists(destination_path):
            os.makedirs(destination_path)
        # copy the image to the destination path
        shutil.copy(image_path, destination_path)
    # select only the rows with valid images
    df = df[df['Image_Available']].copy()
    return df
```

--------------------------------

### Create Train-Test Split Dictionary

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Splits dataframes from a map into training and testing sets and stores them in a dictionary. Assumes `cleaned_df_map` is pre-defined and `specimen_train_test_split` is available.

```python
train_test_dict = {}
for k in cleaned_df_map.keys():
    print(f"\nProcessing {k} dataframe")
    output = specimen_train_test_split(cleaned_df_map[k]) # output is a tuple of (train_df, test_df)
    train_test_dict[k] = output

# join and concatenate the train and test dataframes
train_df = pd.concat([train_test_dict[k][0] for k in train_test_dict.keys()])
test_df = pd.concat([train_test_dict[k][1] for k in train_test_dict.keys()])
```

--------------------------------

### Download Images from DataFrame

Source: https://github.com/veccam/vectorbrain/blob/main/download_data.ipynb

Downloads images from URLs specified in a DataFrame and saves them. It utilizes a multithreaded approach for padding images concurrently.

```python
def download_images_from_df(df, zoomed = True):
    img_column = "full_image" if not zoomed else "image"
    download_dir = FULL_IMAGES_DIR if not zoomed else BEFORE_PADDING_DIR
    download_done_event = Event()
    pad_queue = Queue()
    pad_thread = Thread(target=pad_worker, args=(pad_queue, download_done_event, download_dir, AFTER_PADDING_DIR))
    pad_thread.start()
    for idx, row in tqdm(df.iterrows(), total=len(df), desc="Downloading images..."):
        image_url = row[img_column]
        specimen_id = row['specimenId_unique']
        file_name = f"{download_dir}/{specimen_id}.jpg"
        if os.path.exists(file_name):
            continue
        content = download_image(image_url)
        save_image(content, file_name)
        if zoomed:
            pad_queue.put((idx, row))
    download_done_event.set()
    pad_thread.join()
```

--------------------------------

### Map Species to One-Hot Encoding

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Defines a mapping from raw species names to one-hot encoded integer labels. It then applies this mapping to create the 'species' column in each DataFrame.

```python
unique_morph_species = []
for k in df_map.keys():
    unique_morph_species.extend(df_map[k]['morphSpecies'].unique())

print(f"Unique morph species: {set(unique_morph_species)}")

MORPHSPECIES_TO_ONEHOT_SPECIES = { ## Modify this mapping as needed
    'anopheles funestus': 0,
    'anopheles gambiae': 1,
    'anopheles stephensi': 2,
    'anopheles other': 3,
    'culex': 4,
    'mansonia': 5,
    'other': 5,
    'ziemanni': 3
}

# assert that all species are in the mapping
for k in unique_morph_species:
    assert k in MORPHSPECIES_TO_ONEHOT_SPECIES.keys(), f"{k} not in mapping, add it to MORPHSPECIES_TO_ONEHOT_SPECIES"

# add onehot species column
for k in df_map.keys():
    df_map[k]['species'] = df_map[k]['morphSpecies'].map(MORPHSPECIES_TO_ONEHOT_SPECIES)
```

--------------------------------

### Process and Organize Datasets

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Iterates through the cleaned dataframe map, applying the `identify_and_move_valid_images` function to each dataset. It then prints the number of rows remaining after image validation and movement for each dataset.

```python
for k in cleaned_df_map.keys():
    print(f"Processing {k} dataframe")
    cleaned_df_map[k] = identify_and_move_valid_images(cleaned_df_map[k])
    print(f"{k} has {len(cleaned_df_map[k])} rows 
")
```

--------------------------------

### Read DataFrames from CSV Files

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Loads mosquito image data from multiple CSV files into pandas DataFrames. A dictionary is used to store these DataFrames, keyed by their source or dataset name.

```python
# read data from M7 and M4 
df_M7 = pd.read_csv(os.path.join(CSV_DIR, 'M7_data.csv'))
df_M4 = pd.read_csv(os.path.join(CSV_DIR, 'M4_data.csv'))
df_steph = pd.read_csv(os.path.join(CSV_DIR, 'steph_march.csv')) # colony bred stephensi specimen imaged in March
df_gamb = pd.read_csv(os.path.join(CSV_DIR, 'gambiae_march.csv')) # colony bred gambiae specimen imaged in March

df_map = {}
df_map['M7'] = df_M7
df_map['stephensi_march'] = df_steph
df_map['gambiae_march'] = df_gamb
```

--------------------------------

### Image Padding Worker Function

Source: https://github.com/veccam/vectorbrain/blob/main/download_data.ipynb

A worker function designed to run in a separate thread, processing images from a queue for padding. It handles potential errors during image padding.

```python
def pad_worker(pad_queue, download_done_event, read_path, save_path):
    while not download_done_event.is_set() or not pad_queue.empty():
        if not pad_queue.empty():
            count, row = pad_queue.get()
            name = row['specimenId_unique']
            ind_image_path = os.path.join(read_path, f"{name}.jpg")
            save_image_path = os.path.join(save_path, f"{name}.jpg")
            try:
                padded_image = pad_image(ind_image_path)
                padded_image.save(save_image_path)
            except Exception as e:
                print(f"Error padding image {count}: {e}")
```

--------------------------------

### Select Required Columns from DataFrames

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Defines a function to iterate through a map of dataframes, assert that specified columns exist, and return a new map containing only those columns. This ensures data consistency across different sources.

```python
def select_columns(df_map):
    cleaned_df_map = {}
    for k in df_map.keys():
        for col in COLUMNS_OF_INTEREST:
            assert col in df_map[k].columns, f"{col} not in {k}'s columns"
        cleaned_df_map[k] = df_map[k][COLUMNS_OF_INTEREST].copy()
    return cleaned_df_map
```

--------------------------------

### Clean Species and Specimen Names

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

A utility function to standardize species names and extract unique specimen identifiers from the DataFrame. It handles lowercasing, stripping whitespace, and assigning image names.

```python
def clean_species_specimen_names(df):
    df['morphSpecies'] = df['morphSpecies'].str.lower()
    # remove rows that are nm
    df = df[df['morphSpecies'] != 'nm'].copy()
    # remove extra spaces
    df['morphSpecies'] = df['morphSpecies'].str.strip()

    unique_identifier = 'specimenId_unique' if 'specimenId_unique' in df.columns else 'specimenIdDup'
    df['Image_name'] = df[unique_identifier] + ".jpg"
    df['specimen'] = ""
    for idx, row in df.iterrows():
        #get the string before the last underscore
        if '_' in row['specimenId']:
            df.at[idx,'specimen'] = "".join(row[unique_identifier].split('_')[:-1])
        else:
            df.at[idx,'specimen'] = row[unique_identifier]
    return df

for k in df_map.keys():
    df_map[k]["species"] = -1 # this is to make sure that all species columns are filled 
    df_map[k] = clean_species_specimen_names(df_map[k])
```

--------------------------------

### Assign Specific Species Labels

Source: https://github.com/veccam/vectorbrain/blob/main/train_test_preparation.ipynb

Hardcodes species labels for datasets known to contain only a single species. This ensures correct classification for 'Anopheles stephensi' and 'Anopheles gambiae' in their respective datasets.

```python
# we know that gambiae and stephensi are the only species in their respective datasets
df_map['stephensi_march']['morphSpecies'] = 'anopheles stephensi'
df_map['gambiae_march']['morphSpecies'] = 'anopheles gambiae'
```

--------------------------------

### Filter DataFrame for Rows with Images

Source: https://github.com/veccam/vectorbrain/blob/main/download_data.ipynb

Filters the DataFrame to include only rows where the 'image' column is not null. It then selects the last 20 rows for further processing.

```python
df = df[df['image'].notna()]
# select last 20 rows for testing
df = df.tail(20)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.