TinyLLM

https://github.com/weiserlab/tinyllm
Admin
TinyLLM is a lightweight and customizable framework for efficiently training, fine-tuning, and...
Tokens:7,610
Snippets:38
Trust Score:7
License:Apache-2.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Installing Python Dependencies for TinyLLM

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command installs the necessary Python libraries required for dataset processing and fine-tuning TinyLLM models. It reads dependencies from the `requirements.txt` file. Users must also install a suitable version of PyTorch separately.

```bash
pip install -r requirements.txt
```

--------------------------------

### Example Usage of create_csv_per_user - Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This snippet demonstrates how to invoke the `create_csv_per_user` function. It includes a commented-out example showing placeholder paths for `user_data` and `output_csvs` directories, followed by an active call that uses relative paths (`./` for current directory and `../data/` for a parent's data subdirectory) to process sensor data.

```Python
# create_csv_per_user('/path/to/user_data', '/path/to/output_csvs')
create_csv_per_user('./', '../data/')
```

--------------------------------

### Inferencing TinyLLM with HuggingFace Transformers (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This snippet demonstrates how to perform text generation inference using a fine-tuned TinyLLM model with HuggingFace's transformers library. It initializes a `pipeline` for text generation, specifying the model path, maximum new tokens, repetition penalty, and device mapping. The example then shows how to generate text from a given prompt.

```python
from transformers import pipeline
import torch

path = "./TinyLLM/Fine-tune/results/GPT 2/breathe-0/"
generator = pipeline("text-generation", model=path, max_new_tokens=30, repetition_penalty=1.3, device_map="auto")
prompt = "Your input text here"
print(generator(prompt)[0]['generated_text'])
```

--------------------------------

### Defining Dataset Root Directory (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/SHL/process.ipynb

This line defines the base directory path for the 'User1' dataset. This variable is crucial as it points to the location where the script will start searching for motion data and label files. Users must adjust this path to match their actual dataset location.

```Python
root_dir = 'User1/'
```

--------------------------------

### Parsing CSV Header for Features and Labels (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This function parses the header line of a CSV string (`csv_str`) to extract feature names and label names. It asserts that 'timestamp' is the first column and 'label_source' is the last, and correctly identifies the start of label columns, removing the 'label:' prefix from label names.

```python
def parse_header_of_csv(csv_str):
    # Isolate the headline columns:
    headline = csv_str[:csv_str.index(b'\n')]
    columns = headline.split(b',')

    # The first column should be timestamp:
    assert columns[0] == b'timestamp'
    # The last column should be label_source:
    assert columns[-1] == b'label_source'
    
    # Search for the column of the first label:
    for (ci, col) in enumerate(columns):
        if col.startswith(b'label:'):
            first_label_ind = ci
            break

    # Feature columns come after timestamp and before the labels:
    feature_names = columns[1:first_label_ind]
    # Then come the labels, till the one-before-last column:
    label_names = columns[first_label_ind:-1]
    for (li, label) in enumerate(label_names):
        # In the CSV the label names appear with prefix 'label:', but we don't need it after reading the data:
        assert label.startswith(b'label:')
        label_names[li] = label.replace(b'label:', b'')
    
    return (feature_names, label_names)
```

--------------------------------

### Navigating to TinyLLM Datasets Directory

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command changes the current directory to the `Datasets/` folder, which is the first step before preparing pre-training datasets. All subsequent dataset preparation scripts are expected to be run from this directory.

```bash
cd Datasets/
```

--------------------------------

### Pre-training GPT2 Model with llm.c

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command initiates the pre-training of a GPT2 model using `llm.c`'s `train_gpt2cu` executable. It specifies input training and validation data, an output directory, model depth (`-e`), batch size (`-b`), sequence length (`-t`), and various training parameters. The `-y 1` flag can be used to resume from a checkpoint.

```bash
./train_gpt2cu \
    -i "Datasets/pretraining_data/train*.bin" \
    -j "Datasets/pretraining_data/val*.bin" \
    -o "custom_model" \
    -e "d6" \
    -b 64 -t 1024 \
    -d 524288 \
    -r 1 \
    -z 1 \
    -c 0.1 \
    -l 0.0006 \
    -q 0.0 \
    -u 700 \
    -n 10000 \
    -v 250 -s 20000 \
    -h 1
```

--------------------------------

### Navigating to llm.c Directory for Pre-training

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command changes the current directory to the `llm.c/` folder, which is necessary before initiating the model pre-training process. The `train_gpt2cu` executable resides in this directory.

```bash
cd ../llm.c/
```

--------------------------------

### Running GGUF Model with llama.cpp CLI (Bash)

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This snippet demonstrates how to run inference using a GGUF-formatted model with the `llama.cpp` command-line interface. It executes the `llama-cli` tool, providing the path to the GGUF model, the number of tokens to generate, and the input prompt for text generation.

```bash
./llama-cli -m "../Fine-tune/results/GPT 2/breathe-0/model.gguf" -n 10 -p "Your input prompt"
```

--------------------------------

### Defining Activity Recognition Prompt - Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This snippet defines a multi-line string variable `prompt` that serves as an instruction for an activity recognition model. It details the types of sensor readings provided (e.g., accelerometer, gyroscope, location services) and lists over 50 possible user activities to be identified. This prompt is later used as an 'Instruction' column in the output CSV, providing context for the machine learning task.

```Python
prompt = 'You are provided with sensor readings that include high-frequency motion-reactive sensors (accelerometer, gyroscope, magnetometer, watch accelerometer), location services, audio, watch compass, phone state indicators, and additional low-frequency sensors sampled once per minute. The data may contain NaN values and each sensor value is normalized between 0 and 1. Based on these sensor readings, identify the users activity, which may include one or more of the following:\n"Phone on table", "Sitting", "Indoors", "At home", "Lying down", "Talking", "Sleeping", "At main workplace", "Phone in pocket", "Eating", "Watching TV", "Surfing the internet", "Standing", "Walking", "Outside", "With friends", "Phone in hand", "Computer work", "With co-workers", "Dressing", "Cooking", "Washing dishes", "On a bus", "Grooming", "Drive - Im the driver", "Toilet", "At school", "In a car", "Drinking (alcohol)", "In a meeting", "Drive - Im a passenger", "Bathing - shower", "Strolling", "Singing", "Shopping", "At a restaurant", "Doing laundry", "Running", "Exercise", "Stairs - going up", "Stairs - going down", "Bicycling", "Lab work", "In class", "Cleaning", "At a party", "At a bar", "At the beach", "At the gym", "Elevator", "Phone in bag".'
```

--------------------------------

### Splitting Tokenized Datasets for Pre-training

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command runs `split.py` to divide tokenized datasets into training and validation sets. The default parameters create a 9 billion token dataset with a 98:2 Training:Validation split and 100MB shards, outputting to `./pretraining_data`. Parameters can be adjusted for memory constraints.

```bash
python split.py -d1 0.3 -d2 0.7 -o ./pretraining_data
```

--------------------------------

### Viewing Fine-tuning Loss in Terminal

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command displays the contents of the `loss.txt` file, which contains the training and evaluation loss data generated during the fine-tuning process. This allows users to quickly review the model's performance directly in the terminal.

```bash
cat "results/GPT 2/breathe-0/loss.txt"
```

--------------------------------

### Exporting Pre-trained Model to HuggingFace Format

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command exports the most recently trained model checkpoint from `llm.c` into a HuggingFace-compatible format. It first identifies the latest model binary file and then uses `export_hf.py` to convert it, making the model ready for fine-tuning or deployment with HuggingFace tools.

```bash
lf=$(ls custom_model/model_0*.bin | sort -V | tail -n 1) 
python dev/eval/export_hf.py -i "$lf" -o "custom_model_hf"
```

--------------------------------

### Navigating to TinyLLM Fine-tuning Directory

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command changes the current directory to the `Fine-tune/` folder, which is the prerequisite step before initiating the model fine-tuning process. The `master.py` script for fine-tuning resides in this directory.

```bash
cd ../Fine-tune/
```

--------------------------------

### Fine-tuning TinyLLM Model with master.py

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command executes the `master.py` script to fine-tune a pre-trained model on a specific dataset. It takes the dataset name (`-d`), path to the pre-trained HuggingFace model (`-m`), model type (`-n`), and a parameter file (`-p`) as inputs. Training and evaluation loss plots are saved, and output is logged to `ft_output.log`.

```bash
python master.py \
    -d "breathe" \
    -m "../llm.c/custom_model_hf" \
    -n "gpt2" \
    -p "p-gpt.txt" | tee ft_output.log
```

--------------------------------

### Tokenizing Datasets with TinyLLM's encode.py

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command executes the `encode.py` script to tokenize datasets for pre-training. It supports custom CSV datasets or HuggingFace datasets, and by default processes Fineweb and SHL IoT sensor datasets. Users can modify `datasets_to_tokenize` in `encode.py` for custom data.

```bash
python encode.py
```

--------------------------------

### Importing Libraries for Data Processing - Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/swim/process.ipynb

This snippet imports the necessary Python libraries for the data processing pipeline. `os` is used for interacting with the operating system, `pandas` for data manipulation and analysis, and `sklearn.model_selection.train_test_split` for splitting datasets into training and testing subsets.

```python
import os
import pandas as pd
from sklearn.model_selection import train_test_split
```

--------------------------------

### Converting HuggingFace Model to GGUF Format (Bash)

Source: https://github.com/weiserlab/tinyllm/blob/main/README.md

This command-line snippet illustrates how to convert a HuggingFace model checkpoint into the GGUF format, which is optimized for use with `llama.cpp` and embedded devices. It navigates to the `llama.cpp` directory and executes the `convert_hf_to_gguf.py` script, specifying the input HuggingFace model directory and the desired output GGUF file path.

```bash
cd ../llama.cpp/
python convert_hf_to_gguf.py "../Fine-tune/results/GPT 2/breathe-0/" --outfile "../Fine-tune/results/GPT 2/breathe-0/model.gguf"
```

--------------------------------

### Importing Core Libraries for Data Processing - Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This snippet imports fundamental Python libraries required for various data processing tasks. `os` is used for operating system interactions, `numpy` for numerical operations, `pandas` for data manipulation and analysis, `StringIO` for in-memory text buffering, and `gzip` for handling gzipped files.

```Python
import os
import numpy as np
import pandas as pd
from io import StringIO
import gzip
```

--------------------------------

### Configuring LoRA and Training Parameters (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Fine-tune/p-gpt.txt

This snippet defines key parameters for training a causal language model using the LoRA technique. It sets LoRA's rank (`lora_r`), scaling factor (`lora_alpha`), dropout rate (`lora_dropout`), and bias type (`bias`). Additionally, it specifies training hyperparameters like the task type (`task_type`), per-device batch size (`per_device_train_batch_size`), gradient accumulation steps (`gradient_accumulation_steps`), learning rate (`learning_rate`), maximum training steps (`max_steps`), warmup steps (`warmup_steps`), and the optimizer to be used (`optim`).

```Python
lora_r = 64
lora_alpha = 256
lora_dropout = 0.35
bias = "lora_only"
task_type = "CAUSAL_LM"
per_device_train_batch_size = 1
gradient_accumulation_steps = 6
learning_rate = 6e-4
max_steps = 2050
warmup_steps = 200
optim = "paged_adamw_32bit"
```

--------------------------------

### Defining LoRA and Training Configuration in Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Fine-tune/p-phi3.txt

This snippet defines various hyperparameters for training a Causal Language Model, including LoRA specific parameters (rank, alpha, dropout, bias) and general training parameters like batch size, gradient accumulation, learning rate, max steps, warmup steps, and optimizer. These settings are crucial for controlling the fine-tuning process and model performance.

```Python
lora_r = 16
lora_alpha = 64
lora_dropout = 0.2
bias = "lora_only"
task_type = "CAUSAL_LM"
per_device_train_batch_size = 1
gradient_accumulation_steps = 6
learning_rate = 6e-4
max_steps = 250
warmup_steps = 40
optim = "paged_adamw_8bit"
```

--------------------------------

### Processing Motion Data for All Dates (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/SHL/process.ipynb

This function iterates through all subdirectories (representing different recording dates) within a specified root directory for 'User1'. For each date, it calls `process_motion_data_for_date` to process and export the motion data to a separate CSV file. This allows for batch processing of an entire dataset.

```Python
def process_motion_data_per_date(root_dir):
    """
    Processes motion data files for User1 one by one and exports each date's data to a separate CSV file.
    
    Parameters:
        root_dir (str): The root directory containing the User1 subdirectories.
    """
    # Get all recording session paths (subdirectories) for User1
    record_paths = [os.path.join(root_dir, folder) for folder in os.listdir(root_dir) if os.path.isdir(os.path.join(root_dir, folder))]

    # Process each record path serially
    for record_path in tqdm(record_paths, desc="Processing all dates"):
        process_motion_data_for_date(record_path)
```

--------------------------------

### Defining SHL Fine Label Mappings in Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/SHL/process.ipynb

This Python snippet defines the `FINE_LABEL_MAP` dictionary, which provides a comprehensive mapping from numerical activity labels (0-18) to descriptive string labels. This mapping is essential for interpreting and processing the sensor data from the SHL dataset, allowing for human-readable activity classifications. Label 0 is explicitly mapped to an empty string, indicating an ignored label.

```python
import os
import pandas as pd
from concurrent.futures import ThreadPoolExecutor
from tqdm import tqdm

# Define label mappings for fine labels according to the document provided, using '' for ignored labels
FINE_LABEL_MAP = {
    0: '',
    1: 'Still;Stand;Outside',
    2: 'Still;Stand;Inside',
    3: 'Still;Sit;Outside',
    4: 'Still;Sit;Inside',
    5: 'Walking;Outside',
    6: 'Walking;Inside',
    7: 'Run',
    8: 'Bike',
    9: 'Car;Driver',
    10: 'Car;Passenger',
    11: 'Bus;Stand',
    12: 'Bus;Sit',
    13: 'Bus;Up;Stand',
    14: 'Bus;Up;Sit',
    15: 'Train;Stand',
    16: 'Train;Sit',
    17: 'Subway;Stand',
    18: 'Subway;Sit'
}
```

--------------------------------

### Processing User Sensor Data to CSV - Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This Python function processes gzipped CSV files containing sensor data for individual users. It iterates through user files, reads compressed data, parses headers and body using helper functions (`parse_header_of_csv`, `parse_body_of_csv`), normalizes sensor values while handling NaNs (`normalize_sensor_data_with_nan`), formats the input, and prettifies labels. Finally, it combines this processed data with the predefined `prompt` into a Pandas DataFrame and writes it to a new CSV file, preparing it for machine learning tasks.

```Python
def create_csv_per_user(user_data_dir, output_dir):
    # Iterate over each user file in the provided directory
    for user_file in os.listdir(user_data_dir):
        if user_file.endswith('.csv.gz'):
            print(user_file)
            # Read the compressed CSV file
            with gzip.open(os.path.join(user_data_dir, user_file), 'rb') as f:
                csv_str = f.read()
            
            # Parse the header and body of the CSV
            feature_names, label_names = parse_header_of_csv(csv_str)
            timestamps, X, Y = parse_body_of_csv(csv_str, len(feature_names))
            
            # Normalize sensor data, handling NaNs
            X_norm = normalize_sensor_data_with_nan(X)
            
            # Format the normalized data
            input_data = format_input_data(X_norm)
            
            # Prettify labels
            pretty_labels = prettify_labels(label_names, Y)
            
            # Combine input (formatted normalized sensor data) and response (pretty labels)
            data = {'Instruction': prompt , 'Input': input_data, 'Response': pretty_labels}
            df = pd.DataFrame(data)
            
            # Write to CSV
            output_file = os.path.join(output_dir, f"{user_file[:-7]}.csv")
            df.to_csv(output_file, index=False)
```

--------------------------------

### Reading Fine Labels from File (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/SHL/process.ipynb

This function reads a 'Label.txt' file, extracts fine labels from the third column, and converts them into descriptive text using a predefined mapping (FINE_LABEL_MAP). It filters out any empty or unmapped labels, returning a list of descriptive labels synchronized with time steps. It requires the pandas library.

```Python
def read_fine_labels(label_file_path):
    """
    Reads the Label.txt file and extracts fine labels from the third column, converting them to descriptive text.
    Filters out empty labels.
    
    Parameters:
        label_file_path (str): The path to the Label.txt file.
    
    Returns:
        list: A list of fine descriptive labels corresponding to each time step.
    """
    fine_labels = []
    try:
        # Read the labels from the Label.txt file
        label_data = pd.read_csv(label_file_path, sep="\s+", header=None)
        
        # Convert numeric labels to descriptive text using the provided map
        fine_labels = label_data.iloc[:, 2].apply(lambda x: FINE_LABEL_MAP.get(x, '')).tolist()

        # Keep only non-empty labels
        fine_labels = [label if label else None for label in fine_labels]
    except Exception as e:
        print(f"Error reading {label_file_path}: {e}")

    return fine_labels
```

--------------------------------

### Calling Function to Process Motion Data - Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/SHL/process.ipynb

This snippet demonstrates how to call the `process_motion_data_per_date` function, passing the `root_dir` argument. This function is responsible for processing motion data and exporting it into separate CSV files for each date.

```Python
process_motion_data_per_date(root_dir)
```

--------------------------------

### Configuring LoRA and Training Parameters in Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Fine-tune/p-llama3.txt

This snippet defines various hyperparameters for fine-tuning a Causal Language Model using LoRA. It includes LoRA-specific parameters like rank, alpha, and dropout, along with training parameters such as batch size, learning rate, and optimizer.

```Python
lora_r = 16
lora_alpha = 64
lora_dropout = 0.2
bias = "lora_only"
task_type = "CAUSAL_LM"
per_device_train_batch_size = 1
gradient_accumulation_steps = 6
learning_rate = 6e-4
max_steps = 250
warmup_steps = 40
optim = "paged_adamw_32bit"
```

--------------------------------

### Formatting Normalized Sensor Data (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This function takes normalized sensor data `X_norm` and formats each row into a space-separated string enclosed in square brackets. `NaN` values are explicitly represented as 'nan' strings, preparing the data for specific input formats.

```python
def format_input_data(X_norm):
    # Format the normalized sensor data with all entries in a single set of brackets and separated by spaces
    formatted_data = []
    for row in X_norm:
        formatted_row = "[" + " ".join([f"{value if not np.isnan(value) else 'nan'}" for value in row]) + "]"
        formatted_data.append(formatted_row)
    return formatted_data
```

--------------------------------

### Prettifying Individual Label Names (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This function converts raw, machine-readable label names (e.g., 'FIX_walking') into more human-readable strings (e.g., 'Walking') using a predefined mapping. If a label is not found in the mapping, its original value is returned.

```python
def prettify_label_name(label):
    label_mapping = {
        'FIX_walking': 'Walking',
        'FIX_running': 'Running',
        'LOC_main_workplace': 'At main workplace',
        'OR_indoors': 'Indoors',
        'OR_outside': 'Outside',
        'LOC_home': 'At home',
        'FIX_restaurant': 'At a restaurant',
        'OR_exercise': 'Exercise',
        'LOC_beach': 'At the beach',
        'OR_standing': 'Standing',
        'WATCHING_TV': 'Watching TV'
    }
    return label_mapping.get(label, label)
```

--------------------------------

### Processing and Exporting Motion Data for a Specific Date (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/SHL/process.ipynb

This function orchestrates the processing of motion data files for a given recording session (date). It reads the corresponding labels, processes individual motion files in parallel using a thread pool, and appends the processed sensor data and labels to a single CSV output file. It depends on `read_fine_labels` and `process_motion_data_chunked`.

```Python
def process_motion_data_for_date(record_path):
    """
    Processes motion data files for a specific recording session (date) and exports them to a CSV file.
    
    Parameters:
        record_path (str): The directory path containing the .txt files for a specific date.
    """
    # Output file path
    record_folder = os.path.basename(record_path)
    output_csv = f"{record_folder}_motion_data.csv"
    
    # Read the fine labels from the corresponding Label.txt file
    label_file_path = os.path.join(record_path, 'Label.txt')
    fine_labels = read_fine_labels(label_file_path)

    # Write the header to the CSV file first
    with open(output_csv, 'w') as f:
        f.write('Sensor Data,Label\n')

    # Get the list of _Motion.txt files
    motion_files = [file for file in os.listdir(record_path) if "_Motion.txt" in file]

    # Process each motion data file in parallel
    with ThreadPoolExecutor(max_workers=NUM_THREADS) as executor:
        futures = [executor.submit(process_motion_data_chunked, os.path.join(record_path, file), fine_labels) for file in motion_files]

        for future in tqdm(futures, desc=f"Processing {record_folder}"):
            chunk_results = future.result()

            # Write chunk results to CSV
            with open(output_csv, 'a') as f:
                for sensor_data, label in chunk_results:
                    # Convert list to space-separated string for sensor data
                    sensor_data_str = ' '.join(map(str, sensor_data))
                    f.write(f'"[{sensor_data_str}]",{label}\n')

    print(f"Motion data for {record_folder} exported to {output_csv}")
```

--------------------------------

### Processing Raw Accelerometer Data for Swim Styles - Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/swim/process.ipynb

This extensive snippet defines two core functions, `process_file` and `process_group`, to handle the transformation of raw accelerometer data. `process_file` reads individual CSVs, maps numerical labels to swim styles (e.g., 0 to 'transition'), and groups continuous activity segments. `process_group` then chunks these segments into fixed-size inputs (n=100) for machine learning, handling edge cases where the last chunk is smaller than 'n'. The main loop iterates through swimmer directories, applies these functions, and aggregates all processed data.

```python


# Function to process each file
def process_file(file_path, n):
    data = pd.read_csv(file_path)
    
    # Extract relevant columns
    acc_columns = ['ACC_0', 'ACC_1', 'ACC_2']
    label_column = 'label'
    
    # Map label numbers to actual style names
    label_map = {0: 'transition', 1: 'freestyle', 2: 'breaststroke', 3: 'backstroke', 4: 'butterfly', 5: 'transition'}
    data['label'] = data['label'].map(label_map)
    
    # List to store processed rows
    rows = []
    
    # Group data by continuous labels
    current_label = None
    current_group = []
    
    for i, row in data.iterrows():
        label = row['label']
        
        if label != current_label:
            if current_group:
                rows.extend(process_group(current_group, n, current_label, acc_columns))
            current_group = []
            current_label = label
            
        current_group.append(row)
    
    # Process the last group
    if current_group:
        rows.extend(process_group(current_group, n, current_label, acc_columns))
    
    return rows

# Function to process a group of continuous label data
def process_group(group, n, label, acc_columns):
    rows = []
    num_entries = len(group)
    
    # Convert the group (list of rows) into a DataFrame
    group_df = pd.DataFrame(group)
    
    # Split into chunks of size n
    for i in range(0, num_entries, n):
        chunk = group_df.iloc[i:i+n]
        
        if len(chunk) < n:
            # Handle the case where the last chunk is smaller than n
            previous_chunk = group_df.iloc[i-n:i] if i >= n else group_df.iloc[:i]
            chunk = pd.concat([previous_chunk, chunk]).tail(n)  # Get the last n readings
        
        acc_values = chunk[acc_columns].round(0).astype(int)
        input_str = f"X: {list(acc_values['ACC_0'])}\nY: {list(acc_values['ACC_1'])}\nZ: {list(acc_values['ACC_2'])}"
        rows.append([input_str, label])
    
    return rows

# Directory paths and parameters
data_dir = "Swim"
n = 100

# List to hold all data
all_data = []

# Process each swimmer folder
for swimmer in os.listdir(data_dir):
    swimmer_path = os.path.join(data_dir, swimmer)
    if os.path.isdir(swimmer_path):
        for file in os.listdir(swimmer_path):
            file_path = os.path.join(swimmer_path, file)
            if file.endswith(".csv"):
                all_data.extend(process_file(file_path, n))

# Convert to DataFrame
df = pd.DataFrame(all_data, columns=['Input', 'Label'])
```

--------------------------------

### Normalizing Sensor Data with NaN Handling (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This function normalizes sensor data `X` to a 0-1 range, column by column, while robustly handling `NaN` (Not a Number) values. It calculates min/max only from valid entries and sets columns with all `NaN`s to `NaN` or 0.5 if all valid values are identical, returning the result rounded to 3 decimal places.

```python
def normalize_sensor_data_with_nan(X):
    # Initialize an array to hold normalized data
    X_norm = np.empty_like(X)
    
    for i in range(X.shape[1]):
        # Extract the column, handling NaNs
        column = X[:, i]
        valid_values = column[~np.isnan(column)]
        
        if valid_values.size > 0:
            col_min = valid_values.min()
            col_max = valid_values.max()
            
            if col_max > col_min:  # Avoid division by zero
                X_norm[:, i] = (column - col_min) / (col_max - col_min)
            else:
                X_norm[:, i] = 0.5  # If all valid values are the same, normalize to 0.5
        else:
            X_norm[:, i] = np.nan  # If all values are NaN, keep the column as NaN
    
    return np.round(X_norm, 3)
```

--------------------------------

### Prettifying Active Labels for Each Timestamp (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This function generates a space-separated string of active (true) labels for each timestamp, utilizing the `prettify_label_name` function to convert raw label names into human-readable format. It iterates through the binary label matrix `Y` and corresponding `label_names`.

```python
def prettify_labels(label_names, Y):
    # Create a space-separated string of active labels for each timestamp using the prettified label names
    pretty_labels = []
    for y in Y:
        active_labels = [
            prettify_label_name(label.decode('utf-8')) 
            for label, active in zip(label_names, y) if active
        ]
        pretty_labels.append(" ".join(active_labels))
    return pretty_labels
```

--------------------------------

### Parsing CSV Body for Timestamps, Features, and Labels (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This function reads the body of a CSV string (`csv_str`) into a numeric matrix, extracting timestamps, sensor features (`X`), and binary label values (`Y`). It handles missing labels by converting `NaN` values in the label matrix to `0` and then binarizing them.

```python
def parse_body_of_csv(csv_str, n_features):
    # Read the entire CSV body into a single numeric matrix:
    full_table = np.loadtxt(StringIO(csv_str.decode('utf-8')), delimiter=',', skiprows=1)
    
    # Timestamp is the primary key for the records (examples):
    timestamps = full_table[:, 0].astype(int)
    
    # Read the sensor features:
    X = full_table[:, 1:(n_features+1)]
    
    # Read the binary label values, and the 'missing label' indicators:
    trinary_labels_mat = full_table[:, (n_features+1):-1]  # This should have values of either 0., 1. or NaN
    M = np.isnan(trinary_labels_mat)  # M is the missing label matrix
    Y = np.where(M, 0, trinary_labels_mat) > 0.  # Y is the label matrix
    
    return timestamps, X, Y
```

--------------------------------

### Splitting and Saving Processed Swim Data - Python

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/swim/process.ipynb

This snippet finalizes the data preprocessing by converting the aggregated `all_data` into a pandas DataFrame, dropping any rows with missing values. It then uses `train_test_split` to divide the dataset into training, validation, and test sets, ensuring stratified sampling based on the 'Label' column to maintain class distribution. Finally, these prepared datasets are saved as separate CSV files in a 'data' directory.

```python

df = pd.DataFrame(all_data, columns=['input', 'output'])
df = df.dropna()  # Drop rows with NaN values


# Split into Train, Test, Val
train, test = train_test_split(df, test_size=0.3, stratify=df['Label'])
val, test = train_test_split(test, test_size=2/3, stratify=test['Label'])

# Save to CSV files
train.to_csv('data/train.csv', index=False)
val.to_csv('data/validation.csv', index=False)
test.to_csv('data/test.csv', index=False)

print("Data preprocessing completed!")

```

--------------------------------

### Processing Motion Data in Chunks (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/SHL/process.ipynb

This function processes large motion data files in user-defined chunks to prevent memory overload. It reads sensor data, includes the time column, synchronizes it with corresponding fine labels, and collects valid data points (where labels are not empty). The function returns a list of tuples, each containing sensor data and its associated label. It depends on the pandas library.

```Python
def process_motion_data_chunked(file_path, fine_labels, chunk_size=100000):
    """
    Processes motion data in chunks to avoid memory overload and writes to a CSV file.
    
    Parameters:
        file_path (str): The path to the motion data file.
        fine_labels (list): The list of fine descriptive labels corresponding to each time step.
        chunk_size (int): The number of rows to process in each chunk.
    
    Returns:
        list: List of tuples containing sensor data and the corresponding fine label.
    """
    # Create an empty list to hold chunk results
    chunk_results = []

    try:
        # Read the file in chunks
        for i, chunk in enumerate(pd.read_csv(file_path, sep="\s+", header=None, chunksize=chunk_size)):
            # Convert each row of sensor data to a list including time
            sensor_data = chunk.apply(lambda row: [row[0]] + row[1:].tolist(), axis=1)  # Include time
            
            # Synchronize fine labels with sensor data for the current chunk
            chunk_fine_labels = fine_labels[i * chunk_size: i * chunk_size + len(chunk)]
            
            # Collect the chunk's results, skipping rows where the label is empty
            for data, label in zip(sensor_data, chunk_fine_labels):
                if label is not None:  # Skip rows where the label is empty
                    chunk_results.append((data, label))
    except Exception as e:
        print(f"Error reading {file_path}: {e}")

    return chunk_results
```

--------------------------------

### Configuring Parallel Processing Threads (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/SHL/process.ipynb

This variable defines the maximum number of worker threads to be used for parallel processing tasks, such as processing multiple motion data files concurrently. Users can adjust this value to optimize performance based on system resources.

```Python
NUM_THREADS = 1
```

--------------------------------

### Normalizing Sensor Data (Python)

Source: https://github.com/weiserlab/tinyllm/blob/main/Datasets/Extra_sensory/process.ipynb

This function normalizes sensor data `X` to a 0-1 range across each column, using the minimum and maximum values of each column. The result is rounded to 3 decimal places. This version assumes no `NaN` values or handles them implicitly by `min`/`max` behavior.

```python
def normalize_sensor_data(X):
    # Normalize sensor data between 0 and 1 with 3 decimals precision
    X_min = X.min(axis=0)
    X_max = X.max(axis=0)
    X_norm = (X - X_min) / (X_max - X_min)
    return np.round(X_norm, 3)
```
Tokens:
Raw