LLaVA-Chart

https://github.com/zengxingchen/chartqa-mllm
Admin
[IEEE VIS 2024] LLaVA-Chart: Advancing Multimodal Large Language Models in Chart Question Answering...
Tokens:53,561
Snippets:168
Trust Score:7.5
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install Additional Training Packages

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/README.md

This snippet installs additional Python packages, 'ninja' and 'flash-attn', which are required for specific training functionalities within the project. 'flash-attn' is noted to be installed with no build isolation.

```bash
pip install ninja
pip install flash-attn --no-build-isolation
```

--------------------------------

### Set Up Conda Environment and Install Core Package

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/README.md

This section details the process of creating and activating a Conda environment named 'llava-hr' with Python 3.10. It then proceeds to upgrade pip and install the project's core package using an editable install.

```bash
conda create -n llava-hr python=3.10 -y
conda activate llava-hr
pip install --upgrade pip 
pip install -e .
```

--------------------------------

### PyTorch Model Training Setup

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This Python code snippet demonstrates the setup for training a PyTorch model. It initializes a Mean Squared Error loss function, defines a feature column name, and calls a 'train' function with specified data, features, and the loss criterion. This is a common pattern for initiating supervised learning tasks.

```python
import torch.nn as nn

criterion = nn.MSELoss()
feature_column_name = 'number-annotation' # label, Trend, Layout, number annotation, grouping

features = clip_features
classifier = train(data, features, feature_column_name, criterion)

```

--------------------------------

### Configuration and Logging Setup (Python)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

Initializes logging and imports a clustering utility function. Includes commented-out code for loading YAML configuration parameters, which would typically define parameters for K-means clustering.

```python
import yaml
import random
import numpy as np
import logging
from clustering import compute_centroids

logger = logging.getLogger(__name__) 
logger.addHandler(logging.StreamHandler())

# confg_file = "configs/openclip/clustering_configs.yaml"
# ## -- Load kmeans clustering parameters from configs file
# with open(confg_file, 'r') as y_file:
#     params = yaml.load(y_file, Loader=yaml.FullLoader)
```

--------------------------------

### Progress Bar Initialization in Python

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

Represents an initialized progress bar, typically indicating the start of a task or loop. The format `0%|          | 0/1 [00:00<?, ?it/s]` suggests a task with one iteration that has not yet started.

```python
  0%|          | 0/1 [00:00<?, ?it/s]
```

--------------------------------

### Python Project Setup and Imports

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

Initializes the project environment by setting the current working directory, adding necessary paths to sys.path, and importing required libraries like torch, numpy, pandas, and scikit-learn. It also sets up the device for computation (GPU or CPU) and imports custom utility functions and model classes.

```python
import os
import torch
import numpy as np
import pandas as pd
import sys
os.chdir('~/projects/LLaVA')
current_path = os.getcwd()
sys.path.append(current_path + '/data-engine')

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
from utils import set_env
set_env(seed=1234)

from torch.utils.data import Dataset, DataLoader
from utils import ChartFeatureDataset
from models import MLPClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, accuracy_score
import torch.nn as nn
import numpy as np
from tqdm import tqdm
import torch.nn.functional as F
from utils import FocalLoss
```

--------------------------------

### Run CLI Inference with LLaVA-HR Model

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/README.md

This command provides an example of how to perform command-line interface (CLI) inference with the LLaVA-HR model. It specifies the model path and an image file pattern for processing.

```bash
python -m model.llava_hr.serve.cli \
    --model-path ./checkpoints/llava-hr-ChartInstruction \
    --image-file "*.jpg" 
```

--------------------------------

### Log Job Start and Cluster Range

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

Logs the start of a job and defines the range of clusters it will process. This is a common pattern for sequential data processing or distributed computing tasks.

```text
STARTING JOB:  12
This job will process clusters 12 to  13
This task will process 1 clusters: cluster 12 to cluster 13
This task will process cluster 12 to cluster 13
```

```text
STARTING JOB:  14
This job will process clusters 14 to  15
This task will process 1 clusters: cluster 14 to cluster 15
This task will process cluster 14 to cluster 15
```

```text
STARTING JOB:  15
This job will process clusters 15 to  16
This task will process 1 clusters: cluster 15 to cluster 16
This task will process cluster 15 to cluster 16
```

```text
STARTING JOB:  16
This job will process clusters 16 to  17
This task will process 1 clusters: cluster 16 to cluster 17
This task will process cluster 16 to cluster 17
```

--------------------------------

### Memory-Mapped File Setup (Python)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

Sets up memory-mapped NumPy arrays for efficient storage and retrieval of embeddings and file paths. This is particularly useful for large datasets that may not fit entirely into RAM. It requires pre-defined dataset size and embedding dimensions.

```python
from torch.nn.functional import normalize

path_str_type = 'U200'  # 假设路径字符串最长为200个字符

save_dir = r'./sample_files'

# Setting up memory-mapped files
emb_memory_loc = f'{save_dir}/embeddings_memmap.dat'
paths_memory_loc = f'{save_dir}/paths_memmap.dat'

emd_memmap = np.memmap(emb_memory_loc, dtype='float32', mode='w+', shape=(dataset_size, emb_size))
paths_memmap = np.memmap(paths_memory_loc, dtype=path_str_type, mode='w+', shape=(dataset_size,))
```

--------------------------------

### Fetch and Populate Data from JSON

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/model/llava/eval/webpage/index.html

This JavaScript code fetches data from a 'data.json' file, parses it as JSON, and then uses the data to populate models and questions. It also handles potential errors during the fetch operation. This functionality is essential for dynamically loading content on the evaluation page.

```javascript
fetch('data.json')
  .then(response => response.json())
  .then(json_data => {
    // Populate the models and questions.
    populateModels(json_data.models);
    populateQuestions(json_data.questions);
    displayQuestion(currentQuestionIndex);
  }).catch(error => console.error(error));
```

--------------------------------

### Fetch and Parse JSON Data (JavaScript)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/model/llava_hr/eval/webpage/index.html

This JavaScript code fetches data from a JSON file ('data.json') and then processes the received data to populate models and questions. It handles potential errors during the fetch operation. Dependencies: None, standard browser Fetch API.

```javascript
fetch('data.json')
  .then(response => response.json())
  .then(json_data => {
    populateModels(json_data.models);
    populateQuestions(json_data.questions);
    displayQuestion(currentQuestionIndex);
  }).catch(error => console.error(error));
```

--------------------------------

### Python Model Training Setup with MSE Loss

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This Python code snippet sets up a mean squared error (MSE) loss criterion and defines a feature column name for training a classifier. It then calls a 'train' function with provided data, features, and the specified column name.

```python
criterion = nn.MSELoss()
feature_column_name = 'number-annotation' # label, Trend, Layout, number annotation, grouping

features = resnet_features
classifier = train(data, features, feature_column_name, criterion)
```

--------------------------------

### Progress Tracking and Job Completion (Python)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This snippet demonstrates progress tracking using a progress bar and indicates the completion of a job, along with starting the next one. This is essential for monitoring the overall execution status of a multi-stage process.

```python
DONE cluster:  9
STARTING JOB:  10
This job will process clusters 10 to  11
This task will process 1 clusters: cluster 10 to cluster 11
This task will process cluster 10 to cluster 11


Output:
  0%|          | 0/1 [00:00<?, ?it/s]
```

--------------------------------

### Job Processing Log in Python

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

Logs the start of a new job for a specified range of clusters. This indicates sequential processing of data clusters in a machine learning or data analysis pipeline.

```python
STARTING JOB:  61
This job will process clusters 61 to  62
```

--------------------------------

### Job and Cluster Processing Log

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This log indicates the start of a new job for processing clusters. It specifies the range of clusters that will be handled in this job, suggesting a sequential or batch processing approach for different data segments.

```text
STARTING JOB:  30
This job will process clusters 30 to  31
This task will process 1 clusters: cluster 30 to cluster 31
This task will process cluster 30 to cluster 31


```

--------------------------------

### Python: Process Clusters Sequentially

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This Python code iterates through a range of jobs, initiating a SemDeDupJob for each. It processes clusters in sequential chunks, indicating the start and end cluster for each job. Dependencies include the SemDeDupJob class and arguments defined elsewhere. The output shows the job progress and the clusters being processed.

```python
# 逐个处理cluster
for job in range(total_jobs):
    print('STARTING JOB: ', job)
    exp = SemDeDupJob(
        args=args,
        job_start_cluster=job * clusters_per_job
    )
    exp()
```

--------------------------------

### Clone Repository and Navigate to Project Directory

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/README.md

This snippet demonstrates how to clone the ChartQA-MLLM repository from GitHub and navigate into the newly created directory. This is the initial step for setting up the project environment.

```bash
git clone https://github.com/zengxingchen/ChartQA-MLLM.git
cd ChartQA-MLLM
```

--------------------------------

### Python Configuration and Argument Parsing

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This snippet demonstrates Python configuration for a machine learning project. It initializes variables for dataset size, embedding dimensions, and cluster counts. It then calculates the total number of jobs needed and creates an `argparse.Namespace` object to hold these parameters, making them easily accessible. Dependencies include the `math` module for ceiling division and `argparse` for namespace creation.

```python
import math
from argparse import Namespace

clusters_per_job = 1
num_clusters = 100
# dataset_size, emb_size  = all_features.shape
dataset_size, emb_size  = (611934, 512)

# 计算总的jobs数量
total_jobs = math.ceil(num_clusters / clusters_per_job)

# 定义参数结构
args = Namespace(
    embs_memory_loc=embs_memory_loc,
    save_loc=save_loc,
    sorted_clusters_path=sorted_clusters_path,
    eps_list=eps_list,
    which_to_keep=which_to_keep,
    device=device,
    dataset_size=dataset_size,
    emd_size=emb_size,
    seed=1234,
    clusters_per_job=clusters_per_job,
    num_clusters=num_clusters
)
```

--------------------------------

### Configure Project Settings in Python

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This Python code snippet sets up various configuration parameters for a project, including directories for saving files, memory locations for embeddings, a list of epsilon values for similarity thresholds, and the device to be used (CPU or GPU). It also creates a save directory if it doesn't exist.

```python
import math
import yaml
import os

save_dir = './sample_files'
embs_memory_loc = f'{save_dir}/embeddings_memmap.dat'
save_loc = f"{save_dir}/results"
os.makedirs(save_loc, exist_ok=True)
sorted_clusters_path = f'{save_dir}/sorted_clusters'
eps_list = [0.0001,0.001, 0.01, 0.05, 0.1, 0.2, 0.3, 0.31, 0.32, 0.33, 0.34, 0.35, 0.4, 0.5, 0.9, 0.99]
which_to_keep = 'hard'
device = 'cpu'
```

--------------------------------

### Job and Cluster Processing Status

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

Indicates the start and end of jobs and the processing of clusters. This provides a high-level overview of the workflow execution, including which clusters are being processed and the overall completion status.

```Text
DONE cluster:  16
STARTING JOB:  17
This job will process clusters 17 to  18
This task will process 1 clusters: cluster 17 to cluster 18
This task will process cluster 17 to cluster 18
```

```Text
DONE cluster:  17
STARTING JOB:  18
This job will process clusters 18 to  19
This task will process 1 clusters: cluster 18 to cluster 19
This task will process cluster 18 to cluster 19
```

```Text
DONE cluster:  18
STARTING JOB:  19
This job will process clusters 19 to  20
This task will process 1 clusters: cluster 19 to cluster 20
This task will process cluster 19 to cluster 20
```

```Text
DONE cluster:  19
STARTING JOB:  20
This job will process clusters 20 to  21
This task will process 1 clusters: cluster 20 to cluster 21
This task will process cluster 20 to cluster 21
```

```Text
DONE cluster:  20
STARTING JOB:  21
This job will process clusters 21 to  22
This task will process 1 clusters: cluster 21 to cluster 22
This task will process cluster 21 to cluster 22
```

```Text
DONE cluster:  21
STARTING JOB:  22
```

--------------------------------

### Job Processing Log in Python

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

Logs the start of a job for a specific cluster range and prints a tensor representing some data. This is indicative of batch processing or iterative computations in a machine learning workflow.

```python
STARTING JOB:  56
This job will process clusters 56 to  57
This task will process 1 clusters: cluster 56 to cluster 57
This task will process cluster 56 to cluster 57
===================
tensor([0.0000, 0.1706, 0.3643,  ..., 0.9843, 0.9861, 0.9931])
```

--------------------------------

### Cluster Processing Status in Python

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This snippet indicates the status of cluster processing jobs. It logs when a job starts, which clusters it will process, and when it is done. This is crucial for tracking the workflow and identifying completion of specific tasks.

```python
STARTING JOB:  89
This job will process clusters 89 to  90
This task will process 1 clusters: cluster 89 to cluster 90
This task will process cluster 89 to cluster 90
```

```python
DONE cluster:  89
```

```python
STARTING JOB:  90
This job will process clusters 90 to  91
This task will process 1 clusters: cluster 90 to cluster 91
This task will process cluster 90 to cluster 91
```

```python
DONE cluster:  90
```

```python
STARTING JOB:  91
This job will process clusters 91 to  92
This task will process 1 clusters: cluster 91 to cluster 92
This task will process cluster 91 to cluster 92
```

```python
DONE cluster:  91
```

```python
STARTING JOB:  92
This job will process clusters 92 to  93
This task will process 1 clusters: cluster 92 to cluster 93
This task will process cluster 92 to cluster 93
```

```python
DONE cluster:  92
```

```python
STARTING JOB:  93
This job will process clusters 93 to  94
This task will process 1 clusters: cluster 93 to cluster 94
This task will process cluster 93 to cluster 94
```

```python
DONE cluster:  93
```

```python
STARTING JOB:  94
This job will process clusters 94 to  95
This task will process 1 clusters: cluster 94 to cluster 95
This task will process cluster 94 to cluster 95
```

--------------------------------

### Job Completion and Next Job Log in Python

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

Logs the completion of a cluster job and indicates the start of the next job, specifying the range of clusters to be processed. This signifies sequential execution of tasks in a pipeline.

```python
DONE cluster:  58
STARTING JOB:  59
This job will process clusters 59 to  60
This task will process 1 clusters: cluster 59 to cluster 60
This task will process cluster 59 to cluster 60
```

--------------------------------

### Import Libraries for ML Tasks

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This snippet imports various necessary libraries for machine learning tasks, including math, os, numpy, pandas, submitit, torch, tqdm, pickle, and random. These are commonly used for data manipulation, model training, and progress tracking.

```python
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.

# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

import math
import os
import numpy as np
import pandas as pd
import submitit
import torch
from tqdm import tqdm
import pickle
import random
import math
import time
import pprint
```

--------------------------------

### Display Job and Cluster Processing Status - Generic

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This snippet indicates the completion of a cluster processing job and the start of the next one, along with the range of clusters to be processed. This is common in distributed or iterative processing frameworks.

```generic
DONE cluster:  33
STARTING JOB:  34
This job will process clusters 34 to  35
This task will process 1 clusters: cluster 34 to cluster 35
This task will process cluster 34 to cluster 35
```

```generic
DONE cluster:  34
STARTING JOB:  35
This job will process clusters 35 to  36
This task will process 1 clusters: cluster 35 to cluster 36
This task will process cluster 35 to cluster 36
```

```generic
DONE cluster:  35
STARTING JOB:  36
This job will process clusters 36 to  37
This task will process 1 clusters: cluster 36 to cluster 37
This task will process cluster 36 to cluster 37
```

```generic
DONE cluster:  36
STARTING JOB:  37
This job will process clusters 37 to  38
This task will process 1 clusters: cluster 37 to cluster 38
This task will process cluster 37 to cluster 38
```

```generic
DONE cluster:  37
STARTING JOB:  38
This job will process clusters 38 to  39
This task will process 1 clusters: cluster 38 to cluster 39
This task will process cluster 38 to cluster 39
```

--------------------------------

### Job and Cluster Processing Log

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This log entry indicates the completion of a cluster processing job and the start of a new one. It specifies the range of clusters to be processed in the upcoming job, suggesting an iterative or distributed processing workflow.

```text
DONE cluster:  27
STARTING JOB:  28
This job will process clusters 28 to  29
This task will process 1 clusters: cluster 28 to cluster 29
This task will process cluster 28 to cluster 29


```

--------------------------------

### Python Classifier Training Initialization

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This Python snippet initializes and trains a classifier. It specifies the feature column name, uses FocalLoss as the criterion, and provides the clip features and data for training. The 'train' function is called to perform the classification training.

```python
feature_column_name = 'number-annotation'
criterion = FocalLoss()
features = clip_features
classifier = train(data, features, feature_column_name, criterion)

```

--------------------------------

### Completion of Job and Cluster Processing (Python)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This snippet indicates the completion of a cluster processing job and signals the start of the next job. This is a standard output for batch processing systems to track progress and manage workflow transitions.

```python
DONE cluster:  10
STARTING JOB:  11
```

--------------------------------

### Python Training Function and Execution for ChartQA-MLM

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This Python code defines a `train` function for a machine learning model, likely for chart analysis. It handles data splitting, dataset creation, dataloaders, model initialization (MLPClassifier), optimization, and the training loop over multiple epochs. The function evaluates accuracy on test data and prints a classification report. The subsequent code snippet shows how to initialize the loss criterion and call the train function with specified features and column names.

```python
def train(data, pretrained_features, feature_column_name, criterion):
    # data = pd.read_csv('filtering/annotation_18_30k.csv')
    # You are expected to extract corresponding features with feature_extraction.ipynb
    
    num_classes = len(data[feature_column_name].unique())
    # print(data[feature_column_name].unique())

    train_data, test_data, train_features, test_features = train_test_split(
        data, pretrained_features, test_size=0.2, random_state=1234)
    train_dataset = ChartFeatureDataset(train_data, train_features, feature_column_name)
    test_dataset = ChartFeatureDataset(test_data, test_features, feature_column_name)
    train_dataloader = DataLoader(
        train_dataset, batch_size=2048, shuffle=True, num_workers=8)
    test_dataloader = DataLoader(
        test_dataset, batch_size=2048, shuffle=False, num_workers=8)
    classifier = MLPClassifier(
        input_dim=pretrained_features.shape[1], num_classes=num_classes).to(device)  

    
    optimizer = torch.optim.Adam(classifier.parameters(), lr=1e-3)
    
    epochs = 20
    
    for epoch in tqdm(range(epochs)):
        classifier.train()
        total_loss = 0
        train_preds, train_labels_list = [], []
        for features, labels in train_dataloader:
            labels = labels.to(device)
            features = features.to(device)
            optimizer.zero_grad()
            
            outputs = classifier(features)
            _, predicted = torch.max(outputs, 1)
            loss = criterion(outputs, F.one_hot(labels, num_classes=num_classes).float())
            loss.backward()
            optimizer.step()
            
            total_loss += loss.item()
            train_preds.extend(predicted.cpu().numpy())
            train_labels_list.extend(labels.cpu().numpy())
            
        train_accuracy = accuracy_score(train_labels_list, train_preds)
  
        #eval
        classifier.eval()
        test_preds, test_labels_list = [], []
        with torch.no_grad():
            for features, labels in test_dataloader:
                labels = labels.to(device)
                features = features.to(device)

                outputs = classifier(features)
                _, predicted = torch.max(outputs, 1)
                test_preds.extend(predicted.cpu().numpy())
                test_labels_list.extend(labels.cpu().numpy())
        test_accuracy = accuracy_score(test_labels_list, test_preds)
        print(f"Epoch {epoch+1}, Loss: {total_loss / len(train_dataloader)}, Train Accuracy: {train_accuracy}, Test Accuracy: {test_accuracy}")
        
        # Report
    print("Classification Report on Test Data:")
    print(classification_report(test_labels_list, test_preds, digits=4))
    return classifier
```

```python
criterion = nn.MSELoss()
feature_column_name = 'number-annotation' # label, Trend, Layout, number annotation, grouping
features = convnext_features
classifier = train(data, features, feature_column_name, criterion)
```

--------------------------------

### Alternative Training Progress Logs (Epochs, Loss, Accuracy)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This set of logs shows the training progress for a different model or configuration, also detailing epoch number, loss, training accuracy, and test accuracy. It includes progress bar information and spans several epochs.

```text
 5%|▌         | 1/20 [00:02<00:56,  3.00s/it]
Epoch 1, Loss: 0.5378741438572223, Train Accuracy: 0.7797279013830427, Test Accuracy: 0.8457606734816596

 10%|█         | 2/20 [00:05<00:50,  2.83s/it]
Epoch 2, Loss: 0.14398399797769693, Train Accuracy: 0.8497068550811786, Test Accuracy: 0.8693625977149729

 15%|█▌        | 3/20 [00:08<00:45,  2.69s/it]
Epoch 3, Loss: 0.1156140285042616, Train Accuracy: 0.8688364401683704, Test Accuracy: 0.8656043295249549

 20%|██        | 4/20 [00:11<00:44,  2.75s/it]
```

--------------------------------

### Initial Training Progress Output

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This output shows the initial progress of a machine learning model training, indicating 0% completion and no time elapsed yet. It's typically seen at the very beginning of a training process.

```text
  0%|          | 0/20 [00:00<?, ?it/s]

```

--------------------------------

### Training Progress Logs (Epochs, Loss, Accuracy)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

These logs display the progress of a machine learning model training process over multiple epochs. Each entry shows the epoch number, the loss value, training accuracy, and test accuracy, along with a progress bar indicating the completion percentage.

```text
Epoch 1, Loss: 0.17285589529917791, Train Accuracy: 0.8227600721587492, Test Accuracy: 0.8907095610342755

 10%|█         | 2/20 [00:05<00:52,  2.90s/it]
Epoch 2, Loss: 0.08631693285245162, Train Accuracy: 0.9076217678893566, Test Accuracy: 0.9076969332531569

 15%|█▌        | 3/20 [00:08<00:48,  2.85s/it]
Epoch 3, Loss: 0.070257078569669, Train Accuracy: 0.919986470234516, Test Accuracy: 0.9225796752856283

 20%|██        | 4/20 [00:11<00:45,  2.87s/it]
Epoch 4, Loss: 0.06273202798687495, Train Accuracy: 0.9298331328923632, Test Accuracy: 0.927690920024053

 25%|██▌       | 5/20 [00:14<00:42,  2.85s/it]
Epoch 5, Loss: 0.057956019846292645, Train Accuracy: 0.9367107636800962, Test Accuracy: 0.9349067949488875

 30%|███       | 6/20 [00:16<00:38,  2.77s/it]
Epoch 6, Loss: 0.054347946953315004, Train Accuracy: 0.9420475045099218, Test Accuracy: 0.9374624173180999

 35%|███▌      | 7/20 [00:19<00:36,  2.77s/it]
Epoch 7, Loss: 0.051680562587884754, Train Accuracy: 0.9461815995189417, Test Accuracy: 0.9388153938665064

 40%|████      | 8/20 [00:22<00:33,  2.82s/it]
Epoch 8, Loss: 0.04949346070106213, Train Accuracy: 0.94791040288635, Test Accuracy: 0.9442273000601323

 45%|████▌     | 9/20 [00:25<00:31,  2.84s/it]
Epoch 9, Loss: 0.047596266923042446, Train Accuracy: 0.9506539386650631, Test Accuracy: 0.9422730006013229

 50%|█████     | 10/20 [00:28<00:28,  2.86s/it]
Epoch 10, Loss: 0.0461246740932648, Train Accuracy: 0.9511425135297655, Test Accuracy: 0.9472339146121467

 55%|█████▌    | 11/20 [00:31<00:26,  2.90s/it]
Epoch 11, Loss: 0.04471543717842836, Train Accuracy: 0.952720986169573, Test Accuracy: 0.9508418520745641

 60%|██████    | 12/20 [00:34<00:22,  2.86s/it]
Epoch 12, Loss: 0.04366217524959491, Train Accuracy: 0.953961214672279, Test Accuracy: 0.9427239927841251

 65%|██████▌   | 13/20 [00:37<00:20,  2.94s/it]
Epoch 13, Loss: 0.04294638794202071, Train Accuracy: 0.9542994588093806, Test Accuracy: 0.9482862297053518

 70%|███████   | 14/20 [00:39<00:17,  2.84s/it]
Epoch 14, Loss: 0.041623789530534014, Train Accuracy: 0.9555772699939867, Test Accuracy: 0.9506915213469633

 75%|███████▌  | 15/20 [00:42<00:13,  2.80s/it]
Epoch 15, Loss: 0.04106529945364365, Train Accuracy: 0.9556900180396873, Test Accuracy: 0.952495490078172

 80%|████████  | 16/20 [00:45<00:11,  2.82s/it]
Epoch 16, Loss: 0.040564390329214245, Train Accuracy: 0.9559155141310884, Test Accuracy: 0.953547805171377

 85%|████████▌ | 17/20 [00:48<00:08,  2.82s/it]
Epoch 17, Loss: 0.039436387901122756, Train Accuracy: 0.9572309079975947, Test Accuracy: 0.9488875526157546

 90%|█████████ | 18/20 [00:50<00:05,  2.64s/it]
Epoch 18, Loss: 0.03889151605275961, Train Accuracy: 0.9572684906794949, Test Accuracy: 0.9490378833433554

 95%|█████████▌| 19/20 [00:53<00:02,  2.57s/it]
Epoch 19, Loss: 0.0384263015137269, Train Accuracy: 0.9580201443174985, Test Accuracy: 0.9485868911605532

100%|██████████| 20/20 [00:55<00:00,  2.77s/it]
Epoch 20, Loss: 0.037848459699979194, Train Accuracy: 0.9586966325917018, Test Accuracy: 0.9493385447985568
```

--------------------------------

### Model Training Progress (Second Run)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This section shows the training progress for a different model run, also across multiple epochs. It logs the loss and accuracy metrics, similar to the first log, providing insights into the learning curve of this specific configuration.

```text
5%|▌         | 1/20 [00:02<00:54,  2.85s/it]

Epoch 1, Loss: 0.1843610514815037, Train Accuracy: 0.8177239927841251, Test Accuracy: 0.8541791942273

10%|█         | 2/20 [00:05<00:49,  2.78s/it]

Epoch 2, Loss: 0.11021389353733796, Train Accuracy: 0.86458959711365, Test Accuracy: 0.8672579675285629

15%|█▌        | 3/20 [00:08<00:47,  2.82s/it]

Epoch 3, Loss: 0.08966564329770896, Train Accuracy: 0.88751503307276, Test Accuracy: 0.8926638604930848

20%|██        | 4/20 [00:11<00:46,  2.89s/it]

Epoch 4, Loss: 0.07762728803432904, Train Accuracy: 0.9044272399278412, Test Accuracy: 0.9052916416115454

25%|██▌       | 5/20 [00:14<00:42,  2.84s/it]

Epoch 5, Loss: 0.06947047664568974, Train Accuracy: 0.917280517137703, Test Accuracy: 0.9158147925435959

30%|███       | 6/20 [00:16<00:37,  2.69s/it]

Epoch 6, Loss: 0.06393326417757915, Train Accuracy: 0.92498496692724, Test Accuracy: 0.9221286831028263

35%|███▌      | 7/20 [00:18<00:32,  2.50s/it]
```

--------------------------------

### Process Clusters and Save Results in Python

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This Python code snippet outlines the process of handling clusters, determining which data points to keep based on similarity thresholds (epsilon values), and saving the results to a file. It utilizes libraries like pandas, torch, and pickle for data manipulation and serialization. It also includes timing mechanisms and printing progress updates.

```python
clutser_items_indices = list(range(cluster_size))
            ## -- OR: shuffle cluster to keep random example from each group
            if self.args.which_to_keep.lower() == "random":
                random.shuffle(clutser_items_indices)
                cluster_i = cluster_i[clutser_items_indices]
            ## -- OR: reverse cluster to keep easy examples
            if self.args.which_to_keep.lower() == "easy":
                clutser_items_indices = clutser_items_indices[::-1]
                cluster_i = cluster_i[clutser_items_indices]

            ## -- indices for cluster items in the dataset
            cluster_ids = cluster_i[:, 1].astype("int32")
            cluster_reps = embs[cluster_ids]
            cluster_reps = torch.tensor(cluster_reps)

            M = self.semdedup(cluster_i, cluster_reps, self.args.device)

            points_to_remove_df = pd.DataFrame()
            points_to_remove_df["indices"] = clutser_items_indices

            for eps in self.args.eps_list:
                ## -- 5) We need to remove a point from the dataset when its pairwise similarity to other point is > 1-ebs
                eps_points_to_remove = M > 1 - eps
                points_to_remove_df[f"eps={eps}"] = eps_points_to_remove

            if self.args.save_loc != "":
                ## --save df
                with open(df_file_loc, "wb") as file:
                    pickle.dump(points_to_remove_df, file)

            # step_time.append_cluster(time.time() - step_st)
            step_time.append(time.time() - step_st)
            print("DONE cluster: ", cluster_id)

        # print(
        #     f"DONE in {((time.time()-st)/60):.2f} minutes, Average Step time {(sum(step_time)/len(step_time)):.2f}(s)"
        # )
        return
```

--------------------------------

### Example Usage for Extract Pruned Data Function (Python)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This snippet demonstrates how to call the 'extract_pruned_data' function with specific parameters. It defines the paths for saving the output, pruning tables, and sorted strata, along with epsilon and number of clusters. The function is then invoked to process the data and save the pruned image paths.

```python
output_txt_path = f'{save_dir}/test.txt'
semdedup_pruning_tables_path = f'{save_dir}/results'
sorted_stratums_paths = f'{save_dir}/sorted_stratums'
olds = 0.32
num_clusters = 100
extract_pruned_data(sorted_stratums_paths, semdedup_pruning_tables_path, eps, num_clusters, output_txt_path, retreive_kept_samples=True)
```

--------------------------------

### Displaying Tensor Data and Step Time (Python)

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This snippet shows how to display tensor data and the time taken for a step. It is commonly used for visualizing intermediate results and performance metrics in machine learning tasks. The output typically includes a large tensor representation and a step time in seconds.

```python
tensor([[1.0000, 0.1526,  ..., 0.2037, 0.1829, 0.2327],
        [0.1455, 0.1526, 1.0000,  ..., 0.1701, 0.1166, 0.2125],
        ...,
        [0.1924, 0.2037, 0.1701,  ..., 1.0000, 0.7388, 0.7214],
        [0.1781, 0.1829, 0.1166,  ..., 0.7388, 1.0000, 0.6753],
        [0.2356, 0.2327, 0.2125,  ..., 0.7214, 0.6753, 1.0000]])
Step time: 0.009508609771728516(s)
===================
tensor([0.0000, 0.9803, 0.1526,  ..., 0.9786, 0.7686, 0.8384])
===================
```

--------------------------------

### Task Distribution for Cluster Processing in Python

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This Python code snippet demonstrates how to divide a large number of clusters among different tasks (CPUs) for parallel processing. It calculates the number of clusters each task should handle and defines the start and end cluster IDs for the current task. It also prints informative messages about the assigned cluster range for the task.

```python
# pp = pprint.PrettyPrinter(indent=4)
# pp.pprint(vars(self.args))
job_start_cluster = self.job_start_cluster

print(
    f"This job will process clusters {job_start_cluster} to  {min(self.args.num_clusters, job_start_cluster+self.args.clusters_per_job)}"
)

# job_env = submitit.JobEnvironment()

# print(f"There are {job_env.num_tasks} tasks in this job")

# print(f"I'm the task #{job_env.local_rank} on node {job_env.node}")
# print(f"I'm the task #{job_env.global_rank} in the job")

## divide clusters across tasks (cpus)
num_clusters_per_task = 1 #int(
#     math.ceil(self.args.clusters_per_job / job_env.num_tasks)
# )
task_rank = 0
start_cluster = job_start_cluster + task_rank * num_clusters_per_task
end_cluster = job_start_cluster + (task_rank + 1) * num_clusters_per_task
end_cluster = min(self.args.num_clusters, end_cluster)
end_cluster = min(end_cluster, job_start_cluster + self.args.clusters_per_job)
print(
    f"This task will process {num_clusters_per_task} clusters: cluster {start_cluster} to cluster {end_cluster}"
)
print(
    f"This task will process cluster {start_cluster} to cluster {end_cluster}"
)

self._process_shard(start_cluster, end_cluster)
```

--------------------------------

### Early Stage Training Progress Output

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This output indicates that the model training has progressed to 5% completion, with one out of twenty epochs processed and over 2 seconds elapsed per iteration.

```text
  5%|▌         | 1/20 [00:02<00:51,  2.71s/it]

```

--------------------------------

### Epoch 1 Training Metrics

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This output shows the metrics after the first epoch of training. It includes the loss value, training accuracy, and test accuracy achieved in this initial epoch.

```text
Epoch 1, Loss: 0.41647157302269566, Train Accuracy: 0.8117483463619964, Test Accuracy: 0.8377931449188214

```

--------------------------------

### Initialize KMeans Clustering with FAISS

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/sampling/stratified_sampling.ipynb

This snippet initializes KMeans clustering using FAISS. It prepares memory-mapped files for embeddings and paths, then calls the `compute_centroids` function. Dependencies include numpy, faiss, and a logger object. It takes data, number of centroids, iterations, seed, distance metric, save folder, and verbosity as input.

```python
SEED = 1234
random.seed(SEED)
emb_memory_loc = f'{save_dir}/embeddings_memmap.dat'
paths_memory_loc = f'{save_dir}/paths_memmap.dat'
dataset_size, emb_size = all_features.shape
path_str_type = "U200"

emb_memory = np.memmap(emb_memory_loc, dtype='float32', mode='r', shape=(dataset_size, emb_size))
paths_memory = np.memmap(paths_memory_loc, dtype=path_str_type, mode='r', shape=(dataset_size,))

save_folder = f'{save_dir}/kmeans'
compute_centroids(
    data=emb_memory,
    ncentroids=100,
    niter=100,
    seed=SEED,
    Kmeans_with_cos_dist=True,
    save_folder=save_folder,
    logger=logger,
    verbose=True,
)
```

--------------------------------

### 40% Training Progress Output

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This output shows that the model training has reached 40% completion, with eight out of twenty epochs processed and an average time of 2.71 seconds per iteration.

```text
 40%|████      | 8/20 [00:22<00:32,  2.71s/it]

```

--------------------------------

### Classifier Training with Focal Loss

Source: https://github.com/zengxingchen/chartqa-mllm/blob/main/data_filtering/classification/classification.ipynb

This Python snippet demonstrates training a classifier using ResNet features and a Focal Loss criterion. It defines the feature column name, instantiates the FocalLoss, specifies the features, and then calls the train function with the data and feature column. Ensure 'data', 'resnet_features', and 'FocalLoss' are correctly defined and imported.

```python
feature_column_name = 'number-annotation'
criterion = FocalLoss()
features = resnet_features
classifier = train(data, features, feature_column_name, criterion)
```
Tokens:
Raw