CGMformer

https://github.com/yurunlu/cgmformer
Admin
CGMformer is a pretrained transformer model designed to decode individual glucose dynamics from...
Tokens:13,156
Snippets:62
Trust Score:4.9
License:MIT
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://github.com/yurunlu/cgmformer/blob/main/README.md

Install the required Python packages for the project.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install ipykernel

Source: https://github.com/yurunlu/cgmformer/blob/main/build_vocab.ipynb

Command to install the ipykernel package required for running cells in the CGMformer environment.

```bash
conda install -n CGMformer ipykernel --update-deps --force-reinstall
```

--------------------------------

### Example CGMformer Data Record

Source: https://github.com/yurunlu/cgmformer/blob/main/processing_811_data.ipynb

A sample dictionary representing a single patient record in the CGMformer dataset.

```python
Result:
[{'input_ids': [66.60000000000001,
   66.60000000000001,
   64.8,
   64.8,
   63.0,
   61.2,
   64.8,
   64.8,
   59.4,
   52.2,
   43.2,
   39.6,
   39.6,
   39.6,
   39.6,
   39.6,
   45.0,
   46.8,
   46.8,
   48.6,
   48.6,
   48.6,
   48.6,
   50.4,
   50.4,
   50.4,
   52.2,
   52.2,
   52.2,
   54.0,
   54.0,
   55.8,
   55.8,
   55.8,
   59.4,
   61.2,
   61.2,
   61.2,
   64.8,
   66.60000000000001,
   68.39999999999999,
   70.2,
   72.0,
   73.8,
   75.60000000000001,
   77.39999999999999,
   77.39999999999999,
   79.2,
   82.8,
   84.60000000000001,
   84.60000000000001,
   86.39999999999999,
   86.39999999999999,
   88.2,
   90.0,
   95.4,
   97.2,
   95.4,
   95.4,
   99.0,
   104.4,
   106.2,
   109.8,
   109.8,
   108.0,
   108.0,
   109.8,
   106.2,
   106.2,
   106.2,
   106.2,
   106.2,
   108.0,
   106.2,
   108.0,
   104.4,
   115.2,
   113.4,
   113.4,
   111.6,
   115.2,
   117.0,
   117.0,
   120.6,
   109.8,
   102.6,
   115.2,
   113.4,
   102.6,
   117.0,
   113.4,
   124.2,
   120.6,
   113.4,
   113.4,
   135.0,
   162.0,
   172.79999999999998,
   180.0,
   183.6,
   183.6,
   180.0,
   183.6,
   174.6,
   171.0,
   167.4,
   167.4,
   169.20000000000002,
   167.4,
   171.0,
   171.0,
   163.79999999999998,
   165.6,
   169.20000000000002,
   167.4,
   165.6,
   165.6,
   162.0,
   158.4,
   147.6,
   142.20000000000002,
   140.4,
   144.0,
   154.79999999999998,
   156.6,
   158.4,
   154.79999999999998,
   149.4,
   144.0,
   144.0,
   138.6,
   138.6,
   136.79999999999998,
   135.0,
   133.20000000000002,
   127.8,
   127.8,
   131.4,
   138.6,
   144.0,
   151.20000000000002,
   156.6,
   162.0,
   171.0,
   178.20000000000002,
   183.6,
   196.2,
   201.6,
   203.4,
   205.2,
   207.0,
   210.6,
   210.6,
   207.0,
   207.0,
   210.6,
   207.0,
   210.6,
   212.4,
   214.2,
   217.8,
   221.4,
   221.4,
   216.0,
   198.0,
   189.0,
   189.0,
   185.4,
   185.4,
   183.6,
   174.6,
   185.4,
   187.2,
   185.4,
   187.2,
   183.6,
   178.20000000000002,
   180.0,
   174.6,
   160.20000000000002,
   154.79999999999998,
   156.6,
   149.4,
   144.0,
   133.20000000000002,
   129.6,
   135.0,
   135.0,
   131.4,
   133.20000000000002,
   131.4,
   122.4,
   126.0,
   126.0,
   118.8,
   120.6,
   122.4,
   120.6,
   117.0,
   115.2,
   118.8,
   120.6,
   120.6,
   122.4,
   127.8,
   133.20000000000002,
   135.0,
   140.4,
   144.0,
   154.79999999999998,
   165.6,
   165.6,
   171.0,
   181.8,
   181.8,
   187.2,
   189.0,
   190.8,
   198.0,
   201.6,
   201.6,
   199.8,
   194.4,
   189.0,
   189.0,
   187.2,
   185.4,
   181.8,
   192.6,
   192.6,
   192.6,
   198.0,
   205.2,
   212.4,
   212.4,
   207.0,
   198.0,
   192.6,
   183.6,
   176.4,
   171.0,
   167.4,
   167.4,
   167.4,
   169.20000000000002,
   171.0,
   169.20000000000002,
   165.6,
   160.20000000000002,
   156.6,
   147.6,
   140.4,
   129.6,
   113.4,
   117.0,
   104.4,
   109.8,
   108.0,
   111.6,
   117.0,
   120.6,
   124.2,
   122.4,
   124.2,
   126.0,
   126.0,
   126.0,
   124.2,
   122.4,
   117.0,
   115.2,
   111.6,
   108.0,
   106.2,
   104.4,
   102.6,
   100.8,
   99.0,
   99.0,
   99.0,
   100.8,
   100.8,
   100.8,
   100.8,
   100.8,
   102.6,
   104.4,
   106.2],
  'id': 'Shanghai_T2D_266581',
  'types': 'T2D',
  'age': 63.0,
  'bmi': 23.52365539,
  'hba1c': 6.6,
  'homa-b': nan,
  'homa-is': nan,
  'index': 1440,
  'Fast_s': 0,
  'Fast_e': 72,
  'Dawn_s': 48,
  'Dawn_e': 72,
... [Output truncated]
```

--------------------------------

### Download Checkpoint

Source: https://github.com/yurunlu/cgmformer/blob/main/README.md

Download the pre-trained model weights from Google Drive.

```bash
wget https://drive.google.com/file/d/1SOUkaRoMR7eOGb2EUYBJ-QmXI1Lc0af9
```

--------------------------------

### Load Datasets from Disk

Source: https://github.com/yurunlu/cgmformer/blob/main/rank.ipynb

Initializes various dataset splits from specified file paths using the Hugging Face datasets library.

```python
from itertools import chain
from datasets import load_from_disk
import numpy as np

train_datsset_path = "/share/home/liangzhongming/930/CGMformer/data/Shanghai_train"
train_dataset=load_from_disk(train_datsset_path)

valid_datsset_path = "/share/home/liangzhongming/930/CGMformer/data/Shanghai_valid"
valid_dataset=load_from_disk(valid_datsset_path)

finetune_datsset_path = "/share/home/liangzhongming/930/CGMformer/data/Shanghai_finetune"
finetune_dataset=load_from_disk(train_datsset_path)

test_datsset_path = "/share/home/liangzhongming/930/CGMformer/data/Shanghai_test"
test_dataset=load_from_disk(train_datsset_path)
test_dataset

fold_4_train_path = "/share/home/liangzhongming/930/CGMformer/data/8_2_newData/five_cross_valid/fold_4_train"
fold_4_train=load_from_disk(fold_4_train_path)
fold_4_train['index']
```

--------------------------------

### Create and Save Sampled Dataset

Source: https://github.com/yurunlu/cgmformer/blob/main/rank.ipynb

Constructs a new dataset from sampled input IDs and types, then persists it to disk.

```python
from datasets import Dataset

# Create a new dataset from the sampled data
sampled_dataset = Dataset.from_dict({
    'input_ids': sampled_input_ids,
    'types': sampled_types
})

# Save the new dataset
sampled_dataset.save_to_disk("/share/home/liangzhongming/930/CGMformer/data/Shanghai_total_value")
```

--------------------------------

### Create and Save Dataset

Source: https://github.com/yurunlu/cgmformer/blob/main/processing_811_data.ipynb

Constructs a Hugging Face Dataset from a dictionary and saves it to the specified directory.

```python
new_data = {
    'input_ids': input_ids,
    'types': types,
    'index': index,
    'id': id,
    'p_id': p_id,
    'hba1c': hba1c,
    'filled': filled
}
dataset = Dataset.from_dict(new_data)
dataset.save_to_disk("/share/home/liangzhongming/930/CGMformer/data/Data_downstream/Colas_1028_hfds")
```

```python

new_data = {
    'input_ids': input_ids,

    'index': index,
    'id': id,
    'types': types,
    'cerebrovascular_disease': cerebrovascular_disease,
    'coronary_heart_disease': coronary_heart_disease,
    'peripheral_arterial_disease': peripheral_arterial_disease,
    'nephropathy': nephropathy,
    'neuropathy': neuropathy,
    'retinopathy': retinopathy,
    'macrovascular ': macrovascular ,
    'microvascular': microvascular,
    'complication': complication,
}
dataset = Dataset.from_dict(new_data)
dataset.save_to_disk("/share/home/liangzhongming/930/CGMformer/data/Data_downstream/1017_Zhao_total")
```

--------------------------------

### Generate and Save Vocabulary Mappings

Source: https://github.com/yurunlu/cgmformer/blob/main/build_vocab.ipynb

Creates token-to-ID and ID-to-token dictionaries and saves them as pickle files for model usage.

```python
import pickle
from itertools import count

token2id = {}
id2token = {}

for i in range(39,302):
    token = i
    token2id[token] = i-39
    id2token[i-39] = token


token2id["<mask>"] = max(token2id.values()) + 1
token2id["<pad>"] = max(token2id.values()) + 1
token2id["<cls>"] = max(token2id.values()) + 1
# token2id["<mask>"] = 302
# id2token[302] = "<mask>"

# token2id["<pad>"] = 303  
# id2token[303] = "<pad>"

# token2id["<cls>"] = 304
# id2token[304] = "<cls>"
id2token = {index: token for token, index in token2id.items()}

with open('/share/home/liangzhongming/930/CGMformer/data/8_11_data/token2id.pkl', 'wb') as f:
    pickle.dump(token2id, f)
    
with open('/share/home/liangzhongming/930/CGMformer/data/8_11_data/id2token.pkl', 'wb') as f:  
    pickle.dump(id2token, f)
```

--------------------------------

### Import Dependencies for CGMformer

Source: https://github.com/yurunlu/cgmformer/blob/main/sub_sample.ipynb

Initializes environment variables and imports necessary libraries for data manipulation and model training.

```python
import datetime
import matplotlib as plt

import os
os.environ["NCCL_DEBUG"] = "INFO"
os.environ["OMPI_MCA_opal_cuda_support"] = "true"
os.environ["CONDA_OVERRIDE_GLIBC"] = "2.56"

import pickle
import random
import subprocess

from datasets import Dataset
import pandas as pd
import numpy as np
import pytz
import torch
from datasets import load_dataset
from datasets import load_from_disk
```

--------------------------------

### Load Dataset

Source: https://github.com/yurunlu/cgmformer/blob/main/build_vocab.ipynb

Load a dataset from a specified disk path using the datasets library.

```python
from itertools import chain
from datasets import load_from_disk
import pickle

train_datsset_path = "/share/home/liangzhongming/930/CGMformer/data/8_11_data/downstream/CV_5/downsampled_CV_5_train_96"
train_dataset=load_from_disk(train_datsset_path)

# test_datsset_path = "/share/home/liangzhongming/930/CGMformer/data/8_11_data/downstream/CV_5/downsampled_CV_5_test_96"
# test_dataset=load_from_disk(train_datsset_path)
```

--------------------------------

### Save Mappings

Source: https://github.com/yurunlu/cgmformer/blob/main/build_vocab.ipynb

Serialize the token-to-ID and ID-to-token dictionaries to pickle files.

```python
with open('/share/home/liangzhongming/930/CGMformer/data/8_11_data/downstream/CV_5/CV5_total_token2id.pkl', 'wb') as f:
    pickle.dump(token2id, f)
with open('/share/home/liangzhongming/930/CGMformer/data/8_11_data/downstream/CV_5/CV5_total_id2token.pkl', 'wb') as f:
    pickle.dump(id2token, f)
```

--------------------------------

### Load Sequence Datasets

Source: https://github.com/yurunlu/cgmformer/blob/main/analize_seq.ipynb

Loads datasets from disk and verifies the length of the input sequences.

```python
from datasets import load_from_disk
# dataset1 = load_from_disk("/share/home/liangzhongming/930/CGMformer/data/8_2_newData/Shanghai_714")
# dataset2 = load_from_disk("/share/home/liangzhongming/930/CGMformer/data/8_2_newData/Shanghai_1581_train")

# dataset1 = load_from_disk("/share/home/liangzhongming/930/CGMformer/data/Shanghai_train_subSampleV3")
# dataset2 = load_from_disk("/share/home/liangzhongming/930/CGMformer/data/Shanghai_finetune_subSampleV3")

dataset1 = load_from_disk("/share/home/liangzhongming/930/CGMformer/data/8_8_data/96/Shanghai_downsampled_total_96")
dataset2 = load_from_disk("/share/home/liangzhongming/930/CGMformer/data/8_8_data/96/Shanghai_downsampled_fold_0_train_96")
len(dataset1['input_ids'][0])
```

--------------------------------

### Map Sample IDs to Tokens

Source: https://github.com/yurunlu/cgmformer/blob/main/classify_analysis.ipynb

Iterates through samples to construct a list of dictionaries containing labels, predictions, and token-mapped sample data.

```python
results = []
for i in range(len(samples)):
  result = {
    'sample': [],  
    'label': labels[i],
    'pred': preds[i]
  }
   
  for id in samples[i][1:]:
    result['sample'].append(id2token[id])

  results.append(result)
```

--------------------------------

### Load and Prepare Data

Source: https://github.com/yurunlu/cgmformer/blob/main/processing_811_data.ipynb

Reads time series and label CSV files, validates their lengths, and converts data to list format.

```python
# dir = "/share/home/liangzhongming/930/CGMformer/data/Data_downstream/"
# dir = "/share/home/liangzhongming/930/CGMformer/data/Data_downstream/1017Zhao/"
dir = "/share/home/liangzhongming/930/CGMformer/data/Data_downstream/Colas_1028/"
time_series_df = pd.read_csv(dir + 'Colas_Data.csv', index_col=0)

# splits_df = pd.read_csv(dir + 'Zhao_TrainTestSplit.csv', index_col=0)
labels_df = pd.read_csv(dir + 'Colas_Label.csv', index_col=0)

# time_df = pd.read_csv(dir + 'Shanghai_TimeSplit.csv')
# assert len(time_series_df) == len(labels_df) == len(time_df) == len(splits_df)
assert len(time_series_df) == len(labels_df)

input_ids = time_series_df.iloc[:, :].to_numpy().tolist()
len(input_ids) # 393
len(input_ids[0]) # 288

labels_df

# cerebrovascular_disease = labels_df['cerebrovascular disease'].to_numpy().tolist()
# coronary_heart_disease = labels_df['coronary heart disease'].to_numpy().tolist()
# peripheral_arterial_disease = labels_df['peripheral arterial disease'].to_numpy().tolist()
# nephropathy = labels_df['nephropathy'].to_numpy().tolist()
# neuropathy = labels_df['neuropathy'].to_numpy().tolist()
# retinopathy = labels_df['retinopathy'].to_numpy().tolist()
# macrovascular = labels_df['macrovascular'].to_numpy().tolist()
# microvascular = labels_df['microvascular'].to_numpy().tolist()
# complication = labels_df['complication'].to_numpy().tolist()
```

--------------------------------

### Import Libraries

Source: https://github.com/yurunlu/cgmformer/blob/main/processing_811_data.ipynb

Initial imports required for data manipulation and dataset handling.

```python
import pandas as pd
from datasets import Dataset
```

--------------------------------

### Load Token Dictionary

Source: https://github.com/yurunlu/cgmformer/blob/main/classify_analysis.ipynb

Loads a token dictionary from a pickle file to map IDs to tokens.

```python
import pickle

# TOKEN_DICTIONARY_FILE = "/share/home/liangzhongming/930/CGMformer/data/8_2_newData/Shanghai_vocab_rank.pkl"
# TOKEN_DICTIONARY_FILE = "/share/home/liangzhongming/930/CGMformer/data/Shanghai_inv_vocab_sz10.pkl"
# TOKEN_DICTIONARY_FILE = "/share/home/liangzhongming/930/CGMformer/data/8_8_data/96/Shanghai_total96_token2id.pkl"
# with open(TOKEN_DICTIONARY_FILE, "rb") as f:
#     token_dictionary = pickle.load(f)

TOKEN_DICTIONARY_FILE = '/share/home/liangzhongming/930/CGMformer/data/8_11_data/id2token.pkl'
with open(TOKEN_DICTIONARY_FILE, "rb") as f:
    id2token = pickle.load(f)
```

--------------------------------

### Load Vocabulary

Source: https://github.com/yurunlu/cgmformer/blob/main/build_vocab.ipynb

Load a previously saved vocabulary from a pickle file.

```python
with open('/share/home/liangzhongming/930/CGMformer/data/8_11_data/token2id.pkl', 'rb') as f:
  vocab = pickle.load(f)
# for key, value in vocab.items():
#   print(f"key:{key}, value:{value}")
vocab
```

--------------------------------

### Calculate Token Distribution Ratios

Source: https://github.com/yurunlu/cgmformer/blob/main/mask_strategy.ipynb

This script loads a dataset from disk and iterates through samples to count tokens falling into three predefined ranges. It requires the 'numpy' and 'datasets' libraries.

```python
import numpy as np
from datasets import load_from_disk

train_datsset_path = "/share/home/liangzhongming/930/CGMformer/data/Shanghai_train_subSampleV3"
train_dataset=load_from_disk(train_datsset_path)

def count_tokens(sample):
    tokens = np.array(sample['input_ids'])
    print(tokens)
    count_low = np.sum(tokens < 70)
    count_mid = np.sum(np.logical_and(tokens >= 70, tokens < 180))
    count_high = np.sum(tokens >= 180)
    return count_low, count_mid, count_high

count_low_total = 0
count_mid_total = 0
count_high_total = 0

for sample in train_dataset:
    count_low, count_mid, count_high = count_tokens(sample)
    count_low_total += count_low
    count_mid_total += count_mid
    count_high_total += count_high

count_low_total, count_mid_total, count_high_total
```

--------------------------------

### Run CGMformer_Diet Prediction

Source: https://github.com/yurunlu/cgmformer/blob/main/README.md

Predict glucose levels using embedded vectors and meal nutrition information.

```bash
python PredictGlucose.py
```

--------------------------------

### Visualize Label Distribution

Source: https://github.com/yurunlu/cgmformer/blob/main/mask_strategy.ipynb

Creates a pie chart showing the distribution of labels for samples containing low glucose values.

```python
from collections import Counter

counter = Counter(types_of_samples_with_low_values)

labels, counts = zip(*counter.items())

fig1, ax1 = plt.subplots()
ax1.pie(counts, labels=labels, autopct='%1.1f%%', startangle=90)
ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.
plt.title('Distribution of labels for samples with at least one value below 70')

plt.show()
```

--------------------------------

### Visualize Low Glucose Ratios per Label

Source: https://github.com/yurunlu/cgmformer/blob/main/mask_strategy.ipynb

Generates pie charts for each label showing the proportion of samples with low glucose levels.

```python

colors = ['gold', 'yellowgreen', 'lightcoral']

for label in low_glucose_ratios.keys():
    ratios = [low_glucose_ratios[label], 1-low_glucose_ratios[label]]
    
    plt.figure(figsize=(6, 6))
    plt.pie(ratios, labels=['Low glucose', 'Not low glucose'], colors=colors, autopct='%1.1f%%', startangle=140)
    
    plt.title(f"Proportion of samples with glucose level < 70 for label: {label}")
    
    plt.show()
```

--------------------------------

### Dataset Saving Progress Logs

Source: https://github.com/yurunlu/cgmformer/blob/main/processing_811_data.ipynb

Console output indicating the progress of saving datasets to disk.

```text
Result:
Saving the dataset (0/1 shards):   0%|          | 0/922 [00:00<?, ? examples/s]
```

```text
Result:
Saving the dataset (0/1 shards):   0%|          | 0/292 [00:00<?, ? examples/s]
```

```text
Result:
Saving the dataset (0/1 shards):   0%|          | 0/1039 [00:00<?, ? examples/s]
```

```text
Result:
Saving the dataset (0/1 shards):   0%|          | 0/175 [00:00<?, ? examples/s]
```

--------------------------------

### Visualize Low Sugar Token Proportions

Source: https://github.com/yurunlu/cgmformer/blob/main/mask_strategy.ipynb

Calculates and plots the ratio of low sugar tokens per sample to identify samples with high hypoglycemic content.

```python
low_sugar_token_counts = []
sample_counts = len(samples_with_low_values)
for sample in samples_with_low_values:
    low_sugar_tokens = [token for token in sample['input_ids'] if token <= 70]
    low_sugar_token_ratio = len(low_sugar_tokens) / len(sample['input_ids'])
    low_sugar_token_counts.append(low_sugar_token_ratio)

plt.figure(figsize=(10, 6))
plt.bar(range(sample_counts), low_sugar_token_counts, color='blue')

plt.title('Low Sugar Token Proportions in Samples')
plt.xlabel('Sample Index')
plt.ylabel('Low Sugar Token Proportion')
# plt.legend()

plt.show()
```

--------------------------------

### Inspect Dataset Dimensions

Source: https://github.com/yurunlu/cgmformer/blob/main/sub_sample.ipynb

Loads a dataset and prints its metadata including row count and sequence length.

```python
train_datsset_path = "/share/home/liangzhongming/930/CGMformer/data/8_7_data/Shanghai_downsampled_144"
train_dataset=load_from_disk(train_datsset_path)
train_dataset, len(train_dataset['input_ids']), len(train_dataset['input_ids'][0]), len(train_dataset['types'])
```

```python
train_datsset_path = "/share/home/liangzhongming/930/CGMformer/data/Shanghai_train_subSample"
train_dataset=load_from_disk(train_datsset_path)
train_dataset, len(train_dataset['input_ids']), len(train_dataset['input_ids'][0]), len(train_dataset['types']), train_dataset['types'][0]
```

--------------------------------

### Visualize Token Distribution

Source: https://github.com/yurunlu/cgmformer/blob/main/mask_strategy.ipynb

Generates a pie chart showing the distribution of blood sugar tokens across low, mid, and high ranges.

```python
import matplotlib.pyplot as plt

total = count_low_total + count_mid_total + count_high_total
proportions = [count_low_total / total, count_mid_total / total, count_high_total / total]

labels = ['[0, 70)', '[70, 180)', '[180, )']
plt.pie(proportions, labels=labels, autopct='%1.1f%%')
plt.title('Train_dataset Token Distribution')
# plt.legend()
plt.show()
```

--------------------------------

### Generate Token Mappings

Source: https://github.com/yurunlu/cgmformer/blob/main/build_vocab.ipynb

Extract unique tokens from the dataset and create token-to-ID and ID-to-token dictionaries, including special tokens.

```python
input_ids1 = train_dataset['input_ids']
all_input_ids = list(chain(input_ids1))

unique_tokens = set()

for sequence in all_input_ids:
    for token in sequence:
        integer_token = int(token)
        unique_tokens.add(integer_token)
unique_tokens

sorted_tokens = sorted(list(unique_tokens))

token2id = {token: index for index, token in enumerate(sorted_tokens)}
token2id["<mask>"] = max(token2id.values()) + 1
token2id["<pad>"] = max(token2id.values()) + 1
token2id["<cls>"] = max(token2id.values()) + 1

id2token = {index: token for token, index in token2id.items()}
```

--------------------------------

### Visualize Sample Data

Source: https://github.com/yurunlu/cgmformer/blob/main/classify_analysis.ipynb

Reads a CSV file and generates line plots for each sample's input IDs using matplotlib.

```python
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('/share/home/liangzhongming/930/CGMformer/sample/old288_sz1_5fold4_V2.csv')

df['input_ids'] = df['input_ids'].apply(lambda x: eval(x))

plt.style.use('seaborn-darkgrid')

for i, row in df.iterrows():
    plt.figure(figsize=(10, 6))
    plt.plot(range(1, len(row['input_ids']) + 1), row['input_ids'])
    plt.title(f"Sample {i+1} - Label: {row['label']} - Predict: {row['predict']}")
    plt.xlabel('Time Point')
    plt.ylabel('Blood Glucose Value')
    plt.savefig(f"/share/home/liangzhongming/930/CGMformer/sample/old288_sz1_5fold4_V2/sample_{i+1}_label_{row['label']}_predict_{row['predict']}.png")
```

--------------------------------

### Load and Query Dataset

Source: https://github.com/yurunlu/cgmformer/blob/main/processing_811_data.ipynb

Loads a saved dataset from disk and extracts sequences based on a specific index.

```python
from datasets import load_from_disk
file = "/share/home/liangzhongming/930/CGMformer/data/8_11_data/Shanghai_total"
data = load_from_disk(file)

index_to_extract = 1440
index_list = [i for i, idx in enumerate(data['index']) if idx == index_to_extract]
# if index_list:
#     origined_sequence = [data['input_ids'][i] for i in index_list]
index_list
origined_sequence = [data[i] for i in index_list]
origined_sequence
```

--------------------------------

### Execution Output Logs

Source: https://github.com/yurunlu/cgmformer/blob/main/sub_sample.ipynb

Console output showing warnings and statistical test results.

```text
/share/home/liangzhongming/anaconda3/envs/Geneformer/lib/python3.8/site-packages/scipy/signal/_spectral_py.py:2014: UserWarning: nperseg = 256 is greater than input length  = 96, using nperseg = 96
  warnings.warn('nperseg = {0:d} is greater than input length '
```

```text
KS test result for means: KstestResult(statistic=0.003214638971315529, pvalue=1.0, statistic_location=103.33749999999999, statistic_sign=1)
KS test result for vars: KstestResult(statistic=0.004698318496538081, pvalue=1.0, statistic_location=450.92937500000005, statistic_sign=1)
KS test result for ACFs: KstestResult(statistic=0.27097515564089136, pvalue=0.0, statistic_location=0.1796633494112581, statistic_sign=-1)
KS test result for PSDs: KstestResult(statistic=0.31095972306526637, pvalue=0.0, statistic_location=17.56655374127823, statistic_sign=1)
```

--------------------------------

### Perform Uniform Fixed Sampling

Source: https://github.com/yurunlu/cgmformer/blob/main/sub_sample.ipynb

Splits input sequences into fixed-size sections and creates a new dataset based on these samples.

```python
def uniform_fixed_sampling(data, window_size, num_sections):
    # Split data into sections
    sections = np.array_split(data, num_sections)

    # Initialize list to hold samples
    samples = []

    # For each position within a section
    for pos in range(window_size):
        # Sample the same position from each section
        sample = [section[pos] for section in sections]
        samples.append(sample)

    return samples

sample_path = "/share/home/liangzhongming/930/CGMformer/data/Shanghai_finetune"
save_path = "/share/home/liangzhongming/930/CGMformer/data/Shanghai_finetune_subSampleV3_2cls"

train_dataset=load_from_disk(sample_path)
# Determine the number of sections based on the window size
original_length = len(train_dataset['input_ids'][0])  # assuming all sequences have the same length
window_size = 3
num_sections = original_length // window_size

# Initialize lists to hold the sampled data
sampled_input_ids = []
sampled_types = []      

# Loop over the original data
cnt = 0
for input_ids, type_ in zip(train_dataset['input_ids'], train_dataset['types']):

    # Sample the input_ids sequence
    sampled_ids = uniform_fixed_sampling(input_ids, window_size, num_sections)
    
    # Repeat the type_ for the number of samples
    sampled_type_ = [type_] * len(sampled_ids)
    
    # Append to the lists
    sampled_input_ids.extend(sampled_ids)
    sampled_types.extend(sampled_type_)

# Create a new dataset from the sampled data
sampled_dataset = Dataset.from_dict({
    'input_ids': sampled_input_ids,
    'types': sampled_types
})

# Save the new dataset
sampled_dataset.save_to_disk(save_path)
```

--------------------------------

### Extract Sample Embeddings

Source: https://github.com/yurunlu/cgmformer/blob/main/README.md

Generate embeddings from the model without performing fine-tuning.

```bash
python run_clustering.py --checkpoint_path /path/to/checkpoint --data_path /path/to/data --save_path /path/to/save
```

--------------------------------

### Split and Save Datasets for Cross-Validation

Source: https://github.com/yurunlu/cgmformer/blob/main/processing_811_data.ipynb

Iterates through cross-validation splits, converts data to Hugging Face Dataset objects, and saves them to disk.

```python
from collections import defaultdict
import os

cross_validation_splits = defaultdict(lambda: {'train': defaultdict(list), 'test': defaultdict(list)})

for cv_col in splits_df.columns:
    for idx, is_train in enumerate(splits_df[cv_col]):
        target = 'train' if is_train else 'test'
        
        for key, values in new_data.items():
            cross_validation_splits[cv_col][target][key].append(values[idx])

output_dir = "/share/home/liangzhongming/930/CGMformer/data/8_11_data/downstream/Zhao/"

cross_validation_datasets = {}
for cv, splits in cross_validation_splits.items():
    cross_validation_datasets[cv] = {
        'train': Dataset.from_dict(splits['train']),
        'test': Dataset.from_dict(splits['test']),
    }
    train_dataset = Dataset.from_dict(splits['train'])
    test_dataset = Dataset.from_dict(splits['test'])

    train_path = os.path.join(output_dir, cv, "train")
    test_path = os.path.join(output_dir, cv, "test")

    train_dataset.save_to_disk(train_path)
    test_dataset.save_to_disk(test_path)
```

--------------------------------

### Rank and save dataset

Source: https://github.com/yurunlu/cgmformer/blob/main/rank.ipynb

Sorts dataset input_ids in descending order and saves the resulting dataset to disk.

```python
data_path = "/share/home/liangzhongming/930/CGMformer/data/8_2_newData/Shanghai_non_empty_200finetune"
dataset = load_from_disk(data_path)

dataset

sorted_indices = np.argsort(dataset['input_ids'], kind='stable')
sorted_indices = sorted_indices[::-1]

sorted_indices.shape # (1581, 288)

rank_dataset = {
    'input_ids': sorted_indices.tolist(),
    'types': dataset['types'],
    'age': dataset['age'],
    'bmi': dataset['bmi'],
    'hba1c': dataset['hba1c'],
    'homa-b': dataset['homa-b'],
    'homa-is': dataset['homa-is'],
}

train_dataset = Dataset.from_dict(rank_dataset)
train_dataset.save_to_disk("/share/home/liangzhongming/930/CGMformer/data/8_2_newData/Shanghai_non_empty_200finetune_rank")
```

--------------------------------

### Analyze High Proportion Hypoglycemic Samples

Source: https://github.com/yurunlu/cgmformer/blob/main/mask_strategy.ipynb

Extracts samples where the proportion of low sugar tokens exceeds 40% and calculates basic statistics.

```python
high_proportion_samples = [sample for sample, proportion in zip(samples_with_low_values, low_sugar_token_counts) if proportion >= 0.4]
len(high_proportion_samples) # 26 for 0.4; 101 for 0.2; 3 for 0.8
blood_sugar_values = [sample['input_ids'] for sample in high_proportion_samples]
mean_blood_sugar = np.mean(blood_sugar_values)
std_blood_sugar = np.std(blood_sugar_values)
mean_blood_sugar, std_blood_sugar
high_proportion_samples[13]
```

--------------------------------

### Run Diagnosis Classification

Source: https://github.com/yurunlu/cgmformer/blob/main/README.md

Perform classification tasks using the trained model checkpoint.

```bash
python run_labels_classify.py --checkpoint_path /path/to/checkpoint --train_path /path/to/train_data --test_path /path/to/test_data --output_path /path/to/save
```

--------------------------------

### Sample dataset using StratifiedShuffleSplit

Source: https://github.com/yurunlu/cgmformer/blob/main/rank.ipynb

Uses StratifiedShuffleSplit to extract a balanced subset of data from a Hugging Face dataset.

```python
from sklearn.model_selection import StratifiedShuffleSplit
from collections import Counter
from sklearn.utils import shuffle
import pandas as pd
import numpy as np

data_path = "/share/home/liangzhongming/930/CGMformer/data/8_2_newData/Shanghai_train_1981"
dataset = load_from_disk(data_path)


labels = dataset['types']
label_count = Counter(labels)

X = np.array(dataset['input_ids'])
y = np.array(dataset['types'])

samples = [2000, 1000, 500, 100]

sss = StratifiedShuffleSplit(n_splits=1, test_size=None, random_state=0)

sampled_input_ids = []
sampled_types = []     
for train_index, _ in sss.split(X, y):
    sampled_input_ids = X[train_index[:2000]].tolist()
    sampled_types = y[train_index[:2000]].tolist()
    # sub_dataset = {'input_ids': X[train_index[:2000]].tolist(), 'types': y[train_index[:2000]].tolist()}
    # sub_datasets.append(sub_dataset)

len(sampled_input_ids) # 2000
len(sampled_input_ids[0]) # 288
```

--------------------------------

### Sliding Window Dataset Subsampling

Source: https://github.com/yurunlu/cgmformer/blob/main/sub_sample.ipynb

Implements a sliding window function to generate multiple samples from sequences and saves the resulting dataset to disk.

```python
from datasets import Dataset

def sliding_window_sampling(data, window_size, step_size):
    samples = []
    for i in range(0, len(data) - window_size + 1, step_size):
        sample = data[i:i + window_size]
        samples.append(sample)
    return samples

# Determine step size based on the number of required samples
original_length = len(train_dataset['input_ids'][0])  # assuming all sequences have the same length
num_samples = 10
window_size = 96
step_size = (original_length - window_size) // (num_samples - 1)

# Initialize lists to hold the sampled data
sampled_input_ids = []
sampled_types = []

# Loop over the original data
for input_ids, type_ in zip(train_dataset['input_ids'], train_dataset['types']):
    # Sample the input_ids sequence
    sampled_ids = sliding_window_sampling(input_ids, window_size, step_size)
    
    # Repeat the type_ for the number of samples
    sampled_type_ = [type_] * len(sampled_ids)
    
    # Append to the lists
    sampled_input_ids.extend(sampled_ids)
    sampled_types.extend(sampled_type_)

# Create a new dataset from the sampled data
sampled_dataset = Dataset.from_dict({
    'input_ids': sampled_input_ids,
    'types': sampled_types
})

# Save the new dataset
sampled_dataset.save_to_disk("/share/home/liangzhongming/930/CGMformer/data/Shanghai_train_subSample")
```

--------------------------------

### Compare Dataset Statistics

Source: https://github.com/yurunlu/cgmformer/blob/main/analize_seq.ipynb

Calculates statistical features for two datasets and performs a Kolmogorov-Smirnov test to compare their distributions.

```python
import numpy as np
from scipy.stats import ks_2samp
from statsmodels.tsa.stattools import acf, adfuller
from scipy.signal import welch
from scipy.stats import chisquare


# Calculate statistics for original data
original_means = [np.mean(seq) for seq in dataset1['input_ids']]
original_vars = [np.var(seq) for seq in dataset1['input_ids']]
original_acfs = [acf(seq, nlags=20) for seq in dataset1['input_ids']]
original_p_values = [adfuller(seq)[1] for seq in dataset1['input_ids']]
original_psds = [welch(seq)[1] for seq in dataset1['input_ids']]  # select PSDs

# Calculate statistics for sampled data
sampled_means = [np.mean(seq) for seq in dataset2['input_ids']]
sampled_vars = [np.var(seq) for seq in dataset2['input_ids']]
sampled_acfs = [acf(seq, nlags=20) for seq in dataset2['input_ids']]
sampled_p_values = [adfuller(seq)[1] for seq in dataset2['input_ids']]
sampled_psds = [welch(seq)[1] for seq in dataset2['input_ids']]  # select PSDs

# Compare distributions of means, vars, acfs and psds using Kolmogorov-Smirnov test
ks_result_means = ks_2samp(original_means, sampled_means)
ks_result_vars = ks_2samp(original_vars, sampled_vars)
ks_result_acfs = ks_2samp(np.concatenate(original_acfs), np.concatenate(sampled_acfs))
ks_result_psds = ks_2samp(np.concatenate(original_psds), np.concatenate(sampled_psds))

# Print KS test results
print("KS test result for means:", ks_result_means)
print("KS test result for vars:", ks_result_vars)
print("KS test result for ACFs:", ks_result_acfs)
print("KS test result for PSDs:", ks_result_psds)
```
Tokens:
Raw