EyeFormer

https://github.com/yuejiang-nj/eyeformer-uist2024
Admin
EyeFormer is a deep learning model that predicts human eye scanpaths during free-viewing tasks using...
Tokens:8,588
Snippets:48
Trust Score:8.1
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install ruamel.yaml 0.17.21

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Installs a specific version of ruamel.yaml required for the demo. The output shows successful installation of the package.

```python
!pip install -qq ruamel.yaml==0.17.21
```

```text
Output:
[2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m109.5/109.5 kB[0m [31m3.9 MB/s[0m eta [36m0:00:00[0m
[2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m526.7/526.7 kB[0m [31m13.3 MB/s[0m eta [36m0:00:00[0m
[?25h
```

--------------------------------

### Install PyTorch and Other Dependencies

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Installs PyTorch 1.7.1, torchvision, torchaudio, CUDA 11.0, OpenCV, and other required packages for the demo.

```python
!pip install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit==11.0
!pip install opencv-python==4.5.3.56 Pillow einops multimatch-gaze
!pip install transformers==4.8.1
```

--------------------------------

### Pip install output with errors

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Output from a pip install command showing errors: pytorch==1.7.1 not found, opencv-python build failure, and tokenizers wheel build failure. This indicates dependency issues in the environment.

```text
ERROR: Could not find a version that satisfies the requirement pytorch==1.7.1 (from versions: 0.1.2, 1.0.2)
ERROR: No matching distribution found for pytorch==1.7.1
Collecting opencv-python==4.5.3.56
  Downloading opencv-python-4.5.3.56.tar.gz (89.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.2/89.2 MB 7.7 MB/s eta 0:00:00
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Collecting transformers==4.8.1
  Downloading transformers-4.8.1-py3-none-any.whl.metadata (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.8/48.8 kB 2.6 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.8.1) (3.15.4)
Collecting huggingface-hub==0.0.12 (from transformers==4.8.1)
  Downloading huggingface_hub-0.0.12-py3-none-any.whl.metadata (5.6 kB)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.8.1) (1.26.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from transformers==4.8.1) (24.1)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from transformers==4.8.1) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.8.1) (2024.5.15)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.8.1) (2.31.0)
Collecting sacremoses (from transformers==4.8.1)
  Downloading sacremoses-0.1.1-py3-none-any.whl.metadata (8.3 kB)
Collecting tokenizers<0.11,>=0.10.1 (from transformers==4.8.1)
  Downloading tokenizers-0.10.3.tar.gz (212 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 212.7/212.7 kB 10.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.8.1) (4.66.4)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from huggingface-hub==0.0.12->transformers==4.8.1) (4.12.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.8.1) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.8.1) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.8.1) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.8.1) (2024.7.4)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from sacremoses->transformers==4.8.1) (8.1.7)
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from sacremoses->transformers==4.8.1) (1.4.2)
Downloading transformers-4.8.1-py3-none-any.whl (2.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 43.7 MB/s eta 0:00:00
Downloading huggingface_hub-0.0.12-py3-none-any.whl (37 kB)
Downloading sacremoses-0.1.1-py3-none-any.whl (897 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 897.5/897.5 kB 38.0 MB/s eta 0:00:00
Building wheels for collected packages: tokenizers
  error: subprocess-exited-with-error
  
  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)
```

--------------------------------

### Install timm 0.4.9

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Installs the timm library version 0.4.9 using pip. This is a common step in setting up the EyeFormer project environment.

```python
!pip install timm==0.4.9
```

--------------------------------

### Install dependencies with conda and pip

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/README.md

Commands to create a conda environment and install all required dependencies for EyeFormer. The environment uses Python 3.6.13, PyTorch 1.7.1, and specific versions of transformers, timm, and other packages. The source notes that a requirements.txt file is also provided but advises not to use it for configuration.

```sh
conda create -n your_env python=3.6.13
conda activate your_env
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch
pip install transformers==4.8.1
pip install timm==0.4.9
conda install ruamel_yaml
pip install opencv-python
pip install --upgrade Pillow
pip install einops
pip install multimatch-gaze
pip install multimatch
```

--------------------------------

### Load config, create output directory, and run main

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Instantiates ARGS, loads the YAML config, creates the output directory, dumps the config to the output, and calls main(). Requires the config file to exist.

```python
args = ARGS()

yaml = YAML(typ='rt')
config = yaml.load(open(args.config, 'rt'))

Path(args.output_dir).mkdir(parents=True, exist_ok=True)
yaml.dump(config, open(os.path.join(args.output_dir, 'config.yaml'), 'w'))
main(args, config)
```

--------------------------------

### Download pretrained weights with gdown

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Downloads the pretrained model weights from a Google Drive folder and moves them to the 'weights' directory. Requires the 'gdown' package.

```python
import gdown
url = 'https://drive.google.com/drive/folders/1eae63VRBfQ9XAYj3k2q2N9xDtLmSqQg_'
gdown.download_folder(url, quiet=True)
!mv scanpath_prediction_population_rl weights
```

--------------------------------

### Define main inference function for TrackingTransformer

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Sets up device, seeds, dataset, model, and checkpoint loading, then runs the test function. Distributed mode is commented out. Requires a checkpoint path and output directory.

```python
def main(args, config):
    # utils.init_distributed_mode(args)

    device = torch.device(args.device)

    # fix the seed for reproducibility
    seed = args.seed + utils.get_rank()
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed)
    cudnn.benchmark = True

    #### Dataset ####
    print("Creating dataset")
    datasets = [create_dataset('inference', config)]

    if False:
        num_tasks = utils.get_world_size()
        global_rank = utils.get_rank()
        samplers = create_sampler(datasets, [True], num_tasks, global_rank)
    else:
        samplers = [None]

    data_loader = create_loader(datasets,
                                samplers,
                                batch_size=[config['batch_size_test']],
                                num_workers=[32],
                                is_trains=[False],
                                collate_fns=[None])[0]

    # tokenizer = BertTokenizer.from_pretrained(args.text_encoder)
    tokenizer = None

    #### Model ####
    print("Creating model")
    model = TrackingTransformer(config=config, init_deit=False)

    model = model.to(device)

    if args.checkpoint:
        checkpoint = torch.load(args.checkpoint, map_location='cpu')
        state_dict = checkpoint['model']

        msg = model.load_state_dict(state_dict)
        print('load checkpoint from %s' % args.checkpoint)
        print(msg)

    model_without_ddp = model

    print("Start testing")
    start_time = time.time()

    test(model, data_loader, tokenizer, device, args.output_dir, config)

    # dist.barrier()

    total_time = time.time() - start_time
    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
    print('Testing time {}'.format(total_time_str))
```

--------------------------------

### Sample console output from inference

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Shows the expected console output when running the script without distributed mode: 'Not using distributed mode' and 'Creating dataset'.

```text
Output:
Not using distributed mode
Creating dataset

```

--------------------------------

### Define ARGS class with configuration paths and hyperparameters

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Holds default arguments for the inference script: config file path, checkpoint path, output directory, text encoder, device, seed, and distributed settings. Used to instantiate args for main().

```python
class ARGS:
    config = './configs/Tracking.yaml'
    checkpoint = './weights/checkpoint_19.pth'
    resume = False
    output_dir = 'output/tracking_eval'
    text_encoder = 'bert-base-uncased'
    device ='cuda'
    seed = 42
    world_size = 1
    dist_url = 'env://'
    distributed = True
```

--------------------------------

### Import ruamel.yaml for YAML loading

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Imports the YAML class from ruamel.yaml, used to load and dump configuration files.

```python
from ruamel.yaml import YAML
```

--------------------------------

### Run reinforcement learning stage

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/README.md

Run the reinforcement learning stage. Requires editing data paths in configs/Tracking.yaml before execution. The models are saved in output/scanpath_prediction_population_rl.

```sh
python -m torch.distributed.launch --nproc_per_node=1 --use_env --master_port=xxxx \
mse_tracking.py \
--output_dir output/scanpath_prediction_population_rl \
--checkpoint output/scanpath_prediction_population_pretrain/checkpoint_29.pth \
```

--------------------------------

### Change directory to EyeFormer-UIST2024

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Changes the current working directory to the cloned repository. The output confirms the new path is /content/EyeFormer-UIST2024.

```python
%cd EyeFormer-UIST2024
```

--------------------------------

### Run DTW, TDE, and Eyenalysis evaluation

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/README.md

Command to evaluate with DTW, TDE, and Eyenalysis metrics using the provided ground truth and prediction files.

```sh
python evaluation/eval_xy.py --scanpaths \
--ref_files evaluation/testing_ground_truth.csv \
--pred_files output/scanpath_prediction_population_rl/predicted_result.csv \
```

--------------------------------

### Import Python Libraries

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Imports standard and third-party libraries needed for the EyeFormer demo, including PyTorch, NumPy, and ruamel.yaml.

```python
import argparse
import os
import ruamel.yaml as yaml
import numpy as np
import random
import time
import datetime
import json
from pathlib import Path
import csv

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
import torch.backends.cudnn as cudnn
import torch.distributed as dist
```

--------------------------------

### Sample from Gaussian distribution in model_tracking.py

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/README.md

Replacement for model_tracking.py#L142 to sample from a Gaussian distribution. self.dense predicts the mean, self.dense2 predicts the log standard deviation, and a Normal distribution is constructed and sampled.

```python
pred = self.dense(coord_output.last_hidden_state[:, -1:])
logstd = self.dense2(coord_output.last_hidden_state[:, -1:])
pred_var = torch.exp(logstd)
pred_dist = torch.distributions.Normal(pred, pred_var)
pred = pred_dist.sample()
```

--------------------------------

### Run MultiMatch evaluation

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/README.md

Commands to evaluate with MultiMatch. First change to the evaluation/eval_multipath directory, then run eval_tsv.py with the prediction and ground truth files.

```sh
cd evaluation/eval_multipath

python eval_tsv.py \
--pred_file your_path/output/scanpath_prediction_population_rl/predicted_result.csv \
--gt_file your_path/evaluation/testing_ground_truth.csv \
```

--------------------------------

### Import required modules for inference

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Imports the necessary modules and classes for running the tracking model inference, including TrackingTransformer, interpolate_pos_embed, and dataset utilities.

```python
from models.model_tracking import TrackingTransformer
from models.vit import interpolate_pos_embed

import utils
from dataset import create_dataset, create_sampler, create_loader
import csv
```

--------------------------------

### Run DTWD evaluation

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/README.md

Command to evaluate with the DTWD metric using the provided ground truth and prediction files.

```sh
python evaluation/eval_xyt.py --scanpaths \
--ref_files evaluation/testing_ground_truth.csv \
--pred_files output/scanpath_prediction_population_rl/predicted_result.csv \
```

--------------------------------

### Predict scanpaths with trained model

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/README.md

Run prediction using the trained model. The predicted results are saved as predicted_result.csv in the output directory. To reproduce results, use the checkpoint from the 19th epoch.

```sh
python -m torch.distributed.launch --nproc_per_node=1 --use_env --master_port=xxxx \
eval_tracking.py \
--output_dir output/scanpath_prediction_population_rl \
--checkpoint output/scanpath_prediction_population_rl/checkpoint_18.pth \
```

--------------------------------

### Pretrain scanpath prediction model

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/README.md

Run the pretraining stage for scanpath prediction. Requires editing data paths in configs/Pretrain_tracking.yaml before execution. The model is saved in output/scanpath_prediction_population_pretrain.

```sh
python -m torch.distributed.launch --nproc_per_node=1 --use_env --master_port=xxxx \
mse_tracking_pretrain.py \
--output_dir output/scanpath_prediction_population_pretrain \
```

--------------------------------

### Clone EyeFormer-UIST2024 repository

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Clones the EyeFormer-UIST2024 GitHub repository. The output shows the repository is cloned successfully.

```python
!git clone https://github.com/YueJiang-nj/EyeFormer-UIST2024.git
```

--------------------------------

### FileNotFoundError traceback for missing image directory

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Error traceback showing a FileNotFoundError when the image directory does not exist. The code attempts to list image files from the path '/home/researcher/Documents/saliency_modeling_issues/UEyes_dataset/images'.

```text
Error (truncated):
[0;32m--> 321[0;31m         self.image_list = [os.path.join(image_root, r) for r in os.listdir(image_root)\
[0m[1;32m    322[0m                            if (os.path.isfile(os.path.join(image_root, r)) and (r.endswith(".png") or r.endswith(".jpg")))]
[1;32m    323[0m         [0mself[0m[0;34m.[0m[0mimage_names[0m [0;34m=[0m [0;34m[[0m[0mr[0m[0;34m.[0m[0msplit[0m[0;34m([0m[0;34m"/"[0m[0;34m)[0m[0;34m[[0m[0;34m-[0m[0;36m1[0m[0;34m][0m [0;32mfor[0m [0mr[0m [0;32min[0m [0mself[0m[0;34m.[0m[0mimage_list[0m[0;34m][0m[0;34m[0m[0;34m[0m[0m

[0;31mFileNotFoundError[0m: [Errno 2] No such file or directory: '/home/researcher/Documents/saliency_modeling_issues/UEyes_dataset/images'
```

--------------------------------

### Original mean prediction in model_tracking.py

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/README.md

Original prediction line in model_tracking.py that uses the mean of a Gaussian distribution as the prediction output.

```python
pred = self.dense(coord_output.last_hidden_state[:, -1:])
```

--------------------------------

### Define test function for scanpath prediction

Source: https://github.com/yuejiang-nj/eyeformer-uist2024/blob/main/colab_demo.ipynb

Defines a test function that runs inference on a data loader, collects predicted coordinates, and writes them to a CSV file named 'predicted_result.csv'. The CSV includes columns for image, width, height, x, y, and timestamp.

```python
def test(model, data_loader, tokenizer, device, output_dir, config):
    # train
    model.eval()

    metric_logger = utils.MetricLogger(delimiter="  ")
    header = 'Testing:'

    image_names = []
    results = []
    widths = []
    heights = []
    # user_ids = []
    for i, (image, image_name, width, height) in enumerate(metric_logger.log_every(data_loader, 1, header)):
        image = image.to(device, non_blocking=True)
        # user_id = user_id.to(device, non_blocking=True)

        coord = model.inference(image)
        coord = coord.cpu().numpy().tolist()

        width = width.numpy().tolist()
        height = height.numpy().tolist()

        # user_id = user_id.cpu().numpy().tolist()

        image_names.extend(image_name)
        results.extend(coord)
        widths.extend(width)
        heights.extend(height)
        # user_ids.extend(user_id)

    with open(os.path.join(output_dir, 'predicted_result.csv'), 'w') as wfile:
        writer = csv.writer(wfile)
        writer.writerow(["image", "width", "height", "x", "y", "timestamp"])

        for image, width, height, coord in zip(image_names, widths, heights, results):

            for row in coord:
                x = row[0] * width
                y = row[1] * height
                t = row[2]
                # username = data_loader.dataset.id2user[user_id]
                writer.writerow([image, width, height,
                                x, y, t])

    return
```
Tokens:
Raw