Pupil Core

https://docs.pupil-labs.com/core/
Admin
Official docs for users getting started with their Pupil Labs eye tracking glasses and for...
Tokens:21,656
Snippets:299
Trust Score:9.5
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://docs.pupil-labs.com/core/developer/network-api

Install necessary Python libraries for interacting with the Pupil Core Network API.

```sh
pip install zmq msgpack==0.5.6
```

--------------------------------

### Connect to Publisher Socket Directly

Source: https://docs.pupil-labs.com/core/developer/network-api

Example demonstrating how to connect to a publisher socket directly to write messages to the IPC Backbone.

```python
# continued from above...
# Assumes pupil_remote to be a connected REQ socket

import msgpack
from time import time, sleep

pupil_remote.send_string('PUB_PORT')
pub_port = pupil_remote.recv_string()

publisher = ctx.socket(zmq.PUB)
publisher.connect(f'tcp://{ip}:{pub_port}')

# Wait for the connection to be established...

```

--------------------------------

### Start and Stop Calibration Shortcut

Source: https://docs.pupil-labs.com/core/software/pupil-capture

Press 'c' to start and stop the calibration process in Pupil Capture.

```text
c| Start and stop calibration
```

--------------------------------

### Start and Stop Recording Shortcut

Source: https://docs.pupil-labs.com/core/software/pupil-capture

Press 'r' to start and stop recording within Pupil Capture.

```text
r| Start and stop recording
```

--------------------------------

### Start Calibration Shortcut

Source: https://docs.pupil-labs.com/core/getting-started

Press 'c' on the keyboard or the world screen in Pupil Capture to initiate the screen marker calibration process.

```text
c
```

--------------------------------

### Install USB Drivers on Windows

Source: https://docs.pupil-labs.com/core/software/pupil-capture

This command sequence helps in reinstalling Pupil Cam drivers on Windows by uninstalling existing ones and running Pupil Capture as administrator.

```bash
sudo usermod -a -G plugdev $USER
```

--------------------------------

### Custom Data Example in recent_events

Source: https://docs.pupil-labs.com/core/developer/plugin-api

Example of how to add custom data to the events dictionary within the `recent_events` callback. Custom data should be serializable with msgpack.

```python
from plugin import Plugin

CUSTOM_TOPIC = "custom_topic"


class CustomDataExample(Plugin):
    def recent_events(self, events):
        custom_datum = {
            "topic": CUSTOM_TOPIC,
            "timestamp": self.g_pool.get_timestamp(),  # Timestamp in pupil time
            "custom field": 42,
            # Further fields can be added here.
            # Their values should be serializable with msgpack.
        }
        events[CUSTOM_TOPIC] = [custom_datum]
```

--------------------------------

### Start and Stop Validation Shortcut

Source: https://docs.pupil-labs.com/core/software/pupil-capture

Press 't' to start and stop the validation process in Pupil Capture.

```text
t| Start and stop validation
```

--------------------------------

### Send Notification via Pupil Remote

Source: https://docs.pupil-labs.com/core/developer/network-api

Example script demonstrating how to send notifications to the IPC Backbone using Pupil Remote for reliable transport.

```python
# continued from above
import msgpack
notification = {'subject':'recording.should_start', 'session_name':'my session'}
topic = 'notify.' + notification['subject']
payload = msgpack.dumps(notification)
pupil_remote.send_string(topic, flags=zmq.SNDMORE)
pupil_remote.send(payload)
print(pupil_remote.recv_string())
```

--------------------------------

### Surface Datum Format Example

Source: https://docs.pupil-labs.com/core/developer

This JSON object represents the structure of surface data. It includes surface transformation matrices and lists of gaze and fixation points mapped to the surface. Note that 'gaze_on_surfaces' and 'fixations_on_surfaces' are lists that can contain multiple entries, indicated by '...' in the example.

```json
{
    "topic": "surfaces.surface_name",
    "name": "surface_name",
    "surf_to_img_trans": (
        (-394.2704714040225, 62.996680859974035, 833.0782341017057),
        (24.939461954010476, 264.1698344383364, 171.09768247735033),
        (-0.0031580300961504023, 0.07378146751738948, 1.0),
    ),
    "img_to_surf_trans": (
        (-0.002552357406770253, 1.5534025217146223e-05, 2.1236555655143734),
        (0.00025853538051076233, 0.003973842600569134, -0.8952954577358644),
        (-2.71355412859636e-05, -0.00029314688183396006, 1.0727627809231568),
    ),
    "gaze_on_surfaces": (
        {
            "topic": "gaze.3d.1._on_surface",
            "norm_pos": (-0.6709809899330139, 0.41052111983299255),
            "confidence": 0.5594810076623645,
            "on_surf": False,
            "base_data": ("gaze.3d.1.", 714040.132285),
            "timestamp": 714040.132285,
        },
        ...
    ),
    "fixations_on_surfaces": (
        {
            "topic": "fixations_on_surface",
            "norm_pos": (-0.9006409049034119, 0.7738968133926392),
            "confidence": 0.8663407531808505,
            "on_surf": False,
            "base_data": ("fixations", 714039.771958),
            "timestamp": 714039.771958,
            "id": 27,
            "duration": 306.62299995310605,  # in milliseconds
            "dispersion": 1.4730711610581475,  # in degrees
        },
        ...
    ),
    "timestamp": 714040.103912,
}
```

--------------------------------

### IPC Backbone Notification Message Example

Source: https://docs.pupil-labs.com/core/developer/network-api

Illustrates the structure of a notification message, including its topic and payload, for coordinating activities within Pupil apps.

```python
# message topic:
'notify.recording.should_start'
# message payload, a notification dict
{'subject':'recording.should_start', 'session_name':'my session'}
```

--------------------------------

### Default Recordings Directory Structure

Source: https://docs.pupil-labs.com/core/getting-started

Example of how Pupil Core organizes recordings by date and recording index. Each recording is stored in a unique folder.

```text
recordings/
  2019-09-30/
    000/
    001/
    002/
```

--------------------------------

### Listen to Notifications from Pupil Service

Source: https://docs.pupil-labs.com/core/developer/network-api

This snippet demonstrates how to subscribe to and receive all notifications from Pupil Service. It requires a helper script `zmq_tools.py` and a ZMQ REQ socket to get the SUB_PORT.

```python
from zmq_tools import *

ctx = zmq.Context()
requester = ctx.socket(zmq.REQ)
requester.connect('tcp://localhost:50020') #change ip if using remote machine

# request 'SUB_PORT' for reading data
requester.send_string('SUB_PORT')
ipc_sub_port = requester.recv_string()

monitor = Msg_Receiver(ctx,'tcp://localhost:%s'%ipc_sub_port,topics=('notify.',)) #change ip if using remote machine

while True:
    print(monitor.recv())
```

--------------------------------

### Start/Stop Recording Shortcut

Source: https://docs.pupil-labs.com/core/getting-started

Press the 'r' key or the 'R' button in Pupil Capture to start or stop data recording. Recording data is saved in the 'recordings' folder.

```text
r
```

--------------------------------

### Publish Notifications via IPC Backbone

Source: https://docs.pupil-labs.com/core/developer/network-api

Example of publishing a notification to the IPC Backbone. This uses a ZMQ PUB socket and ensures messages are sent with the `SNDMORE` flag for multi-part messages.

```python
import zmq

# Assuming publisher is already set up and connected
# publisher = ctx.socket(zmq.PUB)
# publisher.connect('tcp://...') 

notification = {'subject':'calibration.should_start'}
topic = 'notify.' + notification['subject']
payload = serializer.dumps(notification) # Assuming serializer is defined elsewhere

publisher.send_string(topic, flags=zmq.SNDMORE)
publisher.send(payload)
```

--------------------------------

### Convert Pupil Time to System Time using Python

Source: https://docs.pupil-labs.com/core/developer

This Python script demonstrates how to convert Pupil Time timestamps to System Time using the recording start times. It calculates the offset and applies it to convert a given Pupil timestamp.

```python
import datetime

start_time_system = 1533197768.2805  # System Time at recording start
start_time_synced = 674439.5502      # Pupil Time at recording start

# Calculate the fixed offset between System and Pupil Time
offset = start_time_system - start_time_synced

# Choose a Pupil timestamp that you want to convert to System Time
# (this can be any or all timestamps of interest)
pupil_timestamp = 674439.4695  # This is a random example of a Pupil timestamp

# Add the fixed offset to the timestamp(s) we wish to convert
pupiltime_in_systemtime = pupil_timestamp + offset

# Using the datetime python module, we can convert timestamps 
# stored as seconds represented by floating point values to a 
# more readable datetime format.
pupil_datetime = datetime.datetime.fromtimestamp(pupiltime_in_systemtime).strftime("%Y-%m-%d %H:%M:%S.%f")

print(pupil_datetime)
# example output: '2018-08-02 15:16:08.199800'

# Hint: you can also copy and paste timestamps into various websites that convert them
# to the readable date time format!
```

--------------------------------

### Custom Pupil Detector Plugin Implementation

Source: https://docs.pupil-labs.com/core/developer/plugin-api

Example of a custom pupil detection plugin subclassing PupilDetectorPlugin. It includes a method to disable other pupil detectors and placeholders for the required pupil_detector and detect methods.

```python
from pupil_detector_plugins import PupilDetectorPlugin

class MyCustomPupilDetectorPlugin(PupilDetectorPlugin):

    def __init__(self, g_pool):
        super().__init__(g_pool)
        # In some cases, it might be desirable to disable other pupil detectors running in the eye process
        # e.g. to increase performance on systems with limited computing resources.
        # In such cases, these pupil detection plugins can be disabled with the helper method:
        self._stop_other_pupil_detectors()

    def _stop_other_pupil_detectors(self):
        # Getting the Plugin_List instance from the g_pool object
        plugin_list = self.g_pool.plugins
        # Iterating over every plugin in the Plugin_List instance
        for plugin in plugin_list:
            # Deactivating every PupilDetectorPlugin instances except self
            if isinstance(plugin, PupilDetectorPlugin) and plugin is not self:
                plugin.alive = False
        # Forcing Plugin_List instance to remove deactivated plugins
        plugin_list.clean()

    @property
    def pupil_detector(self):
        # This read-only property must be implemented by the custom subclass.
        #
        # Returns an instance of pupil detector;
        # See: https://github.com/pupil-labs/pupil-detectors
        pass

    def detect(self, frame, **kwargs):
        # This method must be implemented by the custom subclass.
        #
        # Returns a pupil datum dictionary containing the pupil location and related metadata.
        # See Pupil Datum Format for a list of required keys:
        # https://docs.pupil-labs.com/core/developer/#pupil-datum-format
        pass
```

--------------------------------

### Send Notifications to Pupil Service

Source: https://docs.pupil-labs.com/core/developer/network-api

Use this snippet to send various notifications to Pupil Service, such as starting eye processes, setting calibration methods, or stopping the service. Requires a ZMQ REQ socket connected to Pupil Service.

```python
import zmq, msgpack, time

# create a zmq REQ socket to talk to Pupil Service
ctx = zmq.Context()
pupil_remote = ctx.socket(zmq.REQ)
pupil_remote.connect('tcp://localhost:50020')

# convenience function
def send_recv_notification(n):
    pupil_remote.send_string(f"notify.{n['subject']}", flags=zmq.SNDMORE)
    pupil_remote.send(msgpack.dumps(n))
    return pupil_remote.recv_string()

# set start eye windows
n = {'subject':'eye_process.should_start.0','eye_id': 0, 'args':{}}
print(send_recv_notification(n))
n = {'subject':'eye_process.should_start.1','eye_id':1, 'args':{}}
print(send_recv_notification(n))
time.sleep(2)

# set calibration method to hmd calibration
n = {'subject':'start_plugin','name':'HMD_Calibration', 'args':{}}
print(send_recv_notification(n))
time.sleep(2)

# close Pupil Service
n = {'subject':'service_process.should_stop'}
print(send_recv_notification(n))
```

--------------------------------

### Initialize Plugin with Session Settings

Source: https://docs.pupil-labs.com/core/developer/plugin-api

Demonstrates how to initialize a plugin with custom session settings and store them. The `my_custom_setting` is initialized with a default value of 5 and can be overridden by saved session data.

```python
from plugin import Plugin

class MyCustomPlugin(Plugin):
    def __init__(self, g_pool, my_custom_setting=5):
        self._my_custom_setting = my_custom_setting
    
    def get_init_dict(self):
        return {"my_custom_setting": self._my_custom_setting}
        # return {}  # on next launch, recover plugin with default settings
        # raise NotImplementedError  # on next launch, do not recover plugin
```

--------------------------------

### Construct Notification Topic

Source: https://docs.pupil-labs.com/core/developer/network-api

Shows how to dynamically construct the notification topic string based on a notification dictionary.

```python
topic = f"notify.{notification['subject']}"
```

--------------------------------

### Camera Intrinsic Estimation Snapshot Shortcut

Source: https://docs.pupil-labs.com/core/software/pupil-capture

Press 'i' to take a snapshot of the circle pattern for camera intrinsic estimation.

```text
i| Camera intrinsic estimation: Take snapshot of circle pattern
```

--------------------------------

### Add New Surface Shortcut

Source: https://docs.pupil-labs.com/core/software/pupil-capture

Press 'a' to add a new surface for the surface tracker in Pupil Capture.

```text
a| Surface tracker: Add new surface
```

--------------------------------

### Basic Plugin Structure

Source: https://docs.pupil-labs.com/core/developer/plugin-api

This is the minimal structure for a custom plugin. Inherit from the `Plugin` class to create a new plugin.

```python
from plugin import Plugin

class MyCustomPlugin(Plugin):
    pass
```

--------------------------------

### Export Data Shortcut

Source: https://docs.pupil-labs.com/core/getting-started

Press 'e' on the keyboard or use the down arrow button in Pupil Player to export recorded videos and data.

```text
e
```

--------------------------------

### Send Custom Topic Message via IPC

Source: https://docs.pupil-labs.com/core/developer/network-api

Demonstrates sending a custom-topic message using a PUB socket, including msgpack serialization for the payload.

```python
import zmq
import msgpack

topic = 'your_custom_topic'
payload = {'topic': topic}

# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)

# send payload using custom topic
pub_socket.send_string(topic, flags=zmq.SNDMORE)
pub_socket.send(msgpack.dumps(payload, use_bin_type=True))
```

--------------------------------

### Annotation Hotkey Definitions Format

Source: https://docs.pupil-labs.com/core/software/pupil-player

Stores annotation hotkey definitions for Pupil Player. This JSON structure defines labels and their corresponding hotkeys.

```json
{
    "version": 1,
    "definitions": {
        "<label>": "<hotkey>"
    }
}
```

--------------------------------

### Pupil Remote Commands Overview

Source: https://docs.pupil-labs.com/core/developer/network-api

Reference for available Pupil Remote commands for controlling Pupil Capture and accessing IPC Backbone information.

```python
'R'  # start recording with auto generated session name
'R rec_name'  # start recording named "rec_name"
'r'  # stop recording
'C'  # start currently selected calibration
'c'  # stop currently selected calibration
'T 1234.56'  # resets current Pupil time to given timestamp
't'  # get current Pupil time; returns a float as string.
'v'  # get the Pupil Core software version string

# IPC Backbone communication
'PUB_PORT'  # return the current pub port of the IPC Backbone
'SUB_PORT'  # return the current sub port of the IPC Backbone
```

--------------------------------

### Run Pupil Service with Administrator Privileges on macOS

Source: https://docs.pupil-labs.com/core/software/pupil-capture

Execute this command in the terminal to run Pupil Service with administrator privileges on macOS, necessary for camera access on macOS 12 Monterey and newer.

```zsh
sudo /Applications/Pupil\ Service.app/Contents/MacOS/pupil_service
```

--------------------------------

### Remote Annotation Message Format

Source: https://docs.pupil-labs.com/core/developer/network-api

Format for creating annotation events programmatically and sending them via IPC or Pupil Remote interface.

```python
{
    'topic': "annotation",
    'label': "Hi this is my annotation 1",
    'timestamp': <pupil time>,
    'duration': 1.0,
}
```

--------------------------------

### Connect to Pupil Remote

Source: https://docs.pupil-labs.com/core/developer/network-api

Establish a connection to Pupil Remote using pyzmq's REQ socket.

```python
import zmq

ctx = zmq.Context()
pupil_remote = zmq.Socket(ctx, zmq.REQ)
pupil_remote.connect('tcp://127.0.0.1:50020')
```

--------------------------------

### Request IPC Backbone Ports

Source: https://docs.pupil-labs.com/core/developer/network-api

Obtain the SUB_PORT and PUB_PORT for the IPC Backbone by communicating with Pupil Remote. This is necessary for real-time data access.

```python
import zmq
ctx = zmq.Context()
# The REQ talks to Pupil remote and receives the session unique IPC SUB PORT
pupil_remote = ctx.socket(zmq.REQ)

ip = 'localhost'  # If you talk to a different machine use its IP.
port = 50020  # The port defaults to 50020. Set in Pupil Capture GUI.

pupil_remote.connect(f'tcp://{ip}:{port}')

# Request 'SUB_PORT' for reading data
pupil_remote.send_string('SUB_PORT')
sub_port = pupil_remote.recv_string()

# Request 'PUB_PORT' for writing data
pupil_remote.send_string('PUB_PORT')
pub_port = pupil_remote.recv_string()
```

--------------------------------

### Check XDG Session Type on Linux

Source: https://docs.pupil-labs.com/core/software/pupil-capture

Run this command in the terminal to check if your Linux session is using Wayland or Xorg. If it's Wayland, switch to Xserver for compatibility.

```bash
echo $XDG_SESSION_TYPE
```

--------------------------------

### Transcode Video for Player Compatibility

Source: https://docs.pupil-labs.com/core/developer/recording-format

Use this ffmpeg command to transcode video streams from MJPEG to H264 and change color format from yuvj422p to yuv420p for compatibility with most video players.

```bash
ffmpeg -i world.mp4 -vf format=yuv420p world_compatible.mp4
```

--------------------------------

### Run Pupil Capture with Administrator Privileges on macOS

Source: https://docs.pupil-labs.com/core/software/pupil-capture

Execute this command in the terminal to run Pupil Capture with administrator privileges on macOS, necessary for camera access on macOS 12 Monterey and newer.

```zsh
sudo /Applications/Pupil\ Capture.app/Contents/MacOS/pupil_capture
```

--------------------------------

### Pupil Core Recording File Structure

Source: https://docs.pupil-labs.com/core/software/recording-format

Lists the required and optional files generated during a Pupil Core recording session. The specific files present depend on the headset and enabled features.

```text
# Meta File (required)
info.player.json

# Notifications (required)
notify_timestamps.npy
notify.pldata

# World Video Files (optional in VR/AR recordings)
world_timestamps.npy
world.intrinsics
world.mp4

# Eye Video Files (only required for Offline Pupil Detection)
eye0_timestamps.npy
eye0.mp4

eye1_timestamps.npy
eye1.mp4

# Pupil Data (optional)
pupil_timestamps.npy
pupil.pldata

# Gaze Data (optional)
gaze_timestamps.npy
gaze.pldata

# Annotations (optional)
annotation_timestamps.npy
annotation.pldata

# Audio Capture (optional)
audio_timestamps.npy
audio.mp4

## Blink Detector (optional)
blinks_timestamps.npy
blinks.pldata

# Fixation Detector (optional)
fixations_timestamps.npy
fixations.pldata

# Surface Tracker (optional)
surface_definitions
surface_definitions_v01

# User Info (optional)
user_info.csv
```

--------------------------------

### Subscribe to IPC Backbone Data

Source: https://docs.pupil-labs.com/core/developer/network-api

Connects to the IPC Backbone as a subscriber, subscribes to gaze messages, and continuously receives and prints deserialized data.

```python
#...continued from above
# Assumes `sub_port` to be set to the current subscription port
subscriber = ctx.socket(zmq.SUB)
subscriber.connect(f'tcp://{ip}:{sub_port}')
subscriber.subscribe('gaze.')  # receive all gaze messages

# we need a serializer
import msgpack

while True:
    topic, payload = subscriber.recv_multipart()
    message = msgpack.loads(payload)
    print(f"{topic}: {message}")
```

--------------------------------

### Add Annotation Shortcut

Source: https://docs.pupil-labs.com/core/software/pupil-capture

Press 'x' to add an annotation in Pupil Capture. This is the default keyboard shortcut.

```text
x| Add annotation (default keyboard shortcut)
```

--------------------------------

### BibTeX Citation for UbiComp 2014 Paper

Source: https://docs.pupil-labs.com/core/academic-citation

Use this BibTeX entry to cite the UbiComp 2014 paper when referencing Pupil Core in your academic research.

```bibtex
@inproceedings{Kassner:2014:POS:2638728.2641695,
 author = {Kassner, Moritz and Patera, William and Bulling, Andreas},
 title = {Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction},
 booktitle = {Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing},
 series = {UbiComp '14 Adjunct},
 year = {2014},
 isbn = {978-1-4503-3047-3},
 location = {Seattle, Washington},
 pages = {1151--1160},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2638728.2641695},
 doi = {10.1145/2638728.2641695},
 acmid = {2641695},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {eye movement, gaze-based interaction, mobile eye tracking, wearable computing},
}

```

--------------------------------

### BibTeX Citation for Pupil Technical Report

Source: https://docs.pupil-labs.com/core/academic-citation

Use this BibTeX entry to cite the Pupil Technical Report when referencing the Pupil platform in your academic work.

```bibtex
@article{KassnerPateraBulling:2014,
  author={Kassner, Moritz and Patera, William and Bulling, Andreas},
  title={Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction},
  keywords={Eye Movement, Mobile Eye Tracking, Wearable Computing, Gaze-based Interaction},
  year={2014},
  month={April},
  archivePrefix = "arXiv",
  eprint        = "1405.0006",
  primaryClass  = "cs-cv",
  url = {http://arxiv.org/abs/1405.0006}
}

```

--------------------------------

### Control Pupil Core with Pupil Remote

Source: https://docs.pupil-labs.com/core/developer/network-api

Send commands to Pupil Remote to control recording and calibration. Ensure to receive responses for every sent message to prevent unresponsiveness.

```python
# start recording
pupil_remote.send_string('R')
print(pupil_remote.recv_string())

sleep(5)
pupil_remote.send_string('r')
print(pupil_remote.recv_string())
```

--------------------------------

### Set Region-Of-Interest (ROI) for Pupil Detector Plugin

Source: https://docs.pupil-labs.com/core/developer/network-api

Notification for setting the Region-Of-Interest for any or all pupil detector plugins.

```python
# Notification for setting the Region-Of-Interest (ROI) for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_roi',
    'subject': 'pupil_detector.set_roi',
    'value': <roi: (min_x: int, min_y: int, max_x: int, max_y: int)>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}
```
Tokens:
Raw