### Install Python Libraries

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Installs necessary Python libraries for the project from a requirements file. Supports both standard and macOS installations. Ensure the requirements.txt file is in the current directory before running.

```bash
pip install -r requirements.txt
```

```bash
pip install -r requirements_macos.txt
```

--------------------------------

### Data Structure Example (Text)

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/README.md

Illustrates the expected JSON-like structure for graph data, including adjacency matrices, node features, edge features, labels, and sequence IDs. This format is crucial for processing graph sequences and ensuring proper data splitting.

```text
{
  "a": [AdjMatrixGraph0, AdjMatrixGraph1, ... AdjMatrixGraphN],
  "x": [NodeFeatureMatrixGraph0, NodeFeatureMatrixGraph1, ... NodeFeatureMatrixGraphNN],
  "e": [EdgeFeatureMatrixGraph0, EdgeFeatureMatrixGraph1, ... EdgeFeatureMatrixGraphN]
  "label": [[LabelGraph0], [LabelGraph1]...[LabelGraphN]],
  "id": [SequenceIDGraph0, SequenceIDGraph1, ... SequenceIDGraphN],
  "node_feature_names": ["x", "y", "vx",... "att_team"], 
  "edge_feature_names": ["distance", ...]
}
```

--------------------------------

### Convert Raw Data to Spektral Dataset (Python)

Source: https://context7.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/llms.txt

Transforms raw soccer counterattack data into Spektral's Dataset format for GNN training. This Python class supports various adjacency matrix types and customizable node/edge features, enabling flexible graph representation for GNN models. It includes examples of loading data and inspecting dataset properties.

```python
from spektral.data import Dataset, Graph

class CounterDataset(Dataset):
    def __init__(self, **kwargs):
        self.data = kwargs['data']
        self.matrix_type = kwargs['matrix_type']
        super().__init__(**kwargs)

    def read(self):
        data = self.data
        data_mat = data[self.matrix_type]

        return [
            Graph(x=x, a=a, e=e, y=y) for x, a, e, y in zip(
                data_mat['x'], data_mat['a'], data_mat['e'], data['binary']
            )
        ]

# Load and prepare dataset
data = get_data('women.pkl')

# Select adjacency matrix type:
# - 'normal': attacking players connected to each other, defending to each other, both via ball
# - 'delaunay': Delaunay triangulation connections based on spatial positioning
# - 'dense': all players and ball fully connected
# - 'dense_ap': all attacking players connected to each other and defensive players
# - 'dense_dp': all defending players connected to each other and attacking players

dataset = CounterDataset(data=data, matrix_type='normal')

# Dataset properties:
print(f"Total samples: {len(dataset)}")
print(f"Node features shape: {dataset[0].x.shape}")  # (n_players, n_features)
print(f"Adjacency matrix shape: {dataset[0].a.shape}")  # (n_players, n_players)
print(f"Edge features shape: {dataset[0].e.shape}")  # (n_edges, n_edge_features)
print(f"Label: {dataset[0].y}")  # [0] or [1]

```

--------------------------------

### Download and Load Data from S3

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Fetches specified data files (e.g., 'women.pkl', 'men.pkl') from an S3 bucket if they are not locally present. Displays a progress bar during download and then loads the data using pickle. Handles potential network requests and file I/O.

```python
def get_data(file_name):
    '''
    Fetches the file from the location, loads it into memory and returns the data.
    '''
    if not isfile(file_name):
        url = f"https://ussf-ssac-23-soccer-gnn.s3.us-east-2.amazonaws.com/public/counterattack/{file_name}"

        logger.info(f"Downloading data from {url}...")
        
        r = requests.get(url, stream=True)
        # Fancy code to print progress bar
        block_size = 1024
        n_chunk = 1000
        file_size = int(r.headers.get('Content-Length', None))
        num_bars = np.ceil(file_size / (n_chunk * block_size))
        bar =  progressbar.ProgressBar(maxval=num_bars).start()
        with open(file_name, 'wb') as f:
            for i, chunk in enumerate(r.iter_content(chunk_size=n_chunk * block_size)):
                f.write(chunk)
                bar.update(i+1)
                # Add a little sleep so you can see the bar progress
                time.sleep(0.05)

        logger.info("File downloaded successfully!")
        
    logger.info(f"Opening {file_name}...")
    with open(file_name, 'rb') as handle:
        data = pickle.load(handle)
    return data
```

--------------------------------

### Configure GNN Training Parameters - Python

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Sets hyperparameters for training a Graph Neural Network. Includes learning rate, epochs, batch size, number of hidden channels, and layers. Also prepares the dataset for training and testing by splitting it and creating data loaders.

```python
learning_rate = 1e-3  # Learning rate
epochs = 150  # Number of training epochs
batch_size = 16  # Batch size
channels = 128  # Hidden units for the neural network
layers = 3  # Number of CrystalConv layers

N = max(g.n_nodes for g in dataset) # Number of nodes
F = dataset.n_node_features  # Dimension of node features
S = dataset.n_edge_features  # Dimension of edge features
n_out = dataset.n_labels  # Dimension of the target
n = len(dataset) # Number of samples in the dataset

# Train/test split for the dataset
idxs = np.random.RandomState(seed=15).permutation(len(dataset))
split_va, split_te = int(0.7 * len(dataset)), int(0.69 * len(dataset))
idx_tr, idx_va, idx_te = np.split(idxs, [split_va, split_te])
dataset_tr = dataset[idx_tr]
dataset_te = dataset[idx_te]
loader_tr = DisjointLoader(dataset_tr, batch_size=batch_size, epochs=epochs)
loader_te = DisjointLoader(dataset_te, batch_size=batch_size, epochs=1, shuffle = False)
```

--------------------------------

### Create Dataset Type Selection Widget

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Generates an interactive dropdown widget using ipywidgets to allow the user to select the dataset type for GNN training. Options include 'Women', 'Men', and 'Combined' data.

```python
print("Choose File for training:")
file_widget = Dropdown(
    options=['Women', 'Men', 'Combined'],
    value='Women',
    disabled=False,
)
display(file_widget)
```

--------------------------------

### Download and Load Soccer Counterattack Dataset (Python)

Source: https://context7.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/llms.txt

Downloads counterattack datasets from AWS S3 and loads them into memory using Python. It supports downloading women's, men's, or combined datasets containing graph representations of tracking data frames. The function handles file existence checks and provides progress updates during download.

```python
import pickle
import requests
import numpy as np
import progressbar
from os.path import isfile

def get_data(file_name):
    if not isfile(file_name):
        url = f"https://ussf-ssac-23-soccer-gnn.s3.us-east-2.amazonaws.com/public/counterattack/{file_name}"

        r = requests.get(url, stream=True)
        block_size = 1024
        n_chunk = 1000
        file_size = int(r.headers.get('Content-Length', None))
        num_bars = np.ceil(file_size / (n_chunk * block_size))
        bar = progressbar.ProgressBar(maxval=num_bars).start()

        with open(file_name, 'wb') as f:
            for i, chunk in enumerate(r.iter_content(chunk_size=n_chunk * block_size)):
                f.write(chunk)
                bar.update(i+1)

    with open(file_name, 'rb') as handle:
        data = pickle.load(handle)
    return data

# Load women's dataset (NWSL 2022 + International women's soccer)
women_data = get_data('women.pkl')

# Data structure:
# {
#   "normal": {
#     "a": [AdjacencyMatrix0, AdjacencyMatrix1, ...],  # Adjacency matrices
#     "x": [NodeFeatures0, NodeFeatures1, ...],         # Node features (player/ball attributes)
#     "e": [EdgeFeatures0, EdgeFeatures1, ...]          # Edge features (relationships)
#   },
#   "binary": [[Label0], [Label1], ...]                # Success labels (1=success, 0=fail)
# }

# Available datasets:
# - 'women.pkl': 3,720 graphs from NWSL 2022 + International women's matches
# - 'men.pkl': 17,143 graphs from MLS 2022
# - 'combined.pkl': 20,863 graphs from all competitions

```

--------------------------------

### Model Training and Data Loading in Python

Source: https://context7.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/llms.txt

Sets up data loaders using DisjointLoader from Spektral for training and testing graph neural networks. It defines a training step function using TensorFlow's tf.function for optimization and includes a loop for iterating through training epochs.

```python
# Configuration
learning_rate = 1e-3
epochs = 150
batch_size = 16
channels = 128
layers = 3

# Build and compile model
model = GNN(n_layers=layers, channels=channels, n_out=1)
optimizer = Adam(learning_rate)
loss_fn = BinaryCrossentropy()

# Prepare data loaders
from spektral.data import DisjointLoader

idxs = np.random.RandomState(seed=15).permutation(len(dataset))
split_va, split_te = int(0.7 * len(dataset)), int(0.69 * len(dataset))
idx_tr, idx_va, idx_te = np.split(idxs, [split_va, split_te])

dataset_tr = dataset[idx_tr]
dataset_te = dataset[idx_te]

loader_tr = DisjointLoader(dataset_tr, batch_size=batch_size, epochs=epochs)
loader_te = DisjointLoader(dataset_te, batch_size=batch_size, epochs=1, shuffle=False)

# Training function
@tf.function(input_signature=loader_tr.tf_signature(), experimental_relax_shapes=True)
def train_step(inputs, target):
    with tf.GradientTape() as tape:
        predictions = model(inputs, training=True)
        loss = loss_fn(target, predictions) + sum(model.losses)
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    return loss

# Train model
step = loss = 0
for batch in loader_tr:
    step += 1
    loss += train_step(*batch)
    if step == loader_tr.steps_per_epoch:
        step = 0
        print(f"Loss: {loss / loader_tr.steps_per_epoch:.4f}")
        loss = 0
```

--------------------------------

### Load Selected Dataset

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Loads the selected dataset from S3 based on the user's choice from the `file_widget`. It constructs the filename and calls the `get_data` function to retrieve and load the data.

```python
file_name = file_widget.value.lower() + '.pkl'
# Obtain the data
og_data = get_data(file_name)
```

--------------------------------

### Log Dataset Information

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Logs information about the dataset, including the total number of samples and the percentage of successful samples in the total, training, and testing sets. This helps in understanding the data distribution and potential class imbalance.

```python
logger.info(f"n: {n}")
logger.info(f"Pct successful total: {round(np.asarray([graph.y[0] for graph in dataset]).sum() / n, 2))}")
logger.info(f"Pct successful train: {round(np.asarray([graph.y[0] for graph in dataset_tr]).sum() / (n * .7), 2))}")
logger.info(f"Pct successful test: {round(np.asarray([graph.y[0] for graph in dataset_te]).sum() / (n * .3), 2))}")
```

--------------------------------

### Select Player Type for Feature Importance Testing

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

This code snippet initializes a dropdown menu for selecting player types ('Attacking' or 'Defending') to be used in feature importance testing. It relies on the `ipywidgets` library for the dropdown functionality.

```python
print("Choose Player type for testing feature importance:")
player_type = Dropdown(
    options=['Attacking', 'Defending'],
    value='Attacking',
    disabled=False,
)
display(player_type)
```

--------------------------------

### Import Core Python Libraries for GNN Project

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Imports essential libraries for GNN development, including Spektral for graph data structures, TensorFlow for model building, data manipulation with Pandas and NumPy, and utilities for file handling, logging, and visualization.

```python
from ipywidgets import Checkbox, Dropdown, Accordion, VBox
import sklearn.metrics as metrics
from sklearn.calibration import calibration_curve
from spektral.data import Dataset, Graph, DisjointLoader
from spektral.layers import CrystalConv, GlobalAvgPool
from tensorflow.keras.layers import Dense, Input, Dropout
from tensorflow.keras.losses import BinaryCrossentropy
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import AUC

import copy
import logging
import matplotlib.pyplot as plt
import numpy as np
import pickle
import pandas as pd
import random
import sys
import tensorflow as tf
import requests
import progressbar
from os.path import isfile
import time

# Setting up the logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
stdout_handler = logging.StreamHandler(sys.stdout)
logger.addHandler(stdout_handler)
```

--------------------------------

### Calculate ECE and Bin Data with Pandas

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

This snippet calculates Expected Calibration Error (ECE) by binning probability outputs and computing relevant metrics. It utilizes pandas for data manipulation and assumes the existence of 'ece_df' with 'output_pp' and 'result' columns.

```python
bin_ranges = [(0, 0.1), (0.1, 0.2), (0.2, 0.3), (0.3, 0.4), (0.4, 0.5), (0.5, 0.6), (0.6, 0.7), (0.7, 0.8), (0.8, 0.9), (0.9, 1.0)]
bin_calc = pd.DataFrame(columns = ['bin', 'count', 'accuracy', 'avg_pp', 'acc-conf', 'count_into_acc-conf'])

for i, bin_range in enumerate(bin_ranges):
    # Get the higher and lower end of the bins
    lower, higher = bin_range[0], bin_range[1]
    
    # Get the probability outputs within the range
    bin_calc_temp = ece_df.loc[(ece_df['output_pp'] > lower) & (ece_df['output_pp'] <= higher)]
    count = bin_calc_temp.shape[0]
    
    # Compute parameters needed to calculate ECE 
    if count > 0:
        total_corrects = bin_calc_temp[(bin_calc_temp['result'] == 1)].shape[0]
        accuracy = total_corrects / count
        avg_pp = bin_calc_temp['output_pp'].mean()
        acc_conf = abs(accuracy - avg_pp)

        bin_calc.loc[i] = [bin_range, count, accuracy, avg_pp, acc_conf, count*acc_conf]
        
# Print ECE value    
print("ECE is : " + str(bin_calc['count_into_acc-conf'].sum() / bin_calc['count'].sum()))
```

--------------------------------

### Organize Soccer GNN Features with Accordion (Python)

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Creates an Accordion widget in Python to organize the previously defined edge and node feature selections. It sets titles for each section ('Edge Features' and 'Node Features') for better user interface organization.

```python
accordion = Accordion(children=[edge_f_box, node_f_box])
accordion.set_title(0, 'Edge Features')
accordion.set_title(1, 'Node Features')
accordion
```

--------------------------------

### GNN Model Training Step

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Defines a single training step for the GNN model using TensorFlow's `tf.function` for optimization. It calculates the loss, computes gradients, and applies them to update model weights. The loop iterates through the training loader to print the loss at each epoch.

```python
@tf.function(input_signature=loader_tr.tf_signature(), experimental_relax_shapes=True)
def train_step(inputs, target):
    with tf.GradientTape() as tape:
        predictions = model(inputs, training=True)
        loss = loss_fn(target, predictions) + sum(model.losses)
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    return loss


# Print loss at each step of training.
step = loss = 0
for batch in loader_tr:
    step += 1
    loss += train_step(*batch)
    if step == loader_tr.steps_per_epoch:
        step = 0
        print("Loss: {}".format(loss / loader_tr.steps_per_epoch))
        loss = 0
```

--------------------------------

### Choose Soccer GNN Node Features (Python)

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Defines checkboxes for selecting various node features in a soccer GNN model, including player coordinates, velocity components, and game-specific attributes like distance to goal and attacking team flag. These selections are organized into a VBox widget.

```python
x_coord = Checkbox(
    value=True,
    description='x coordinate',
    disabled=False
)

y_coord = Checkbox(
    value=True,
    description='y coordinate',
    disabled=False
)

vx = Checkbox(
    value=True,
    description='vX',
    disabled=False
)


vy = Checkbox(
    value=True,
    description='vY',
    disabled=False
)


v = Checkbox(
    value=True,
    description='Velocity',
    disabled=False
)


velocity_angle = Checkbox(
    value=True,
    description='Velocity Angle',
    disabled=False
)

dist_goal = Checkbox(
    value=True,
    description='Distance to Goal',
    disabled=False
)

goal_angle = Checkbox(
    value=True,
    description='Angle with Goal',
    disabled=False
)


dist_ball = Checkbox(
    value=True,
    description='Distance to Ball',
    disabled=False
)

ball_angle = Checkbox(
    value=True,
    description='Angle with Ball',
    disabled=False
)


is_attacking = Checkbox(
    value=True,
    description='Attacking Team Flag',
    disabled=False
)

potential_receiver = Checkbox(
    value=True,
    description='Potential Receiver',
    disabled=False
)

# Add the user selection of node features in node_f_box list.
node_f_box = VBox([x_coord, y_coord, vx, vy, v, velocity_angle, dist_goal, goal_angle, dist_ball, ball_angle, 
                is_attacking, potential_receiver])
```

--------------------------------

### Convert Data to Spektral Graphs - Python

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Defines a `CounterDataset` class to convert raw graph data into Spektral's `Graph` representation. It utilizes the `spektral` library and selects the adjacency matrix type. The constructor takes `data` and `matrix_type` as input. The `read` method processes the data and returns a list of `Graph` objects.

```python
from spektral.data import Dataset, Graph
from spektral.data.loaders import DisjointLoader
import numpy as np
from loguru import logger

class CounterDataset(Dataset):
    """
    Convert raw Graph data to a CounterDataset with Dataset class from the spektral library to include node features, 
    edge features and the adjacency matrix.
    """
    def __init__(self, **kwargs):
        """
        Constructor to load parameters.
        """
        self.data = kwargs['data']
        self.matrix_type = kwargs['matrix_type']
        super().__init__(**kwargs)
        
    def read(self):
        """
        Overriding the read function - to return a list of Graph objects
        """
        logger.info("Loading Pass Dataset.")
        
        data = self.data
        # Choosing the data with the required matrix type.
        data_mat = data[self.matrix_type]
        
        # Print Graph information
        logger.info(
            f"""
node_features (x): {data_mat['x'][0].shape}
             \n adj_matrix (a): {data_mat['a'][0].shape}
             \n edge_features (e): {data_mat['e'][0].shape}
             """
        )
        
        # Return a list of Graph objects
        return [
            Graph(x=x, a=a, e=e, y=y) for x, a, e, y in zip(
                data_mat['x'], data_mat['a'], data_mat['e'], data['binary']
            )
        ]

# Load the dataset in the CounterDataset format.
dataset = CounterDataset(data = data, matrix_type = adj_matrix.value)
```

--------------------------------

### Calibration Curve and ECE Calculation (Python)

Source: https://context7.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/llms.txt

This snippet calculates and visualizes the calibration curve and the Expected Calibration Error (ECE) for a model. It involves processing predictions across the entire dataset, binning probabilities, and comparing predicted confidence with actual accuracy. Dependencies include scikit-learn, pandas, and matplotlib.

```python
from sklearn.calibration import calibration_curve
import pandas as pd
import matplotlib.pyplot as plt

# Assuming 'loader_full' is a loader for the full dataset and 'model' is your trained model
loader_full = DisjointLoader(dataset, batch_size=1, epochs=1, shuffle=False)

predictions_df = pd.DataFrame(columns=['output_pp', 'predicted', 'target', 'result'])

for batch in loader_full:
    inputs, target = batch
    p = model(inputs, training=False)
    original_prediction = p.numpy()[0][0]
    predicted_value = 1 if original_prediction >= 0.5 else 0
    predictions_df.loc[len(predictions_df)] = [
        original_prediction, predicted_value, target[0][0],
        1 if predicted_value == target[0][0] else 0
    ]

# Calculate Expected Calibration Error (ECE)
bin_ranges = [(i/10, (i+1)/10) for i in range(10)]
bin_calc = pd.DataFrame(columns=['bin', 'count', 'accuracy', 'avg_pp', 'acc-conf', 'weighted'])

for i, (lower, higher) in enumerate(bin_ranges):
    bin_data = predictions_df[(predictions_df['output_pp'] > lower) &
                              (predictions_df['output_pp'] <= higher)]
    count = len(bin_data)

    if count > 0:
        accuracy = bin_data[bin_data['result'] == 1].shape[0] / count
        avg_pp = bin_data['output_pp'].mean()
        acc_conf = abs(accuracy - avg_pp)
        bin_calc.loc[i] = [(lower, higher), count, accuracy, avg_pp, acc_conf, count * acc_conf]

ece = bin_calc['weighted'].sum() / bin_calc['count'].sum()
print(f"Expected Calibration Error (ECE): {ece:.4f}")

# Plot calibration curve
cal_y, cal_x = calibration_curve(predictions_df['target'], predictions_df['output_pp'], n_bins=10)

plt.figure(figsize=(8, 6))
plt.plot(cal_x, cal_y, marker='.', label='Model')
plt.plot([0, 1], [0, 1], ls='--', color='green', label='Ideal calibration')
plt.legend(loc='upper left')
plt.xlabel('Average Predicted Probability in each bin')
plt.ylabel('Ratio of positives')
plt.title("Calibration Curve")
plt.show()
```

--------------------------------

### Build GNN Model with Spektral

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Defines a Graph Neural Network (GNN) model using the Spektral library. The model consists of multiple convolutional layers (CrystalConv), a global pooling layer (GlobalAvgPool), and dense layers for classification. The `call` method outlines the forward pass of the network.

```python
class GNN(Model):
    '''
    Building the Graph Neural Network configuration with Model as the parent class 
    from spektral library.
    '''
    def __init__(self, n_layers):
        '''
        Constructor code for setting up the layers needed for training the model.
        '''
        super().__init__()
        self.conv1 = CrystalConv()
        self.convs = []
        for _ in range(1, n_layers):
            self.convs.append(
                CrystalConv()
            )
        self.pool = GlobalAvgPool()
        self.dense1 = Dense(channels, activation="relu")
        self.dropout = Dropout(0.5)
        self.dense2 = Dense(channels, activation="relu")
        self.dense3 = Dense(n_out, activation="sigmoid")

    def call(self, inputs):
        '''
        Build the neural network.
        '''
        x, a, e, i = inputs
        x = self.conv1([x, a, e])
        for conv in self.convs:
            x = conv([x, a, e])
        x = self.pool([x, i])
        x = self.dense1(x)
        x = self.dropout(x)
        x = self.dense2(x)
        x = self.dropout(x)
        return self.dense3(x)


# Build model
model = GNN(layers)
# Setup the optimizer
optimizer = Adam(learning_rate)
# Set up the logloss function
loss_fn = BinaryCrossentropy()
```

--------------------------------

### Plot Calibration Curve with Scikit-learn and Matplotlib

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

This snippet generates a calibration curve to visualize model calibration. It uses scikit-learn's `calibration_curve` function and matplotlib for plotting, showing the model's predicted probabilities against the actual proportion of positives. Assumes 'ece_df' contains 'target' and 'output_pp' columns.

```python
# Use the sklearn calibration_curve() function to obtain calibration values for the model.
cal_y, cal_x = calibration_curve(ece_df['target'], ece_df['output_pp'], n_bins = 10)

# Plot the calibration curve.
fig, ax = plt.subplots()
plt.plot(cal_x, cal_y, marker = '.')
plt.plot([0, 1], [0, 1], ls = '--', color = 'green', label = 'Ideal calibration')
leg = plt.legend(loc = 'upper left')
plt.xlabel('Average Predicted Probability in each bin')
plt.ylabel('Ratio of positives')
plt.title("Calibration Curve")
plt.show()
```

--------------------------------

### ROC-AUC Evaluation Imports in Python

Source: https://context7.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/llms.txt

Imports necessary libraries from scikit-learn and matplotlib for evaluating model performance using Receiver Operating Characteristic curves and Area Under the Curve metrics. These are standard tools for binary classification assessment.

```python
import sklearn.metrics as metrics
import matplotlib.pyplot as plt
```

--------------------------------

### Choose Soccer GNN Edge Features (Python)

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Defines checkboxes for selecting various edge features in a soccer GNN model, including player distance, speed difference, and positional/velocity angles. These selections are then organized into a VBox widget.

```python
player_dist = Checkbox(
    value=True,
    description='Player Distance',
    disabled=False
)

speed_diff_matrix = Checkbox(
    value=True,
    description='Speed Difference',
    disabled=False
)

pos_sin_angle = Checkbox(
    value=True,
    description='Positional Sine angle',
    disabled=False
)

pos_cos_angle = Checkbox(
    value=True,
    description='Positional Cosine angle',
    disabled=False
)

vel_sin_angle = Checkbox(
    value=True,
    description='Velocity Sine angle',
    disabled=False
)

vel_cos_angle = Checkbox(
    value=True,
    description='Velocity Cosine angle',
    disabled=False
)

# Add the user selection of edge features in edge_f_box list.
edge_f_box = VBox([player_dist, speed_diff_matrix, pos_sin_angle, pos_cos_angle, vel_sin_angle, vel_cos_angle])
```

--------------------------------

### Evaluate GNN Model Calibration

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Evaluates the calibration of the GNN model by calculating the Expected Calibration Error (ECE). It reloads the dataset, computes predictions, and stores them along with the predicted and true class labels in a pandas DataFrame for further analysis.

```python
# Reload the dataset in the CounterDataset format.
dataset_c = CounterDataset(data=data, matrix_type = adj_matrix.value)

# Setup the loader.
loader = DisjointLoader(dataset_c, batch_size=1, epochs=1, shuffle = False)

# Set up an empty pandas dataframe.
ece_df = pd.DataFrame(columns = ['output_pp', 'predicted', 'target', 'result'])

# Compute the predictions and save them in the Pandas DataFrame.
for batch in loader:
    inputs, target = batch
    p = model(inputs, training=False)
    original_prediction = p.numpy()[0][0]
    
    # Threshold set to 0.5
    predicted_value = 1 if original_prediction >= 0.5 else 0 
    ece_df.loc[len(ece_df)] = [original_prediction, predicted_value, target[0][0], 1 if predicted_value == target[0][0] else 0]
```

--------------------------------

### Create Adjacency Matrix Selection Widget

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

Creates a dropdown widget for selecting the type of adjacency matrix to be used in the GNN. Available options include 'normal', 'delaunay', 'dense', 'dense_ap', and 'dense_dp'.

```python
print("Choose Adjacency Matrix:")
adj_matrix = Dropdown(
    options=['normal', 'delaunay', 'dense', 'dense_ap', 'dense_dp'],
    value='normal',
    disabled=False,
)
display(adj_matrix)
```

--------------------------------

### Shuffle Node Features and Calculate Importance

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

This Python code shuffles node features to assess their importance by calculating Area Under the Curve (AUC) metrics. It iterates through each feature, shuffles it, splits data into training and testing sets, trains a model, and computes AUC. It requires libraries like `numpy`, `sklearn.metrics`, and custom dataset/loader classes.

```python
aucs = [] # Empty list to store the AUC values
feature_dict = {} # Feature dictionary to store the feature change values
flag_found = False # Flag to check if Attacking Team feature is included in training
iterations = 10 # Number of iterations

# Check if Attacking Team Flag exists
for flag_index, feature in enumerate(node_features):
    if feature == 'Attacking Team Flag':
        flag_found = True
        break

# If Attacking Team Flag does not exist - feature importance can't be performed.
if not flag_found:
    print("Attacking team Flag not included in node features. Can't compute feature importance.")
    
else:
    for i in range(len(node_features)): 
        mini_auc = []
        for _ in range(iterations):
            # Get shuffled data
            shuffle_data = ShuffledCounterDataset(data = data,
                                          node_feature_shuffle = i, player_type = player_type.value.lower(), 
                                          matrix_type = adj_matrix.value, flag_index = flag_index)
            
            # Split between training and testing data
            idxs = np.random.RandomState(seed=35).permutation(len(shuffle_data))
            split_va, split_te = int(0.7 * len(dataset)), int(0.7 * len(shuffle_data))
            idx_tr, idx_va, idx_te = np.split(idxs, [split_va, split_te])
            dataset_tr = shuffle_data[idx_tr]
            dataset_va = shuffle_data[idx_va]
            dataset_te = shuffle_data[idx_te]
            loader_te = DisjointLoader(dataset_te, batch_size=16, epochs=1)

            # Obtain the model metrics
            y_true = []
            y_pred = []

            for batch in loader_te:
                inputs, target = batch
                p = model(inputs, training=False)
                y_true.append(target)
                y_pred.append(p.numpy())

            y_true = np.vstack(y_true)
            y_pred = np.vstack(y_pred)
            fpr, tpr, threshold = metrics.roc_curve(y_true, y_pred)
            roc_auc = metrics.auc(fpr, tpr)
            # Store the AUC's at all iterations
            mini_auc.append(roc_auc)

        # Perform the error calculation and store it in the aucs list.
        errors = [1 - auc_1 for auc_1 in mini_auc]
        main_error = 1 - roc_auc
        errors_ = [100*(error - main_error) for error in errors]
        feature_dict[node_features[i]] = errors_
        aucs.append(sum(mini_auc) / len(mini_auc))
```

--------------------------------

### Custom Dataset for Permutation Feature Importance (Python)

Source: https://context7.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/llms.txt

This snippet defines a custom `Dataset` class, `ShuffledCounterDataset`, for use with Spektral. It's designed to facilitate permutation feature importance by allowing specific node features to be shuffled for either attacking or defending players within graph data. It takes graph data, the feature to shuffle, player type, matrix type, and flag index as input.

```python
import copy
import random
from spektral.data import Dataset, Graph

class ShuffledCounterDataset(Dataset):
    def __init__(self, **kwargs):
        self.data = kwargs['data']
        self.node_feature_shuffle = kwargs['node_feature_shuffle']
        self.player_type = kwargs['player_type']  # 'attacking' or 'defending'
        self.matrix_type = kwargs['matrix_type']
        self.flag_index = kwargs['flag_index']  # index of attacking team flag
        super().__init__(**kwargs)

    def read(self):
        data = copy.deepcopy(self.data)
        data_mat = data[self.matrix_type]

        for i in range(len(data_mat['x'])):
            arr = data_mat['x'][i]

            # Separate attacking and defending players
            atts = arr[:-1][arr[:-1, self.flag_index] == 1].copy()
            defs = arr[:-1][arr[:-1, self.flag_index] == 0].copy()

            # Shuffle selected feature for chosen player type
            if self.player_type == 'attacking':
                random.shuffle(atts[:, self.node_feature_shuffle])
            else:
                random.shuffle(defs[:, self.node_feature_shuffle])

            # Reassign shuffled values
            arr[:-1][arr[:-1, self.flag_index] == 1] = atts
            arr[:-1][arr[:-1, self.flag_index] == 0] = defs

        return [
            Graph(x=x, a=a, e=e, y=y) for x, a, e, y in
            zip(data_mat['x'], data_mat['a'], data_mat['e'], data['binary'])
        ]
```

--------------------------------

### BibTeX Citation for Soccer GNN Paper

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/README.md

This snippet provides the BibTeX entry for citing the research paper associated with the Soccer GNN project. It includes title, authors, book title, pages, and year of publication.

```text
@inproceedings{sahasrabudhe2023graph,
title={A Graph Neural Network deep-dive into successful counterattacks},
author={Sahasrabudhe, Amod and Bekkers, Joris},
booktitle={17th Annual MIT Sloan Sports Analytics Conference. Boston, MA, USA: MIT},
pages={15},
year={2023}
}
```

--------------------------------

### GNN Crystal Convolutional Graph Neural Network in Python

Source: https://context7.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/llms.txt

Implements a Graph Neural Network using CrystalConv layers for spatial feature learning, followed by global average pooling and dense layers for binary classification. It utilizes TensorFlow and Spektral libraries. The model takes node, adjacency, edge features, and batch index as input.

```python
from spektral.layers import CrystalConv, GlobalAvgPool
from tensorflow.keras.layers import Dense, Input, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import BinaryCrossentropy
import tensorflow as tf

class GNN(Model):
    def __init__(self, n_layers, channels=128, n_out=1):
        super().__init__()
        self.conv1 = CrystalConv()
        self.convs = []
        for _ in range(1, n_layers):
            self.convs.append(CrystalConv())
        self.pool = GlobalAvgPool()
        self.dense1 = Dense(channels, activation="relu")
        self.dropout = Dropout(0.5)
        self.dense2 = Dense(channels, activation="relu")
        self.dense3 = Dense(n_out, activation="sigmoid")

    def call(self, inputs):
        x, a, e, i = inputs  # nodes, adjacency, edges, batch_index
        x = self.conv1([x, a, e])
        for conv in self.convs:
            x = conv([x, a, e])
        x = self.pool([x, i])
        x = self.dense1(x)
        x = self.dropout(x)
        x = self.dense2(x)
        x = self.dropout(x)
        return self.dense3(x)
```

--------------------------------

### Custom Dataset for Permutation Feature Importance

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

This Python code defines a custom `ShuffledCounterDataset` class inheriting from `spektral.data.Dataset`. It's designed to facilitate permutation feature importance by allowing specific features of attacking or defending players to be shuffled within graph data. The `read` method overrides the base functionality to perform the permutation.

```python
class ShuffledCounterDataset(Dataset):
    '''
    Convert raw Graph data to a ShuffledCounterDataset with Dataset class from the spektral library to include node features, 
    edge features and the adjacency matrix.
    '''
    def __init__(self, **kwargs):
        '''
        Constructor to load the parameters.
        '''
        self.data = kwargs['data']
        self.node_feature_shuffle = kwargs['node_feature_shuffle']
        self.player_type = kwargs['player_type']
        self.matrix_type = kwargs['matrix_type']
        self.flag_index = kwargs['flag_index']
        super().__init__(**kwargs)
        
    def read(self):
        '''
        Overriding the read function - to return a list of Graph objects.
        Permuting the features in this function.
        '''
        data = copy.deepcopy(self.data)
        data_mat = data[self.matrix_type]
        
        # Check the type of players to be shuffled
        if self.player_type == 'attacking':
            for i in range(len(data_mat['x'])):
                arr = data_mat['x'][i]
                  
                # Get the appropriate type of player
                atts = arr[0:-1][arr[:-1, self.flag_index] == 1].copy()
                defs = arr[0:-1][arr[:-1, self.flag_index] == 0].copy()
                ball = arr[-1].copy()

                # Shuffle the feature requested
                random.shuffle(atts[:, self.node_feature_shuffle])

                # Assign back the shuffled values to the correct place they came from
                arr[0:-1][arr[:-1, self.flag_index] == 1] = atts
                arr[0:-1][arr[:-1, self.flag_index] == 0] = defs
           
        else:
            for i in range(len(data_mat['x'])):
                arr = data_mat['x'][i]
                    
                # Get the appropriate type of player
                atts = arr[0:-1][arr[:-1, self.flag_index] == 1].copy()
                defs = arr[0:-1][arr[:-1, self.flag_index] == 0].copy()
                ball = arr[-1].copy()

                # Shuffle the feature requested
                random.shuffle(defs[:, self.node_feature_shuffle])

                # Assign back the shuffled values to the correct place they came from
                arr[0:-1][arr[:-1, self.flag_index] == 1] = atts
                arr[0:-1][arr[:-1, self.flag_index] == 0] = defs
        
        
        return [
            Graph(x=x, a=a, e=e, y=y) for x, a, e, y in zip(data_mat['x'], data_mat['a'], data_mat['e'], data['binary'])
        ]
```

--------------------------------

### Plot Feature Importances Using Box Plot

Source: https://github.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/blob/main/counterattack.ipynb

This Python code generates a box plot to visualize feature importances calculated in the previous step. It uses `matplotlib.pyplot` to create the plot, defines custom properties for the box plot, and sets labels and title for clarity. Requires `matplotlib.pyplot` and `numpy`.

```python
plt.subplots(figsize=(10,25))

box_plot = plt.boxplot(feature_dict.values(),
                               positions=np.array(
    np.arange(len(feature_dict.keys())))*2.0+0.3,
                               widths=0.6, vert = False)
    
    
def define_box_properties(plot_name, color_code = 'black', label = ''):
    '''
    Define Box plot properties.
    '''
    for k, v in plot_name.items():
        plt.setp(plot_name.get(k), color=color_code)
         
    # use plot function to draw a small line to name the legend.
    plt.plot([], c=color_code, label=label)
    plt.legend()
    
define_box_properties(box_plot)

plt.yticks(np.arange(0, len(feature_dict.keys()) * 2, 2), feature_dict.keys())
plt.xlabel("Feature Importance")
plt.title('Feature Importance - Box Plot')
```

--------------------------------

### Predict on Test Set and Calculate ROC-AUC (Python)

Source: https://context7.com/ussoccerfederation/ussf_ssac_23_soccer_gnn/llms.txt

This snippet demonstrates how to iterate through a test data loader, obtain model predictions, and calculate the Receiver Operating Characteristic Area Under the Curve (ROC-AUC). It also includes plotting the ROC curve for visualization. Dependencies include TensorFlow/Keras for the model and scikit-learn for metrics.

```python
from sklearn import metrics
import matplotlib.pyplot as plt
import numpy as np

# Assuming 'loader_te' is your test data loader and 'model' is your trained model
y_true = []
y_pred = []

for batch in loader_te:
    inputs, target = batch
    predictions = model(inputs, training=False)
    y_true.append(target)
    y_pred.append(predictions.numpy())

y_true = np.vstack(y_true)
y_pred = np.vstack(y_pred)

# Calculate ROC-AUC
fpr, tpr, threshold = metrics.roc_curve(y_true, y_pred)
roc_auc = metrics.auc(fpr, tpr)

# Plot ROC curve
plt.figure(figsize=(8, 6))
plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label=f'AUC = {roc_auc:.2f}')
plt.legend(loc='lower right')
plt.plot([0, 1], [0, 1], 'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()

print(f"Model ROC-AUC: {roc_auc:.4f}")
# Expected output for well-trained model: ROC-AUC > 0.75
```