### Initialize and Run IGNNK Model Source: https://context7.com/kaimaoge/ignnk/llms.txt Demonstrates the setup of the IGNNK model using bidirectional random walk matrices for signal reconstruction. ```python import torch import numpy as np from basic_structure import IGNNK from utils import calculate_random_walk_matrix # Initialize the IGNNK model # h: time dimension (number of timesteps) # z: hidden dimension for graph convolution # K: diffusion convolution order (actual steps = K+1) h = 24 # 24 time steps z = 100 # hidden dimension K = 1 # diffusion order model = IGNNK(h, z, K) # Prepare input data # X: Input tensor of shape (batch_size, num_timesteps, num_nodes) batch_size = 4 num_nodes = 150 X = torch.randn(batch_size, h, num_nodes) # Create adjacency matrix and compute random walk matrices A = np.random.rand(num_nodes, num_nodes) A = (A + A.T) / 2 # Make symmetric np.fill_diagonal(A, 0) A = np.exp(-A) # Distance-based weights # Compute forward and backward random walk matrices A_q = torch.from_numpy(calculate_random_walk_matrix(A).T.astype('float32')) A_h = torch.from_numpy(calculate_random_walk_matrix(A.T).T.astype('float32')) # Forward pass - reconstruct signals at all nodes output = model(X, A_q, A_h) print(f"Input shape: {X.shape}") # (4, 24, 150) print(f"Output shape: {output.shape}") # (4, 24, 150) ``` -------------------------------- ### End-to-End METR-LA Kriging Setup Source: https://context7.com/kaimaoge/ignnk/llms.txt Initializes data loading, splitting, and model configuration for traffic speed kriging. ```python import torch import torch.nn as nn import torch.optim as optim import numpy as np import random from basic_structure import IGNNK from utils import load_metr_la_rdata, calculate_random_walk_matrix, test_error # Hyperparameters n_o_n_m = 150 # Sampled nodes per batch h = 24 # Time window z = 100 # Hidden dimension K = 1 # Diffusion order n_m = 50 # Masked nodes N_u = 50 # Unknown test locations max_epochs = 100 learning_rate = 0.0001 E_maxvalue = 80 batch_size = 4 # Load data A, X = load_metr_la_rdata() X = X[:, 0, :] # Speed only # Train/test split split = int(X.shape[1] * 0.7) training_set = X[:, :split].T test_set = X[:, split:].T # Define unknown locations rand = np.random.RandomState(0) unknow_set = set(rand.choice(207, N_u, replace=False)) know_set = set(range(207)) - unknow_set # Get training subset (known locations only) training_set_s = training_set[:, list(know_set)] A_s = A[np.ix_(list(know_set), list(know_set))] # Initialize model model = IGNNK(h, z, K) criterion = nn.MSELoss() optimizer = optim.Adam(model.parameters(), lr=learning_rate) ``` -------------------------------- ### Train IGNNK via Command Line Source: https://context7.com/kaimaoge/ignnk/llms.txt Executes the training pipeline for various datasets using command-line arguments. ```bash # Train on METR-LA traffic data python IGNNK_train.py "metr" \ --n_o 150 \ --h 24 \ --n_m 50 \ --n_u 50 \ --max_iter 750 \ --learning_rate 0.0001 \ --batch_size 4 # Train on NREL solar data python IGNNK_train.py "nrel" \ --n_o 100 \ --h 24 \ --n_m 30 \ --n_u 30 \ --max_iter 750 # Train on USHCN weather data (larger hidden dimension) python IGNNK_train.py "ushcn" \ --n_o 900 \ --h 6 \ --n_m 300 \ --n_u 300 \ --max_iter 750 \ --z 350 # Train on Seattle traffic data python IGNNK_train.py "sedata" \ --n_o 240 \ --h 24 \ --n_m 80 \ --n_u 80 \ --max_iter 750 ``` -------------------------------- ### Train IGNNK on NREL Dataset Source: https://github.com/kaimaoge/ignnk/blob/master/README.md Command to initiate training of the IGNNK model using the NREL solar energy dataset. Key parameters include --n_o, --h, --n_m, --n_u, and --max_iter. ```python python IGNNK_train.py "nrel" --n_o 100 --h 24 --n_m 30 --n_u 30 --max_iter 750 ``` -------------------------------- ### Train IGNNK on USHCN Dataset Source: https://github.com/kaimaoge/ignnk/blob/master/README.md Execute this command for training the IGNNK model on the USHCN weather condition dataset. Note the additional --z parameter for specific configurations. ```python python IGNNK_train.py "ushcn" --n_o 900 --h 6 --n_m 300 --n_u 300 --max_iter 750 --z 350 ``` -------------------------------- ### Load and preprocess data for IGNNK Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_METR-LA.ipynb Loads the METR-LA dataset, splits it into training and testing sets, and prepares observed and unknown node sets. ```python A, X = load_metr_la_rdata() split_line1 = int(X.shape[2] * 0.7) training_set = X[:, 0, :split_line1].transpose() test_set = X[:, 0, split_line1:].transpose() # split the training and test period rand = np.random.RandomState(0) # Fixed random output, just an example when seed = 0. unknow_set = rand.choice(list(range(0,X.shape[0])),N_u,replace=False) unknow_set = set(unknow_set) full_set = set(range(0,207)) know_set = full_set - unknow_set training_set_s = training_set[:, list(know_set)] # get the training data in the sample time period A_s = A[:, list(know_set)][list(know_set), :] # get the observed adjacent matrix from the full adjacent matrix, # the adjacent matrix are based on pairwise distance, # so we need not to construct it for each batch, we just use index to find the dynamic adjacent matrix ``` -------------------------------- ### Define NREL Model Parameters Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_NREL.ipynb Configuration of hyperparameters and model initialization for the NREL case study. ```python n_o_n_m = 100 #sampled space dimension h = 16 #sampled time dimension z = 100 #hidden dimension for graph convolution K = 1 #If using diffusion convolution, the actual diffusion convolution step is K+1 n_m = 30 #number of mask node during training N_u = 30 #target locations, N_u locations will be deleted from the training data Max_episode = 750 #max training episode learning_rate = 0.0001 #the learning_rate for Adam optimizer batch_size = 8 STmodel = IGNNK(h, z, K) STmodel = torch.load('model/nrel_ignnk_sigmaA_cap_20210621.pth') ``` -------------------------------- ### Implement Kipf Graph Convolution (K_GCN) Source: https://context7.com/kaimaoge/ignnk/llms.txt Sets up the standard Kipf & Welling graph convolution layer, recommended for use with SELU activation. ```python import torch from basic_structure import K_GCN from utils import get_normalized_adj import numpy as np # Initialize Kipf GCN layer in_channels = 24 out_channels = 64 kgcn = K_GCN(in_channels, out_channels, activation='selu') # Prepare input: (batch_size, num_nodes, num_timesteps) batch_size = 4 num_nodes = 50 X = torch.randn(batch_size, num_nodes, in_channels) # Create normalized adjacency matrix A = np.random.rand(num_nodes, num_nodes).astype('float32') A = (A + A.T) / 2 A_hat = torch.from_numpy(get_normalized_adj(A).astype('float32')) # Forward pass output = kgcn(X, A_hat) print(f"K_GCN output shape: {output.shape}") # (4, 50, 64) ``` -------------------------------- ### Load and Preprocess NREL Data Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_NREL.ipynb Data loading, adjacency matrix construction using Gaussian kernels, and temporal filtering. ```python # _, X, files_info = load_nerl_data() X = np.load('data/nrel/nerl_X.npy') files_info = pd.read_pickle('data/nrel/nerl_file_infos.pkl') dist_mx = loadmat('data/nrel/nrel_dist_mx_lonlat.mat') dist_mx = dist_mx['nrel_dist_mx_lonlat'] dis = dist_mx/1e3 A = np.exp( -0.5* np.power( dis/14 ,2) ) # We only use 7:00am to 7:00pm as the target data, because otherwise the 0-values of periods without sunshine will greatly influence the results time_used_base = np.arange(84,228) time_used = np.array([]) for i in range(365): time_used = np.concatenate((time_used,time_used_base + 24*12* i)) X=X[:,time_used.astype(np.int)] capacities = np.array(files_info['capacity']) capacities = capacities.astype('float32') E_maxvalue = capacities.max() X = X.transpose()/capacities print(X.shape,E_maxvalue) split_line1 = int(X.shape[0] * 0.7) training_set = X[:split_line1, :] test_set = X[split_line1:, :] # split the training and test period rand = np.random.RandomState(0) # Fixed random output, just an example when seed = 0. unknow_set = rand.choice(list(range(0,X.shape[1])),N_u,replace=False) unknow_set = set(unknow_set) full_set = set(range(0,X.shape[1])) know_set = full_set - unknow_set training_set_s = training_set[:, list(know_set)] # get the training data in the sample time period A_s = A[:, list(know_set)][list(know_set), :] # get the observed adjacent matrix from the full adjacent matrix, # the adjacent matrix are based on pairwise distance, # so we need not to construct it for each batch, we just use index to find the dynamic adjacent matrix ``` -------------------------------- ### Load IGNNK Model and Evaluate Performance Source: https://github.com/kaimaoge/ignnk/blob/master/IGNNK_U_Central_missing.ipynb Loads a pre-trained IGNNK model and its parameters. It then evaluates the model's performance using the kNN imputation function and prints the resulting MAE, RMSE, and MAPE values for both methods. ```python # Load IGNNC model for central missing space_dim = 900 # randomly set 50 of them are missing, training with dynamic graph time_dim = 6 hidden_dim_s = 100 hidden_dim_t = 15 rank_s = 20 rank_t = 4 K=1 E_maxvalue = 80 STmodel = IGNNK(time_dim,hidden_dim_s,K) params_old = torch.load('model/GCNmodel_udata_carea.pth') params_new = {'GNN1.Theta1':params_old['SC1.Theta1'], 'GNN1.bias':params_old['SC1.bias'], 'GNN2.Theta1':params_old['SC2.Theta1'], 'GNN2.bias':params_old['SC2.bias'], 'GNN3.Theta1':params_old['SC3.Theta1'], 'GNN3.bias':params_old['SC3.bias']} # Keys redefined, does not influece the result STmodel.load_state_dict(params_new) udata_sim = joblib.load('model/udata_sim.joblib') MAE_t, RMSE_t, MAPE_t, udata_ignnk= test_error_cap(STmodel, unknow_set_central, full_set, X, A,time_dim,capacities) MAE, RMSE, MAPE, udata_knn = kNN(udata_sim, X, full_set, unknow_set_central) print(MAE_t, RMSE_t, MAPE_t) print(MAE, RMSE, MAPE) ``` -------------------------------- ### Import Dependencies Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_NREL.ipynb Required libraries and custom modules for IGNNK model operations. ```python import torch from torch.autograd import Variable import numpy as np import torch.nn.functional as F import torch.optim as optim from torch import nn import matplotlib.pyplot as plt from utils import load_nerl_data, get_normalized_adj, get_Laplace, calculate_random_walk_matrix,test_error_cap import random import copy import scipy from scipy.io import loadmat import math import pandas as pd from basic_structure import D_GCN, C_GCN, K_GCN,IGNNK ``` -------------------------------- ### Train the IGNNK model Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_METR-LA.ipynb Defines the loss function and optimizer, then iterates through training episodes, preparing batches, calculating loss, and performing backpropagation. Includes periodic testing and error reporting. ```python criterion = nn.MSELoss() optimizer = optim.Adam(STmodel.parameters(), lr=learning_rate) RMSE_list = [] MAE_list = [] MAPE_list = [] for epoch in range(Max_episode): for i in range(training_set.shape[0]//(h * batch_size)): #using time_length as reference to record test_error t_random = np.random.randint(0, high=(training_set_s.shape[0] - h), size=batch_size, dtype='l') know_mask = set(random.sample(range(0,training_set_s.shape[1]),n_o_n_m)) #sample n_o + n_m nodes feed_batch = [] for j in range(batch_size): feed_batch.append(training_set_s[t_random[j]: t_random[j] + h, :][:, list(know_mask)]) #generate 8 time batches inputs = np.array(feed_batch) inputs_omask = np.ones(np.shape(inputs)) inputs_omask[inputs == 0] = 0 # We found that there are irregular 0 values for METR-LA, so we treat those 0 values as missing data, # For other datasets, it is not necessary to mask 0 values missing_index = np.ones((inputs.shape)) for j in range(batch_size): missing_mask = random.sample(range(0,n_o_n_m),n_m) #Masked locations missing_index[j, :, missing_mask] = 0 Mf_inputs = inputs * inputs_omask * missing_index / E_maxvalue #normalize the value according to experience Mf_inputs = torch.from_numpy(Mf_inputs.astype('float32')) mask = torch.from_numpy(inputs_omask.astype('float32')) #The reconstruction errors on irregular 0s are not used for training A_dynamic = A_s[list(know_mask), :][:, list(know_mask)] #Obtain the dynamic adjacent matrix A_q = torch.from_numpy((calculate_random_walk_matrix(A_dynamic).T).astype('float32')) A_h = torch.from_numpy((calculate_random_walk_matrix(A_dynamic.T).T).astype('float32')) outputs = torch.from_numpy(inputs/E_maxvalue) #The label optimizer.zero_grad() X_res = STmodel(Mf_inputs, A_q, A_h) #Obtain the reconstruction loss = criterion(X_res*mask, outputs*mask) loss.backward() optimizer.step() #Errors backward MAE_t, RMSE_t, MAPE_t, metr_ignnk_res = test_error(STmodel, unknow_set, test_set, A,E_maxvalue, True) RMSE_list.append(RMSE_t) MAE_list.append(MAE_t) MAPE_list.append(MAPE_t) if epoch%50 == 0: print(epoch, MAE_t, RMSE_t, MAPE_t) idx = MAE_list == min(MAE_list) print('Best model result:',np.array(MAE_list)[idx],np.array(RMSE_list)[idx],np.array(MAPE_list)[idx]) #torch.save(STmodel.state_dict(), 'model/IGNNK.pth') # Save the model ``` -------------------------------- ### Define IGNNK model parameters Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_METR-LA.ipynb Sets up key hyperparameters for the IGNNK model, such as spatial and temporal dimensions, hidden dimensions, and training configurations. ```python n_o_n_m = 150 #sampled space dimension h = 24 #sampled time dimension z = 100 #hidden dimension for graph convolution K = 1 #If using diffusion convolution, the actual diffusion convolution step is K+1 n_m = 50 #number of mask node during training N_u = 50 #target locations, N_u locations will be deleted from the training data Max_episode = 750 #max training episode learning_rate = 0.0001 #the learning_rate for Adam optimizer E_maxvalue = 80 #the max value from experience batch_size = 4 ``` -------------------------------- ### Initialize and Run GAT Layer Source: https://context7.com/kaimaoge/ignnk/llms.txt Implements graph attention mechanism for learning adaptive edge weights based on node features. ```python import torch from basic_structure import GAT import numpy as np # Initialize GAT layer in_channels = 24 alpha = 0.2 # LeakyReLU negative slope threshold = 0.1 # Connection threshold gat = GAT(in_channels, alpha, threshold, concat=True) # Prepare input: (batch_size, num_nodes, in_channels) batch_size = 4 num_nodes = 50 X = torch.randn(batch_size, num_nodes, in_channels) # Adjacency matrix for masking attention adj = torch.rand(num_nodes, num_nodes) # Forward pass output = gat(X, adj) print(f"GAT output shape: {output.shape}") # (4, 50, 24) ``` -------------------------------- ### Initialize the IGNNK model Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_METR-LA.ipynb Instantiates the IGNNK model with the specified hidden dimensions and graph convolution parameters. ```python STmodel = IGNNK(h, z, K) # The graph neural networks ``` -------------------------------- ### Execute IGNNK Training Loop Source: https://context7.com/kaimaoge/ignnk/llms.txt Iterates through epochs to train the model, performing batch sampling, dynamic adjacency calculation, and loss optimization. Includes periodic evaluation and model state saving. ```python # Training loop for epoch in range(max_epochs): model.train() epoch_loss = 0 for i in range(training_set.shape[0] // (h * batch_size)): # Sample random time windows t_random = np.random.randint(0, training_set_s.shape[0] - h, size=batch_size) # Sample random nodes know_mask = list(random.sample(range(training_set_s.shape[1]), n_o_n_m)) # Create batch batch = np.array([ training_set_s[t:t+h, know_mask] for t in t_random ]) # Mask some nodes for reconstruction inputs_omask = np.ones(batch.shape) inputs_omask[batch == 0] = 0 missing_index = np.ones(batch.shape) for j in range(batch_size): mask_idx = random.sample(range(n_o_n_m), n_m) missing_index[j, :, mask_idx] = 0 # Prepare tensors Mf_inputs = torch.from_numpy( (batch * inputs_omask * missing_index / E_maxvalue).astype('float32') ) mask = torch.from_numpy(inputs_omask.astype('float32')) outputs = torch.from_numpy((batch / E_maxvalue).astype('float32')) # Dynamic adjacency for sampled nodes A_dynamic = A_s[np.ix_(know_mask, know_mask)] A_q = torch.from_numpy(calculate_random_walk_matrix(A_dynamic).T.astype('float32')) A_h = torch.from_numpy(calculate_random_walk_matrix(A_dynamic.T).T.astype('float32')) # Forward pass optimizer.zero_grad() X_res = model(Mf_inputs, A_q, A_h) loss = criterion(X_res * mask, outputs * mask) # Backward pass loss.backward() optimizer.step() epoch_loss += loss.item() # Evaluate if epoch % 10 == 0: model.eval() MAE, RMSE, R2, _ = test_error( model, unknow_set, test_set, A, E_maxvalue, True, test_set ) print(f"Epoch {epoch}: Loss={epoch_loss:.4f}, MAE={MAE:.2f}, RMSE={RMSE:.2f}") # Save trained model torch.save(model.state_dict(), 'model/IGNNK_metr_trained.pth') print("Model saved!") ``` -------------------------------- ### Train IGNNK on METR-LA Dataset Source: https://github.com/kaimaoge/ignnk/blob/master/README.md Use this command to train the IGNNK model on the METR-LA traffic dataset. Adjust parameters like --n_o, --h, --n_m, --n_u, and --max_iter for different configurations. ```python python IGNNK_train.py "metr" --n_o 150 --h 24 --n_m 50 --n_u 50 --max_iter 750 ``` -------------------------------- ### Train IGNNK on SeData Dataset Source: https://github.com/kaimaoge/ignnk/blob/master/README.md Command for training the IGNNK model on the SeData traffic dataset. Parameters such as --n_o, --h, --n_m, --n_u, and --max_iter can be adjusted. ```python python IGNNK_train.py "sedata" --n_o 240 --h 24 --n_m 80 --n_u 80 --max_iter 750 ``` -------------------------------- ### Download and Extract USHCN Data (Linux) Source: https://github.com/kaimaoge/ignnk/blob/master/data/ushcn/readme.txt Use wget to download the compressed data file and then tar to extract it. Ensure your tar command supports decompression, or use gzip -d followed by tar -xvf. ```bash wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tmax.latest.FLs.52i.tar.gz tar -zxvf ushcn.tmax.latest.FLs.52i.tar.gz ``` ```bash wget ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tmax.latest.FLs.52i.tar.gz gzip -d ushcn.tmax.latest.FLs.52i.tar.gz tar -xvf ushcn.tmax.latest.FLs.52i.tar.gz ``` -------------------------------- ### Import necessary libraries for IGNNK Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_METR-LA.ipynb Imports all required libraries for the IGNNK project, including PyTorch, NumPy, and custom utility functions. ```python from __future__ import division import torch import numpy as np import torch.optim as optim from torch import nn import matplotlib.pyplot as plt from utils import load_metr_la_rdata, get_normalized_adj, get_Laplace, calculate_random_walk_matrix,test_error import random import pandas as pd from basic_structure import D_GCN, C_GCN, K_GCN,IGNNK import geopandas as gp import matplotlib as mlt ``` -------------------------------- ### Load Baseline Results and Print Shapes Source: https://github.com/kaimaoge/ignnk/blob/master/IGNNK_U_Central_missing.ipynb Loads true data and results from a baseline method (GLTL). It then prints the shapes of the true data, GLTL results, IGNNK imputed data, and kNN imputed data to verify dimensions. ```python # Load baselines udata_true = X gltl = scipy.io.loadmat('model/gltl_result_udata_miss_area_central519_mu5_sigma0.05.mat') udata_gltl = gltl['sol_cokriging'] print(udata_gltl.shape) print(udata_ignnk.shape) print(udata_knn.shape) print(udata_true.shape) ``` -------------------------------- ### Load NREL Solar Energy Data Source: https://context7.com/kaimaoge/ignnk/llms.txt Loads NREL solar power generation data and retrieves station capacity information. ```python from utils import load_nerl_data # Load NREL dataset A, X, files_info = load_nerl_data() print(f"Adjacency matrix shape: {A.shape}") # (137, 137) print(f"Data shape: {X.shape}") # (137, 105120) - stations x time # Get station capacities for normalization capacities = files_info['capacity'].values.astype('float32') print(f"Number of stations: {len(capacities)}") print(f"Station info columns: {files_info.columns.tolist()}") # ['type', 'latitude', 'longitude', 'year', 'pv_type', 'capacity', 'time_interval', 'file_name'] ``` -------------------------------- ### Implement Chebyshev Graph Convolution (C_GCN) Source: https://context7.com/kaimaoge/ignnk/llms.txt Initializes the Chebyshev spectral graph convolution layer using graph Laplacian polynomials. ```python import torch from basic_structure import C_GCN from utils import get_Laplace import numpy as np # Initialize Chebyshev GCN layer in_channels = 24 out_channels = 64 orders = 3 # Chebyshev polynomial order cgcn = C_GCN(in_channels, out_channels, orders, activation='relu') ``` -------------------------------- ### Training progress output Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_METR-LA.ipynb Displays the Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Mean Absolute Percentage Error (MAPE) at specified training intervals. ```text 0 14.214141101715395 18.745514121355807 0.3673466003037573 50 7.6207559868909085 10.601529965891634 0.7976476629673228 100 7.140504566502854 10.312907491686309 0.8085156088490497 150 6.946970704095781 10.307552345120284 0.8087144200466962 200 7.083858814378775 10.310239258583136 0.8086146805989787 ``` -------------------------------- ### Load USHCN Weather Data Source: https://context7.com/kaimaoge/ignnk/llms.txt Loads USHCN weather data and reshapes it for kriging operations. ```python from utils import load_udata # Load USHCN dataset A, X, Omissing = load_udata() print(f"Adjacency matrix shape: {A.shape}") # (1218, 1218) print(f"Data shape: {X.shape}") # (1218, 120, 12, 2) - stations x years x months x features print(f"Missing mask shape: {Omissing.shape}") # Same as X, 1=observed, 0=missing # Reshape for kriging: (stations, time) X_flat = X[:, :, :, 0].reshape(1218, 120*12) # Precipitation only print(f"Flattened data shape: {X_flat.shape}") # (1218, 1440) ``` -------------------------------- ### Load Seattle Traffic Data Source: https://context7.com/kaimaoge/ignnk/llms.txt Loads Seattle loop detector traffic data. ```python from utils import load_sedata # Load Seattle dataset A, X = load_sedata() print(f"Adjacency matrix shape: {A.shape}") # (323, 323) print(f"Data shape: {X.shape}") # Time x nodes print(f"Data type: {X.dtype}") ``` -------------------------------- ### Import Libraries for Kriging Analysis Source: https://github.com/kaimaoge/ignnk/blob/master/IGNNK_U_Central_missing.ipynb Imports necessary libraries for data manipulation, plotting, and machine learning, including PyTorch, GeoPandas, NumPy, Pandas, Matplotlib, and custom utility functions. Sets up plotting configurations and warning filters. ```python from __future__ import division from torch import nn import geopandas as gp import numpy as np import pandas as pd from matplotlib import pyplot as plt import matplotlib as mlt from mpl_toolkits.axes_grid1 import make_axes_locatable import torch import random import copy import scipy.sparse as sp from utils import * import warnings warnings.filterwarnings('ignore') from basic_structure import D_GCN, C_GCN, K_GCN,IGNNK %matplotlib inline import seaborn as sns import heapq plt.rcParams['figure.figsize'] = (20, 10) ``` -------------------------------- ### Execute C_GCN Forward Pass Source: https://context7.com/kaimaoge/ignnk/llms.txt Demonstrates the forward pass for a Graph Convolutional Network using a Laplacian matrix. ```python # Prepare input: (batch_size, num_nodes, num_timesteps) batch_size = 4 num_nodes = 50 X = torch.randn(batch_size, num_nodes, in_channels) # Create Laplacian matrix A = np.random.rand(num_nodes, num_nodes).astype('float32') A = (A + A.T) / 2 A_laplace = torch.from_numpy(get_Laplace(A).astype('float32')) # Forward pass output = cgcn(X, A_laplace) print(f"C_GCN output shape: {output.shape}") # (4, 50, 64) ``` -------------------------------- ### Load METR-LA Traffic Data Source: https://context7.com/kaimaoge/ignnk/llms.txt Loads the METR-LA traffic speed dataset and splits it into training and test sets. ```python from utils import load_metr_la_rdata import numpy as np # Load METR-LA dataset A, X = load_metr_la_rdata() print(f"Adjacency matrix shape: {A.shape}") # (207, 207) print(f"Data shape: {X.shape}") # (207, 2, 34272) - nodes x features x time # Extract speed data and split train/test X_speed = X[:, 0, :] # First feature is speed split_line = int(X_speed.shape[1] * 0.7) training_set = X_speed[:, :split_line].T # (time, nodes) test_set = X_speed[:, split_line:].T print(f"Training samples: {training_set.shape[0]}") print(f"Test samples: {test_set.shape[0]}") ``` -------------------------------- ### Visualize Spatial Node Data with Matplotlib Source: https://github.com/kaimaoge/ignnk/blob/master/IGNNK_U_Central_missing.ipynb Generates a 2x4 grid of subplots to compare node estimation methods across different crowd scenarios. Requires pre-defined variables like meta_locations, udata_true, and various model-specific data arrays. ```python fig,axes = plt.subplots(2,4,figsize = (42,7)) lng_div = 0.02 lat_div = 0.01 crowd = [180,200] ylbs = ['High','Low'] s_known = 80 s_unknow = 120 x_max = 30 for row in range(2): for col in range(4): ax = axes[row,col] map_us.boundary.plot(ax=ax,color='black') ax.set_xlim((np.min(meta_locations['longitude'])-lng_div,np.max(meta_locations['longitude'])+lng_div)) ax.set_ylim((np.min(meta_locations['latitude'])-lat_div,np.max(meta_locations['latitude'])+lat_div)) ax.set_xticks([]) ax.set_yticks([]) if col == 0: cax=ax.scatter(meta_locations['longitude'][list(know_set)],meta_locations['latitude'][list(know_set)],s=s_known,cmap=plt.cm.RdYlGn_r, c = udata_true[crowd[row],list(know_set)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20 ),alpha=0.6,label='Known nodes') cax2=ax.scatter(meta_locations['longitude'][list(unknow_set_central)],meta_locations['latitude'][list(unknow_set_central)],s=s_unknow,cmap=plt.cm.RdYlGn_r,c=udata_true[crowd[row],list(unknow_set_central)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=1,marker='*',label = 'Unknown nodes') ax.set_ylabel(ylbs[row],fontsize=34) if row == 0: ax.set_title('True',fontsize = 36) elif col == 1: ax.scatter(meta_locations['longitude'][list(know_set)],meta_locations['latitude'][list(know_set)],s=s_known,cmap=plt.cm.RdYlGn_r, c = udata_true[crowd[row],list(know_set)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=0.6) ax.scatter(meta_locations['longitude'][list(unknow_set_central)],meta_locations['latitude'][list(unknow_set_central)],s=s_unknow,cmap=plt.cm.RdYlGn_r,c=udata_ignnc[crowd[row],list(unknow_set_central)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=1,marker='*') if row == 0: ax.set_title('IGNNK',fontsize = 36) elif col == 2: ax.scatter(meta_locations['longitude'][list(know_set)],meta_locations['latitude'][list(know_set)],s=s_known,cmap=plt.cm.RdYlGn_r, c = udata_true[crowd[row],list(know_set)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=0.6) ax.scatter(meta_locations['longitude'][list(unknow_set_central)],meta_locations['latitude'][list(unknow_set_central)],s=s_unknow,cmap=plt.cm.RdYlGn_r,c=udata_knn[crowd[row],list(unknow_set_central)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=1,marker='*') if row == 0: ax.set_title('kNN',fontsize = 36) else: ax.scatter(meta_locations['longitude'][list(know_set)],meta_locations['latitude'][list(know_set)],s=s_known,cmap=plt.cm.RdYlGn_r, c = udata_true[crowd[row],list(know_set)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=0.6) ax.scatter(meta_locations['longitude'][list(unknow_set_central)],meta_locations['latitude'][list(unknow_set_central)],s=s_unknow,cmap=plt.cm.RdYlGn_r,c=udata_gltl[crowd[row],list(unknow_set_central)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=1,marker='*') if row == 0: ax.set_title('GLTL',fontsize = 36) fig.tight_layout() fig.subplots_adjust(right = 0.87,hspace=0.01,wspace =0.01,bottom=0,top=1) l = 0.87 b = 0.03 w = 0.01 h = 0.70 rect = [l,b,w,h] cbar_ax = fig.add_axes(rect) cbar = fig.colorbar(cax, cax=cbar_ax) cbar.ax.tick_params(labelsize=30) plt.figlegend(handles=(cax,cax2),labels=('Known nodes','Unknown nodes'),bbox_to_anchor=(0.98, 1), loc=1, borderaxespad=0.,markerscale =3 ,fontsize = 30) plt.savefig('fig/spatial_ushcn_full_carea_noon{:}_evening{:}.pdf'.format(crowd[0],crowd[1])) plt.show() ``` -------------------------------- ### Plotting Spatial Data with Multiple Algorithms Source: https://github.com/kaimaoge/ignnk/blob/master/IGNNK_U_Central_missing.ipynb This code generates a grid of plots to visualize spatial data. It iterates through different algorithms (True, IGNNC, kNN, GLTL) and displays known and unknown nodes with varying sizes and color maps. Use this for comparing the performance of different imputation or prediction methods on spatial datasets. ```python fig,axes = plt.subplots(2,4,figsize = (42,7)) lng_div = 0.02 lat_div = 0.01 crowd = [180,200] ylbs = ['High','Low'] s_known = 80 s_unknow = 120 x_max = 30 for row in range(2): for col in range(4): ax = axes[row,col] map_us.boundary.plot(ax=ax,color='black') ax.set_xlim((np.min(meta_locations['longitude'].iloc[list(unknow_set_central)])-lng_div,np.max(meta_locations['longitude'].iloc[list(unknow_set_central)])+lng_div)) ax.set_ylim((np.min(meta_locations['latitude'].iloc[list(unknow_set_central)])-lat_div,np.max(meta_locations['latitude'].iloc[list(unknow_set_central)])+lat_div)) ax.set_xticks([]) ax.set_yticks([]) if col == 0: cax=ax.scatter(meta_locations['longitude'][list(know_set)],meta_locations['latitude'][list(know_set)],s=s_known,cmap=plt.cm.RdYlGn_r, c = udata_true[crowd[row],list(know_set)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20 ),alpha=0.6,label='Known nodes') cax2=ax.scatter(meta_locations['longitude'][list(unknow_set_central)],meta_locations['latitude'][list(unknow_set_central)],s=s_unknow,cmap=plt.cm.RdYlGn_r,c=udata_true[crowd[row],list(unknow_set_central)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=1,marker='*',label = 'Unknown nodes') ax.set_ylabel(ylbs[row],fontsize=34) if row == 0: ax.set_title('True',fontsize = 36) elif col == 1: ax.scatter(meta_locations['longitude'][list(know_set)],meta_locations['latitude'][list(know_set)],s=s_known,cmap=plt.cm.RdYlGn_r, c = udata_true[crowd[row],list(know_set)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=0.6) ax.scatter(meta_locations['longitude'][list(unknow_set_central)],meta_locations['latitude'][list(unknow_set_central)],s=s_unknow,cmap=plt.cm.RdYlGn_r,c=udata_ignnc[crowd[row],list(unknow_set_central)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=1,marker='*') if row == 0: ax.set_title('IGNNK',fontsize = 36) elif col == 2: ax.scatter(meta_locations['longitude'][list(know_set)],meta_locations['latitude'][list(know_set)],s=s_known,cmap=plt.cm.RdYlGn_r, c = udata_true[crowd[row],list(know_set)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=0.6) ax.scatter(meta_locations['longitude'][list(unknow_set_central)],meta_locations['latitude'][list(unknow_set_central)],s=s_unknow,cmap=plt.cm.RdYlGn_r,c=udata_knn[crowd[row],list(unknow_set_central)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=1,marker='*') if row == 0: ax.set_title('kNN',fontsize = 36) else: ax.scatter(meta_locations['longitude'][list(know_set)],meta_locations['latitude'][list(know_set)],s=s_known,cmap=plt.cm.RdYlGn_r, c = udata_true[crowd[row],list(know_set)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=0.6) ax.scatter(meta_locations['longitude'][list(unknow_set_central)],meta_locations['latitude'][list(unknow_set_central)],s=s_unknow,cmap=plt.cm.RdYlGn_r,c=udata_gltl[crowd[row],list(unknow_set_central)], norm=mlt.colors.Normalize(vmin=X.min(), vmax = 20),alpha=1,marker='*') if row == 0: ax.set_title('GLTL',fontsize = 36) fig.tight_layout() fig.subplots_adjust(right = 0.87,hspace=0.01,wspace =0.01,bottom=0,top=1) l = 0.87 b = 0.03 w = 0.01 h = 0.70 rect = [l,b,w,h] cbar_ax = fig.add_axes(rect) cbar = fig.colorbar(cax, cax=cbar_ax) cbar.ax.tick_params(labelsize=30) plt.figlegend(handles=(cax,cax2),labels=('Known nodes','Unknown nodes'),bbox_to_anchor=(0.98, 1), loc=1, borderaxespad=0.,markerscale =3 ,fontsize = 30) plt.savefig('fig/spatial_ushcn_central_carea_noon{:}_evening{:}.pdf'.format(crowd[0],crowd[1])) plt.show() ``` -------------------------------- ### Evaluate IGNNK Model Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_NREL.ipynb Calculate error metrics using the trained model and test dataset. ```python MAE_t, RMSE_t, R2_t, nrel_ignnk_res = test_error_cap(STmodel, unknow_set, full_set,test_set, A,h,capacities) print('Best model', MAE_t, RMSE_t, R2_t) ``` -------------------------------- ### Evaluate with Capacity Normalization Source: https://context7.com/kaimaoge/ignnk/llms.txt Evaluates model performance for datasets where normalization depends on station-specific capacities. ```python from utils import test_error_cap, load_nerl_data, calculate_random_walk_matrix from basic_structure import IGNNK import numpy as np # Load NREL data with capacities A, X, files_info = load_nerl_data() capacities = files_info['capacity'].values.astype('float32') # Normalize by capacity X_norm = X / capacities[:, None] test_set = X_norm[:, int(X.shape[1]*0.7):].T # Define unknown stations unknow_set = set(np.random.choice(137, 30, replace=False)) full_set = set(range(137)) # Create model model = IGNNK(h=24, z=100, k=1) time_dim = 24 # Evaluate with capacity scaling MAE, RMSE, R2, predictions = test_error_cap( STmodel=model, unknow_set=unknow_set, full_set=full_set, test_set=test_set, A=A, time_dim=time_dim, capacities=capacities ) print(f"NREL MAE: {MAE:.4f} MW") print(f"NREL RMSE: {RMSE:.4f} MW") ``` -------------------------------- ### Draw Temporal Information of METR-LA Kriging Source: https://github.com/kaimaoge/ignnk/blob/master/Demo_METR-LA.ipynb Compares the true traffic speed with INGNK predictions over a specific time period for a chosen station. Saves the plot to 'fig/metr_ignnk_temporal.pdf'. ```python fig,ax = plt.subplots(figsize = (16,5)) s = int(6400-64 ) e = int(s + 24*60/5+1) station = list(unknow_set)[24] ax.plot(test_set[s:e,station],label='True',linewidth=3) ax.plot(metr_ignnk_res[s:e,station],label='IGNNK',linewidth = 3) ax.set_ylabel('mile/h',fontsize=20) ax.tick_params(axis="x", labelsize=14) ax.tick_params(axis="y", labelsize=14) ax.set_xticks(range(0,350,50)) ax.set_xticklabels(['0:00\nMar 3rd','4:00','8:00','12:00','16:00','20:00','0:00\nMar 4th']) ax.legend(bbox_to_anchor=(1, 1), loc=0, borderaxespad=0,fontsize=16) plt.tight_layout() plt.savefig('fig/metr_ignnk_temporal.pdf') plt.show() ``` -------------------------------- ### Compute Random Walk Matrix Source: https://context7.com/kaimaoge/ignnk/llms.txt Computes the row-normalized random walk transition matrix for diffusion graph convolutions. ```python from utils import calculate_random_walk_matrix import numpy as np # Create sample adjacency matrix num_nodes = 100 A = np.random.rand(num_nodes, num_nodes).astype('float32') A = (A + A.T) / 2 # Symmetric A = np.exp(-A) # Distance-based weights # Compute random walk matrix: D^{-1} * A rw_matrix = calculate_random_walk_matrix(A) print(f"Random walk matrix shape: {rw_matrix.shape}") # (100, 100) print(f"Row sums (should be ~1): {rw_matrix.sum(axis=1)[:5]}") # For bidirectional diffusion, compute both directions A_forward = calculate_random_walk_matrix(A).T A_backward = calculate_random_walk_matrix(A.T).T ``` -------------------------------- ### Compute Symmetric Normalized Adjacency Source: https://context7.com/kaimaoge/ignnk/llms.txt Computes the symmetric normalized adjacency matrix for Kipf GCN. ```python from utils import get_normalized_adj import numpy as np # Create sample adjacency matrix num_nodes = 100 A = np.random.rand(num_nodes, num_nodes).astype('float32') A = (A + A.T) / 2 ``` -------------------------------- ### Display Matplotlib Figure Output Source: https://github.com/kaimaoge/ignnk/blob/master/IGNNK_U_Central_missing.ipynb Shows the standard output representation of the generated Matplotlib figure. ```text
```