### Install Dependencies
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Installs the necessary libraries, seaborn and ncps, using pip.
```bash
pip install seaborn ncps
```
--------------------------------
### Install Dependencies
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/torch_first_steps.rst
Installs necessary libraries including seaborn, ncps, torch, and pytorch-lightning using pip.
```bash
pip install seaborn ncps torch pytorch-lightning
```
--------------------------------
### Install Dependencies
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/mujoco_pomdp.rst
Installs necessary packages including ncps, tensorflow, ray[rllib], and gymnasium[mujoco] for reinforcement learning tasks.
```bash
pip3 install ncps tensorflow "ray[rllib]" "gymnasium[mujoco]"
```
--------------------------------
### Install Dependencies (PyTorch)
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_bc.rst
Installs necessary packages for PyTorch, including ncps, torch, ale-py, ray, and gym.
```bash
pip3 install ncps torch "ale-py==0.7.4" "ray[rllib]==2.1.0" "gym[atari,accept-rom-license]==0.23.1"
```
--------------------------------
### Install Neural Circuit Policies
Source: https://github.com/mlech26l/ncps/blob/master/docs/index.rst
Installs the Neural Circuit Policies (NCPs) package using pip. The `-U` flag ensures that the package is upgraded to the latest version if it's already installed.
```bash
pip3 install -U ncps
```
--------------------------------
### Installation
Source: https://github.com/mlech26l/ncps/blob/master/README.md
Provides the command to install the ncps package using pip.
```bash
pip install ncps
```
--------------------------------
### Install Dependencies for Atari RL
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_ppo.rst
Installs necessary Python packages for Atari reinforcement learning, including ncps, tensorflow, ale-py, ray[rllib], and gym[atari].
```bash
pip3 install ncps tensorflow "ale-py==0.7.4" "ray[rllib]==2.1.0" "gym[atari,accept-rom-license]==0.23.1"
```
--------------------------------
### Install Dependencies (TensorFlow)
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_bc.rst
Installs necessary packages for TensorFlow, including ncps, tensorflow, gymnasium, and ray. Note the compatibility notes regarding older versions.
```bash
pip3 install -U ncps tensorflow "gymnasium[atari,accept-rom-license]" "ray[rllib]"
pip3 install ncps tensorflow "ale-py==0.7.4" "ray[rllib]==2.1.0" "gym[atari,accept-rom-license]==0.23.1"
```
--------------------------------
### Example Training Output
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_ppo.rst
This is an example of the text output generated during the training of the Atari PPO agent. It shows the elapsed time, number of sampled steps, policy reward (mean), and the path to the saved checkpoint at various stages of the training process.
```text
> Ran 0.0 hours
> sampled 4k steps
> policy reward: nan
> saved checkpoint 'rl_ckpt/ALE/Breakout-v5/checkpoint-1'
> Ran 0.1 hours
> sampled 52k steps
> policy reward: 1.9
> saved checkpoint 'rl_ckpt/ALE/Breakout-v5/checkpoint-13'
> Ran 0.2 hours
> sampled 105k steps
> policy reward: 2.6
> saved checkpoint 'rl_ckpt/ALE/Breakout-v5/checkpoint-26'
> Ran 0.3 hours
> sampled 157k steps
> policy reward: 3.4
> saved checkpoint 'rl_ckpt/ALE/Breakout-v5/checkpoint-39'
> Ran 0.4 hours
> sampled 210k steps
> policy reward: 6.7
> saved checkpoint 'rl_ckpt/ALE/Breakout-v5/checkpoint-52'
> Ran 0.4 hours
> sampled 266k steps
> policy reward: 8.7
> saved checkpoint 'rl_ckpt/ALE/Breakout-v5/checkpoint-66'
> Ran 0.5 hours
> sampled 323k steps
> policy reward: 10.5
> saved checkpoint 'rl_ckpt/ALE/Breakout-v5/checkpoint-80'
> Ran 0.6 hours
> sampled 379k steps
> policy reward: 10.7
> saved checkpoint 'rl_ckpt/ALE/Breakout-v5/checkpoint-94'
```
--------------------------------
### PyTorch CfC Network Example
Source: https://github.com/mlech26l/ncps/blob/master/docs/index.rst
Demonstrates the creation and usage of a fully connected CfC (Contractive Functionally Connected) network in PyTorch. It shows how to initialize the CfC layer and pass input data through it.
```python
from ncps.torch import CfC
import torch
# a fully connected CfC network
rnn = CfC(input_size=20, units=50)
x = torch.randn(2, 3, 20) # (batch, time, features)
h0 = torch.zeros(2,50) # (batch, units)
output, hn = rnn(x,h0)
```
--------------------------------
### PyTorch Training Output Example
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_bc.rst
Example output from the PyTorch training script, showing progress bars for data loading, epoch-wise validation loss and accuracy, and the mean return from closed-loop evaluations.
```text
> loss=0.4349: 100%|██████████| 938/938 [01:35<00:00, 9.83it/s]
> Epoch 1, val_loss=1.67, val_acc=31.94%
> Mean return 0.2 (n=10)
> loss=0.2806: 100%|██████████| 938/938 [01:30<00:00, 10.33it/s]
> Epoch 2, val_loss=0.43, val_acc=83.51%
> Mean return 3.7 (n=10)
> loss=0.223: 100%|██████████| 938/938 [01:31<00:00, 10.30it/s]
> Epoch 3, val_loss=0.2349, val_acc=91.43%
> Mean return 4.9 (n=10)
> loss=0.1951: 100%|██████████| 938/938 [01:31<00:00, 10.26it/s]
```
--------------------------------
### Tensorflow LTC Model with AutoNCP Wiring
Source: https://github.com/mlech26l/ncps/blob/master/docs/index.rst
Provides an example of building a Tensorflow Keras model using the LTC (Liquid Time-Constant) layer with AutoNCP sparse wiring. This setup is suitable for tasks requiring recurrent neural network capabilities with biologically inspired connectivity.
```python
# Tensorflow example
from ncps.tf import LTC
from ncps.wirings import AutoNCP
import tensorflow as tf
wiring = AutoNCP(28, 4) # 28 neurons, 4 outputs
model = tf.keras.models.Sequential(
[
tf.keras.layers.InputLayer(input_shape=(None, 2)),
# LTC model with NCP sparse wiring
LTC(wiring, return_sequences=True),
]
)
```
--------------------------------
### Import Libraries
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/torch_first_steps.rst
Imports core libraries for numerical operations, neural networks, NCP models, and PyTorch Lightning.
```python
import numpy as np
import torch.nn as nn
from ncps.wirings import AutoNCP
from ncps.torch import LTC
import pytorch_lightning as pl
import torch
import torch.utils.data as data
```
--------------------------------
### PPO Atari Configuration and Training
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_ppo.rst
Sets up the PPO algorithm for Atari environments, including environment registration, hyperparameter configuration, and the main training loop. It also handles checkpoint loading.
```python
import argparse
import os
import gym
from ray.tune.registry import register_env
from ray.rllib.algorithms.ppo import PPO
import time
import ale_py
from ray.rllib.env.wrappers.atari_wrappers import wrap_deepmind
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--env", type=str, default="ALE/Breakout-v5")
parser.add_argument("--cont", default="")
parser.add_argument("--render", action="store_true")
parser.add_argument("--hours", default=4, type=int)
args = parser.parse_args()
register_env("atari_env", lambda env_config: wrap_deepmind(gym.make(args.env)))
config = {
"env": "atari_env",
"preprocessor_pref": None,
"gamma": 0.99,
"num_gpus": 1,
"num_workers": 16,
"num_envs_per_worker": 4,
"create_env_on_driver": True,
"lambda": 0.95,
"kl_coeff": 0.5,
"clip_rewards": True,
"clip_param": 0.1,
"vf_clip_param": 10.0,
"entropy_coeff": 0.01,
"rollout_fragment_length": 100,
"sgd_minibatch_size": 500,
"num_sgd_iter": 10,
"batch_mode": "truncate_episodes",
"observation_filter": "NoFilter",
"model": {
"vf_share_layers": True,
"custom_model": "cfc",
"max_seq_len": 20,
"custom_model_config": {
"cell_size": 64,
},
},
"framework": "tf2",
}
algo = PPO(config=config)
os.makedirs(f"rl_ckpt/{args.env}", exist_ok=True)
if args.cont != "":
algo.load_checkpoint(f"rl_ckpt/{args.env}/checkpoint-{args.cont}")
if args.render:
run_closed_loop(
algo,
config,
)
else:
start_time = time.time()
last_eval = 0
while True:
info = algo.train()
if time.time() - last_eval > 60 * 5: # every 5 minutes print some stats
print(f"Ran {(time.time()-start_time)/60/60:0.1f} hours")
print(
f" sampled {info['info']['num_env_steps_sampled']/1000:0.0f}k steps"
)
```
--------------------------------
### Model Summary
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Provides a summary of the constructed sequential model, detailing the layers, output shapes, and parameter counts.
```text
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
ltc (LTC) (None, None, 1) 350
=================================================================
Total params: 350
Trainable params: 350
Non-trainable params: 0
_________________________________________________________________
```
--------------------------------
### Import Libraries
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Imports essential libraries for building and training the NCP model, including numpy, tensorflow, and ncps components.
```python
import numpy as np
import os
from tensorflow import keras
from ncps import wirings
from ncps.tf import LTC
```
--------------------------------
### Configure LTC Model and Trainer
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/torch_first_steps.rst
Sets up the LTC model with an AutoNCP wiring, specifying 16 neurons and 1 output feature. It then creates a SequenceLearner instance and configures a PyTorch Lightning Trainer with a CSV logger, a maximum of 400 epochs, and gradient clipping for stability.
```python
out_features = 1
in_features = 2
wiring = AutoNCP(16, out_features) # 16 units, 1 motor neuron
ltc_model = LTC(in_features, wiring, batch_first=True)
learn = SequenceLearner(ltc_model, lr=0.01)
trainer = pl.Trainer(
logger=pl.loggers.CSVLogger("log"),
max_epochs=400,
gradient_clip_val=1, # Clip gradient to stabilize training
)
```
--------------------------------
### Visualize Prediction Before Training
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Shows the initial prediction of the LTC model on the synthetic data before any training has occurred, comparing it to the target output.
```python
# Let's visualize how LTC initialy performs before the training
sns.set()
prediction = model(data_x).numpy()
plt.figure(figsize=(6, 4))
plt.plot(data_y[0, :, 0], label="Target output")
plt.plot(prediction[0, :, 0], label="NCP output")
plt.ylim((-1, 1))
plt.title("Before training")
plt.legend(loc="upper right")
plt.show()
```
--------------------------------
### NCPS RNN Model Forward Pass and Initial State
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/mujoco_pomdp.rst
Implements the forward pass for the RNN model and provides a method to get the initial state. The forward pass utilizes the `rnn_model` to compute outputs and states, while `get_initial_state` returns a zero-initialized state vector.
```python
@override(RecurrentNetwork)
def forward_rnn(self, inputs, state, seq_lens):
model_out, self._value_out, h = self.rnn_model([inputs, seq_lens] + state)
return model_out, [h]
@override(ModelV2)
def get_initial_state(self):
return [
np.zeros(self.cell_size, np.float32),
]
```
--------------------------------
### Train the Model (Python)
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/torch_first_steps.rst
This code snippet demonstrates how to train the model using a trainer object and a dataloader. The training is set for 400 epochs, which also corresponds to the number of training steps.
```python
# Train the model for 400 epochs (= training steps)
trainer.fit(learn, dataloader)
```
--------------------------------
### Draw NCP Wiring Diagram
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Visualizes the wiring diagram of the NCP network using seaborn and matplotlib, with labels and neuron colors for clarity.
```python
sns.set_style("white")
plt.figure(figsize=(6, 4))
legend_handles = wiring.draw_graph(draw_labels=True, neuron_colors={"command": "tab:cyan"})
plt.legend(handles=legend_handles, loc="upper center", bbox_to_anchor=(1, 1))
sns.despine(left=True, bottom=True)
plt.tight_layout()
plt.show()
```
--------------------------------
### Train the LTC Model
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Trains the LTC model for 400 epochs using the generated synthetic data. The training progress, including the loss at each epoch, is displayed.
```python
# Train the model for 400 epochs (= training steps)
hist = model.fit(x=data_x, y=data_y, batch_size=1, epochs=400,verbose=1)
```
--------------------------------
### Generate Synthetic Sinusoidal Data
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Generates synthetic training data consisting of sine and cosine waves for input and a double-frequency sine wave for the target output. It also visualizes this data.
```python
import matplotlib.pyplot as plt
import seaborn as sns
N = 48 # Length of the time-series
# Input feature is a sine and a cosine wave
data_x = np.stack(
[np.sin(np.linspace(0, 3 * np.pi, N)), np.cos(np.linspace(0, 3 * np.pi, N))], axis=1
)
data_x = np.expand_dims(data_x, axis=0).astype(np.float32) # Add batch dimension
# Target output is a sine with double the frequency of the input signal
data_y = np.sin(np.linspace(0, 6 * np.pi, N)).reshape([1, N, 1]).astype(np.float32)
print("data_x.shape: ", str(data_x.shape))
print("data_y.shape: ", str(data_y.shape))
# Let's visualize the training data
sns.set()
plt.figure(figsize=(6, 4))
plt.plot(data_x[0, :, 0], label="Input feature 1")
plt.plot(data_x[0, :, 1], label="Input feature 1")
plt.plot(data_y[0, :, 0], label="Target output")
plt.ylim((-1, 1))
plt.title("Training data")
plt.legend(loc="upper right")
plt.show()
```
--------------------------------
### PyTorch Lightning Sequence Learner Module
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/torch_first_steps.rst
Defines a custom PyTorch Lightning module for training sequence models. It includes methods for handling training, validation, and testing steps, calculating Mean Squared Error (MSE) loss, and configuring the Adam optimizer.
```python
# LightningModule for training a RNNSequence module
class SequenceLearner(pl.LightningModule):
def __init__(self, model, lr=0.005):
super().__init__()
self.model = model
self.lr = lr
def training_step(self, batch, batch_idx):
x, y = batch
y_hat, _ = self.model.forward(x)
y_hat = y_hat.view_as(y)
loss = nn.MSELoss()(y_hat, y)
self.log("train_loss", loss, prog_bar=True)
return {"loss": loss}
def validation_step(self, batch, batch_idx):
x, y = batch
y_hat, _ = self.model.forward(x)
y_hat = y_hat.view_as(y)
loss = nn.MSELoss()(y_hat, y)
self.log("val_loss", loss, prog_bar=True)
return loss
def test_step(self, batch, batch_idx):
# Here we just reuse the validation_step for testing
return self.validation_step(batch, batch_idx)
def configure_optimizers(self):
return torch.optim.Adam(self.model.parameters(), lr=self.lr)
```
--------------------------------
### Training Progress
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Logs the training progress, showing the loss for each epoch during the model training process.
```text
Epoch 1/400
1/1 [==============================] - 6s 6s/step - loss: 0.4980
Epoch 2/400
1/1 [==============================] - 0s 55ms/step - loss: 0.4797
Epoch 3/400
1/1 [==============================] - 0s 54ms/step - loss: 0.4686
Epoch 4/400
1/1 [==============================] - 0s 54ms/step - loss: 0.4590
```
--------------------------------
### Define and Compile LTC Model
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Defines the LTC model using AutoNCP wiring with 8 neurons and 1 output. The model is then compiled with the Adam optimizer and mean squared error loss.
```python
wiring = wirings.AutoNCP(8,1) # 8 neurons in total, 1 output (motor neuron)
model = keras.models.Sequential(
[
keras.layers.InputLayer(input_shape=(None, 2)),
# here we could potentially add layers before and after the LTC network
LTC(wiring, return_sequences=True),
]
)
model.compile(
optimizer=keras.optimizers.Adam(0.01),
loss='mean_squared_error'
)
model.summary()
```
--------------------------------
### Compare Model Prediction After Training
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Compares the trained model's prediction against the target output for a sinusoidal function. It plots both the target output and the model's output, requiring the model, data_x, and data_y to be defined.
```python
import matplotlib.pyplot as plt
# How does the trained model now fit to the sinusoidal function?
prediction = model(data_x).numpy()
plt.figure(figsize=(6, 4))
plt.plot(data_y[0, :, 0], label="Target output")
plt.plot(prediction[0, :, 0], label="LTC output",linestyle="dashed")
plt.ylim((-1, 1))
plt.legend(loc="upper right")
plt.title("After training")
plt.show()
```
--------------------------------
### BibTeX Entry
Source: https://github.com/mlech26l/ncps/blob/master/README.md
BibTeX citation for the 'Neural circuit policies enabling auditable autonomy' paper.
```bibtex
@article{lechner2020neural,
title={Neural circuit policies enabling auditable autonomy},
author={Lechner, Mathias and Hasani, Ramin and Amini, Alexander and Henzinger, Thomas A and Rus, Daniela and Grosu, Radu},
journal={Nature Machine Intelligence},
volume={2},
number={10},
pages={642--652},
year={2020},
publisher={Nature Publishing Group}
}
```
--------------------------------
### Augmentation Utilities
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
Provides utility functions for performing shadow augmentation and sample weighting, shared across both active and passive training pipelines.
```python
import augmentation_utils
# Example usage:
augmented_data = augmentation_utils.apply_shadow_augmentation(data)
weighted_samples = augmentation_utils.apply_sample_weighting(data, weights)
```
--------------------------------
### Saliency Widget Visualization
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
HTML visualization tool to inspect the attention maps of all active test recordings. This allows for interactive exploration of where the model is 'looking' during the active steering tests.
```html
Saliency Map Viewer
Saliency Map Inspection
Original Image
```
--------------------------------
### Wirings API Documentation
Source: https://github.com/mlech26l/ncps/blob/master/docs/api/index.rst
API documentation for the Wirings module in the ncps project. This section explains how to define and manage connections and configurations.
```APIDOC
wirings:
This section provides the API reference for the Wirings module.
It details the methods for configuring and managing system wirings and connections.
Key functionalities include defining network topologies and component interdependencies.
```
--------------------------------
### PyTorch CfC and LTC with AutoNCP Wiring
Source: https://github.com/mlech26l/ncps/blob/master/README.md
Demonstrates initializing PyTorch CfC and LTC models using the AutoNCP wiring strategy, which automatically determines the network structure.
```python
from ncps.torch import CfC, LTC
from ncps.wirings import AutoNCP
wiring = AutoNCP(28, 4) # 28 neurons, 4 outputs
input_size = 20
rnn = CfC(input_size, wiring)
rnn = LTC(input_size, wiring)
```
--------------------------------
### End-to-End RNN Wrapper (models/e2e_rnn.py)
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
This script provides a wrapper to make various RNN models (implemented in models/rnn_models.py) compatible with the project's training pipeline. It ensures that different RNN architectures can be used interchangeably within the training framework.
```python
from models.e2e_rnn import E2E_RNN_Wrapper
# Example usage:
wrapped_gru = E2E_RNN_Wrapper(gru_model_instance)
```
--------------------------------
### PyTorch CfC and LTC Model Initialization
Source: https://github.com/mlech26l/ncps/blob/master/README.md
Shows how to initialize both the Closed-Form Continuous-Time (CfC) and Liquid Time-Constant (LTC) models in PyTorch with specified input size and number of units.
```python
from ncps.torch import CfC, LTC
input_size = 20
units = 28 # 28 neurons
rnn = CfC(input_size, units)
rnn = LTC(input_size, units)
```
--------------------------------
### Plot Training Loss
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Visualizes the training loss over the training steps using a line plot. It requires the 'seaborn' and 'matplotlib.pyplot' libraries and assumes a 'hist' object containing training history is available.
```python
import seaborn as sns
import matplotlib.pyplot as plt
# Let's visualize the training loss
sns.set()
plt.figure(figsize=(6, 4))
plt.plot(hist.history["loss"], label="Training loss")
plt.legend(loc="upper right")
plt.xlabel("Training steps")
plt.show()
```
--------------------------------
### Run and Compare MLP and CfC Policies
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/mujoco_pomdp.rst
This code snippet registers custom models and environments, runs training for both MLP and CfC policies, and then plots their performance, including reward and reward with noise. It highlights the comparison between the two architectures.
```python
if __name__ == "__main__":
ModelCatalog.register_custom_model("cfc_rnn", CustomRNN)
register_env("my_env", lambda env_config: make_partial_observation_cheetah())
ray.init(num_cpus=24, num_gpus=1)
cfc_result = run_algo("cfc_rnn", 1000)
ray.shutdown()
ModelCatalog.register_custom_model("cfc_rnn", CustomRNN)
register_env("my_env", lambda env_config: make_partial_observation_cheetah())
ray.init(num_cpus=24, num_gpus=1)
mlp_result = run_algo("default", 1000)
fig, ax = plt.subplots(figsize=(10, 6))
ax.plot(
mlp_result["iteration"], mlp_result["reward"], label="MLP", color="tab:orange"
)
ax.plot(
cfc_result["iteration"], cfc_result["reward"], label="CfC", color="tab:blue"
)
ax.plot(
mlp_result["iteration"],
mlp_result["reward_noise"],
label="MLP (noise)",
color="tab:orange",
ls="--",
)
ax.plot(
cfc_result["iteration"],
cfc_result["reward_noise"],
label="CfC (noise)",
color="tab:blue",
ls="--",
)
ax.legend(loc="upper left")
fig.tight_layout()
plt.savefig("cfc_vs_mlp.png")
```
--------------------------------
### Atari PPO Training Loop and Checkpointing
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_ppo.rst
This Python code snippet demonstrates the core training loop for a Proximal Policy Optimization (PPO) agent on Atari environments. It includes logic for periodic evaluation, saving checkpoints, and monitoring training progress based on elapsed time.
```python
ckpt = algo.save_checkpoint(f"rl_ckpt/{args.env}")
print(f" saved checkpoint '{ckpt}'")
elapsed = (time.time() - start_time) / 60 # in minutes
if elapsed > args.hours * 60:
break
```
--------------------------------
### PyTorch AtariCloningDataset
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_bc.rst
Loads and prepares the Atari cloning dataset for PyTorch using `AtariCloningDataset`. It shows how to create training and validation datasets and then load them into PyTorch DataLoaders with specified batch sizes and number of workers.
```python
import torch
from torch.utils.data import Dataset
import torch.optim as optim
from ncps.datasets.torch import AtariCloningDataset
train_ds = AtariCloningDataset("breakout", split="train")
val_ds = AtariCloningDataset("breakout", split="val")
trainloader = torch.utils.data.DataLoader(
train_ds, batch_size=32, num_workers=4, shuffle=True
)
valloader = torch.utils.data.DataLoader(val_ds, batch_size=32, num_workers=4)
```
--------------------------------
### Visualize Trained Model Performance (Python)
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/torch_first_steps.rst
This snippet visualizes the performance of the LTC model after training. It plots the target output against the model's prediction, similar to the pre-training visualization, to show the improvement. It uses seaborn, torch, and matplotlib, with the y-axis limited to -1 to 1.
```python
# How does the trained model now fit to the sinusoidal function?
sns.set()
with torch.no_grad():
prediction = ltc_model(data_x)[0].numpy()
plt.figure(figsize=(6, 4))
plt.plot(data_y[0, :, 0], label="Target output")
plt.plot(prediction[0, :, 0], label="NCP output")
plt.ylim((-1, 1))
plt.title("After training")
plt.legend(loc="upper right")
plt.show()
```
--------------------------------
### Visualize LTC Model Performance Before Training (Python)
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/torch_first_steps.rst
This snippet visualizes the initial performance of the LTC model by plotting the target output against the model's prediction before any training occurs. It uses libraries like seaborn, torch, and matplotlib for plotting and model inference. The output is limited to a y-axis range of -1 to 1.
```python
# Let's visualize how LTC initialy performs before the training
sns.set()
with torch.no_grad():
prediction = ltc_model(data_x)[0].numpy()
plt.figure(figsize=(6, 4))
plt.plot(data_y[0, :, 0], label="Target output")
plt.plot(prediction[0, :, 0], label="NCP output")
plt.ylim((-1, 1))
plt.title("Before training")
plt.legend(loc="upper right")
plt.show()
```
--------------------------------
### End-to-End Worm Pilot Implementation (models/e2e_worm_pilot.py)
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
A wrapper script that makes the NCP model (implemented in wormflow3.py) compatible with the project's training pipeline. This allows the NCP model to be used seamlessly within the training and evaluation framework.
```python
from models.e2e_worm_pilot import E2E_WormPilot
# Example usage:
e2e_ncp_wrapper = E2E_WormPilot(ncp_model_instance)
```
--------------------------------
### Atari Environment Wrapper
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_bc.rst
Demonstrates how to wrap a gym Atari environment using `wrap_deepmind` from Ray RLlib. This process includes downscaling frames to 84x84, converting to grayscale, and stacking 4 consecutive frames for a complete observation. The resulting observation is an 84x84 image with 4 channels.
```python
import gym
import ale_py
from ray.rllib.env.wrappers.atari_wrappers import wrap_deepmind
import numpy as np
env = gym.make("ALE/Breakout-v5")
# We need to wrap the environment with the Deepmind helper functions
env = wrap_deepmind(env)
```
--------------------------------
### Create Partial Observation Environment
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/mujoco_pomdp.rst
Factory function to create a HalfCheetah-v4 environment with a partial observation space, excluding joint velocities.
```python
def make_partial_observation_cheetah():
return PartialObservation(
gymnasium.make("HalfCheetah-v4"), [0, 1, 2, 3, 8, 9, 10, 11, 12]
)
```
--------------------------------
### Visualize Atari Game Play
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_bc.rst
Visualizes the trained model playing an Atari game (Breakout) in a human-readable mode. It uses the `gym` library to create the environment and `wrap_deepmind` for environment wrapping.
```python
# Visualize Atari game and play endlessly
env = gym.make("ALE/Breakout-v5", render_mode="human")
env = wrap_deepmind(env)
run_closed_loop(model, env)
```
--------------------------------
### Synthetic Data Shapes
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/tf_first_steps.rst
Displays the shapes of the generated input and target data.
```text
data_x.shape: (1, 48, 2)
data_y.shape: (1, 48, 1)
```
--------------------------------
### Active Test Data Provider
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
Loads the training data specifically for the active test scenario. This script is part of the active training pipeline.
```python
import active_data_provider
# Example usage:
data = active_data_provider.load_data()
```
--------------------------------
### Training Progress Output (Text)
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/torch_first_steps.rst
This represents the typical output logged during the model training process, showing the progress of training steps, loss values, and validation numbers.
```text
.... 1/1 [00:00<00:00, 4.91it/s, loss=0.000459, v_num=0, train_loss=0.000397]
```
--------------------------------
### Perspective Transformation
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
Contains code for cropping and adjusting input images before they are processed by the neural network models. This is a shared utility.
```python
import perspective_transformation
# Example usage:
processed_image = perspective_transformation.crop_and_adjust(image)
```
--------------------------------
### PyTorch CfC Model Initialization and Usage
Source: https://github.com/mlech26l/ncps/blob/master/README.md
Demonstrates how to initialize a Closed-Form Continuous-Time (CfC) model in PyTorch and pass input data through it. It shows the expected input and output shapes for the RNN layer.
```python
import torch
from ncps.torch import CfC
rnn = CfC(20,50) # (input, hidden units)
x = torch.randn(2, 3, 20) # (batch, time, features)
h0 = torch.zeros(2,50) # (batch, units)
output, hn = rnn(x,h0)
```
--------------------------------
### RNN Models Implementation (models/rnn_models.py)
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
Provides implementations for various Recurrent Neural Network (RNN) architectures, including Vanilla RNN, CT-RNN, GRU, and CT-GRU. These models are designed to be integrated into the training pipeline.
```python
from models.rnn_models import VanillaRNN, CTGRU
# Example usage:
vanilla_rnn = VanillaRNN(input_size=10, hidden_size=20)
ct_gru = CTGRU(input_size=10, hidden_size=20)
```
--------------------------------
### Partial Observation Wrapper
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/mujoco_pomdp.rst
Defines a custom Gymnasium ObservationWrapper to create a partially observable environment by filtering observations. It selects specific indices from the original observation space.
```python
import gymnasium
from gymnasium import spaces
import ray
from ray.tune.registry import register_env
from ray.rllib.models import ModelCatalog
from ray.rllib.algorithms.ppo import PPO
import time
import numpy as np
from ray.rllib.models.modelv2 import ModelV2
from ray.rllib.models.tf.recurrent_net import RecurrentNetwork
from ray.rllib.utils.annotations import override
import tensorflow as tf
import ncps.tf
import matplotlib.pyplot as plt
class PartialObservation(gymnasium.ObservationWrapper):
def __init__(self, env: gymnasium.Env, obs_indices: list):
gymnasium.ObservationWrapper.__init__(self, env)
obsspace = env.observation_space
self.obs_indices = obs_indices
self.observation_space = spaces.Box(
low=np.array([obsspace.low[i] for i in obs_indices]),
high=np.array([obsspace.high[i] for i in obs_indices]),
dtype=np.float32,
)
self._env = env
def observation(self, observation):
filter_observation = self._filter_observation(observation)
return filter_observation
def _filter_observation(self, observation):
observation = np.array([observation[i] for i in self.obs_indices])
return observation
```
--------------------------------
### Wiring Class Documentation
Source: https://github.com/mlech26l/ncps/blob/master/docs/api/wirings.rst
Provides the base class for all wiring strategies in ncps.wirings. It defines the fundamental interface and common attributes for creating custom wiring configurations.
```APIDOC
class ncps.wirings.Wiring
"""Base class for all wiring strategies."""
__init__(self, n_in: int, n_out: int, n_hidden: int, **kwargs)
Initializes the Wiring object.
Parameters:
n_in: Number of input neurons.
n_out: Number of output neurons.
n_hidden: Number of hidden neurons.
__call__(self, n_in: int, n_out: int, n_hidden: int, **kwargs)
Abstract method to define the wiring logic.
Should be implemented by subclasses.
Parameters:
n_in: Number of input neurons.
n_out: Number of output neurons.
n_hidden: Number of hidden neurons.
Returns:
A tuple containing the connection matrix (torch.Tensor), input indices (torch.Tensor), and output indices (torch.Tensor).
get_connections(self, n_in: int, n_out: int, n_hidden: int, **kwargs)
Helper method to generate connections based on the wiring strategy.
Parameters:
n_in: Number of input neurons.
n_out: Number of output neurons.
n_hidden: Number of hidden neurons.
Returns:
A tuple containing the connection matrix (torch.Tensor), input indices (torch.Tensor), and output indices (torch.Tensor).
```
--------------------------------
### Closed-Loop Policy Visualization
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/atari_ppo.rst
Defines a function to visualize the trained policy's interaction with the Atari environment. It handles environment rendering, state initialization, and action computation using the trained algorithm.
```python
def run_closed_loop(algo, config):
env = gym.make(args.env, render_mode="human")
env = wrap_deepmind(env)
rnn_cell_size = config["model"]["custom_model_config"]["cell_size"]
obs = env.reset()
state = init_state = [np.zeros(rnn_cell_size, np.float32)]
while True:
action, state, _ = algo.compute_single_action(
obs, state=state, explore=False, policy_id="default_policy"
)
obs, reward, done, _ = env.step(action)
if done:
obs = env.reset()
state = init_state
```
--------------------------------
### Train Policy Network (PPO)
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/mujoco_pomdp.rst
Trains a policy network using the PPO algorithm. This function configures the training process, allowing for either a 'cfc_rnn' model or a 'default' MLP baseline. It includes parameters for environment, optimizer, and network architecture, and logs performance metrics during training.
```python
def run_algo(model_name, num_iters):
config = {
"env": "my_env",
"gamma": 0.99,
"num_gpus": 1,
"num_workers": 16,
"num_envs_per_worker": 4,
"lambda": 0.95,
"kl_coeff": 1.0,
"num_sgd_iter": 64,
"lr": 0.0005,
"vf_loss_coeff": 0.5,
"clip_param": 0.1,
"sgd_minibatch_size": 4096,
"train_batch_size": 65536,
"grad_clip": 0.5,
"batch_mode": "truncate_episodes",
"observation_filter": "MeanStdFilter",
"framework": "tf",
}
rnn_cell_size = None
if model_name == "cfc_rnn":
rnn_cell_size = 64
config["model"] = {
"vf_share_layers": True,
"custom_model": "cfc_rnn",
"custom_model_config": {
"cell_size": rnn_cell_size,
},
}
elif model_name == "default":
pass
else:
raise ValueError(f"Unknown model type {model_name}")
algo = PPO(config=config)
history = {"reward": [], "reward_noise": [], "iteration": []}
for iteration in range(1, num_iters + 1):
algo.train()
if iteration % 10 == 0 or iteration == 1:
history["iteration"].append(iteration)
history["reward"].append(run_closed_loop(algo, rnn_cell_size))
history["reward_noise"].append(
run_closed_loop(algo, rnn_cell_size, pertubation_level=0.1)
)
print(
```
--------------------------------
### Tensorflow NCP Layers
Source: https://github.com/mlech26l/ncps/blob/master/README.md
Demonstrates the instantiation of various Neural Circuit Policies (NCP) layers, including CfC (Closed-Form Continuous-time) and LTC (Leaky-Integrate-and-Fire with Continuous-time dynamics), with different wiring configurations using the ncps.tf module.
```python
from ncps.tf import CfC, LTC
from ncps.wirings import AutoNCP
units = 28
wiring = AutoNCP(28, 4) # 28 neurons, 4 outputs
input_size = 20
rnn1 = LTC(units) # fully-connected LTC
rnn2 = CfC(units) # fully-connected CfC
rnn3 = LTC(wiring) # NCP wired LTC
rnn4 = CfC(wiring) # NCP wired CfC
```
--------------------------------
### Draw NCP Wiring Diagram
Source: https://github.com/mlech26l/ncps/blob/master/docs/examples/torch_first_steps.rst
Visualizes the network's wiring diagram using seaborn and matplotlib. It displays the connections between neurons, highlighting the command neurons in cyan, and includes a legend for clarity.
```python
sns.set_style("white")
plt.figure(figsize=(6, 4))
legend_handles = wiring.draw_graph(draw_labels=True, neuron_colors={"command": "tab:cyan"})
plt.legend(handles=legend_handles, loc="upper center", bbox_to_anchor=(1, 1))
sns.despine(left=True, bottom=True)
plt.tight_layout()
plt.show()
```
--------------------------------
### Passive Test Data Provider
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
Loads the training data for the passive test and handles the splitting required for cross-testing evaluation. This script is unique to the passive training pipeline.
```python
import passive_test_data_provider
# Example usage:
data, labels = passive_test_data_provider.load_and_split_data()
```
--------------------------------
### Train Active Test
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
The main script for training models intended for the active test. It utilizes the active data provider.
```python
import train_active_test
# Example usage:
train_active_test.train_model()
```
--------------------------------
### List Passive Results
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
Summarizes the results from passive evaluation runs by averaging over experiment IDs. It processes logs located in the 'passive_sessions' directory.
```python
python3 list_passive_results.py
```
--------------------------------
### Torch API Documentation
Source: https://github.com/mlech26l/ncps/blob/master/docs/api/index.rst
Detailed API documentation for Torch functionalities within the ncps project. This section covers the various functions, classes, and modules available for Torch integration.
```APIDOC
torch:
This section details the Torch API available in the ncps project.
It includes functions for tensor operations, neural network layers, and optimization algorithms.
Refer to individual sub-sections for specific details on each component.
```
--------------------------------
### CNN Model Implementation (models/cnn_model.py)
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
Implements a baseline feedforward convolutional neural network (CNN) model. This implementation is compatible with the project's training pipeline.
```python
from models.cnn_model import CNNModel
# Example usage:
cnn = CNNModel(input_shape=(64, 64, 3), num_classes=10)
output = cnn(image)
```
--------------------------------
### Perturbation Analysis - SSIM and Crash Plot
Source: https://github.com/mlech26l/ncps/blob/master/reproducibility/README.md
Computes the Structural Similarity Index (SSIM) for saliency maps with increasing input noise and plots the number of crashes against input noise variance for RNNs.
```matlab
SSIM_Crash_analysis/ssim_and_crash_plot.m
```