### Build and start the database container

Source: https://ubc-provenance.github.io/PIDSMaker/ten-minute-install

Builds and starts the PostgreSQL container using docker compose. Remember to source .env after modifying variables.

```bash
docker compose -p postgres -f compose-postgres.yml up -d --build
```

--------------------------------

### Start containers and enter pidsmaker shell

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Starts the postgres and pidsmaker containers and opens a shell inside the pidsmaker container to verify that the databases are built and filled.

```bash
docker compose -p postgres -f compose-postgres.yml up -d --build
docker compose -f compose-pidsmaker.yml up -d --build
docker compose exec pids bash

```

--------------------------------

### Run the visualizer server: pidsmaker.vizgen.web.viz_server

Source: https://ubc-provenance.github.io/PIDSMaker/features/visualization

Starts the Flask server inside the container. Open http://127.0.0.1:5000 afterwards. The last viewed run loads automatically; use the Run Browser to pick another. If localhost is refused but 127.0.0.1 works, bind to IPv4 only; start with --host :: to serve both.

```shell
python -m pidsmaker.vizgen.web.viz_server

```

--------------------------------

### Set up database with Makefile

Source: https://ubc-provenance.github.io/PIDSMaker/singularity_install

Run the full setup from the scripts/apptainer directory. This downloads and runs a PostgreSQL container via Singularity/Apptainer and loads database dumps using load_dumps.sh.

```bash
cd ./scripts/apptainer
make full-setup
```

--------------------------------

### Install NVIDIA Container Toolkit for Docker CUDA support

Source: https://ubc-provenance.github.io/PIDSMaker/ten-minute-install

Adds the NVIDIA package repository, installs the nvidia-container-toolkit, and restarts Docker services to enable GPU support.

```bash
# Add the NVIDIA package repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Update and install
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Restart services
sudo systemctl restart docker
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```

--------------------------------

### Install dependencies with conda

Source: https://ubc-provenance.github.io/PIDSMaker/singularity_install

Create a conda environment from the provided environment.yml file and activate it.

```bash
conda env create -f ./scripts/apptainer/environment.yml
conda activate pids
```

--------------------------------

### Manage container with make down and make up

Source: https://ubc-provenance.github.io/PIDSMaker/singularity_install

Stop the container with 'make down' and start it again with 'make up'.

```bash
make down
```

```bash
make up
```

--------------------------------

### Cross-dataset tuning with --tuning_file_path

Source: https://ubc-provenance.github.io/PIDSMaker/features/tuning

Commands to apply the same tuning configuration across all datasets. The first runs on all datasets, the second on a single dataset. The path passed to --tuning_file_path must start from systems/.

```bash
./run_all_datasets.py my_system \
    --tuning_mode=hyperparameters \
    --tuning_file_path=systems/default/tuning_my_system_all_datasets

```

```bash
./run.sh my_system CLEARSCOPE_E3 \
    --tuning_mode=hyperparameters \
    --tuning_file_path=systems/default/tuning_my_system_all_datasets

```

--------------------------------

### Install gdown for dataset download

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Installs gdown, the Google Drive download utility used to fetch the official public DARPA dataset files.

```bash
pip install gdown

```

--------------------------------

### Start and enter PIDSMaker container manually

Source: https://ubc-provenance.github.io/PIDSMaker/ten-minute-install

Starts the PIDSMaker container in detached mode with build and opens a bash shell inside it. Use this manual approach instead of the VSCode Devcontainer extension.

```bash
docker compose -f compose-pidsmaker.yml up -d --build
docker compose exec pids bash

```

--------------------------------

### Run PIDSmaker on DARPA OpTC optc_h051

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Executes PIDSmaker on the optc_h051 dataset, which includes a malicious Notepad++ update installing Meterpreter, SYSTEM escalation, LSASS migration for Mimikatz, persistence, timestomping, and RDP admin account creation.

```bash
python pidsmaker/main.py SYSTEM optc_h051
```

--------------------------------

### Configure run_n_times experiment in run_n_times.yml

Source: https://ubc-provenance.github.io/PIDSMaker/features/instability

Configuration for the `run_n_times` experiment, defined in `config/experiments/uncertainty/run_n_times.yml`. `iterations` sets the number of runs (default 5), and `restart_from` starts each iteration from the featurization stage, reusing earlier stages like construction.

```yaml
training_loop:
  run_evaluation: each_epoch

experiment:
  used_method: uncertainty
  uncertainty:
    deep_ensemble:
      iterations: 5        # number of runs
      restart_from: featurization  # task to restart from

```

--------------------------------

### Define dataset configuration in pidsmaker/config/config.py

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Add a new dataset by defining its configuration in the DATASET_DEFAULT_CONFIG dictionary. The configuration includes raw directory, database names, number of node and edge types, start and end dates, and lists of dates for training, validation, testing, and unused dates. It also specifies the ground truth relative path and attack-to-time-window mappings.

```python
DATASET_DEFAULT_CONFIG = {
    "MY_DATASET": {
        "raw_dir": "",
        "database": "my_database_name",
        "database_all_file": "my_database_name",
        "num_node_types": 3, # Number of node types in the dataset __format__ (i.e., in pidsmaker/utils/dataset_utils.py)
        "num_edge_types": 10, # Number of edge types in the dataset __format__ (i.e., in pidsmaker/utils/dataset_utils.py)
        "start_date": "2018-04-02", # Start date (Inclusive)
        "end_date": "2018-04-14", # End date (Exclusive)
        "train_dates": [
            # Dates/graphs used for training (i.e., benign activity)
            "2018-04-02",
            "2018-04-03",
            "2018-04-04",
            "2018-04-05",
            "2018-04-07",
            "2018-04-08",
            "2018-04-09",
        ],
        "val_dates": [
            # Dates/graphs used for validation/threshold calibration (i.e., benign activity)
            "2018-04-10"
        ],
        "test_dates": [
            # Dates/graphs used for testing (i.e., contains both benign and attack activity)
            "2018-04-06",
            "2018-04-11",
            "2018-04-12",
            "2018-04-13"
        ],
        "unused_dates": [
            # Any unused dates/graphs that should be ignored
             "2018-04-14"
        ],
        "ground_truth_relative_path": ["MY_DATASET/labels.csv"],
        "attack_to_time_window": [
            ["MY_DATASET/labels.csv", "2018-04-11 10:00:00", "2018-04-12 12:00:00"],
        ],
    },
}

```

--------------------------------

### Get a shell into the postgres container

Source: https://ubc-provenance.github.io/PIDSMaker/ten-minute-install

Opens a bash shell inside the running postgres container.

```bash
docker compose -p postgres exec postgres bash
```

--------------------------------

### Run the framework

Source: https://ubc-provenance.github.io/PIDSMaker/singularity_install

Execute the main script with the required arguments: SYSTEM, DATASET, and artifact directory, specifying the database host as localhost.

```bash
python pidsmaker/main.py SYSTEM DATASET --artifact_dir ./artifacts/ --database_host localhost
```

--------------------------------

### Launch hyperparameter tuning with --tuning_mode=hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/features/tuning

Launches hyperparameter tuning for my_system on the CADETS_E3 dataset. Requires prior `wandb login` in the container's shell. Automatically loads the tuning configuration based on dataset and system names, falling back to the default tuning configuration if no dataset-specific file exists.

```bash
./run.sh my_system CADETS_E3 --tuning_mode=hyperparameters

```

--------------------------------

### Create database for OpTC datasets (create_database_optc.py)

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Fills the database for a DARPA OpTC dataset (optc_h051, optc_h201, optc_h501). Must be run inside the pidsmaker container shell.

```bash
python dataset_preprocessing/optc/create_database_optc.py orthrus DATASET

```

--------------------------------

### Download datasets via CLI with download_datasets.sh

Source: https://ubc-provenance.github.io/PIDSMaker/ten-minute-install

Downloads pre-processed DARPA TC/OpTC database dumps from the command line. Requires a Google API access token obtained via the OAuth 2.0 Playground. Use lowercase dataset names, either as a comma-separated list or 'all' for every dataset.

```bash
# Example 1: Comma-separated datasets
./download_datasets.sh cadets_e3,optc_h201,clearscope_e5 YOUR_ACCESS_TOKEN

# Example 2: All datasets
./download_datasets.sh all YOUR_ACCESS_TOKEN

```

--------------------------------

### Copy .env.local to .env

Source: https://ubc-provenance.github.io/PIDSMaker/ten-minute-install

Creates the .env file from the local template. Set INPUT_DIR to the data folder path and optionally ARTIFACTS_DIR for generated files.

```bash
cp .env.local .env
```

--------------------------------

### Run PIDSMaker with W&B monitoring

Source: https://ubc-provenance.github.io/PIDSMaker/introduction

Run the framework in the shell with W&B monitoring.

```bash
python pidsmaker/main.py SYSTEM DATASET --wandb

```

--------------------------------

### Format all files with pre-commit

Source: https://ubc-provenance.github.io/PIDSMaker/contributing

Run this command before opening a pull request to format all files with pre-commit.

```bash
pre-commit run --all-files

```

--------------------------------

### Configure custom_system run with batching, training, and encoder options

Source: https://ubc-provenance.github.io/PIDSMaker/tutorial

Three command-line invocations for configuring a custom system run. The first removes node type features to keep only the word2vec embedding. The second increases node embedding size. The third reduces the number of GNN layers.

```bash
./run.sh custom_system CADETS_E3 --project=test_custom_system \
    --batching.node_features=node_emb
```

```bash
./run.sh custom_system CADETS_E3 --project=test_custom_system \
    --training.node_hid_dim=256 \
    --training.node_out_dim=256
```

```bash
./run.sh custom_system CADETS_E3 --project=test_custom_system \
    --training.encoder.custom_encoder.num_layers=2
```

--------------------------------

### Restart pipeline from scratch in isolated folder (--restart_from_scratch flag)

Source: https://ubc-provenance.github.io/PIDSMaker/pipeline

Restarts the entire pipeline in an isolated environment by using a randomly generated hash for the output folder. All generated files are deleted at the end of the pipeline to avoid unnecessary disk usage.

```bash
python pidsmaker/main.py orthrus CADETS_E3 --restart_from_scratch

```

--------------------------------

### Load all database dumps at once

Source: https://ubc-provenance.github.io/PIDSMaker/ten-minute-install

Runs the load_dumps.sh script to load all datasets from the data folder if enough space is available.

```bash
./scripts/load_dumps.sh
```

--------------------------------

### Create database for TC E5 datasets (create_database_e5.py)

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Fills the database for a DARPA TC E5 dataset (CLEARSCOPE_E5, CADETS_E5, THEIA_E5). Must be run inside the pidsmaker container shell. Large storage capacity is needed; a single run can generate more than 15GB of artifact files on E3 datasets, and much more with E5 datasets.

```bash
python dataset_preprocessing/darpa_tc/create_database_e5.py orthrus DATASET

```

--------------------------------

### Run pretrained_supervised on THEIA_E3 with custom attack edges

Source: https://ubc-provenance.github.io/PIDSMaker/features/supervised

Run the supervised objective on THEIA_E3, pointing attack_edges_path to its YAML file. The backslash continues the command on the next line.

```bash
python pidsmaker/main.py pretrained_supervised THEIA_E3 --featurization.pretrained.weights_path=/home/pids/weights/spider \
    --training.decoder.predict_edge_supervised.attack_edges_path=config/attack_edges/theia_e3.yml
```

--------------------------------

### Run run_n_times with --training.stable_optim=True

Source: https://ubc-provenance.github.io/PIDSMaker/features/instability

Enables stable optimization (`training.stable_optim: True`) to reduce variance between runs, then executes the `run_n_times` experiment. Use this to measure the effect on the `*_std` metrics. `training.weight_decay` is ignored when this is enabled.

```bash
./run.sh orthrus CADETS_E3 --training.stable_optim=True --experiment=run_n_times

```

--------------------------------

### Log in to Weights & Biases CLI

Source: https://ubc-provenance.github.io/PIDSMaker/ten-minute-install

Logs into Weights & Biases from the CLI by pasting your API key. Required to use the W&B interface for visualizing and historizing experiments.

```bash
wandb login

```

--------------------------------

### Run pidsmaker/main.py to verify dataset

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Runs the PIDSMaker main tool inside the pidsmaker container to verify that the database is built and filled with data.

```bash
python pidsmaker/main.py orthrus DATASET

```

--------------------------------

### Run consistency with SPIDER weights (pretrained_velox)

Source: https://ubc-provenance.github.io/PIDSMaker/features/pretrained_encoders

Runs the pretrained_velox pipeline on CADETS_E3 with the SPIDER weights folder. Adjust the weights path to the actual container location, which is /home/pids/weights/spider after the download and extraction steps.

```bash
python pidsmaker/main.py pretrained_velox CADETS_E3 --featurization.pretrained.weights_path=/home/pids/weights/spider
```

--------------------------------

### Create DATA_FOLDER directory

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Creates the DATA_FOLDER directory that will hold all raw dataset files. Make sure the folder is empty before downloading and parsing raw data, and remove all old files before downloading a new dataset.

```bash
DATA_FOLDER=./data
mkdir ${DATA_FOLDER}

```

--------------------------------

### Create database for TC E3 datasets (create_database_e3.py)

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Fills the database for a DARPA TC E3 dataset (CLEARSCOPE_E3, CADETS_E3, THEIA_E3). Must be run inside the pidsmaker container shell. Large storage capacity is needed; a single run can generate more than 15GB of artifact files on E3 datasets.

```bash
python dataset_preprocessing/darpa_tc/create_database_e3.py orthrus DATASET

```

--------------------------------

### Run tuned system with --tuned

Source: https://ubc-provenance.github.io/PIDSMaker/features/tuning

Runs a system with its best hyperparameters on a particular dataset. Searches for config/tuned_baselines/cadets_e3/tuned_my_system.yml and overrides the default system config with the best hyperparameters.

```bash
./run.sh my_system CADETS_E3 --tuned

```

--------------------------------

### Kairos training command for optc_h201

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs Kairos on the optc_h201 dataset with dropout 0.3, learning rate 0.001, node hidden/out dim 64, 12 epochs, and embedding dim 32.

```bash
python pidsmaker/main.py kairos optc_h201 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=64 --training.node_out_dim=64 --training.num_epochs=12 --featurization.emb_dim=32
```

--------------------------------

### NodLink training command for optc_h201

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs NodLink on the optc_h201 dataset with dropout 0.3, learning rate 0.0001, node hidden/out dim 320, 12 epochs, and embedding dim 128.

```bash
python pidsmaker/main.py nodlink optc_h201 --training.encoder.dropout=0.3 --training.lr=0.0001 --training.node_hid_dim=320 --training.node_out_dim=320 --training.num_epochs=12 --featurization.emb_dim=128
```

--------------------------------

### NodLink training command for CLEARSCOPE_E5

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs NodLink on the CLEARSCOPE_E5 dataset with dropout 0.3, learning rate 0.0001, node hidden/out dim 128, 12 epochs, and embedding dim 256.

```bash
python pidsmaker/main.py nodlink CLEARSCOPE_E5 --training.encoder.dropout=0.3 --training.lr=0.0001 --training.node_hid_dim=128 --training.node_out_dim=128 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Run PIDSMaker with arguments

Source: https://ubc-provenance.github.io/PIDSMaker/introduction

Basic command to run the PIDSMaker framework. SYSTEM must match a YML file in config/ and DATASET must be defined in DATASET_DEFAULT_CONFIG in config/config.py.

```bash
python pidsmaker/main.py SYSTEM DATASET --arg1=x --arg2=y

```

--------------------------------

### Kairos training command for CLEARSCOPE_E5

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs Kairos on the CLEARSCOPE_E5 dataset with dropout 0.3, learning rate 0.001, node hidden/out dim 64, 12 epochs, and embedding dim 32.

```bash
python pidsmaker/main.py kairos CLEARSCOPE_E5 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=64 --training.node_out_dim=64 --training.num_epochs=12 --featurization.emb_dim=32
```

--------------------------------

### Run Velox on CLEARSCOPE_E5 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Velox system on the CLEARSCOPE_E5 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.001, hidden and output dimensions 128, 12 epochs, and embedding dimension 128.

```bash
python pidsmaker/main.py velox CLEARSCOPE_E5 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=128 --training.node_out_dim=128 --training.num_epochs=12 --featurization.emb_dim=128
```

--------------------------------

### NodLink training command for optc_h501

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs NodLink on the optc_h501 dataset with dropout 0.3, learning rate 0.0001, node hidden/out dim 128, 12 epochs, and embedding dim 256.

```bash
python pidsmaker/main.py nodlink optc_h501 --training.encoder.dropout=0.3 --training.lr=0.0001 --training.node_hid_dim=128 --training.node_out_dim=128 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Run pipeline multiple times with --experiment=run_n_times

Source: https://ubc-provenance.github.io/PIDSMaker/features/instability

Runs the pipeline N times with the same configuration, each run using a different random seed. Requires the `--experiment=run_n_times` flag.

```bash
./run.sh orthrus CADETS_E3 --experiment=run_n_times

```

--------------------------------

### Run Velox on optc_h201 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Velox system on the optc_h201 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.001, hidden and output dimensions 128, 12 epochs, and embedding dimension 256.

```bash
python pidsmaker/main.py velox optc_h201 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=128 --training.node_out_dim=128 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### NodLink training command for optc_h051

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs NodLink on the optc_h051 dataset with dropout 0.3, learning rate 0.0001, node hidden/out dim 256, 12 epochs, and embedding dim 256.

```bash
python pidsmaker/main.py nodlink optc_h051 --training.encoder.dropout=0.3 --training.lr=0.0001 --training.node_hid_dim=256 --training.node_out_dim=256 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Kairos training command for optc_h051

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs Kairos on the optc_h051 dataset with dropout 0.3, learning rate 0.001, node hidden/out dim 128, 12 epochs, and embedding dim 32.

```bash
python pidsmaker/main.py kairos optc_h051 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=128 --training.node_out_dim=128 --training.num_epochs=12 --featurization.emb_dim=32
```

--------------------------------

### PIDSMaker project structure (config/ and pidsmaker/ layout)

Source: https://ubc-provenance.github.io/PIDSMaker/pipeline

Directory layout showing where configuration files and the main entry point reside. The pidsmaker/main.py file is the sole entry point, and each tasks/*.py module corresponds to one pipeline stage from construction through triage.

```plaintext
config/                         # existing systems are defined by their own YML file
├── orthrus.yml
├── kairos.yml
└── ...
pidsmaker/
├── main.py                     # only entry point of the framework
│── config/
│   ├── config.py               # available arguments to use in YML files
│   └── pipeline.py             # pipeline code
├── tasks/                      
│   ├── construction.py         # stage 1. parse raw provenance + graph construction
│   ├── transformation.py       # stage 2. graph transformations
│   ├── featurization.py        # stage 3. text embedding featurization (Word2Vec, Doc2Vec, ...)
│   ├── batching.py             # stage 4. batch construction, neighbor sampling, etc
│   ├── training.py             # stage 5. GNN training + inference loop
│   ├── evaluation.py           # stage 6. metrics calculation + plots
│   └── triage.py               # stage 7. optional post-processing attack tracing
```

--------------------------------

### Enable viz pre-saving during training: --save_for_viz

Source: https://ubc-provenance.github.io/PIDSMaker/features/visualization

Opt in to pre-saving extra artifacts during training for the fastest, most complete viz generation. These artifacts (per-epoch model checkpoints and a cached copy of test graphs) are large and off by default. Without it, the featurization space works but the GNN encoder space needs the checkpoints (skipped with a warning if absent), and test graphs are recomputed on demand.

```shell
python pidsmaker/main.py SYSTEM DATASET --save_for_viz

```

--------------------------------

### Run Velox on CLEARSCOPE_E3 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Velox system on the CLEARSCOPE_E3 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.001, hidden and output dimensions 64, 12 epochs, embedding dimension 128, and time window size 1.

```bash
python pidsmaker/main.py velox CLEARSCOPE_E3 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=64 --training.node_out_dim=64 --training.num_epochs=12 --featurization.emb_dim=128 --construction.time_window_size=1
```

--------------------------------

### Kairos training command for optc_h501

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs Kairos on the optc_h501 dataset with dropout 0.3, learning rate 0.001, node hidden/out dim 64, 12 epochs, and embedding dim 16.

```bash
python pidsmaker/main.py kairos optc_h501 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=64 --training.node_out_dim=64 --training.num_epochs=12 --featurization.emb_dim=16
```

--------------------------------

### NodLink training command for CLEARSCOPE_E3

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs NodLink on the CLEARSCOPE_E3 dataset with dropout 0.3, learning rate 0.0001, node hidden/out dim 64, 12 epochs, embedding dim 128, and time window 1.

```bash
python pidsmaker/main.py nodlink CLEARSCOPE_E3 --training.encoder.dropout=0.3 --training.lr=0.0001 --training.node_hid_dim=64 --training.node_out_dim=64 --training.num_epochs=12 --featurization.emb_dim=128 --construction.time_window_size=1
```

--------------------------------

### Download DARPA OpTC dataset using download_dataset.py

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Downloads raw DARPA OpTC dataset files into the specified DATA_FOLDER. DATASET can be optc_h051, optc_h201, or optc_h501. DATA_FOLDER must be the absolute path and must be empty before downloading.

```bash
python ./dataset_preprocessing/optc/download_dataset.py DATASET ${DATA_FOLDER}

```

--------------------------------

### Run PIDSMaker in background with W&B

Source: https://ubc-provenance.github.io/PIDSMaker/introduction

Run the framework in the background with W&B monitoring, ideal for multiple parallel runs. Logs can be watched with `tail -f nohup.out`.

```bash
./run.sh SYSTEM DATASET

```

--------------------------------

### Assign sweep name and project with --exp and --project

Source: https://ubc-provenance.github.io/PIDSMaker/features/tuning

Recommended command for better experiment historization: assigns a name to the sweep with --exp and a dedicated project name with --project.

```bash
./run.sh my_system CADETS_E3 \
    --tuning_mode=hyperparameters \
    --exp=bench_fasttext_word2vec \
    --project=best_featurization_method

```

--------------------------------

### Run Velox on optc_h501 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Velox system on the optc_h501 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.001, hidden and output dimensions 320, 12 epochs, and embedding dimension 256.

```bash
python pidsmaker/main.py velox optc_h501 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=320 --training.node_out_dim=320 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Run Velox on THEIA_E5 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Velox system on the THEIA_E5 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.0001, hidden and output dimensions 128, 12 epochs, embedding dimension 128, and time window size 15.0.

```bash
python pidsmaker/main.py velox THEIA_E5 --training.encoder.dropout=0.3 --training.lr=0.0001 --training.node_hid_dim=128 --training.node_out_dim=128 --training.num_epochs=12 --featurization.emb_dim=128 --construction.time_window_size=15.0
```

--------------------------------

### NodLink training command for THEIA_E5

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs NodLink on the THEIA_E5 dataset with dropout 0.3, learning rate 0.001, node hidden/out dim 320, 12 epochs, embedding dim 128, featurization epochs 20, time window 15.0, and no transformation methods.

```bash
python pidsmaker/main.py nodlink THEIA_E5 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=320 --training.node_out_dim=320 --training.num_epochs=12 --featurization.emb_dim=128 --featurization.epochs=20 --construction.time_window_size=15.0 --transformation.used_methods="none"
```

--------------------------------

### Reproduce SPIDER paper experiments with run.sh

Source: https://ubc-provenance.github.io/PIDSMaker/features/pretrained_encoders

Run the SPIDER paper experiments from the scripts directory after downloading the weights. The ARGS variable sets the experiment type and the pretrained weights path. Frozen encoder runs train on each dataset for a fixed number of epochs (CADETS_E3 uses 22, others default). Continued pretraining runs use the fine-tuned SPIDER weights with the provided continue_pretrain flags and training epochs.

```bash
cd scripts
ARGS="--experiment=run_n_times --featurization.pretrained.weights_path=/home/pids/weights/spider"

# Frozen encoder
./run.sh pretrained_velox CADETS_E3 $ARGS --training.num_epochs=22
./run.sh pretrained_velox THEIA_E3 $ARGS
./run.sh pretrained_velox THEIA_E5 $ARGS
./run.sh pretrained_velox CLEARSCOPE_E5 $ARGS --training.node_hid_dim=128 --training.node_out_dim=128
./run.sh pretrained_velox optc_h051 $ARGS --training.node_hid_dim=256 --training.node_out_dim=256

# Continued pretraining (fine-tuned SPIDER in the paper)
./run.sh pretrained_velox CADETS_E3 $ARGS --feat_inference.continue_pretrain=True --feat_inference.continue_pretrain_epochs=10
./run.sh pretrained_velox THEIA_E3 $ARGS --feat_inference.continue_pretrain=True --feat_inference.continue_pretrain_epochs=10
./run.sh pretrained_velox THEIA_E5 $ARGS --feat_inference.continue_pretrain=True --feat_inference.continue_pretrain_epochs=5
./run.sh pretrained_velox CLEARSCOPE_E5 $ARGS --feat_inference.continue_pretrain=True --feat_inference.continue_pretrain_epochs=1
./run.sh pretrained_velox optc_h051 $ARGS --feat_inference.continue_pretrain=True --feat_inference.continue_pretrain_epochs=10
```

--------------------------------

### NodLink training command for CADETS_E3

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs NodLink on the CADETS_E3 dataset with specified hyperparameters: dropout 0.3, learning rate 0.001, node hidden/out dim 320, 12 epochs, embedding dim 128, featurization epochs 20, time window 15.0, and no transformation methods.

```bash
python pidsmaker/main.py nodlink CADETS_E3 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=320 --training.node_out_dim=320 --training.num_epochs=12 --featurization.emb_dim=128 --featurization.epochs=20 --construction.time_window_size=15.0 --transformation.used_methods="none"
```

--------------------------------

### Run pidsmaker pipeline in background with W&B

Source: https://ubc-provenance.github.io/PIDSMaker/tutorial

Run the framework in background for practical experimentation, monitoring logs, figures, and metrics in Weights & Biases (W&B). Ensure you have logged in with wandb login before running.

```bash
cd scripts
./run.sh custom_system CADETS_E3 --project=test_custom_system

```

--------------------------------

### Tuning configuration YAML (tuning_my_system.yml)

Source: https://ubc-provenance.github.io/PIDSMaker/features/tuning

YAML tuning configuration for a system named my_system on the THEIA_E3 dataset. Defines a grid search over learning rate, hidden dimension, and featurization method. Stored at config/experiments/tuning/systems/theia_e3/tuning_my_system.yml.

```yaml
method: grid

parameters:
  training.lr:
    values: [0.001, 0.0001]
  training.node_hid_dim:
    values: [32, 64, 128, 256]
  featurization.used_method:
    values: [fasttext, word2vec]

```

--------------------------------

### Run Velox on optc_h051 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Velox system on the optc_h051 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.001, hidden and output dimensions 320, 12 epochs, and embedding dimension 256.

```bash
python pidsmaker/main.py velox optc_h051 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=320 --training.node_out_dim=320 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Run PIDSMaker on CLEARSCOPE_E5 dataset

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Command to run PIDSMaker with the CLEARSCOPE_E5 dataset, an Android device with APK-based attacks. Use this to process the dataset for APT detection research.

```bash
python pidsmaker/main.py SYSTEM CLEARSCOPE_E5
```

--------------------------------

### Run CyberGFM pretrained baseline on CADETS_E3

Source: https://ubc-provenance.github.io/PIDSMaker/features/pretrained_encoders

Run the CyberGFM baseline with BERT pretraining and link prediction fine-tuning on the CADETS_E3 dataset. Available via the pretrained detector mode when training.used_method is pretrained and finetune_mode is lp.

```bash
python pidsmaker/main.py cybergfm CADETS_E3

```

--------------------------------

### Kairos training command for CLEARSCOPE_E3

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs Kairos on the CLEARSCOPE_E3 dataset with dropout 0.3, learning rate 0.0001, node hidden/out dim 256, 12 epochs, embedding dim 128, and time window 1.

```bash
python pidsmaker/main.py kairos CLEARSCOPE_E3 --training.encoder.dropout=0.3 --training.lr=0.0001 --training.node_hid_dim=256 --training.node_out_dim=256 --training.num_epochs=12 --featurization.emb_dim=128 --construction.time_window_size=1
```

--------------------------------

### NodLink training command for THEIA_E3

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs NodLink on the THEIA_E3 dataset with dropout 0.3, learning rate 0.0001, node hidden/out dim 320, 12 epochs, embedding dim 128, featurization epochs 20, time window 15.0, and no transformation methods.

```bash
python pidsmaker/main.py nodlink THEIA_E3 --training.encoder.dropout=0.3 --training.lr=0.0001 --training.node_hid_dim=320 --training.node_out_dim=320 --training.num_epochs=12 --featurization.emb_dim=128 --featurization.epochs=20 --construction.time_window_size=15.0 --transformation.used_methods="none"
```

--------------------------------

### Run pretrained_velox with GPT-2 or DeepWalk encoder

Source: https://ubc-provenance.github.io/PIDSMaker/features/pretrained_encoders

Run the pretrained_velox PIDS with an alternative encoder. The first uses GPT-2 with the small model size via the --featurization.pretrained.model_type and --featurization.pretrained.model_size flags. The second selects DeepWalk with only the model_type flag. Both target the CADETS_E3 dataset and use SPIDER by default unless another encoder is specified.

```bash
python pidsmaker/main.py pretrained_velox CADETS_E3 --featurization.pretrained.model_type=gpt2_pretrained --featurization.pretrained.model_size=small
```

```bash
python pidsmaker/main.py pretrained_velox CADETS_E3 --featurization.pretrained.model_type=deepwalk
```

--------------------------------

### Tuned hyperparameters YAML (tuned_my_system.yml)

Source: https://ubc-provenance.github.io/PIDSMaker/features/tuning

YAML file containing the best hyperparameters for a tuned system, stored at config/tuned_baselines/{dataset}/tuned_my_system.yml. Obtained from the best W&B run by sorting on the desired metric.

```yaml
featurization:
  used_method: word2vec

training:
  lr: 0.0001
  node_hid_dim: 128

```

--------------------------------

### Kairos training command for THEIA_E5

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs Kairos on the THEIA_E5 dataset with dropout 0.3, learning rate 0.001, node hidden/out dim 256, 12 epochs, and embedding dim 256.

```bash
python pidsmaker/main.py kairos THEIA_E5 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=256 --training.node_out_dim=256 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Select ground truth version with evaluation.ground_truth_version (CLI and YAML)

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Selects the threatrace ground-truth source, either via a CLI flag or a YAML config key. The default is `orthrus`; switching only affects evaluation labels, not training or graphs.

```bash
python pidsmaker/main.py SYSTEM CADETS_E3 --evaluation.ground_truth_version threatrace
```

```yaml
evaluation:
  ground_truth_version: threatrace

```

--------------------------------

### Run Velox on THEIA_E3 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Velox system on the THEIA_E3 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.001, hidden and output dimensions 320, 12 epochs, and embedding dimension 256.

```bash
python pidsmaker/main.py velox THEIA_E3 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=320 --training.node_out_dim=320 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Run PIDSMaker on THEIA_E5 dataset

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Command to run PIDSMaker with the THEIA_E5 dataset, an Ubuntu host with Firefox exploitation. Use this to process the dataset for APT detection research.

```bash
python pidsmaker/main.py SYSTEM THEIA_E5
```

--------------------------------

### Run Velox on CADETS_E3 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Velox system on the CADETS_E3 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.0001, hidden and output dimensions 256, 12 epochs, and embedding dimension 256.

```bash
python pidsmaker/main.py velox CADETS_E3 --training.encoder.dropout=0.3 --training.lr=0.0001 --training.node_hid_dim=256 --training.node_out_dim=256 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Kairos training command for THEIA_E3

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs Kairos on the THEIA_E3 dataset with dropout 0.3, learning rate 0.001, node hidden/out dim 320, 12 epochs, and embedding dim 256.

```bash
python pidsmaker/main.py kairos THEIA_E3 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=320 --training.node_out_dim=320 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Kairos training command for CADETS_E3

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Runs Kairos on the CADETS_E3 dataset with dropout 0.3, learning rate 0.001, node hidden/out dim 256, 12 epochs, and embedding dim 256.

```bash
python pidsmaker/main.py kairos CADETS_E3 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=256 --training.node_out_dim=256 --training.num_epochs=12 --featurization.emb_dim=256
```

--------------------------------

### Download DARPA TC dataset with download_DATASET.sh

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Downloads raw DARPA TC dataset files into the specified DATA_FOLDER. DATASET can be clearscope_e3, cadets_e3, theia_e3, clearscope_e5, cadets_e5, or theia_e5. DATA_FOLDER must be the absolute path and must be empty before downloading.

```bash
./dataset_preprocessing/darpa_tc/scripts/download_DATASET.sh ${DATA_FOLDER}

```

--------------------------------

### Export database dump with pg_dump

Source: https://ubc-provenance.github.io/PIDSMaker/create-db-from-scratch

Exports the database as a dump file for sharing. Uses the postgres password and connects to the postgres service on port 5432, dumping the DATASET database to DATASET.dump.

```bash
PGPASSWORD=postgres pg_dump -U postgres -h postgres -p 5432 -F c -d DATASET -f DATASET.dump

```

--------------------------------

### Run PIDSMaker locally without W&B

Source: https://ubc-provenance.github.io/PIDSMaker/introduction

Run the framework in the shell without W&B, using the same settings as `./run.sh`.

```bash
./run_local.sh SYSTEM DATASET

```

--------------------------------

### Run Orthrus non-snooped on CLEARSCOPE_E5 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Orthrus non-snooped system on the CLEARSCOPE_E5 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.001, hidden and output dimensions 256, 12 epochs, and embedding dimension 128.

```bash
python pidsmaker/main.py orthrus_non_snooped CLEARSCOPE_E5 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=256 --training.node_out_dim=256 --training.num_epochs=12 --featurization.emb_dim=128
```

--------------------------------

### Run PIDSMaker on THEIA_E3 dataset

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Command to run PIDSMaker with the THEIA_E3 dataset, an Ubuntu host with Firefox exploitation. Use this to process the dataset for APT detection research.

```bash
python pidsmaker/main.py SYSTEM THEIA_E3
```

--------------------------------

### Load a single database dump with pg_restore

Source: https://ubc-provenance.github.io/PIDSMaker/ten-minute-install

Loads one database dump individually when space is limited. Replace DATASET_NAME with the lowercase dataset name (e.g., cadets_e3).

```bash
pg_restore -U postgres -h localhost -p 5432 -d DATASET_NAME /data/DATASET_NAME.dump
```

--------------------------------

### Run PIDSMaker on CLEARSCOPE_E3 dataset

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Command to run PIDSMaker with the CLEARSCOPE_E3 dataset, an Android device with Firefox exploitation. Use this to process the dataset for APT detection research.

```bash
python pidsmaker/main.py SYSTEM CLEARSCOPE_E3
```

--------------------------------

### Create config/custom_system.yml for custom PIDS pipeline

Source: https://ubc-provenance.github.io/PIDSMaker/tutorial

Configuration file for custom system, named config/custom_system.yml. It includes and overrides the base orthrus configuration, specifying parameters for construction, featurization, batching, training, and evaluation. This file defines the logic of the new PIDS pipeline.

```yaml
_include_yml: orthrus 

construction:
  time_window_size: 15.0 
  node_label_features: 
    subject: type, path, cmd_line
    file: type, path
    netflow: type, remote_ip, remote_port

featurization:
  emb_dim: 128
  epochs: 50
  used_method: word2vec 
  word2vec:
    alpha: 0.025
    window_size: 5
    min_count: 1
    use_skip_gram: True
    num_workers: 1
    compute_loss: True
    negative: 5
    decline_rate: 30

batching:
  node_features: node_emb,node_type 
  edge_features: none 
  intra_graph_batching:
    used_methods: none 

training:
  lr: 0.0001 
  node_hid_dim: 128
  node_out_dim: 128

  encoder:
    dropout: 0.3
    used_methods: custom_encoder 
    custom_encoder:
      activation: relu 
      num_layers: 3

  decoder:
    used_methods: predict_edge_type 
    predict_edge_type:
      decoder: edge_mlp 
      edge_mlp:
        src_dst_projection_coef: 2 
        architecture_str: linear(0.5) | relu 

evaluation:
  used_method: node_evaluation 
  node_evaluation:
    threshold_method: max_val_loss 
    use_kmeans: False 

```

--------------------------------

### Run Orthrus non-snooped on optc_h201 with tuned hyperparameters

Source: https://ubc-provenance.github.io/PIDSMaker/tuned_systems

Command to run the Orthrus non-snooped system on the optc_h201 dataset using the best hyperparameters from grid search. Uses dropout 0.3, learning rate 0.001, hidden and output dimensions 128, 12 epochs, and embedding dimension 128.

```bash
python pidsmaker/main.py orthrus_non_snooped optc_h201 --training.encoder.dropout=0.3 --training.lr=0.001 --training.node_hid_dim=128 --training.node_out_dim=128 --training.num_epochs=12 --featurization.emb_dim=128
```

--------------------------------

### Run pretrained_supervised with SPIDER on CADETS_E3

Source: https://ubc-provenance.github.io/PIDSMaker/features/supervised

Run Velox with SPIDER embeddings and the supervised objective on CADETS_E3. The attack labels are embedded during the feat_inference task; if it already ran with the same settings, add --force_restart=feat_inference.

```bash
python pidsmaker/main.py pretrained_supervised CADETS_E3 --featurization.pretrained.weights_path=/home/pids/weights/spider
```

--------------------------------

### Run PIDSMaker on CADETS_E3 dataset

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Command to run PIDSMaker with the CADETS_E3 dataset, a FreeBSD host with Nginx server exploitation. Use this to process the dataset for APT detection research.

```bash
python pidsmaker/main.py SYSTEM CADETS_E3
```

--------------------------------

### Run PIDsMaker on CARBANAKv2 EDR dataset

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Command to run the PIDsMaker tool on the CARBANAKv2 EDR dataset. The dataset contains Carbon Black Cloud EDR telemetry from a multi-host testbed with a Carbanak APT emulation.

```bash
python pidsmaker/main.py SYSTEM CARBANAKV2_EDR

```

--------------------------------

### Run PIDSmaker on DARPA OpTC optc_h501

Source: https://ubc-provenance.github.io/PIDSMaker/datasets

Executes PIDSmaker on the optc_h501 dataset, which involves a phishing email launching PowerShell Empire stager, DeathStar escalation, WMI persistence, RDP tunneling, and file exfiltration.

```bash
python pidsmaker/main.py SYSTEM optc_h501
```

--------------------------------

### Run PIDSMaker in shell without W&B

Source: https://ubc-provenance.github.io/PIDSMaker/introduction

Run the framework in the shell without W&B monitoring.

```bash
python pidsmaker/main.py SYSTEM DATASET

```