### Install Ubuntu Dependencies

Source: https://github.com/anza-xyz/agave/blob/master/README.md

Installs necessary development libraries on Ubuntu for building Agave.

```bash
sudo apt-get update
sudo apt-get install libssl-dev libudev-dev pkg-config zlib1g-dev llvm clang cmake make libprotobuf-dev protobuf-compiler libclang-dev
```

--------------------------------

### Check Core Pattern

Source: https://github.com/anza-xyz/agave/wiki/General-Debugging

Verify the system's configuration for handling core dump patterns. This example shows a typical systemd-coredump setup.

```bash
$ cat /proc/sys/kernel/core_pattern
```

--------------------------------

### Install Rust Toolchain

Source: https://github.com/anza-xyz/agave/blob/master/README.md

Installs Rust, Cargo, and Rustfmt. Ensure to source the environment variables after installation.

```bash
curl https://sh.rustup.rs -sSf | sh
source $HOME/.cargo/env
rustup component add rustfmt
```

--------------------------------

### Install Ghostscript and Graphviz

Source: https://github.com/anza-xyz/agave/wiki/General-Debugging

Install necessary tools for creating graph outputs from profile data. These are required for generating visual representations of the profile.

```bash
sudo apt install ghostscript graphviz
```

--------------------------------

### Install Agave v2.0.13

Source: https://github.com/anza-xyz/agave/wiki/2024-10-09-Testnet-Rollback-and-Restart

Use this command to install Agave v2.0.13, which is necessary for creating the correct snapshot.

```bash
agave-install init v2.0.13
```

--------------------------------

### Install Agave Version

Source: https://github.com/anza-xyz/agave/wiki/2025-07-03-Devnet-rollback-and-restart

Install the specified Agave version to ensure the correct snapshot can be created in the subsequent step.

```bash
agave-install init v2.3.2
```

--------------------------------

### Install Fedora Dependencies

Source: https://github.com/anza-xyz/agave/blob/master/README.md

Installs necessary development libraries on Fedora for building Agave.

```bash
sudo dnf install openssl-devel systemd-devel pkg-config zlib-devel llvm clang cmake make protobuf-devel protobuf-compiler perl-core libclang-dev
```

--------------------------------

### Enable Metrics Collection in Shell

Source: https://github.com/anza-xyz/agave/blob/master/metrics/scripts/README.md

Source this script in any shell where you want to collect metrics from applications. For example, before starting a Solana validator.

```bash
$ source ./enable.sh
```

--------------------------------

### Start Validator with Snapshot Arguments

Source: https://github.com/anza-xyz/agave/wiki/2024-01-02-Testnet-Rollback-and-Restart

Add these arguments to your validator startup script to specify snapshot loading and waiting parameters. Ensure these match your network's requirements.

```bash
--wait-for-supermajority 244604256 \
    --expected-shred-version 14676 \
    --expected-bank-hash 2ZHZpzSpBhkbfqsENGybfLbXSZ2hZiTq79qHCM4TWBpi \
```

--------------------------------

### Install Agave v2.1.9

Source: https://github.com/anza-xyz/agave/wiki/2025-01-14-Testnet-Rollback-and-Restart

Install Agave version 2.1.9, which is necessary for creating the correct snapshot.

```bash
agave-install init v2.1.9
```

--------------------------------

### Enable Solana Logger Setup

Source: https://github.com/anza-xyz/agave/wiki/General-Debugging

Call this at the top of your test or executable to route log statements to the terminal.

```rust
solana_logger::setup()
```

--------------------------------

### Install Rust Nightly for Benchmarking

Source: https://github.com/anza-xyz/agave/blob/master/README.md

Installs the nightly build of Rust, which is required for running benchmarks that use unstable features.

```bash
rustup install nightly
```

--------------------------------

### Install Agave Version

Source: https://github.com/anza-xyz/agave/wiki/2024-12-11-Testnet-Restart

Use this command to install Agave version v2.1.5. This is necessary for creating the correct snapshot.

```bash
agave-install init v2.1.5
```

--------------------------------

### Install Agave Version

Source: https://github.com/anza-xyz/agave/wiki/2024-10-16-Testnet-Rollback-and-Restart

Use this command to install the specific Agave version required for the testnet restart.

```bash
agave-install init v1.18.26
```

--------------------------------

### Update Validator Startup Configuration

Source: https://github.com/anza-xyz/agave/wiki/2024-10-09-Testnet-Rollback-and-Restart

Add these arguments to your validator startup script to ensure it starts with the correct snapshot and waits for supermajority.

```bash
--wait-for-supermajority 296876256 \
    --expected-shred-version 10323 \
    --expected-bank-hash Ea1SMrWMbGnCiYV7cAZP3gWTeFeu5UkEBb7oDp8VKaLr \
```

--------------------------------

### Instantiate TransactionBatchProcessor and Execute Transactions

Source: https://github.com/anza-xyz/agave/blob/master/svm/doc/spec.md

Example of how to instantiate TransactionBatchProcessor and call load_and_execute_sanitized_transactions. This requires setting up slot, epoch, program_cache, callbacks, and other necessary configurations.

```rust
let mut processor = TransactionBatchProcessor::new(
            slot,
            epoch,
            program_cache.clone(),
            builtin_program_ids.clone(),
            sysvar_cache.clone(),
        );

        let _ = processor
            .load_and_execute_sanitized_transactions(
                &mut callbacks,
                &sanitized_txs, // slice of sanitized transactions
                &mut check_results, // mutable slice of transaction check results
                environment, // runtime environment
                config, // transaction processing config
            )
            .await;

```

--------------------------------

### Expected Snapshot Creation Output

Source: https://github.com/anza-xyz/agave/wiki/2024-12-11-Testnet-Restart

This is an example of the expected output upon successful snapshot creation. Verify that the bank hash, second slot number, and shred version match the required values.

```text
Successfully created snapshot for slot 306450862, hash BiGFLfFewfTB2asBRLjwRL6z7VNfuvYraS3H7RfQNCrf: /home/sol/ledger-snapshots/incremental-snapshot-<BASE_SLOT>-306450862-<SNAPSHOT_HASH>.tar.zst
Shred version: 64506
```

--------------------------------

### Start Local Metric Services

Source: https://github.com/anza-xyz/agave/blob/master/metrics/scripts/README.md

Execute this script to start the local InfluxDB and Grafana services. Metrics collection is enabled on a per-shell basis.

```bash
$ ./start.sh
```

--------------------------------

### Update Validator Startup Configuration

Source: https://github.com/anza-xyz/agave/wiki/2025-07-03-Devnet-rollback-and-restart

Add these arguments to your validator startup script to configure the validator to wait for a supermajority, use the expected shred version, and the expected bank hash upon starting.

```bash
--wait-for-supermajority 391843994 \
    --expected-shred-version 29062 \
    --expected-bank-hash TvMCp3TJJTQmpYyXh5E36wLkH9DqpmKZqXy882S8PnK \
```

--------------------------------

### Validator Startup Arguments for Testnet Restart

Source: https://github.com/anza-xyz/agave/wiki/2024-10-16-Testnet-Rollback-and-Restart

Add these arguments to your validator startup script to ensure it starts correctly with the new snapshot and waits for supermajority.

```bash
--wait-for-supermajority 296877184 \
    --expected-shred-version 27799 \
    --expected-bank-hash ABuvd5grMYhbrjeLmu3qCNqiD8DaDRw6oyboMgZfaQS5 \

```

--------------------------------

### Check Agave Ledger Tool Version

Source: https://github.com/anza-xyz/agave/wiki/2025-12-11-Testnet-rollback-and-restart

Verify that the agave-ledger-tool is installed with the required version v3.1.4. Build instructions are available separately.

```bash
agave-ledger-tool --version
```

--------------------------------

### Snapshot File Naming Convention

Source: https://github.com/anza-xyz/agave/wiki/2025-12-11-Testnet-rollback-and-restart

These are examples of snapshot file names. Ensure the base slot numbers match between full and incremental snapshots.

```text
snapshot-<BASE_SLOT>-<BASE_SNAPSHOT_HASH>.tar.zst
incremental-snapshot-<BASE_SLOT>-374301609-<SNAPSHOT_HASH>.tar.zst
```

--------------------------------

### Agave Validator Startup Arguments

Source: https://github.com/anza-xyz/agave/wiki/2026-01-22-Testnet-Restart

Arguments to add to your validator startup script for Agave. These ensure the validator starts with the correct parameters after a testnet restart.

```bash
--wait-for-supermajority 383520372 \
    --expected-shred-version 27350 \
    --expected-bank-hash 3zk4WMwk6wCTVJXu9UAk2dYWMedCKooDs15XL5u6FkvE \
```

--------------------------------

### Verify Solana Ledger Locally

Source: https://github.com/anza-xyz/agave/wiki/Debugging-Consensus-Failures

Use `solana-ledger-tool verify` to replay a ledger and halt at a specific slot. This requires significant system resources and is best performed on a validator setup.

```bash
$ solana-ledger-tool verify --ledger LEDGER_DIR --halt-at-slot FAIL_SLOT
```

--------------------------------

### Create Agave Testnet Snapshot

Source: https://github.com/anza-xyz/agave/wiki/2025-12-03-Testnet-rollback-and-restart

Use this command to create a snapshot for the Agave testnet. Ensure you have the correct ledger tool version installed. This command may remove some activated feature gate accounts.

```bash
agave-ledger-tool --ledger <ledger-path> create-snapshot \
    --fix-testnet-ed25519-precompile-account \
    --incremental \
    --snapshots <snapshot-path> \
    --hard-fork 374301609 \
    --deactivate-feature-gate \
        ENTRYnPAoT5Swwx73YDGzMp3XnNH1kxacyvLosRHza1i \
    --enable-capitalization-change \
    --  374301608 <snapshot-path>
```

--------------------------------

### Configure Startup Script for Snapshot Download

Source: https://github.com/anza-xyz/agave/wiki/2025-10-01-Testnet-rollback-and-restart

Add these flags to your validator's startup script to download a snapshot from a known validator. Ensure the `--no-snapshot-fetch` flag is removed when initially downloading.

```bash
--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
    --expected-shred-version 41708 \

```

--------------------------------

### Download Snapshot from Known Validator

Source: https://github.com/anza-xyz/agave/wiki/2024-08-26-Testnet-Restart

Add these flags to your startup script if you need to download a snapshot from a known validator instead of creating one locally. Ensure `--no-snapshot-fetch` is removed if present.

```bash
--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
    --expected-shred-version 4084 \
```

--------------------------------

### Install Patched Agave Version

Source: https://github.com/anza-xyz/agave/wiki/2024-08-26-Testnet-Restart

Install the patched version of the Agave validator software.

```bash
agave-install init v2.0.7
```

--------------------------------

### Download Snapshot from Known Validator

Source: https://github.com/anza-xyz/agave/wiki/2025-07-03-Devnet-rollback-and-restart

If you are unable to generate a snapshot locally, add these lines to your startup script to download a snapshot from a known validator. Ensure the `--no-snapshot-fetch` flag is removed if present.

```bash
--known-validator dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 \
    --expected-shred-version 29062 \
```

--------------------------------

### Install Agave v2.0.22

Source: https://github.com/anza-xyz/agave/wiki/2025-01-14-Testnet-Rollback-and-Restart

Install Agave version 2.0.22, which is the target version for restarting the cluster.

```bash
agave-install init v2.0.22
```

--------------------------------

### Download Snapshot from Known Validator

Source: https://github.com/anza-xyz/agave/wiki/2025-12-11-Testnet-rollback-and-restart

If local snapshot creation fails, use these startup arguments to download a snapshot from a known validator. Ensure `--no-snapshot-fetch` is removed if present.

```bash
--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
--expected-shred-version 9604 \

```

--------------------------------

### Configure Validator Startup

Source: https://github.com/anza-xyz/agave/wiki/2025-01-14-Testnet-Rollback-and-Restart

Configure validator startup with specific arguments including wait-for-supermajority, expected shred version, and expected bank hash. This ensures the validator loads the correct snapshot and waits for network consensus.

```bash
--wait-for-supermajority 312256121 \
    --expected-shred-version 64475 \
    --expected-bank-hash 6A7U1X8WqnWRXWtqaxF15sSYHLT66j1ycBEyFigGwr2Z \
```

--------------------------------

### Download Snapshot from Known Validator

Source: https://github.com/anza-xyz/agave/wiki/2024-10-16-Testnet-Rollback-and-Restart

If local snapshot creation fails, add these flags to your startup script to download a snapshot from a known validator. Ensure `--no-snapshot-fetch` is removed if present.

```bash
--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
    --expected-shred-version 27799 \

```

--------------------------------

### Install Agave Validator Version

Source: https://github.com/anza-xyz/agave/wiki/2025-10-01-Testnet-rollback-and-restart

Installs the specified version of the Agave validator. This is the version required to restart the cluster.

```bash
agave-install init v2.3.11
```

--------------------------------

### Download Snapshot from Known Validator

Source: https://github.com/anza-xyz/agave/wiki/2025-12-03-Testnet-rollback-and-restart

If you cannot generate a snapshot locally, add these flags to your startup script to download a snapshot from a known validator. Ensure the `--no-snapshot-fetch` flag is removed if present.

```bash
--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
    --expected-shred-version 24207 \

```

--------------------------------

### Snapshot Error: Ending Slot Less Than Starting Slot

Source: https://github.com/anza-xyz/agave/wiki/2025-12-11-Testnet-rollback-and-restart

This error occurs when the blockstore cannot be processed due to an invalid slot range. It implies that the ending slot is earlier than the starting slot.

```text
Unable to process blockstore from starting slot <slot> to 374301608; the ending slot is less than the starting slot. The starting slot will be the latest snapshot slot, or genesis if the --no-snapshot flag is specified or if no snapshots are found.
```

--------------------------------

### Add Known Validator and Shred Version Flags

Source: https://github.com/anza-xyz/agave/wiki/2024-12-11-Testnet-Restart

If you are unable to generate a snapshot locally, add these flags to your startup script to download a snapshot from a known validator and specify the expected shred version. Ensure the `--no-snapshot-fetch` flag is removed if present.

```bash
--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
    --expected-shred-version 64506 \
```

--------------------------------

### Update Shred Version

Source: https://github.com/anza-xyz/agave/wiki/2024-08-26-Testnet-Restart

Update your shred version to the expected value before starting normally.

```bash
--expected-shred-version 4084 \
```

--------------------------------

### Update Dockerfile Packages

Source: https://github.com/anza-xyz/agave/blob/master/ci/docker/README.md

Modify this Dockerfile to install or remove additional packages in the Docker image.

```docker
ci/docker/Dockerfile
```

--------------------------------

### Download Snapshot from Known Validator

Source: https://github.com/anza-xyz/agave/wiki/2024-01-02-Testnet-Rollback-and-Restart

Add these arguments to your startup script if you are unable to generate a snapshot locally. This tells the validator to use a known validator's snapshot and disables fetching.

```bash
--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
    --expected-shred-version 14676 \
```

--------------------------------

### Update Expected Shred Version

Source: https://github.com/anza-xyz/agave/wiki/2025-10-01-Testnet-rollback-and-restart

When nodes haven't joined the cluster, they need to update their shred version before starting normally.

```bash
--expected-shred-version 41708 \
```

--------------------------------

### Analyze Heap Profile Data with jeprof (Single File)

Source: https://github.com/anza-xyz/agave/wiki/General-Debugging

Use the jeprof tool to analyze collected heap profile data and generate a PDF output. Specify the binary path, the heap profile file, and the output format.

```bash
jeprof --show_bytes /path/to/binary /path/to/heap/profile/file --pdf > heap.pdf
```

--------------------------------

### Off-CPU Profiling with Perf

Source: https://github.com/anza-xyz/agave/wiki/General-Debugging

Instructions for building a patched `perf` tool and using it to record off-CPU events for a running Solana validator process. This requires specific build dependencies and a patched `perf` source.

```bash
~/work/linux/tools/perf/perf record  --call-graph fp --off-cpu -e dummy --proc-map-timeout 30000 -p $PID_OF_RUNNING_SOLANA_VALIDATOR
```

```bash
sudo ~/work/linux/tools/perf/perf script -F +pid > ./perf-offcpu.perf
```

--------------------------------

### Example recipients.csv format

Source: https://github.com/anza-xyz/agave/blob/master/tokens/README.md

CSV file format for specifying recipients and their token amounts. Amounts should be in raw format (no decimals).

```text
recipient,amount
CYRJWqiSjLitBAcRxPvWpgX3s5TvmN2SuRY3eEYypFvT,75400
C56nwrDVFpPrqwGYsTgQxv1ZraTh81H14PV4RHvZe36s,10000
7aHDubg5FBYj1SgmyBgU3ZJdtfuqYCQsJQK2pTR5JUqr,42100
7qQPmVAQxEQ5djPDCtiEUrxaPf8wKtLG1m6SB1brejJ1,20000
```

--------------------------------

### Analyze Heap Profile Data with jeprof (Multiple Files)

Source: https://github.com/anza-xyz/agave/wiki/General-Debugging

Analyze multiple heap profile files by providing a wildcard path to the jeprof tool. This is useful when a process generates several profile files.

```bash
jeprof --show_bytes /path/to/binary /path/to/heap/profile/directory/jeprof.* --pdf > heap.pdf
```

--------------------------------

### Downgrade Solana Software Version

Source: https://github.com/anza-xyz/agave/wiki/2024-01-02-Testnet-Rollback-and-Restart

Install version v1.16.24 of the Solana software. This step is crucial for ensuring compatibility with the testnet rollback procedure.

```bash
solana-install init v1.16.24
```

--------------------------------

### Update Expected Shred Version

Source: https://github.com/anza-xyz/agave/wiki/2026-01-22-Testnet-Restart

When nodes haven't joined the cluster, they need to update their shred version. This is a command-line argument for starting a node.

```bash
--expected-shred-version 27350 \
```

--------------------------------

### Enable Apport Service for Core Dumps

Source: https://github.com/anza-xyz/agave/wiki/General-Debugging

Ensure the 'apport' service is enabled and running to handle core dumps. Run as root if necessary.

```bash
# Check service status
$ systemctl status apport

# Enable service (must be run as root)
$ sudo systemctl start apport
```

--------------------------------

### Query InfluxDB for Airdrop Measurements

Source: https://github.com/anza-xyz/agave/blob/master/metrics/scripts/README.md

Use the InfluxDB CLI to pull all measurements from the 'faucet-airdrop' measurement in the 'testnet' database. Requires InfluxDB client installation.

```sh
$ influx -database testnet -username read -password read -execute 'SELECT * FROM "faucet-airdrop"'
```

--------------------------------

### Update Shred Version for Nodes Joining Cluster

Source: https://github.com/anza-xyz/agave/wiki/2024-12-11-Testnet-Restart

Nodes that have not yet joined the cluster need to update their shred version to 64506 before starting normally.

```bash
--expected-shred-version 64506 \
```

--------------------------------

### Add Known Validator Flags

Source: https://github.com/anza-xyz/agave/wiki/2025-07-02-Testnet-Rollback-and-Restart

Add these flags to your startup script to download a snapshot from a known validator. Ensure the `--no-snapshot-fetch` flag is removed if present.

```bash
--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
    --expected-shred-version 9065 \

```

--------------------------------

### Configure Core File Limits

Source: https://github.com/anza-xyz/agave/wiki/General-Debugging

Check and configure the system's limits for creating core dump files. 'unlimited' allows for large core dumps.

```bash
$ ulimit -c
```

```bash
* soft core unlimited
* hard core unlimited
```

```bash
[Service]
... 
LimitCORE=infinity
```

--------------------------------

### Agave Validator Startup Arguments

Source: https://github.com/anza-xyz/agave/wiki/2025-12-11-Testnet-rollback-and-restart

Add these arguments to your validator startup script to configure it to use a specific snapshot and wait for network conditions.

```bash
--wait-for-supermajority 374301609 \
--expected-shred-version 9604 \
--expected-bank-hash EJMzxv7JscF8WNZfDYqzsAyALCDCS52HuihabVgyz5mf \

```

--------------------------------

### Expected Shred Version Argument

Source: https://github.com/anza-xyz/agave/wiki/2024-10-16-Testnet-Rollback-and-Restart

This argument is used when starting a validator to specify the expected shred version. It is relevant for nodes that haven't yet joined the cluster.

```bash
--expected-shred-version 27799 \
```

--------------------------------

### Add Known Validator Flags to Startup Script

Source: https://github.com/anza-xyz/agave/wiki/2025-12-06-Testnet-rollback-and-restart

Add these flags to your validator's startup script to download a snapshot from a specified known validator. Ensure the `--expected-shred-version` matches the snapshot's version.

```bash
--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
    --expected-shred-version 6045 \

```

--------------------------------

### Update Shred Version and Bank Hash

Source: https://github.com/anza-xyz/agave/wiki/2024-01-02-Testnet-Rollback-and-Restart

Update your shred version and bank hash before starting the normal process. These values are specific to the current testnet state.

```bash
--expected-shred-version 14676 \
    --expected-bank-hash 2ZHZpzSpBhkbfqsENGybfLbXSZ2hZiTq79qHCM4TWBpi \

```

--------------------------------

### Updated recipients.csv for dry-run simulation

Source: https://github.com/anza-xyz/agave/blob/master/tokens/README.md

An example of an updated CSV file with new amounts for simulating a distribution. This is used with the `--dry-run` or `solana-tokens balances` commands.

```text
recipient,amount
CYRJWqiSjLitBAcRxPvWpgX3s5TvmN2SuRY3eEYypFvT,100000
C56nwrDVFpPrqwGYsTgQxv1ZraTh81H14PV4RHvZe36s,100000
7aHDubg5FBYj1SgmyBgU3ZJdtfuqYCQsJQK2pTR5JUqr,100000
7qQPmVAQxEQ5djPDCtiEUrxaPf8wKtLG1m6SB1brejJ1,100000
```