Substreams

https://github.com/streamingfast/substreams
Admin
Substreams is a high-performance blockchain indexing technology that enables developers to write...
Tokens:264,210
Snippets:2,187
Trust Score:8
License:Apache-2.0
Update:4 weeks ago
Show doc for...
Tokens:
Raw
### Clone and run the connect-web-example

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/community/other-sinks/kv.md

Commands to clone the repository, install dependencies, and start the development server.

```bash
git clone git@github.com:streamingfast/substreams-sink-kv
cd substreams-sink-kv/connect-web-example
npm install
npm run dev
```

--------------------------------

### foundational-store CLI commands

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/foundational-store-reference.md

Lists the available commands for the foundational-store binary: completion, get, help, and server. The 'get' command retrieves a value via gRPC, and 'server' starts the gRPC server.

```bash
foundational-store [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  get         Get a value from the foundational-store using gRPC
  help        Help about any command
  server      Start the gRPC server
```

--------------------------------

### Run Web application with ViteJS

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/stream/javascript.md

Starts the ViteJS development server for the Web example. After running, navigate to https://localhost:5173 to receive streamed data.

```bash
npm run dev
```

--------------------------------

### Install Foundry with curl and foundryup

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/evm/local-development/foundry.md

Installs Foundry if not already installed.

```bash
curl -L https://foundry.paradigm.xyz | bash
foundryup
```

--------------------------------

### Install dependencies with npm

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/stream/javascript.md

Run this command in the project directory to install all dependencies listed in package.json.

```bash
npm install
```

--------------------------------

### Install PubSub CLI

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/pubsub.md

Installs the PubSub CLI using Go, which is required to deploy Substreams to the PubSub service.

```bash
go install ./cmd/substreams-sink-pubsub
```

--------------------------------

### Install Substreams CLI from source (Linux)

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/cli/installing-the-cli.md

Clones the Substreams repository and builds the CLI using Go. After installation, add $HOME/go/bin to the system path if it's not already present.

```bash
git clone https://github.com/streamingfast/substreams
cd substreams
go install -v ./cmd/substreams
```

--------------------------------

### YAML example for my-module on Ethereum Mainnet

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/substreams-components/modules/parameterized-modules.md

Concrete example showing how to set a parameter for the my-module module on the mainnet network. Replace the placeholder value with the actual parameter value.

```yaml
...

networks:
    mainnet:
        params: 
            my-module: “value”

...
```

--------------------------------

### PostgreSQL DSN examples

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/sql/dsn-reference.md

Examples of PostgreSQL DSNs for basic connections, SSL disabled, specific schema, and multiple options.

```bash
# Basic connection
postgres://myuser:mypassword@localhost:5432/mydatabase
```

```bash
# With SSL disabled
postgres://myuser:mypassword@localhost:5432/mydatabase?sslmode=disable
```

```bash
# With specific schema
postgres://myuser:mypassword@localhost:5432/mydatabase?schema=public
```

```bash
# Multiple options
postgres://myuser:mypassword@localhost:5432/mydatabase?sslmode=disable&schema=analytics
```

--------------------------------

### Example manifest file with sink config

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Example YAML manifest showing a sink configuration for a SQL sink, including schema file and frontend options.

```yaml
[...]
network: mainnet

sink:
  module: main:db_out
  type: sf.substreams.sink.sql.v1.Service
  config:
    schema: "./schema.sql"
    wire_protocol_access: true
    postgraphile_frontend:
      enabled: true
    pgweb_frontend:
      enabled: true
```

--------------------------------

### Build, auth, and stream with GUI

Source: https://github.com/streamingfast/substreams/blob/develop/sink/sql/db_proto/test/substreams/full-block/README.md

Commands to build the Substreams, authenticate, and start streaming via the GUI. Run from the project root.

```bash
substreams build
substreams auth
substreams gui       			  # Get streaming!
```

--------------------------------

### Extract proto with SQL annotations and setup Postgres sink

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/sql/proto-annotations.md

Commands to extract the module's output proto with SQL annotations commented out, then uncomment the options and run the setup with the proto-file-override. The same override must be passed to the run and constraints apply, otherwise they derive the unannotated schema again.

```bash
substreams tools extract-proto --sql substreams.yaml map_events
# uncomment the options that describe the schema, then:
substreams sink postgres setup substreams.yaml --proto-file-override=./events.proto --dsn=$DSN
```

--------------------------------

### Example store state for get mode

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/substreams-components/modules/types.md

Example store state used to illustrate the behavior of the `get_first`, `get_last`, and `get_at` methods in `get mode`. The store contains a block number and a list of deltas with ordinals, keys, types, and old/new values.

```rust
let store = StoreUSDPrice {
   Block: #1000,
   Deltas: [
      Ord: 1, Key: "usd", Type: UPDATE, OldValue: 1.45, NewValue: 1.54,
      Ord: 2, Key: "usd", Type: DELETE, OldValue: 1.54, NewValue: <nil>,
      Ord: 3, Key: "usd", Type: INSERT, OldValue: <nil>, NewValue: 1.47,
      Ord: 4, Key: "usd", Type: UPDATE, OldValue: 1.47, NewValue: 1.65,
   ]
}
```

--------------------------------

### Manifest imports example

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/substreams-components/manifests.md

Example showing how to import third-party Substreams packages using the imports section, with a comment showing the alternative source input.

```yaml
imports:
  sol: https://spkg.io/streamingfast/solana-explorer-v0.2.0.spkg
  # or:
  ethereum: substreams-ethereum-v1.0.0.spkg
  token: ../eth-token/substreams.yaml

...

modules:
...
    inputs:
      - map: sol:map_block_without_votes
# replacing:
#    inputs:
#      - source: sf.solana.type.v1.Block
```

--------------------------------

### Build on the Ubuntu machine

Source: https://github.com/streamingfast/substreams/blob/develop/sink/sql/db_proto/benchmarks/README.md

Installs Go 1.26 via snap (apt ships 1.22), git, clones the repo, and runs the benchmark. Requires outbound HTTPS and Docker access.

```bash
sudo snap install go --classic          # or install the official tarball
sudo apt-get install -y git
git clone <repo> && cd substreams/sink/sql/db_proto/benchmarks
./live-benchmark.sh
```

--------------------------------

### Example of block keys

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/substreams-components/modules/indexes.md

Shows an example of the keys list for a block, containing event addresses.

```text
Block 32443
--------------------------
keys = {'evt_addr:0xa34', 'evt_addr:0xba7', 'evt_addr:0x99a'}
```

--------------------------------

### Install Substreams CLI via Homebrew (macOS)

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/cli/installing-the-cli.md

Installs the Substreams CLI using Homebrew on macOS. Requires Homebrew to be installed.

```bash
brew install streamingfast/tap/substreams
```

--------------------------------

### Install Ubuntu dependencies for benchmark script

Source: https://github.com/streamingfast/substreams/blob/develop/sink/sql/db_proto/benchmarks/README.md

Installs docker.io, python3, and ca-certificates on Ubuntu. python3 and ca-certificates are absent from minimal/container images; without certificates HTTPS calls fail with 'x509: certificate signed by unknown authority'.

```bash
sudo apt-get update
sudo apt-get install -y docker.io python3 ca-certificates
sudo usermod -aG docker "$USER"   # then log out and back in, or: newgrp docker
```

--------------------------------

### ClickHouse DSN examples

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/sql/dsn-reference.md

Examples of ClickHouse DSNs for local, authenticated, cloud, and with additional options.

```bash
# Local ClickHouse (unencrypted)
clickhouse://default:@localhost:9000/default
```

```bash
# ClickHouse with authentication
clickhouse://myuser:mypassword@localhost:9000/mydatabase
```

```bash
# ClickHouse Cloud (encrypted)
clickhouse://default:mypassword@abc123.clickhouse.cloud:9440/default?secure=true
```

```bash
# With additional options
clickhouse://myuser:mypassword@localhost:9000/mydatabase?compress=true&debug=true
```

--------------------------------

### Output from substreams inspect

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/cli/command-line-interface.md

A example output shows the structure of proto_files and module definitions.

```bash
proto_files
...
modules {
  modules {
    name: "my_module_name"
...
}
```

--------------------------------

### Start PostgreSQL database with Docker

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/sql/relational-mappings.md

Starts a PostgreSQL 15 container named `postgres-db` with password `password`, database `substreams`, and port mapping 5432. Use this before running the sink.

```bash
docker run --name postgres-db -e POSTGRES_PASSWORD=password -e POSTGRES_DB=substreams -p 5432:5432 -d postgres:15
```

--------------------------------

### Complete descriptorSets example with all fields

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/substreams-components/manifests.md

Complete example showing all available fields for descriptorSets: module, version, symbols, and localPath. The symbols array restricts imported types; if omitted, all types are available. localPath specifies a local cache path for the descriptor set.

```yaml
protobuf:
  descriptorSets:
    - module: buf.build/streamingfast/substreams-sink-sql
      version: v1.0.0
      symbols:
        - sf.substreams.sink.sql.v1.Service
        - sf.substreams.sink.sql.v1.Table
      localPath: ./proto-cache/sink-sql.binpb
    - module: buf.build/streamingfast/substreams-entity-change@v1.3.0
      symbols:
        - sf.substreams.entity.v1.EntityChanges
```

--------------------------------

### Start the environment with Docker Compose

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/evm/local-development/hardhat.md

Starts the local Ethereum environment in detached mode. To reset all data, use `docker compose down --volumes`.

```bash
docker compose up -d
```

--------------------------------

### Install substreams-sink-prometheus globally via npm

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/community/other-sinks/prometheus.md

Installs the CLI tool globally using npm.

```bash
$ npm install -g substreams-sink-prometheus
```

--------------------------------

### Output from substreams info

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/cli/command-line-interface.md

Example output showing the package name, version, description, and module details including names, initial blocks, kinds, output types, hashes, and update policies.

```bash
Package name: solana_spl_transfers
Version: v0.5.2
Doc: Solana SPL Token Transfers stream

  Stream SPL token transfers to the nearest human being.

Modules:
----
Name: spl_transfers
Initial block: 130000000
Kind: map
Output Type: proto:solana.spl.v1.TokenTransfers
Hash: 2b59e4e840f814f4154a688c2935da9c3b61dc61

Name: transfer_store
Initial block: 130000000
Kind: store
Value Type: proto:solana.spl.v1.TokenTransfers
Update Policy: UPDATE_POLICY_SET
Hash: 11fd70768029bebce3741b051c15191d099d2436
```

--------------------------------

### SQL Sink Progress and Performance Logs

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/sql/progress-info.md

Example output showing the one-line stream summary followed by the multi-line performance panel.

```text
INFO (substreams) substreams stream stats {"data_msg_rate": "5386.833 msg/s (246075 total)", ...}
INFO (substreams) -----------------------------------
INFO (substreams) Stats {"block_count": 264390, ...}
INFO (substreams)    Wait Duration Between Blocks {...}
INFO (substreams)       Block Processing Duration {...}
INFO (substreams)          Unmarshalling Duration {...}
INFO (substreams)            Spool Write Duration {...}
INFO (substreams)           Message Walk Duration {...}
INFO (substreams)                  Flush duration {...}
INFO (substreams)               Pipeline progress {...}
INFO (substreams)                   Spool applier {...}
INFO (substreams)                Spool throughput {...}
INFO (substreams)                  Spool pressure {...}
```

--------------------------------

### Start Anchor shell

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/solana/local-development/anchor.md

Starts the Anchor shell to interact with the deployed Counter program.

```bash
anchor shell
```

--------------------------------

### Example Functions in vtproto_stream_example_test.go

Source: https://github.com/streamingfast/substreams/blob/develop/storage/store/marshaller/STREAMING_README.md

These functions are available in the test file for reference and benchmarking. They demonstrate basic streaming, smart file reading, and smart reader processing. Use them as a reference for implementing custom streaming logic.

```go
// Basic streaming (preserved in tests)
func (p *VTproto) UnmarshalStream(reader io.Reader) (*StoreData, uint64, error)
```

```go
// Smart file reading (preserved in tests)
func (p *VTproto) UnmarshalFromFile(filename string, maxMemoryThreshold int64) (*StoreData, uint64, error)
```

```go
// Smart reader processing (preserved in tests)
func (p *VTproto) UnmarshalFromReader(reader io.Reader, maxBufferSize int64) (*StoreData, uint64, error)
```

--------------------------------

### Example sink output

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/community/other-sinks/kv.md

Sample log output observed when running the sink.

```text
2023-01-12T10:08:31.803-0500 INFO (sink-kv) starting prometheus metrics server {"listen_addr": "localhost:9102"}
2023-01-12T10:08:31.803-0500 INFO (sink-kv) sink to kv {"dsn": "badger3:///Users/stepd/repos/substreams-sink-kv/badger_data.db", "endpoint": "mainnet.eth.streamingfast.io:443", "manifest_path": "https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.4.0/substreams-eth-block-meta-v0.4.0.spkg", "output_module_name": "kv_out", "block_range": ""}
2023-01-12T10:08:31.803-0500 INFO (sink-kv) starting pprof server {"listen_addr": "localhost:6060"}
2023-01-12T10:08:31.826-0500 INFO (sink-kv) reading substreams manifest {"manifest_path": "https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.4.0/substreams-eth-block-meta-v0.4.0.spkg"}
2023-01-12T10:08:32.186-0500 INFO (sink-kv) validating output store {"output_store": "kv_out"}
2023-01-12T10:08:32.186-0500 INFO (sink-kv) resolved block range {"start_block": 0, "stop_block": 0}
2023-01-12T10:08:32.186-0500 INFO (sink-kv) starting to listen on {"addr": "localhost:8000"}
2023-01-12T10:08:32.186-0500 INFO (sink-kv) starting stats service {"runs_each": "2s"}
2023-01-12T10:08:32.186-0500 INFO (sink-kv) no block data buffer provided. since undo steps are possible, using default buffer size {"size": 12}
2023-01-12T10:08:32.186-0500 INFO (sink-kv) starting stats service {"runs_each": "2s"}
2023-01-12T10:08:32.186-0500 INFO (sink-kv) ready, waiting for signal to quit
2023-01-12T10:08:32.186-0500 INFO (sink-kv) launching server {"listen_addr": "localhost:8000"}
2023-01-12T10:08:32.187-0500 INFO (sink-kv) serving plaintext {"listen_addr": "localhost:8000"}
2023-01-12T10:08:32.278-0500 INFO (sink-kv) session init {"trace_id": "a3c59bd7992c433402b70f9541565d2d"}
2023-01-12T10:08:34.186-0500 INFO (sink-kv) substreams sink stats {"db_flush_rate": "10.500 flush/s (21 total)", "data_msg_rate": "0.000 msg/s (0 total)", "progress_msg_rate": "0.000 msg/s (0 total)", "block_rate": "0.000 blocks/s (0 total)", "flushed_entries": 0, "last_block": "None"}
2023-01-12T10:08:34.186-0500 INFO (sink-kv) substreams sink stats {"progress_msg_rate": "16551.500 msg/s (33103 total)", "block_rate": "10941.500 blocks/s (21883 total)", "last_block": "#291883 (66d03f819dde948b297c8d582889246d7ba11a5b947335497f8716a7b608f78e)"}
```

--------------------------------

### URL encoding examples

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/sql/dsn-reference.md

Examples of DSNs with special characters in password and username that must be URL-encoded.

```bash
# Special characters in password
postgres://myuser:my%40password@localhost:5432/mydatabase
```

```bash
# Special characters in username  
postgres://my%2Buser:mypassword@localhost:5432/mydatabase
```

--------------------------------

### Install Rust for Substreams development

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/cli/installing-the-cli.md

Installs Rust using the official rustup script and configures the current shell. Required for building Substreams modules.

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env # to configure your current shell
```

--------------------------------

### Webhook sink log output example

Source: https://github.com/streamingfast/substreams/blob/develop/sink/webhook/README.md

Example log lines showing the webhook sink's logging behavior, including successful calls, failures, and retries.

```text
INFO calling webhook block=12345 url=https://api.example.com/webhook
WARN webhook call failed for block 12345: webhook returned server error status 503 for block 12345
INFO calling webhook block=12345 url=https://api.example.com/webhook (retry 1/3)
INFO webhook call successful block=12345 url=https://api.example.com/webhook
```

--------------------------------

### Setup SQL sink tables

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/sql/db_out.md

Creates the necessary tables from a given schema.sql file for a db_out module. Replace <SUBSTREAMS_PACKAGE> with the path to your Substreams package and <DSN> with your database connection string.

```bash
substreams sink postgres setup <SUBSTREAMS_PACKAGE> --dsn <DSN>
```

--------------------------------

### Run the Go sinker CLI with endpoint, package, and module

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/stream/go.md

Command structure to run the example CLI using `go run . sink <ENDPOINT> <SPKG> <MODULE_NAME>`. The parameters are the Substreams endpoint, the package, and the module to execute.

```bash
go run . sink <ENDPOINT> <SPKG> <MODULE_NAME>
```

```bash
go run . sink mainnet.eth.streamingfast.io:443 https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.5.1/substreams-eth-block-meta-v0.5.1.spkg db_out
```

--------------------------------

### Setup sink tables with DSN

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/sql/db_out.md

Configures the sink to create the necessary tables from the schema.sql file specified in the substreams.yaml.

```bash
substreams sink postgres setup ./substreams.yaml --dsn $DSN
```

--------------------------------

### sink postgres command surface

Source: https://github.com/streamingfast/substreams/blob/develop/plans/2026-08-12-coherent-sink-sql-flags.md

Shows the full command surface for sink postgres, including the run action and subcommands for setup, constraints, and tools. The root command is the run action; setup creates the schema, constraints apply/drop manage constraints, and tools are unchanged.

```text
sink postgres [<manifest> [<module>]]      run
sink postgres setup <manifest>             create the schema
sink postgres constraints apply <manifest> create constraints on a loaded database
sink postgres constraints drop <manifest>  drop them again
sink postgres tools ...                    unchanged
```

--------------------------------

### CLI override examples

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/sql/sink-config.md

Commands demonstrating how to use the sink with the YAML configuration, override the module name, and use a published package.

```bash
# Use sink config from substreams.yaml
substreams sink postgres ./substreams.yaml --dsn $DSN
```

```bash
# Override module name (ignores sink config)
substreams sink postgres ./substreams.yaml my_custom_module --dsn $DSN
```

```bash
# Use published package (no local config needed)
substreams sink postgres https://github.com/example/package.spkg --dsn $DSN
```

--------------------------------

### Override manifest example

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Example of an override manifest for porting a substreams to another network, using the `deriveFrom` field to reference the original substreams.

```yaml
deriveFrom: path/to/mainnet-substreams.spkg #this can also be a remote url

package:
  name: "polygon-substreams"
  version: "100.0.0"

network: polygon

initialBlocks:
  module1: 17500000
params:
  module1: "address=2a75ca72679cf1299936d6104d825c9654489058"
```

--------------------------------

### Create transfer table

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/sql/reorg-handling.md

Defines the example 'transfer' table used throughout the walkthrough.

```sql
CREATE TABLE transfer (
    id     varchar PRIMARY KEY,
    "from" varchar,
    "to"   varchar,
    value  bigint
);
```

--------------------------------

### Protobuf definition with load_from_file and zip_from_folder options

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Example protobuf definition showing how to use the `load_from_file` and `zip_from_folder` field options in sink configs. The `schema` field loads content from a file, while `extra_config_files` creates a zip archive.

```protobuf
import "sf/substreams/v1/options.proto";

message HostedPostgresDatabase {
  bytes schema = 1 [ (sf.substreams.v1.options).load_from_file = true ];
  bytes extra_config_files = 2 [ (sf.substreams.v1.options).zip_from_folder = true ];
}
```

--------------------------------

### Web package.json

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/sinks/stream/javascript.md

Defines the Web example's dependencies, including @substreams/core, @substreams/manifest, and @connectrpc/connect-web. Includes Vite scripts for development and building. Use this for client-side JavaScript applications.

```json
{
  "name": "substreams-js-web-example",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "dependencies": {
    "@substreams/core": "^0.1.19",
    "@substreams/manifest": "^0.0.9",
    "@connectrpc/connect-web": "1.4.0",
    "@connectrpc/connect": "1.4.0"
  },
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "preview": "vite preview"
  },
  "devDependencies": {
    "vite": "^5.1.6"
  }
}
```

--------------------------------

### Build and run commands

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/evm/eth-calls.md

The `build` command compiles the Substreams. The `run` command executes it against the mainnet, with the given start and stop blocks.

```bash
substreams build
```

```bash
substreams run -e mainnet.eth.streamingfast.io:443 substreams.yaml map_events --start-block 12292922 --stop-block +1
```

--------------------------------

### Run solana_common blocks_without_votes module with Substreams CLI

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/solana/migrate-from-yellowstone.md

Runs the `blocks_without_votes` module from the `solana_common-v0.3.3.spkg` package using the Substreams GUI. The `--start-block=320100000` flag specifies the starting block. Press `Enter` to start streaming, `Tab` to move across tabs, `o`/`p` to move across blocks, and `q` to exit.

```bash
substreams gui https://spkg.io/streamingfast/solana_common-v0.3.3.spkg blocks_without_votes --start-block=320100000
```

--------------------------------

### Run Substreams with CLI

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/solana/explore-solana/dex-trades.md

Use the Substreams CLI to run the Substreams against the Solana mainnet endpoint. The command starts the GUI, specifies the endpoint, the manifest file, the output map, and the start block with a range of +1 block.

```bash
substreams gui -e mainnet.sol.streamingfast.io:443 \
    substreams.yaml map_block -s 138616676 -t +1
```

--------------------------------

### Environment variable substitution example

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/sql/dsn-reference.md

Shows how to use environment variables in a DSN string and pass it to the sink command.

```bash
# Using environment variables
export DB_USER="myuser"
export DB_PASS="mypassword"
export DSN="postgres://${DB_USER}:${DB_PASS}@localhost:5432/mydatabase"

substreams sink postgres ./substreams.yaml --dsn $DSN
```

--------------------------------

### Initialize RpcBatch

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/evm/eth-calls.md

Starts a new RpcBatch instance to group multiple RPC calls into one request. This is the first step in batching eth_calls.

```rust
fn get_calls() {
    let batch = substreams_ethereum::rpc::RpcBatch::new();

}
```

--------------------------------

### Basic webhook sink usage

Source: https://github.com/streamingfast/substreams/blob/develop/sink/webhook/README.md

Shows the basic command to call a webhook with default settings and with a custom start block.

```bash
# Basic webhook call with default settings
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name

# With custom start block
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name --start-block 12345
```

--------------------------------

### Build and deploy prebuilt binary

Source: https://github.com/streamingfast/substreams/blob/develop/sink/sql/db_proto/benchmarks/README.md

Builds a CGO-free Linux binary on any machine with Go, copies it with the benchmark scripts to the Ubuntu host, then runs the benchmark. Requires Go on the build machine and SUBSTREAMS_BIN on the host.

```bash
# on any machine with Go:
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o substreams ./cmd/substreams
scp substreams live-benchmark.sh live-report.py ubuntu-host:~/bench/

# on the Ubuntu host:
cd ~/bench && SUBSTREAMS_BIN=./substreams ./live-benchmark.sh
```

--------------------------------

### npm install && npm run generate && npm run deploy-local && npm run remove-local

Source: https://github.com/streamingfast/substreams/blob/develop/codegen/templates/subgraph/README.md

These bash commands handle the local development lifecycle. Install dependencies, generate the subgraph, deploy it to the local graph node, and remove it to restart the process.

```bash
npm install
npm run generate
npm run deploy-local

# rince and repeat
npm run remove-local
```

--------------------------------

### Verify authentication with substreams gui

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/cli/authentication.md

Runs a test Substreams against Ethereum Mainnet. If the stream starts without an authentication error, credentials are in place.

```bash
substreams gui ethereum-common@v0.3.1 all_events --start-block=15000000
```

--------------------------------

### Start foundational-store server with configuration flags

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/foundational-stores/hosting-foundational-stores.md

Runs the foundational-store server using specified as Badger storage path, and other configuration flags for the server: --dsn, --type-url, --manifest-path, --output-module-name, --endpoint, --start-block. This command assumes a specific backend is set up.

```bash
./foundational-store server \
  --dsn "badger:///path/to/data" \
  --type-url "your.module.Type" \
  --manifest-path "../your-substreams-module/substreams.yaml" \
  --output-module-name "your_output_module" \
  --endpoint "your-blockchain.streamingfast.io:443" \
  --start-block 1000000
```

--------------------------------

### Run solana_common blocks_without_votes module with JavaScript (Node)

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/solana/migrate-from-yellowstone.md

Runs the `blocks_without_votes` module from the `solana_common-v0.3.3.spkg` package using the JavaScript example from the Yellowstone Examples repository. The command connects to the StreamingFast Solana mainnet endpoint and starts at block 320876956.

```bash
node index.js https://mainnet.sol.streamingfast.io:443 https://spkg.io/streamingfast/solana_common-v0.3.3.spkg blocks_without_votes 320876956
```

--------------------------------

### Manifest excerpt: package

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/substreams-components/manifests.md

Excerpt from the example Substreams manifest showing the package section with name, version, and doc fields.

```yaml
package:
  name: module_name_for_project
  version: v0.5.0
  doc: |
    Documentation heading for the package.

    More detailed documentation for the package.
```

--------------------------------

### Check Substreams CLI Version

Source: https://github.com/streamingfast/substreams/blob/develop/codegen/templates/sql/README.md

Run this command to verify that the `substreams` CLI (v1.7.2 or above) is installed and available.

```bash
substreams --version
```

--------------------------------

### Build foundational-store from source

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/foundational-store-reference.md

Clones the repository and builds the foundational-store binary using Go.

```bash
git clone https://github.com/streamingfast/substreams-foundational-store
cd substreams-foundational-store
go build -o foundational-store ./cmd/foundational-store
```

--------------------------------

### Clone the Substreams Explorers repository

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/evm/exploring-ethereum/exploring-ethereum.md

Clones the substreams-explorers GitHub repository to access all available Substreams Explorers. Run this command in your terminal to get started.

```bash
git clone https://github.com/streamingfast/substreams-explorers
```

--------------------------------

### Declare a module using all_transactions as input

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/cosmos/injective/foundational.md

Example manifest declaration of a custom module that uses the all_transactions foundational module as input. This is the required setup to consume the foundational module's output in your own Substreams.

```yaml
- name: my_test_module
  kind: map
  inputs:
    - map: all_transactions
  output:
    type: proto:sf.test.MyOutputObject
```

--------------------------------

### Run Substreams CLI to get account changes history

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/solana/migrate-from-yellowstone.md

Use the Substreams CLI to stream filtered account changes from the solana-accounts-foundational package. The command specifies the package URL, module name, and a filter parameter for the account 5Q544fKrFoe6tsEbD7S8EmxGTJYAKtTVhAW5Q5pge4j1, starting at block 327404502.

```bash
substreams gui https://spkg.io/streamingfast/solana_accounts_foundational-v0.1.1.spkg filtered_accounts -p "filtered_accounts=account:5Q544fKrFoe6tsEbD7S8EmxGTJYAKtTVhAW5Q5pge4j1" --start-block=327404502
```

--------------------------------

### Run JavaScript (Node) script to get account changes history

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/solana/migrate-from-yellowstone.md

Use a Node.js script to stream filtered account changes from the solana-accounts-foundational package. The command specifies the StreamingFast endpoint, package URL, module name, start block, and filter parameter for the account 5Q544fKrFoe6tsEbD7S8EmxGTJYAKtTVhAW5Q5pge4j1.

```bash
node index.js https://accounts.mainnet.sol.streamingfast.io:443 https://spkg.io/streamingfast/solana_accounts_foundational-v0.1.1.spkg filtered_accounts 327404502 filtered_accounts=account:5Q544fKrFoe6tsEbD7S8EmxGTJYAKtTVhAW5Q5pge4j1
```

--------------------------------

### Initialize HardHat Project with npx hardhat --init

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/evm/local-development/hardhat.md

Initialize a new HardHat project. Follow the interactive prompts to choose Hardhat 3 Beta, use '.' as the relative path, select a TypeScript project with Node Test Runner and Viem, and install dependencies.

```bash
npx hardhat --init
```

--------------------------------

### Command-line parameter override example

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Shows how to pass module parameters via the command line using the `-p` flag. Multiple parameters can be specified with separate `-p` flags.

```bash
substreams run -p module=value -p "module2=other value" ...
```

--------------------------------

### Run Substreams with default parameters

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/solana/explore-solana/token-tracker.md

Runs the Substreams using the Substreams CLI with the default manifest parameters, which track USDC. The command uses the mainnet Solana endpoint and specifies a start block and a stop block of +1.

```bash
substreams gui ./substreams.yaml map_solana_token_events -e mainnet.sol.streamingfast.io:443  --start-block 158558168 --stop-block +1
```

--------------------------------

### Clone and Load Substreams Skills Locally in Claude Code

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/generic/agent-skills.md

Alternative installation: clones the repository and run Claude Code with the plugin directory pointing to the cloned repo.

```bash
git clone https://github.com/streamingfast/substreams-skills.git
claude --plugin-dir ./substreams-skills
```

--------------------------------

### Initialize Substreams project with substreams init

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/evm/eth-calls.md

Run `substreams init` to scaffold a new project. Then provide the project name, protocol, chain, and the USDT contract address `0xdac17f958d2ee523a2206206994597c13d831ec7` when prompted.

```bash
substreams init
```

```bash
Project name (lowercase, numbers, underscores): usdttracker
Protocol: Ethereum
Ethereum chain: Mainnet
Contract address to track (leave empty to use "Bored Ape Yacht Club"): 0xdac17f958d2ee523a2206206994597c13d831ec7
Would you like to track another contract? (Leave empty if not):
Retrieving Ethereum Mainnet contract information (ABI & creation block)
Fetched contract ABI for dac17f958d2ee523a2206206994597c13d831ec7
Fetched initial block 4634748 for dac17f958d2ee523a2206206994597c13d831ec7 (lowest 4634748)
Generating ABI Event models for
  Generating ABI Events for AddedBlackList (_user)
  Generating ABI Events for Approval (owner,spender,value)
  Generating ABI Events for Deprecate (newAddress)
  Generating ABI Events for DestroyedBlackFunds (_blackListedUser,_balance)
  Generating ABI Events for Issue (amount)
  Generating ABI Events for Params (feeBasisPoints,maxFee)
  Generating ABI Events for Redeem (amount)
  Generating ABI Events for RemovedBlackList (_user)
  Generating ABI Events for Transfer (from,to,value)
Writing project files
Generating Protobuf Rust code
```

--------------------------------

### Validate Substreams CLI installation with --version

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/cli/installing-the-cli.md

Checks that the Substreams CLI is installed correctly by running the --version flag. A successful installation prints the version, e.g., 'substreams version dev'.

```bash
substreams --version
```

```bash
substreams version dev
```

--------------------------------

### Run live-bmark.sh with environment variables

Source: https://github.com/streamingfast/substreams/blob/develop/sink/sql/db_proto/benchmarks/README.md

Shows how to configure and run the live-benchmark.sh script. Set SUBSTREAMS_API_KEY to your API key, optionally override SIZES and ENDPOINT, and use WARM=1 to benchmark an unwarmed range. The script builds the binary, starts a PostgreSQL container, runs both variants at each size, and appends results to results.tsv.

```bash
export SUBSTREAMS_API_KEY=...

cd sink/sql/db_proto/benchmarks
./live-benchmark.sh                                    # 10k and 50k blocks
SIZES="10000 50000 200000" ./live-benchmark.sh
ENDPOINT=https://mainnet.eth.ca.streamingfast.io ./live-benchmark.sh
WARM=1 ./live-benchmark.sh                             # range nobody has streamed yet
```

--------------------------------

### Launch Sinker with Run

Source: https://github.com/streamingfast/substreams/blob/develop/sink/README.md

Launches the sinker by calling Run with a context, cursor, and handlers. The Run method blocks until the sinker is stopped or encounters an error. The cursor can start from the beginning or load from a checkpoint.

```go
ctx := context.Background()
cursor, err := sink.NewCursor("") // Start from beginning, or load from checkpoint
if err != nil {
    return fmt.Errorf("creating cursor: %w", err)
}

// Run the sinker with your handlers
err = sinker.Run(ctx, cursor, handlers)
if err != nil {
    return fmt.Errorf("sinker failed: %w", err)
}
```

--------------------------------

### Module parameterization manifest example

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Defines a module with a parameter and provides parameter overrides in the manifest. The `params` section maps module names to parameter values, including imported modules.

```yaml
module:
  name: my_module
  inputs:
    params: string
  ...

params:
  my_module: "0x123123"
  "imported:module": override value from imported module
```

--------------------------------

### x-api-key metadata example

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/hosted-services/hosted-store-substreams-feed.md

Shows the gRPC metadata header used to authenticate calls to the hosted store. The API key is created automatically when the store is created; manage keys at thegraph.market/api-keys.

```text
x-api-key: <api-key>
```

--------------------------------

### Install Substreams CLI from pre-compiled binary (macOS)

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/cli/installing-the-cli.md

Downloads and extracts the latest Substreams CLI binary for macOS using curl and awk to determine the correct download URL.

```bash
LINK=$(curl -s https://api.github.com/repos/streamingfast/substreams/releases/latest | awk "/download.url.*$(uname -s | tr '[:upper:]' '[:lower:]')_$(uname -m)/ {print \$2}" | sed 's/"//g')
curl -L  $LINK  | tar zxf -
```

--------------------------------

### Update Substreams CLI via curl or Homebrew

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/generic/local-development/troubleshooting.md

Update the Substreams CLI to the latest version using either the curl installer or Homebrew. The curl command downloads and runs the official installer, while the Homebrew command installs the tap and package.

```bash
# Using curl
curl -sSf https://substreams.streamingfast.io/install | bash

# Or using Homebrew
brew install streamingfast/tap/substreams
```

--------------------------------

### Build command proto generation status messages

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/cli/command-line-interface.md

Example outputs from the build command showing when proto generation is skipped or run based on hash-based caching. The hash is stored in a .last_generated_hash file.

```bash
# Example output when proto definitions haven't changed:
Proto definitions unchanged, skipping buf generate (hash: a1b2c3d4e5f6)
```

```bash
# Example output when proto definitions have changed:
Proto definitions changed or no previous generation found, running buf generate (hash: f6e5d4c3b2a1)
```

--------------------------------

### Manifest binary type before enabling wasm-bindgen-shims

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Original Substreams manifest binary type without the `+wasm-bindgen-shims` suffix. Used as the starting point for enabling the feature.

```yaml
binaries:
  default:
    type: wasm/rust-v1
    file: <some_file>
```

--------------------------------

### Install Substreams CLI from pre-compiled binary (Linux)

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/cli/installing-the-cli.md

Downloads and extracts the latest Substreams CLI binary for Linux using curl and awk to determine the correct download URL. Ensure you use the correct binary for your platform.

```bash
# Use correct binary for your platform
LINK=$(curl -s https://api.github.com/repos/streamingfast/substreams/releases/latest | awk "/download.url.*linux_$(uname -m)/ {print \$2}" | sed 's/"//g')
curl -L  $LINK  | tar zxf -
```

--------------------------------

### Create project directory

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/solana/local-development/anchor.md

Creates and enters the project directory for the local Solana development environment.

```bash
mkdir substreams-solana-local
cd substreams-solana-local
```

--------------------------------

### Override substreams.yaml values with pack -c flag

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Example YAML for the `substreams pack -c` flag to override package name, network, initial blocks, and params in substreams.yaml.

```yaml
package:
  name: my_custom_package_name

network: arbitrum-one
initialBlocks:
  module_name_1: 123123123
params:
  mod1: "custom_parameter"
```

--------------------------------

### Run map_filter_transactions from source with GUI

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/solana/explore-solana/filter-transactions.md

Runs the `map_filter_transactions` module using the Substreams GUI, filtering for the example signature. The `--start-block` and `--stop-block` flags limit the range to a single block.

```bash
substreams gui ./substreams.yaml \
    map_filter_transactions -e mainnet.sol.streamingfast.io:443 \
    --start-block 153000028 --stop-block +1
```

--------------------------------

### Clickhouse sink manifest with REST interface (YAML)

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Example manifest segment for a clickhouse sink with REST interface enabled. The `rest_frontend.enabled` flag turns on the REST API, while `postgraphile_frontend` and `pgweb_frontend` are disabled.

```yaml
[...]

sink:
  module: db_out
  type: sf.substreams.sink.sql.v1.Service
  config:
    schema: "./schema.clickhouse.sql"
    wire_protocol_access: true
    engine: clickhouse
    postgraphile_frontend:
      enabled: false
    pgweb_frontend:
      enabled: false
    rest_frontend:
      enabled: true
```

--------------------------------

### Example output of gravatar_updates module

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/cli/command-line-interface.md

Example output from the `gravatar_updates` module when run with `-o json`. The output is an indented stream of data, not displaying progress information or logs.

```bash
{
  "updates": [
    {
      "id": "39",
      "owner": "0xaadcc13071fdf9c73cfbb8d97639ea68aa6fd1d2",
      "displayName": "alex | OpenSea",
      "imageUrl": "https://ucarecdn.com/13a67247-cb89-417a-92d2-50a7d7aa481c/-/crop/382x382/0,0/-/preview/"
    }
  ]
}
...
```

--------------------------------

### YAML: Configure buf build URL in substreams manifest

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Example of using a buf.build URL in the protobuf configuration of a substreams manifest. This allows the manifest to reference remote protobuf definitions from the Buf Schema Registry.

```yaml
protobuf:
  buf_build:
    - buf.build/streamingfast/firehose-cosmos

```

--------------------------------

### Initialize Substreams Project

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/evm/local-development/hardhat.md

Run `substreams init` and follow the interactive prompts to generate a Substreams module for an EVM contract. The prompts require selecting the EVM protocol, the evm-events-calls generator, and providing contract details like address, ABI path, and initial block.

```bash
substreams init
```

--------------------------------

### Sink manifest with DBT configuration (YAML)

Source: https://github.com/streamingfast/substreams/blob/develop/docs/release-notes/change-log.md

Example manifest segment for a sink server deployment with DBT configuration. The `dbt.files` path points to a folder containing the dbt project, and `run_interval_seconds` controls how often the dbt models are run.

```yaml
[...]

sink:
  module: db_out
  type: sf.substreams.sink.sql.v1.Service
  config:
    schema: "./schema.sql"
    wire_protocol_access: true
    postgraphile_frontend:
      enabled: true
    pgweb_frontend:
      enabled: true
    dbt:
      files: "./dbt"
      run_interval_seconds: 60
```

--------------------------------

### Install Substreams Skills Plugin in Claude Code

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/generic/agent-skills.md

Adds the Substreams skills plugin from the marketplace. Then enable via plugin manager and install the first. Used with Claude Code marketplace.

```bash
claude plugin marketplace add https://github.com/streamingfast/substreams-skills
```

--------------------------------

### Initialize a Rust project with cargo init --lib

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/substreams-components/modules/setting-up-handlers.md

Use this command to create an empty Rust library project suitable for WASM compilation.

```bash
# Creates a empty Rust project suitable for WASM compilation
cargo init --lib
```

--------------------------------

### Container name conflict error example

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/generic/local-development/troubleshooting.md

Example error message when a container name conflict occurs. It indicates the conflicting container name and ID, and instructs to remove or rename the container.

```text
Error response from daemon: Conflict. The container name "/ethereum-dev-node" is already in use by container "10d7ad16b7895f4a0d714778b2daa222686517444e81e289ac142dc32f5e42d6". You have to remove (or rename) that container to be able to reuse that name.
```

--------------------------------

### Full Migration: Before

Source: https://github.com/streamingfast/substreams/blob/develop/sink/MIGRATION.md

The original implementation prior to adopting the new API standards.

```go
// Before
import "github.com/streamingfast/substreams-sink"

func setupFlags(flags *pflag.FlagSet) {
    sink.AddFlagsToSet(flags, sink.FlagIgnore(sink.FlagFinalBlocksOnly))
}

func createSinker(cmd *cobra.Command) (*sink.Sinker, error) {
    return sink.NewFromViper(
        cmd,
        "sf.substreams.database.v1.DatabaseChanges",
        endpoint,
        manifestPath,
        moduleName,
        blockRange,
        zlog,
        tracer,
    )
}
```

--------------------------------

### Define a `store` input with default `get` mode

Source: https://github.com/streamingfast/substreams/blob/develop/docs/references/substreams-components/modules/inputs.md

Defines a `store` input in the Substreams manifest. Store modules are set to `get` mode by default, which provides a read-only key-value store in sync with the block being processed.

```yaml
  inputs:
    - store: my_store # defaults to mode: get
```

--------------------------------

### Troubleshoot container startup with docker compose commands

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/develop-your-own-substreams/generic/local-development/troubleshooting.md

Use these commands to diagnose and resolve container startup failures. Check logs and status first, then restart services; use --volumes for a clean restart that removes volumes.

```bash
# Check container logs
docker compose logs <service-name>

# Check all container statuses
docker compose ps

# Restart services
docker compose down
docker compose up -d

# Clean restart (removes volumes)
docker compose down --volumes
docker compose up -d
```

--------------------------------

### Run substreams auth

Source: https://github.com/streamingfast/substreams/blob/develop/docs/how-to-guides/cli/authentication.md

Opens a browser to sign in and select an organization and API key. On success writes .substreams.env with SUBSTREAMS_API_TOKEN.

```bash
substreams auth
```
Tokens:
Raw