Shovel

https://indexsupply.com/shovel/docs
Admin
Shovel is a program that indexes data from Ethereum nodes into a Postgres database, supporting...
Tokens:11,412
Snippets:58
Trust Score:5.5
Update:1 year ago
Show doc for...
Tokens:
Raw
### Shovel Installation and Execution Commands

Source: https://indexsupply.com/shovel/docs/index

Commands to install Shovel, set up PostgreSQL, download a demo configuration, and run the Shovel executable. Includes package manager commands, database setup, file downloads, and execution with a configuration file.

```bash
brew install postgresql@16
brew services start postgresql@16
createdb shovel

curl -LO --silent https://raw.githubusercontent.com/indexsupply/code/main/cmd/shovel/demo.json
curl -LO --silent https://indexsupply.net/bin/1.6/darwin/arm64/shovel
chmod +x shovel

./shovel -config demo.json
```

--------------------------------

### Shovel Log Output Example

Source: https://indexsupply.com/shovel/docs/index

Example log output from running the Shovel executable. Shows informational messages about task processing, starting at the latest block, and convergence progress with details on RPC requests and block processing.

```text
l=info  v=1.6 msg=new-task ig=usdc-transfer src=mainnet
l=info  v=1.6 msg=new-task ig=usdc-transfer src=base
l=info  v=1.6 msg=prune-task n=0
l=info  v=1.6 msg=start at latest ig=usdc-transfer src=mainnet num=19793295
l=info  v=1.6 msg=start at latest ig=usdc-transfer src=base num=13997369
l=info  v=1.6 msg=converge ig=usdc-transfer host=base-rpc.publicnode.com src=base req=15507896/1 n=15507896 h=edadbe50 nrows=11 nrpc=3 nblocks=1 elapsed=817.485916ms
l=info  v=1.6 msg=converge ig=usdc-transfer host=ethereum-rpc.publicnode.com src=mainnet req=20043416/1 n=20043416 h=949c4c7e nrows=3 nrpc=3 nblocks=1 elapsed=984.472375ms
```

--------------------------------

### Install Shovel Config TypeScript Package

Source: https://indexsupply.com/shovel/docs/index

Installs the necessary TypeScript package for Shovel configuration management using Bun.

```Shell
bun install @indexsupply/shovel-config
```

--------------------------------

### Install Shovel on Linux

Source: https://indexsupply.com/shovel/docs/index

Downloads the Shovel binary for Linux (amd64 architecture) using curl and makes it executable.

```Shell
curl -LO https://indexsupply.net/bin/1.6/linux/amd64/shovel
chmod +x shovel
```

--------------------------------

### Example Shovel JSON Configuration

Source: https://indexsupply.com/shovel/docs/index

A basic JSON configuration structure for Shovel, specifying enabled status, Ethereum sources, and table details for indexing.

```json
{
    "enabled": true,
    "sources": [{"name": "mainnet"}],
    "table": {
      "name": "tx",
      "columns": [
        {"name": "tx_input", "type": "bytea"}
      ]
    },
    "block": [
      {
        "name": "tx_input",
        "column": "tx_input",
        "filter_op": "contains",
        "filter_arg": ["a0712d68"]
      }
    ]
  }
```

--------------------------------

### Install Shovel on Mac

Source: https://indexsupply.com/shovel/docs/index

Downloads the Shovel binary for macOS (arm64 architecture) using curl and makes it executable.

```Shell
curl -LO https://indexsupply.net/bin/1.6/darwin/arm64/shovel
chmod +x shovel
```

--------------------------------

### Install and Run Shovel with Demo Config (Bash)

Source: https://indexsupply.com/shovel/docs/shovel

This sequence of bash commands demonstrates how to set up and run Shovel. It includes creating a PostgreSQL database named 'shovel', downloading a demo configuration file, downloading the Shovel executable, making it executable, and then running Shovel with the downloaded configuration.

```Bash
# assumes you have pg running
createdb shovel

# download demo config file
curl -LO https://raw.githubusercontent.com/indexsupply/code/main/cmd/shovel/demo.json

# download shovel
curl -LO https://indexsupply.net/bin/1.6/darwin/arm64/shovel
chmod +x shovel
./shovel -config demo.json

```

--------------------------------

### TypeScript Shovel Configuration Example

Source: https://indexsupply.com/shovel/docs/index

Demonstrates how to define a Shovel configuration using TypeScript, including data sources, tables, and event integrations. The output is converted to JSON for Shovel to use.

```TypeScript
import { makeConfig, toJSON } from "@indexsupply/shovel-config";
import type { Source, Table, Integration } from "@indexsupply/shovel-config";

const table: Table = {
  name: "transfers",
  columns: [
    { name: "log_addr", type: "bytea" },
    { name: "from", type: "bytea" },
    { name: "to", type: "bytea" },
    { name: "amount", type: "numeric" },
  ],
};

const mainnet: Source = {
  name: "mainnet",
  chain_id: 1,
  url: "https://ethereum-rpc.publicnode.com",
};

let integrations: Integration[] = [
  {
    enabled: true,
    name: "transfers",
    sources: [{ name: mainnet.name, start: 0n }],
    table: table,
    block: [
      {
        name: "log_addr",
        column: "log_addr",
        filter_op: "contains",
        filter_arg: ["0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"],
      },
    ],
    event: {
      type: "event",
      name: "Transfer",
      inputs: [
        { indexed: true, name: "from", type: "address", column: "from" },
        { indexed: true, name: "to", type: "address", column: "to" },
        { indexed: false, name: "amount", type: "uint256", column: "amount" },
      ],
    },
  },
];

const config = makeConfig({
  pg_url: "postgres:///shovel",
  sources: [mainnet],
  integrations: integrations,
});

console.log(toJSON(config));

```

--------------------------------

### Shovel Event Filter Example

Source: https://indexsupply.com/shovel/docs/index

This example shows a Shovel event filter configuration for a 'Transfer' event. It includes input fields like 'from', 'to', and 'value', with specific column mappings and a filter for the 'from' address.

```json
{
    "integrations": [
        {
            "event": {
                "name": "Transfer",
                "type": "event",
                "anonymous": false,
                "inputs": [
                    {
                        "indexed": true,
                        "name": "from",
                        "type": "address",
                        "column": "f",
                        "filter_op": "contains",
                        "filter_arg": ["0x0000000000000000000000000000000000000000"]
                    },
                    {
                        "indexed": true,
                        "name": "to",
                        "type": "address",
                        "column": "t"
                    },
                    {
                        "indexed": false,
                        "name": "value",
                        "type": "uint256",
                        "column": "v"
                    }
                ]
            }
        }
    ]
}
```

--------------------------------

### Shovel Dashboard Configuration - Enable Loopback Authentication

Source: https://indexsupply.com/shovel/docs/index

Example JSON configuration for the Shovel dashboard, enabling loopback authentication. This is useful for testing authentication code locally.

```json
{
  "...": "...",
  "dashboard": {
    "...": "...",
    "enable_loopback_authn": true
  }
}
```

--------------------------------

### Shovel Block Filter Example

Source: https://indexsupply.com/shovel/docs/index

This example demonstrates how to configure a block filter in Shovel. It specifies a filter for the 'log_addr' column, checking if it contains the address '0xabc'.

```json
{
    "integrations": [
        {
            "block": [
                {
                    "name": "log_addr",
                    "column": "log_addr",
                    "filter_op": "contains",
                    "filter_arg": ["0xabc"]
                }
            ]
        }
    ]
}
```

--------------------------------

### Configure Backfilling with Concurrency

Source: https://indexsupply.com/shovel/docs/index

This configuration enables efficient backfilling of Ethereum data by setting concurrency and batch size parameters. It defines an integration named 'fast' with a concurrency of 10 and a batch size of 100, starting from block 1.

```json
{
  "pg_url": "postgres:///shovel",
  "eth_sources": [
    {
      "name": "fast",
      "chain_id": 1,
      "urls": ["https://ethereum-rpc.publicnode.com"],
      "concurrency": 10,
      "batch_size": 100
    }
  ],
  "integrations": [{
    "name": "fast",
    "enabled": true,
    "sources": [{"name": "fast", "start": 1}],
    "table": {"name": "fast", "columns": []},
    "block": [],
    "event": {}
  }]
}
```

--------------------------------

### Shovel Filter Operations Example

Source: https://indexsupply.com/shovel/docs/index

Demonstrates how to use Shovel's filter operations for binary data and strings. It shows the 'contains', '!contains', 'eq', 'ne', 'gt', and 'lt' operations with their respective input and filter_arg formats.

```json
filter_op: contains, !contains
    input: binary data
    filter_arg: json array of hex encoded, 0x-prefixed bytes

    input: string
    filter_arg: json array of utf8 encoded strings

filter_op: eq, ne
    input: binary data
    filter_arg: json array of hex encoded, 0x-prefixed bytes

    input: string
    filter_arg: json array of utf8 encoded strings

    input: int/uint
    filter_arg: json array of a single 256bit number encoded as decimal

filter_op: lt, gt
    input: int/uint
    filter_arg: json array of a single 256bit number encoded as decimal
```

--------------------------------

### Shovel Dashboard Configuration - Root Password

Source: https://indexsupply.com/shovel/docs/index

Example JSON configuration for the Shovel dashboard, showing how to set or unset the root password. An empty string indicates a new password will be generated on startup.

```json
{
  "...": "...",
  "dashboard": {
    "...": "...",
    "root_password": ""
  }
}
```

--------------------------------

### Shovel Block Filter with Filter Reference

Source: https://indexsupply.com/shovel/docs/index

This example demonstrates a Shovel block filter using `filter_ref`. The 'from' address in the 'Transfer' event is filtered based on the 'addr' column of the 'factory_contracts' integration.

```json
{
    "integrations": [
        {
            "name": "factory_contracts",
            "table": {
                "name": "factory_contracts",
                "columns": [
                    {"name": "log_addr", "type": "bytea"},
                    {"name": "addr", "type": "bytea"}
                ]
            }
        },
        {
            "event": {
                "name": "Transfer",
                "type": "event",
                "anonymous": false,
                "inputs": [
                    {
                        "indexed": true,
                        "name": "from",
                        "type": "address",
                        "column": "f",
                        "filter_ref": {
                            "integration": "factory_contracts",
                            "column": "addr"
                        }
                    },
                    {
                        "indexed": true,
                        "name": "to",
                        "type": "address",
                        "column": "t"
                    },
                    {
                        "indexed": false,
                        "name": "value",
                        "type": "uint256",
                        "column": "v"
                    }
                ]
            }
        }
    ]
}
```

--------------------------------

### Shovel Dashboard Configuration - Disable Authentication

Source: https://indexsupply.com/shovel/docs/index

Example JSON configuration for the Shovel dashboard, disabling authentication entirely. Use with caution.

```json
{
  "...": "...",
  "dashboard": {
    "...": "...",
    "disable_authn": true
  }
}
```

--------------------------------

### Shovel Error Logging

Source: https://indexsupply.com/shovel/docs/index

This snippet shows an example of an error message logged by Shovel when an Ethereum node provider is unsynchronized. It indicates a problem with retrieving logs for a specific block.

```Go
error=getting logs: eth backend missing logs for block
```

--------------------------------

### Build Shovel from Source

Source: https://indexsupply.com/shovel/docs/index

Builds and runs the Shovel application from source code using Go.

```Go
cd shovel
go run ./cmd/shovel
```

--------------------------------

### Shovel Binary Download URLs

Source: https://indexsupply.com/shovel/docs/index

Direct URLs for downloading Shovel binaries for different operating systems and architectures. Includes links for the latest stable version (1.6) and the latest version from the main branch.

```url
https://indexsupply.net/bin/1.6/darwin/arm64/shovel
```

```url
https://indexsupply.net/bin/1.6/linux/amd64/shovel
```

```url
https://indexsupply.net/bin/main/darwin/arm64/shovel
```

```url
https://indexsupply.net/bin/main/linux/amd64/shovel
```

--------------------------------

### Test Shovel Version

Source: https://indexsupply.com/shovel/docs/index

Executes the Shovel binary to display its version information, including a git tag and commit hash.

```Shell
./shovel -version
```

--------------------------------

### Print Shovel Schema

Source: https://indexsupply.com/shovel/docs/index

This command-line instruction tells Shovel to parse the configuration file and output the derived schema to standard output. It's useful for testing and understanding the data structure without needing a database connection.

```Shell
shovel -config demo.json --print-schema
```

--------------------------------

### Run TypeScript Configuration and Generate JSON

Source: https://indexsupply.com/shovel/docs/index

Executes the TypeScript configuration file and redirects its JSON output to a file named 'config.json', which is then used by the Shovel command-line tool.

```Shell
bun run shovel-config.ts > config.json
shovel -config config.json
```

--------------------------------

### Clone Shovel Repository

Source: https://indexsupply.com/shovel/docs/index

Clones the Index Supply Shovel repository from GitHub using git clone.

```Shell
git clone https://github.com/indexsupply/code.git shovel
```

--------------------------------

### Shovel Batch Size and Concurrency Calculation

Source: https://indexsupply.com/shovel/docs/index

Defines how Shovel computes data deltas and indexes blocks using batch size and concurrency. It explains how block requests are batched for RPC calls based on integration type (logs, headers, blocks) or made individually.

```go
let db_latest = query_task_updates_table()
let next_to_process = max(db_latest, integrations[].sources[].start)
```

--------------------------------

### Shovel Integration with eth_getLogs and Address Filtering

Source: https://indexsupply.com/shovel/docs/index

Configures a Shovel integration to use `eth_getLogs` and filters logs by a specific contract address to optimize data retrieval.

```JSON
{
  "pg_url": "...",
  "eth_sources": [...],
  "integrations": [{
    ...
    "block": [
      {
        "name": "log_addr",
        "column": "log_addr",
        "filter_op": "contains",
        "filter_arg": [
          "0xa0b86991c6218b36c1d19d4a2e9eb0cE3606eB48",
          "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913"
        ]
      }
    ],
    "event": {
      ...
    }
  }]

}
```

--------------------------------

### Shovel ERC20 Transfer Configuration

Source: https://indexsupply.com/shovel/docs/index

A basic Shovel configuration to index ERC20 token transfers. It defines the PostgreSQL table structure, columns, and maps event inputs to these columns. It also specifies the Ethereum network source and the ERC20 Transfer event ABI.

```json
{
  "pg_url": "postgres:///shovel",
  "eth_sources": [
    {
      "name": "mainnet",
      "chain_id": 1,
      "urls": ["https://ethereum-rpc.publicnode.com"]
    }
  ],
  "integrations": [{
    "name": "erc20_transfers",
    "enabled": true,
    "sources": [{"name": "mainnet"}],
    "table": {
      "name": "erc20_transfers",
      "columns": [
        {"name": "block_num", "type": "numeric"},
        {"name": "tx_hash", "type": "bytea"},
        {"name": "from", "type": "bytea"},
        {"name": "to", "type": "bytea"},
        {"name": "value", "type": "bytea"}
      ]
    },
    "block": [
      {"name": "block_num", "column": "block_num"},
      {"name": "tx_hash", "column": "tx_hash"}
    ],
    "event": {
      "name": "Transfer",
      "type": "event",
      "anonymous": false,
      "inputs": [
        {"indexed": true, "name": "from", "type": "address", "column": "from"},
        {"indexed": true, "name": "to", "type": "address", "column": "to"},
        {"name": "value", "type": "uint256", "column": "value"}
      ]
    }
  }]
}
```

--------------------------------

### Optimize Shovel Concurrency for eth_getLogs

Source: https://indexsupply.com/shovel/docs/index

Sets the `batch_size` to 2000 and `concurrency` to 1 for Shovel's Ethereum sources when primarily indexing data from logs.

```JSON
"eth_sources": [{..., "batch_size": 2000, "concurrency": 1}],

```

--------------------------------

### Describe Public Minimal Table

Source: https://indexsupply.com/shovel/docs/index

This SQL command describes the 'public.minimal' table, outlining its essential columns: integration name, source name, block number, and transaction index. It also specifies a unique index to enforce data uniqueness based on these columns.

```SQL
                Table "public.minimal"
  Column   |  Type   | Collation | Nullable | Default
-----------+---------+-----------+----------+---------
 ig_name   | text    |           |          |
 src_name  | text    |           |          |
 block_num | numeric |           |          |
 tx_idx    | integer |           |          |
Indexes:
    "u_minimal" UNIQUE, btree (ig_name, src_name, block_num, tx_idx)
```

--------------------------------

### Internal ETH Transfers Integration Configuration

Source: https://indexsupply.com/shovel/docs/index

Configuration for the 'internal-eth-transfers' integration, which indexes internal Ethereum transfers via traces. It specifies data sources, table schema, and block-level filtering criteria.

```json
{
    "pg_url": "postgres:///shovel",
    "eth_sources": [{"name": "mainnet", "chain_id": 1, "urls": ["https://ethereum-rpc.publicnode.com"]}],
    "integrations": [
        {
            "name": "internal-eth-transfers",
            "enabled": true,
            "sources": [{"name": "mainnet", "start": 19737332, "stop": 19737333}],
            "table": {
                "name": "internal_eth_transfers",
                "columns": [
                    {"name": "block_hash", "type": "bytea"},
                    {"name": "block_num", "type": "numeric"},
                    {"name": "tx_hash", "type": "bytea"},
                    {"name": "tx_idx", "type": "int"},
                    {"name": "call_type", "type": "text"},
                    {"name": "from", "type": "bytea"},
                    {"name": "to", "type": "bytea"},
                    {"name": "value", "type": "numeric"}
                ]
            },
            "block": [
                {"name": "block_hash", "column": "block_hash"},
                {"name": "block_num", "column": "block_num"},
                {"name": "tx_hash", "column": "tx_hash"},
                {
                    "name": "trace_action_call_type",
                    "column": "call_type",
                    "filter_op": "eq",
                    "filter_arg": ["call"]
                },
                {"name": "trace_action_from", "column": "from"},
                {"name": "trace_action_to", "column": "to"},
                {
                    "name": "trace_action_value",
                    "column": "value",
                    "filter_op": "gt",
                    "filter_arg": ["0"]
                }
            ]
        }
    ]
}
```

--------------------------------

### Shovel Prometheus Metrics

Source: https://indexsupply.com/shovel/docs/index

This section details the Prometheus metrics exposed by Shovel. These metrics provide insights into the Shovel's operational status, including block processing, database ping times, and RPC API request performance.

```Prometheus
# HELP shovel_latest_block_local last block processed
# TYPE shovel_latest_block_local gauge
shovel_latest_block_local{src="mainnet"} 19648035

# HELP shovel_pg_ping number of ms to make basic status query
# TYPE shovel_pg_ping gauge
shovel_pg_ping 0

# HELP shovel_pg_ping_error number of errors in making basic status query
# TYPE shovel_pg_ping_error gauge
shovel_pg_ping_error 0

# HELP shovel_latest_block_remote latest block height from rpc api
# TYPE shovel_latest_block_remote gauge
shovel_latest_block_remote{src="mainnet"} 19648035

# HELP shovel_rpc_ping number of ms to make a basic http request to rpc api
# TYPE shovel_rpc_ping gauge
shovel_rpc_ping{src="mainnet"} 127

# HELP shovel_rpc_ping_error number of errors in making basic rpc api request
# TYPE shovel_rpc_ping_error gauge
shovel_rpc_ping_error{src="mainnet"} 0

# HELP shovel_delta number of blocks between the source and the shovel database
# TYPE shovel_delta gauge
shovel_delta{src="mainnet"} 0
```

--------------------------------

### Describe Shovel Task Updates Table

Source: https://indexsupply.com/shovel/docs/index

This command displays the structure of the 'shovel.task_updates' table, detailing its columns such as block number, hash, integration name, source name, source hash, source number, number of blocks, number of rows, latency, and insert timestamp. It also shows the unique index on integration name, source name, and block number.

```SQL
shovel=# \d shovel.task_updates
```

--------------------------------

### Shovel Table Indexing Configuration

Source: https://indexsupply.com/shovel/docs/index

Details how to configure B-Tree and unique indexes for Shovel tables. It explains how to specify columns for indexing, sort order, and the default unique index behavior, including disabling it.

```sql
{..., "index": [["from", "block_num ASC"]], ...}
```

--------------------------------

### Enable Localhost Authentication for Shovel Dashboard

Source: https://indexsupply.com/shovel/docs/index

Enables authentication specifically for requests originating from localhost, allowing developers to test authentication features.

```JSON
{
  "...": "...",
  "dashboard": {
    "enable_loopback_authn": true
  }
}
```

--------------------------------

### Create USDC Table Schema

Source: https://indexsupply.com/shovel/docs/index

This SQL statement defines the schema for a 'usdc' table, including columns like log address, block time, sender, receiver, value, integration name, source name, block number, transaction index, log index, and ABI index. It also includes a unique index to ensure data integrity.

```SQL
create table if not exists usdc(
	log_addr bytea,
	block_time numeric,
	"from" bytea,
	"to" bytea,
	value numeric,
	ig_name text,
	src_name text,
	block_num numeric,
	tx_idx int,
	log_idx int,
	abi_idx int2
);
create unique index if not exists u_usdc on usdc (
	ig_name,
	src_name,
	block_num,
	tx_idx,
	log_idx,
	abi_idx
);
```

--------------------------------

### Seaport Orders Table Schema

Source: https://indexsupply.com/shovel/docs/index

Defines the schema for the 'seaport_orders' table in the Shovel PostgreSQL database. It includes columns for order hash, indexer and source names, block number, transaction index, and ABI index.

```sql
shovel=# \d seaport_orders
             Table "public.seaport_orders"
   Column   |   Type   | Collation | Nullable | Default
------------+----------+-----------+----------+---------
 order_hash | bytea    |           |          |
 ig_name    | text     |           |          |
 src_name   | text     |           |          |
 block_num  | numeric  |           |          |
 tx_idx     | integer  |           |          |
 log_idx    | smallint |           |          |
 abi_idx    | smallint |           |          |
Indexes:
    "u_seaport_orders" UNIQUE, btree (ig_name, src_name, block_num, tx_idx, log_idx, abi_idx)
```

--------------------------------

### Shovel Dashboard Command Line Flag

Source: https://indexsupply.com/shovel/docs/index

Demonstrates how to specify the listening address for the Shovel dashboard using a command-line flag. The dashboard listens on port 8546 by default.

```bash
./shovel -l :8080
```

--------------------------------

### Filter Transactions by Input Data

Source: https://indexsupply.com/shovel/docs/index

This configuration demonstrates how to ingest Ethereum transactions and filter them based on specific byte patterns within the transaction input data. It specifies the PostgreSQL URL, Ethereum RPC sources, and defines an integration to capture transactions containing 'a1671295' in their input.

```json
{
  "pg_url": "postgres:///shovel",
  "eth_sources": [
    {
      "name": "mainnet",
      "chain_id": 1,
      "urls": ["https://ethereum-rpc.publicnode.com"]
    }
  ],
  "integrations": [
    {
      "name": "specific-tx",
      "enabled": true,
      "sources": [
        {
          "name": "mainnet"
        }
      ],
      "table": {
        "name": "tx",
        "columns": [
          {
            "name": "tx_input",
            "type": "bytea"
          }
        ]
      },
      "block": [
        {
          "name": "tx_input",
          "column": "tx_input",
          "filter_op": "contains",
          "filter_arg": [
            "a1671295"
          ]
        }
      ]
    }
  ]
}
```

--------------------------------

### Shovel Integration Table Schema Definition

Source: https://indexsupply.com/shovel/docs/index

Specifies the structure of PostgreSQL tables used by Shovel integrations. It details required columns, optional columns for specific data types (logs, traces), and how to define custom columns.

```sql
[
  {"name": "ig_name", "type": "text"},
  {"name": "src_name", "type": "text"},
  {"name": "block_num", "type": "numeric"},
  {"name": "tx_idx", "type": "numeric"}
]
```

```sql
{"name": "log_idx", "type": "int"}
```

```sql
{"name": "abi_idx", "type": "int"}
```

```sql
{"name": "trace_action_idx", "type": "int"}
```

--------------------------------

### Configure ERC20 Transfers with Shovel (JSON)

Source: https://indexsupply.com/shovel/docs/shovel

This JSON configuration defines how Shovel should process ERC20 'Transfer' events from Ethereum mainnet and Sepolia testnet. It maps event data to columns in a 'transfers' table in PostgreSQL, including filtering for a specific token address.

```JSON
{
  "pg_url": "postgres:///shovel",
  "eth_sources": [
    {
        "name": "mainnet",
        "chain_id": 1,
        "urls": ["https://ethereum-rpc.publicnode.com"]
    },
    {
        "name": "sepolia",
        "chain_id": 11155111,
        "urls": ["https://ethereum-sepolia-rpc.publicnode.com"]
    }
  ],
  "integrations": [
    {
      "name": "tokens",
      "enabled": true,
      "sources": [{"name": "mainnet"}, {"name": "sepolia"}],
      "table": {
        "name": "transfers",
          "columns": [
            {"name": "log_addr", "type": "bytea"},
            {"name": "block_time", "type": "numeric"},
            {"name": "from", "type": "bytea"},
            {"name": "to", "type": "bytea"},
            {"name": "value", "type": "numeric"}
          ]
      },
      "block": [
        {"name": "block_time", "column": "block_time"},
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_arg": ["a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"]
        }
      ],
      "event": {
        "name": "Transfer",
        "type": "event",
        "anonymous": false,
        "inputs": [
          {"indexed": true, "name": "from", "type": "address", "column": "from"},
          {"indexed": true, "name": "to", "type": "address", "column": "to"},
          {"indexed": false, "name": "value", "type": "uint256", "column": "value"}
        ]
      }
    }
  ]
}
```

--------------------------------

### Shovel Docker Image URL

Source: https://indexsupply.com/shovel/docs/index

URL for the Shovel Docker image hosted on Docker Hub. Specifies the available image tags for different operating systems and architectures.

```url
https://hub.docker.com/r/indexsupply/shovel
```

--------------------------------

### Decode Seaport Sales with Complex ABI

Source: https://indexsupply.com/shovel/docs/index

This configuration demonstrates how to ingest and decode Seaport sales events from the Ethereum blockchain. It specifies the integration to capture 'OrderFulfilled' events, mapping ABI inputs like 'orderHash' to database columns.

```json
{
  "pg_url": "postgres:///shovel",
  "eth_sources": [
    {
      "name": "mainnet",
      "chain_id": 1,
      "urls": ["https://ethereum-rpc.publicnode.com"]
    }
  ],
  "integrations": [
    {
      "name": "seaport-orders",
      "enabled": true,
      "sources": [
        {
          "name": "mainnet"
        }
      ],
      "table": {
        "name": "seaport_orders",
        "columns": [
          {
            "name": "order_hash",
            "type": "bytea"
          }
        ]
      },
      "block": [],
      "event": {
        "name": "OrderFulfilled",
        "type": "event",
        "anonymous": false,
        "inputs": [
          {
            "indexed": false,
            "internalType": "bytes32",
            "name": "orderHash",
            "column": "order_hash",
            "type": "bytes32"
          },
          {

```

--------------------------------

### Seaport Order Offer Structure

Source: https://indexsupply.com/shovel/docs/index

Defines the structure of the 'offer' component within a Seaport order, detailing item type, token, identifier, and amount.

```json
{
            "indexed": false,
            "internalType": "struct SpentItem[]",
            "name": "offer",
            "type": "tuple[]",
            "components": [
              {
                "internalType": "enum ItemType",
                "name": "itemType",
                "type": "uint8"
              },
              {
                "internalType": "address",
                "name": "token",
                "type": "address"
              },
              {
                "internalType": "uint256",
                "name": "identifier",
                "type": "uint256"
              },
              {
                "internalType": "uint256",
                "name": "amount",
                "type": "uint256"
              }
            ]
          }
```

--------------------------------

### Increase Shovel Throughput with Higher Concurrency

Source: https://indexsupply.com/shovel/docs/index

Increases Shovel's throughput by setting a larger `batch_size` (4000) and `concurrency` (2) for Ethereum sources.

```JSON
"eth_sources": [{..., "batch_size": 4000, "concurrency": 2}],

```

--------------------------------

### Optimize Shovel Concurrency for Block Headers or Bodies

Source: https://indexsupply.com/shovel/docs/index

Reduces `batch_size` to 100 and keeps `concurrency` at 1 for Shovel's Ethereum sources when indexing data from block headers or bodies.

```JSON
"eth_sources": [{..., "batch_size": 100, "concurrency": 1}],

```

--------------------------------

### Shovel Config: Environment Variable Usage

Source: https://indexsupply.com/shovel/docs/index

Demonstrates how Shovel utilizes environment variables for configuration values. Values prefixed with '$' are read from the operating system's environment. If an environment variable is not found for a '$'-prefixed value, Shovel will exit with an error.

```json
{
  "pg_url": "$PG_URL", ...
}
```

--------------------------------

### Track USDC Transfers on Multiple Chains

Source: https://indexsupply.com/shovel/docs/index

This configuration captures USDC token transfers from both Ethereum mainnet and Goerli testnet. It defines the table schema for storing transfer data, including chain ID, token addresses, block times, and sender/receiver information, and filters for transfers involving a specific USDC contract address.

```json
{
  "pg_url": "postgres:///shovel",
  "eth_sources": [
    {"name": "mainnet", "chain_id": 1, "urls": ["https://ethereum-rpc.publicnode.com"]},
    {"name": "goerli",  "chain_id": 11155111, "urls": ["https://ethereum-sepolia-rpc.publicnode.com"]
  ],
  "integrations": [
    {
      "name": "tokens",
      "enabled": true,
      "sources": [{"name": "mainnet"}, {"name": "goerli"}],
      "table": {
        "name": "transfers",
          "columns": [
            {"name": "chain_id",    "type": "numeric"},
            {"name": "log_addr",    "type": "bytea"},
            {"name": "block_time",  "type": "numeric"},
            {"name": "f",           "type": "bytea"},
            {"name": "t",           "type": "bytea"},
            {"name": "v",           "type": "numeric"}
          ]
      },
      "block": [
        {"name": "chain_id", "column": "chain_id"},
        {"name": "block_time", "column": "block_time"},
        {
          "name": "log_addr",
          "column": "log_addr",
          "filter_op": "contains",
          "filter_arg": ["a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"]
        }
      ],
      "event": {
        "name": "Transfer",
        "type": "event",
        "anonymous": false,
        "inputs": [
          {"indexed": true,  "name": "from",  "type": "address", "column": "f"},
          {"indexed": true,  "name": "to",    "type": "address", "column": "t"},
          {"indexed": false, "name": "value", "type": "uint256", "column": "v"}
        ]
      }
    }
  ]
}
```

--------------------------------

### Shovel Notification Configuration

Source: https://indexsupply.com/shovel/docs/index

Configures Shovel to send notifications with specific data from indexed blocks. The `columns` list defines which block fields are serialized and included in the notification payload.

```bash
NOTIFY "mainnet-foo", '$block_num,$a,$b'
```

--------------------------------

### View Git Commit Stats

Source: https://indexsupply.com/shovel/docs/index

Displays the statistics for a specific Git commit, showing changes made to files. This is useful for understanding the impact of a particular build.

```Shell
git show d80f --stat
```

--------------------------------

### Query ERC20 Transfers from Shovel Database (SQL)

Source: https://indexsupply.com/shovel/docs/shovel

This SQL query retrieves a single record from the 'transfers' table, which is populated by Shovel. It displays various columns including block time, sender and receiver addresses (as bytea), value, and the indexed token's address.

```SQL
shovel=# select * from transfers limit 1;
-[ RECORD 1 ]------------------------------------------
block_time | 1699912391
f          | \x66a89e05525bc2d4de6973ed2f30015f8ac4f165
t          | \x7e40bfd3d80e060cdd7b8970ec967eb8adf121f9
v          | 227300000
ig_name    | tokens
src_name   | mainnet
block_num  | 18565786
tx_idx     | 17
log_idx    | 0
abi_idx    | 0
log_addr   | \xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48
```

--------------------------------

### Seaport Order Event Signature

Source: https://indexsupply.com/shovel/docs/index

Represents the ABI signature for a Seaport order event, including indexed parameters like offerer and zone, and non-indexed parameters like recipient, offer, and consideration.

```json
{
            "indexed": true,
            "internalType": "address",
            "name": "offerer",
            "type": "address"
          },
          {
            "indexed": true,
            "internalType": "address",
            "name": "zone",
            "type": "address"
          },
          {
            "indexed": false,
            "internalType": "address",
            "name": "recipient",
            "type": "address"
          }
```

--------------------------------

### PostgreSQL Table Schema for Transfers

Source: https://indexsupply.com/shovel/docs/index

This SQL snippet shows the schema for the 'transfers' table, designed to store token transfer events. It includes columns for chain ID, token address, block timestamp, sender and receiver addresses, value, and metadata for indexing.

```sql
shovel=# \d transfers
                Table "public.transfers"
   Column   |   Type   | Collation | Nullable | Default
------------+----------+-----------+----------+---------
 chain_id   | numeric  |           |          |
 log_addr   | bytea    |           |          |
 block_time | numeric  |           |          |
 f          | bytea    |           |          |
 t          | bytea    |           |          |
 v          | numeric  |           |          |
 ig_name    | text     |           |          |
 src_name   | text     |           |          |
 block_num  | numeric  |           |          |
 tx_idx     | integer  |           |          |
 log_idx    | smallint |           |          |
 abi_idx    | smallint |           |          |
Indexes:
    "u_transfers" UNIQUE, btree (ig_name, src_name, block_num, tx_idx, log_idx, abi_idx)
```

--------------------------------

### Shovel Config: PostgreSQL URL

Source: https://indexsupply.com/shovel/docs/index

Specifies the PostgreSQL database connection URL for Shovel. It can be a direct URL or use an environment variable for sensitive information like passwords.

```json
{
  "pg_url": "postgres:///shovel", ...
}
```

```json
{
  "pg_url": "$DATABASE_URL",...
}
```

--------------------------------

### Query Shovel's Latest Block

Source: https://indexsupply.com/shovel/docs/index

This SQL query retrieves the most recent block indexed by Shovel, grouped by the Integration's Source name. It helps identify the latest progress for different data sources.

```SQL
select * from shovel.latest ;
```

--------------------------------

### PostgreSQL Table Schema for Transactions

Source: https://indexsupply.com/shovel/docs/index

This SQL snippet displays the schema for the 'tx' table, which stores transaction data. It includes columns like 'tx_input', 'ig_name', 'src_name', 'block_num', and 'tx_idx', with a unique index defined on several of these fields.

```sql
shovel=# \d tx
                  Table "public.tx"
  Column   |  Type   | Collation | Nullable | Default
-----------+---------+-----------+----------+---------
 tx_input  | bytea   |           |          |
 ig_name   | text    |           |          |
 src_name  | text    |           |          |
 block_num | numeric |           |          |
 tx_idx    | integer |           |          |
Indexes:
    "u_tx" UNIQUE, btree (ig_name, src_name, block_num, tx_idx)
```

--------------------------------

### Configure Shovel Dashboard Password

Source: https://indexsupply.com/shovel/docs/index

Sets a root password for the Shovel dashboard by defining the 'root_password' within the dashboard configuration in the config file.

```JSON
{
  "...": "...",
  "dashboard": {
    "root_password": ""
  }
}
```

--------------------------------

### Index Transactions Calling mint(uint256)

Source: https://indexsupply.com/shovel/docs/index

Configuration to index transactions that specifically call the `mint(uint256)` function. This is achieved using a 'contains' filter operation on the transaction input data.

```json
{
  "pg_url": "postgres:///shovel",
  "eth_sources": [
    {"name": "mainnet", "chain_id": 1, "urls": ["https://ethereum-rpc.publicnode.com"]}  ],
  "integrations": [{
    "name": "specific-tx",
    "enabled": true,
    "sources": [{"name": "mainnet", "start": 19737332}],
    "table": {
      "name": "specific_tx",
      "columns": [
        {"name": "block_hash", "type": "bytea"},
        {"name": "block_num", "type": "numeric"},
        {"name": "tx_hash", "type": "bytea"},
        {"name": "tx_idx", "type": "int"},
        {"name": "call_type", "type": "text"},
        {"name": "from", "type": "bytea"},
        {"name": "to", "type": "bytea"},
        {"name": "value", "type": "numeric"}
      ]
    },
    "block": [
      {"name": "block_hash", "column": "block_hash"},
      {"name": "block_num", "column": "block_num"},
      {"name": "tx_hash", "column": "tx_hash"},
      {
        "name": "trace_action_call_type",
        "column": "call_type",
        "filter_op": "eq",
        "filter_arg": ["call"]
      },
      {"name": "trace_action_from", "column": "from"},
      {"name": "trace_action_to", "column": "to"},
      {
        "name": "trace_action_value",
        "column": "value",
        "filter_op": "gt",
        "filter_arg": ["0"]
      }
    ]
  }]
}
```

--------------------------------

### Seaport Order Consideration Structure

Source: https://indexsupply.com/shovel/docs/index

Defines the structure of the 'consideration' component within a Seaport order, detailing item type, token, identifier, amount, and recipient.

```json
{
            "indexed": false,
            "internalType": "struct ReceivedItem[]",
            "name": "consideration",
            "type": "tuple[]",
            "components": [
              {
                "internalType": "enum ItemType",
                "name": "itemType",
                "type": "uint8"
              },
              {
                "internalType": "address",
                "name": "token",
                "type": "address"
              },
              {
                "internalType": "uint256",
                "name": "identifier",
                "type": "uint256"
              },
              {
                "internalType": "uint256",
                "name": "amount",
                "type": "uint256"
              },
              {
                "internalType": "address payable",
                "name": "recipient",
                "type": "address"
              }
            ]
          }
```

--------------------------------

### Disable Shovel Dashboard Authentication

Source: https://indexsupply.com/shovel/docs/index

Disables all authentication for the Shovel dashboard, making it accessible without a password.

```JSON
{
  "...": "...",
  "dashboard": {
    "disable_authn": true
  }
}
```
Tokens:
Raw