Firehose Ethereum

https://github.com/streamingfast/firehose-ethereum
Admin
Firehose for Ethereum provides a high-performance data pipeline for processing Ethereum blockchain...
Tokens:3,632
Snippets:29
Trust Score:8
License:Apache-2.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Starting Reader and Merger for Batch Processing (Shell)

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This command starts the `reader` and `merger` applications simultaneously using `fireeth start`. It is used for producing merged blocks in batch reprocessing jobs. Additional flags (`--...`) are required to configure paths, block ranges (`--merger-stop-block`), and other settings.

```Shell
fireeth start reader,merger --...
```

--------------------------------

### Starting Combined Index Builder with fireeth (Shell)

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This command launches the `combined-index-builder` application using the `fireeth start` command. This builder creates new combined indexes from upgraded block data, replacing older index generation tools. It can be run in parallel to cover a specific block range.

```Shell
fireeth start combined-index-builder
```

--------------------------------

### Install Firehose Ethereum from Source - Shell

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md

Installs the Firehose for Ethereum binary by compiling it from source using the Go toolchain. Requires Go (version 1.16+) to be installed and the source code cloned.

```Shell
go install ./cmd/fireeth
```

--------------------------------

### Starting Combined Index Generation Batch Job (Shell)

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This command initiates a batch job to generate combined block indices using `fireeth start generate-combined-index`. It requires configuring input/output storage URLs, index size, start block, optional stop block, and the gRPC listen address. This app replaces the older `sfeth tools generate-...` command.

```Shell
fireeth start generate-combined-index --common-blocks-store-url=/path/to/blocks --common-index-store-url=/path/to/index --combined-index-builder-index-size=10000 --combined-index-builder-start-block=0 [--combined-index-builder-stop-block=10000] --combined-index-builder-grpc-listen-addr=:9000
```

--------------------------------

### Starting Devnet Synchronization (Shell)

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-devnet12/README.md

Executes the `start.sh` script to initiate the synchronization process for the devnet12 environment. This script is the primary command for bringing up the necessary components (presumably `geth` and `fireeth`) to sync the network. It assumes the script is executable and located in the current directory, and that required dependencies like `geth` and `fireeth` are installed and accessible via the system's PATH.

```Shell
./start.sh
```

--------------------------------

### Starting Firehose Ethereum Stack - Bash

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-mainnet/README.md

This command starts the full Firehose on Ethereum stack using the specified configuration file. It assumes the Lighthouse consensus client is already synced and the Firehose-instrumented Geth binary is available in your PATH. The `fireeth` process will supervise and manage Geth by default.

```bash
fireeth -c sync-mainnet.yaml start
```

--------------------------------

### Starting Lighthouse Consensus Client - Bash

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-mainnet/README.md

This command executes the script to start the Lighthouse Ethereum consensus client. It's a prerequisite for syncing the chain with Firehose and the execution client. Ensure the `consensus.sh` script is executable and the `lighthouse` binary is accessible via PATH or `LIGHTHOUSE_BIN`.

```bash
./consensus.sh
```

--------------------------------

### Obsolete node App Configuration - YAML

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This YAML snippet shows the configuration for the now-removed `node` app. If your configuration includes `node` in the `start.args` or flags starting with `node-`, you must remove them as the `node` app is no longer supported by fireeth.

```yaml
start:
  args:
  - ...
  - node
  - ...
  flags:
    ...
    node-...
```

--------------------------------

### Install Firehose Ethereum using Homebrew - Shell

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md

Installs the Firehose for Ethereum binary using the Homebrew package manager. Requires Homebrew to be installed on macOS or Linux.

```Shell
brew install tap/streamingfast/firehose-ethereum
```

--------------------------------

### New index-builder App Configuration - YAML

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This YAML snippet shows the updated configuration for the renamed `index-builder` app. The app name and flags starting with `combined-index-builder-` have been replaced by `index-builder` and flags starting with `index-builder-`. Operators should update their configurations accordingly.

```yaml
start:
  args:
  - ...
  - index-builder
  - ...
  flags:
    ...
    index-builder-grpc-listen-addr: ":9999"
    index-builder-index-size: 10000
    index-builder-start-block: 0
    index-builder-stop-block: 0
    ...
```

--------------------------------

### Configuring Substreams State Store URL with Version in Path - YAML

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

Example demonstrating the previous configuration format where the version tag was included directly in the `substreams-state-store-url` flag path. This method is being replaced by the `substreams-state-store-default-tag` flag to support dynamic state store selection via client headers.

```YAML
start:
  ...
  flags:
    substreams-state-store-url: /<some>/<path>/v3
```

--------------------------------

### Configuring Substreams State Store URL with Default Tag - YAML

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

Example demonstrating the new recommended configuration format for the Substreams state store. The base path is set with `substreams-state-store-url`, and the default version tag is specified separately using the new `substreams-state-store-default-tag` flag, enabling client-side state store selection.

```YAML
start:
  ...
  flags:
    substreams-state-store-url: /<some>/<path>
    substreams-state-store-default-tag: v3
```

--------------------------------

### Starting Firehose Ethereum in Stdin Mode - Bash

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-mainnet/README.md

This command pipeline starts the Firehose Ethereum stack in `stdin` mode, where the Firehose-instrumented Geth execution client is manually launched and its output is piped into `fireeth`. This requires modifying the `sync-mainnet.yaml` to use `reader-node-stdin` and removing Geth arguments from the config.

```bash
geth --mainnet --datadir=./firehose-data/reader/data --ipcpath=./firehose-data/reader/ipc --port=30305 --http --http.api=eth,net,web3 --http.port=8547 --http.addr=0.0.0.0 "--http.vhosts=*" --authrpc.jwtsecret=./jwt.txt --authrpc.addr=0.0.0.0 --authrpc.port=9551 "--authrpc.vhosts=*" --http.addr=0.0.0.0 --http.port=9545 "--http.vhosts=*" --ws.port=9546 --port=40303 --firehose-enabled | fireeth -c sync-mainnet.yaml start
```

--------------------------------

### Starting Geth and Piping to Firehose in stdin Mode - Bash

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-sepolia/README.md

This command demonstrates running the Geth execution client manually with specific Sepolia configuration flags and then piping its standard output directly to the `fireeth` binary. This requires `fireeth` to be configured in `stdin` mode (by modifying `sync-sepolia.yaml`) where it consumes data piped from another process rather than managing Geth itself. A Firehose-instrumented Geth binary, a `jwt.txt` file, and a synced Beacon Chain client are necessary.

```bash
geth --sepolia --datadir=./firehose-data/reader/data --ipcpath=./firehose-data/reader/ipc --port=30305 --http --http.api=eth,net,web3 --http.port=8547 --http.addr=0.0.0.0 "--http.vhosts=*" --authrpc.jwtsecret=./jwt.txt --authrpc.addr=0.0.0.0 --authrpc.port=9551 "--authrpc.vhosts=*" --http.addr=0.0.0.0 --http.port=9545 "--http.vhosts=*" --ws.port=9546 --port=40303 --firehose-enabled | fireeth -c sync-sepolia.yaml start
```

--------------------------------

### Querying Latest Block with fireeth firehose-client (Shell)

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This command uses the `fireeth tools firehose-client` to connect to a specified Firehose endpoint and fetch only the latest block. The `--limit=1` flag restricts the output to a single block, and the `-- -1 0` arguments specify the range starting from the latest block (-1). This is useful for quickly verifying the current head of the chain.

```Shell
fireeth tools firehose-client <endpoint> --limit=1 -- -1 0
```

--------------------------------

### Starting Beacon Chain Client (Lighthouse) - Bash

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-sepolia/README.md

This command executes the `consensus.sh` script to launch the Lighthouse Beacon Chain client. It is required to sync the consensus layer of the Ethereum network. The script uses the `lighthouse` binary, which should be available in the system's PATH or specified by the `LIGHTHOUSE_BIN` environment variable. The `CHECKPOINT_SYNC_URL` environment variable can be used to specify a checkpoint for faster bootstrapping.

```bash
./consensus.sh
```

--------------------------------

### Starting Firehose Ethereum Stack - Bash

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-sepolia/README.md

This command launches the full Firehose on Ethereum stack using the specified YAML configuration file (`sync-sepolia.yaml`). It orchestrates the necessary components, including supervising the Firehose-instrumented Geth execution client, to sync both execution data and generate Firehose blocks. A synced Beacon Chain client is a prerequisite for this step.

```bash
fireeth -c sync-sepolia.yaml start
```

--------------------------------

### Create Docker Alias for Firehose Ethereum - Shell

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md

Creates a convenient shell alias to run the Firehose for Ethereum Docker image. Requires Docker to be installed and access to the ghcr.io/streamingfast/firehose-ethereum image. Assumes SF_API_TOKEN environment variable is set.

```Shell
alias fireeth='docker run --rm -it -e="SF_API_TOKEN=${SF_API_TOKEN}" ghcr.io/streamingfast/firehose-ethereum'
```

--------------------------------

### Run GitHub Workflow for Docker Image (Latest Release) - Shell

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md

Triggers a GitHub Actions workflow (`docker.yml`) to build a bundled Docker image using the Firehose V2 instrumentation and referencing a specific tag (e.g., v1.2.2). Requires GitHub CLI (`gh`) to be installed and configured.

```Shell
gh workflow run docker.yml -f geth_version=fh2 --ref v1.2.2
```

--------------------------------

### Run GitHub Workflow for Docker Image (Develop Branch) - Shell

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md

Triggers a GitHub Actions workflow (`docker.yml`) to build a bundled Docker image using the Firehose V2 instrumentation and referencing the `develop` branch for development builds. Requires GitHub CLI (`gh`) to be installed and configured.

```Shell
gh workflow run docker.yml -f geth_version=fh2 --ref develop
```

--------------------------------

### Old combined-index-builder App Configuration - YAML

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This YAML snippet illustrates the deprecated configuration for the `combined-index-builder` app. This app has been renamed, and its flags have been changed. Configurations using this name and flag prefix should be updated to the new format.

```yaml
start:
  args:
  - ...
  - combined-index-builder
  - ...
  flags:
    ...
    combined-index-builder-grpc-listen-addr: ":9999"
    combined-index-builder-index-size: 10000
    combined-index-builder-start-block: 0
    combined-index-builder-stop-block: 0
    ...
```

--------------------------------

### Before Upgrade: Reader Node Configuration YAML

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This snippet shows the configuration for the `reader-node` component in the YAML file *before* the specified upgrade. It includes the use of the `+` prefix for `reader-node-arguments` to append flags to automatically provided defaults, and deprecated flags like `reader-node-enforce-peers` and `reader-node-log-to-zap`.

```YAML
flags:
    ...
    reader-node-bootstrap-data-url: ./reader/genesis.json
    reader-node-enforce-peers: localhost:13041
    reader-node-arguments: +--firehose-genesis-file=./reader/genesis.json --authrpc.port=8552
    reader-node-log-to-zap: false
    ...
```

--------------------------------

### Checking Merged Blocks for Corruption - Shell

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This shell command uses the `tools check merged-blocks-batch` utility to identify potentially corrupted merged-blocks files within a specified block range. Operators can run this command to diagnose issues like duplicate or out-of-range blocks in their merged block store.

```shell
tools check merged-blocks-batch <merged-blocks-store> <start> <stop>
```

--------------------------------

### Upgrading Merged Blocks with fireeth Tool (Shell)

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This command upgrades merged blocks from an older version (v2 folders) to a newer version (v3 folders) using the `fireeth tools` command. It requires specifying the source and destination paths for the block data and the start/stop block numbers for the range to upgrade. Multiple commands can be run in parallel.

```Shell
fireeth tools upgrade-merged-blocks /path/to/v2 /path/to/v3 {start} {stop}
```

--------------------------------

### Fixing Bloated Merged Blocks - Shell

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This shell command utilizes the `tools fix-bloated-merged-blocks` utility to attempt to repair corrupted merged-blocks files. It processes a specified block range from a source store, writes the fixed files to an output store, which can then be used to replace the original corrupted files.

```shell
tools fix-bloated-merged-blocks <merged-blocks-store> <output-store> <start>:<stop>
```

--------------------------------

### After Upgrade: Reader Node Configuration YAML

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This snippet shows the updated configuration for the `reader-node` component in the YAML file *after* the specified upgrade. It removes the `+` prefix, requires all necessary `geth` flags (including `--firehose-enabled`) to be explicitly listed under `reader-node-arguments`, and demonstrates the use of templating variables like `{node-data-dir}` and `{data-dir}`. Deprecated flags are removed.

```YAML
flags:
    ...
    reader-node-bootstrap-data-url: ./reader/genesis.json
    reader-node-arguments:
      --networkid=1515
      --datadir={node-data-dir}
      --ipcpath={data-dir}/reader/ipc
      --port=30305
      --http
      --http.api=eth,net,web3
      --http.port=8547
      --http.addr=0.0.0.0
      --http.vhosts=*
      --firehose-enabled
      --firehose-genesis-file=./reader/genesis.json
      --authrpc.port=8552
    ...
```

--------------------------------

### Fixing Polygon Index Data - Shell

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md

This specific shell command uses the `fireeth tools fix-polygon-index` utility to address a data issue found in Polygon/Mumbai blocks where system transactions had incorrect indices. Operators running these chains with older data may need to use this tool to fix corrupted blocks.

```shell
fireeth tools fix-polygon-index /your/merged/blocks /temporary/destination 0 48200000
```

--------------------------------

### Run Release Script - Shell

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md

Executes the project's release script, automating the process of creating a new release. Runs in dry-run mode by default. Requires execution permissions on the script.

```Shell
./bin/release.sh
```

--------------------------------

### Developer Certificate of Origin Text - Plaintext

Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CONTRIBUTING.md

This snippet contains the full text of the Developer Certificate of Origin (DCO) version 1.1. By contributing, developers certify that they have the right to submit their work under the project's open source license, ensuring legal clarity and proper provenance for contributions.

```Plaintext
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.
```
Tokens:
Raw