### Starting Reader and Merger for Batch Processing (Shell) Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This command starts the `reader` and `merger` applications simultaneously using `fireeth start`. It is used for producing merged blocks in batch reprocessing jobs. Additional flags (`--...`) are required to configure paths, block ranges (`--merger-stop-block`), and other settings. ```Shell fireeth start reader,merger --... ``` -------------------------------- ### Starting Combined Index Builder with fireeth (Shell) Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This command launches the `combined-index-builder` application using the `fireeth start` command. This builder creates new combined indexes from upgraded block data, replacing older index generation tools. It can be run in parallel to cover a specific block range. ```Shell fireeth start combined-index-builder ``` -------------------------------- ### Install Firehose Ethereum from Source - Shell Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md Installs the Firehose for Ethereum binary by compiling it from source using the Go toolchain. Requires Go (version 1.16+) to be installed and the source code cloned. ```Shell go install ./cmd/fireeth ``` -------------------------------- ### Starting Combined Index Generation Batch Job (Shell) Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This command initiates a batch job to generate combined block indices using `fireeth start generate-combined-index`. It requires configuring input/output storage URLs, index size, start block, optional stop block, and the gRPC listen address. This app replaces the older `sfeth tools generate-...` command. ```Shell fireeth start generate-combined-index --common-blocks-store-url=/path/to/blocks --common-index-store-url=/path/to/index --combined-index-builder-index-size=10000 --combined-index-builder-start-block=0 [--combined-index-builder-stop-block=10000] --combined-index-builder-grpc-listen-addr=:9000 ``` -------------------------------- ### Starting Devnet Synchronization (Shell) Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-devnet12/README.md Executes the `start.sh` script to initiate the synchronization process for the devnet12 environment. This script is the primary command for bringing up the necessary components (presumably `geth` and `fireeth`) to sync the network. It assumes the script is executable and located in the current directory, and that required dependencies like `geth` and `fireeth` are installed and accessible via the system's PATH. ```Shell ./start.sh ``` -------------------------------- ### Starting Firehose Ethereum Stack - Bash Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-mainnet/README.md This command starts the full Firehose on Ethereum stack using the specified configuration file. It assumes the Lighthouse consensus client is already synced and the Firehose-instrumented Geth binary is available in your PATH. The `fireeth` process will supervise and manage Geth by default. ```bash fireeth -c sync-mainnet.yaml start ``` -------------------------------- ### Starting Lighthouse Consensus Client - Bash Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-mainnet/README.md This command executes the script to start the Lighthouse Ethereum consensus client. It's a prerequisite for syncing the chain with Firehose and the execution client. Ensure the `consensus.sh` script is executable and the `lighthouse` binary is accessible via PATH or `LIGHTHOUSE_BIN`. ```bash ./consensus.sh ``` -------------------------------- ### Obsolete node App Configuration - YAML Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This YAML snippet shows the configuration for the now-removed `node` app. If your configuration includes `node` in the `start.args` or flags starting with `node-`, you must remove them as the `node` app is no longer supported by fireeth. ```yaml start: args: - ... - node - ... flags: ... node-... ``` -------------------------------- ### Install Firehose Ethereum using Homebrew - Shell Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md Installs the Firehose for Ethereum binary using the Homebrew package manager. Requires Homebrew to be installed on macOS or Linux. ```Shell brew install tap/streamingfast/firehose-ethereum ``` -------------------------------- ### New index-builder App Configuration - YAML Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This YAML snippet shows the updated configuration for the renamed `index-builder` app. The app name and flags starting with `combined-index-builder-` have been replaced by `index-builder` and flags starting with `index-builder-`. Operators should update their configurations accordingly. ```yaml start: args: - ... - index-builder - ... flags: ... index-builder-grpc-listen-addr: ":9999" index-builder-index-size: 10000 index-builder-start-block: 0 index-builder-stop-block: 0 ... ``` -------------------------------- ### Configuring Substreams State Store URL with Version in Path - YAML Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md Example demonstrating the previous configuration format where the version tag was included directly in the `substreams-state-store-url` flag path. This method is being replaced by the `substreams-state-store-default-tag` flag to support dynamic state store selection via client headers. ```YAML start: ... flags: substreams-state-store-url: ///v3 ``` -------------------------------- ### Configuring Substreams State Store URL with Default Tag - YAML Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md Example demonstrating the new recommended configuration format for the Substreams state store. The base path is set with `substreams-state-store-url`, and the default version tag is specified separately using the new `substreams-state-store-default-tag` flag, enabling client-side state store selection. ```YAML start: ... flags: substreams-state-store-url: // substreams-state-store-default-tag: v3 ``` -------------------------------- ### Starting Firehose Ethereum in Stdin Mode - Bash Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-mainnet/README.md This command pipeline starts the Firehose Ethereum stack in `stdin` mode, where the Firehose-instrumented Geth execution client is manually launched and its output is piped into `fireeth`. This requires modifying the `sync-mainnet.yaml` to use `reader-node-stdin` and removing Geth arguments from the config. ```bash geth --mainnet --datadir=./firehose-data/reader/data --ipcpath=./firehose-data/reader/ipc --port=30305 --http --http.api=eth,net,web3 --http.port=8547 --http.addr=0.0.0.0 "--http.vhosts=*" --authrpc.jwtsecret=./jwt.txt --authrpc.addr=0.0.0.0 --authrpc.port=9551 "--authrpc.vhosts=*" --http.addr=0.0.0.0 --http.port=9545 "--http.vhosts=*" --ws.port=9546 --port=40303 --firehose-enabled | fireeth -c sync-mainnet.yaml start ``` -------------------------------- ### Starting Geth and Piping to Firehose in stdin Mode - Bash Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-sepolia/README.md This command demonstrates running the Geth execution client manually with specific Sepolia configuration flags and then piping its standard output directly to the `fireeth` binary. This requires `fireeth` to be configured in `stdin` mode (by modifying `sync-sepolia.yaml`) where it consumes data piped from another process rather than managing Geth itself. A Firehose-instrumented Geth binary, a `jwt.txt` file, and a synced Beacon Chain client are necessary. ```bash geth --sepolia --datadir=./firehose-data/reader/data --ipcpath=./firehose-data/reader/ipc --port=30305 --http --http.api=eth,net,web3 --http.port=8547 --http.addr=0.0.0.0 "--http.vhosts=*" --authrpc.jwtsecret=./jwt.txt --authrpc.addr=0.0.0.0 --authrpc.port=9551 "--authrpc.vhosts=*" --http.addr=0.0.0.0 --http.port=9545 "--http.vhosts=*" --ws.port=9546 --port=40303 --firehose-enabled | fireeth -c sync-sepolia.yaml start ``` -------------------------------- ### Querying Latest Block with fireeth firehose-client (Shell) Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This command uses the `fireeth tools firehose-client` to connect to a specified Firehose endpoint and fetch only the latest block. The `--limit=1` flag restricts the output to a single block, and the `-- -1 0` arguments specify the range starting from the latest block (-1). This is useful for quickly verifying the current head of the chain. ```Shell fireeth tools firehose-client --limit=1 -- -1 0 ``` -------------------------------- ### Starting Beacon Chain Client (Lighthouse) - Bash Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-sepolia/README.md This command executes the `consensus.sh` script to launch the Lighthouse Beacon Chain client. It is required to sync the consensus layer of the Ethereum network. The script uses the `lighthouse` binary, which should be available in the system's PATH or specified by the `LIGHTHOUSE_BIN` environment variable. The `CHECKPOINT_SYNC_URL` environment variable can be used to specify a checkpoint for faster bootstrapping. ```bash ./consensus.sh ``` -------------------------------- ### Starting Firehose Ethereum Stack - Bash Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-sepolia/README.md This command launches the full Firehose on Ethereum stack using the specified YAML configuration file (`sync-sepolia.yaml`). It orchestrates the necessary components, including supervising the Firehose-instrumented Geth execution client, to sync both execution data and generate Firehose blocks. A synced Beacon Chain client is a prerequisite for this step. ```bash fireeth -c sync-sepolia.yaml start ``` -------------------------------- ### Create Docker Alias for Firehose Ethereum - Shell Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md Creates a convenient shell alias to run the Firehose for Ethereum Docker image. Requires Docker to be installed and access to the ghcr.io/streamingfast/firehose-ethereum image. Assumes SF_API_TOKEN environment variable is set. ```Shell alias fireeth='docker run --rm -it -e="SF_API_TOKEN=${SF_API_TOKEN}" ghcr.io/streamingfast/firehose-ethereum' ``` -------------------------------- ### Run GitHub Workflow for Docker Image (Latest Release) - Shell Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md Triggers a GitHub Actions workflow (`docker.yml`) to build a bundled Docker image using the Firehose V2 instrumentation and referencing a specific tag (e.g., v1.2.2). Requires GitHub CLI (`gh`) to be installed and configured. ```Shell gh workflow run docker.yml -f geth_version=fh2 --ref v1.2.2 ``` -------------------------------- ### Run GitHub Workflow for Docker Image (Develop Branch) - Shell Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md Triggers a GitHub Actions workflow (`docker.yml`) to build a bundled Docker image using the Firehose V2 instrumentation and referencing the `develop` branch for development builds. Requires GitHub CLI (`gh`) to be installed and configured. ```Shell gh workflow run docker.yml -f geth_version=fh2 --ref develop ``` -------------------------------- ### Old combined-index-builder App Configuration - YAML Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This YAML snippet illustrates the deprecated configuration for the `combined-index-builder` app. This app has been renamed, and its flags have been changed. Configurations using this name and flag prefix should be updated to the new format. ```yaml start: args: - ... - combined-index-builder - ... flags: ... combined-index-builder-grpc-listen-addr: ":9999" combined-index-builder-index-size: 10000 combined-index-builder-start-block: 0 combined-index-builder-stop-block: 0 ... ``` -------------------------------- ### Before Upgrade: Reader Node Configuration YAML Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This snippet shows the configuration for the `reader-node` component in the YAML file *before* the specified upgrade. It includes the use of the `+` prefix for `reader-node-arguments` to append flags to automatically provided defaults, and deprecated flags like `reader-node-enforce-peers` and `reader-node-log-to-zap`. ```YAML flags: ... reader-node-bootstrap-data-url: ./reader/genesis.json reader-node-enforce-peers: localhost:13041 reader-node-arguments: +--firehose-genesis-file=./reader/genesis.json --authrpc.port=8552 reader-node-log-to-zap: false ... ``` -------------------------------- ### Checking Merged Blocks for Corruption - Shell Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This shell command uses the `tools check merged-blocks-batch` utility to identify potentially corrupted merged-blocks files within a specified block range. Operators can run this command to diagnose issues like duplicate or out-of-range blocks in their merged block store. ```shell tools check merged-blocks-batch ``` -------------------------------- ### Upgrading Merged Blocks with fireeth Tool (Shell) Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This command upgrades merged blocks from an older version (v2 folders) to a newer version (v3 folders) using the `fireeth tools` command. It requires specifying the source and destination paths for the block data and the start/stop block numbers for the range to upgrade. Multiple commands can be run in parallel. ```Shell fireeth tools upgrade-merged-blocks /path/to/v2 /path/to/v3 {start} {stop} ``` -------------------------------- ### Fixing Bloated Merged Blocks - Shell Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This shell command utilizes the `tools fix-bloated-merged-blocks` utility to attempt to repair corrupted merged-blocks files. It processes a specified block range from a source store, writes the fixed files to an output store, which can then be used to replace the original corrupted files. ```shell tools fix-bloated-merged-blocks : ``` -------------------------------- ### After Upgrade: Reader Node Configuration YAML Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This snippet shows the updated configuration for the `reader-node` component in the YAML file *after* the specified upgrade. It removes the `+` prefix, requires all necessary `geth` flags (including `--firehose-enabled`) to be explicitly listed under `reader-node-arguments`, and demonstrates the use of templating variables like `{node-data-dir}` and `{data-dir}`. Deprecated flags are removed. ```YAML flags: ... reader-node-bootstrap-data-url: ./reader/genesis.json reader-node-arguments: --networkid=1515 --datadir={node-data-dir} --ipcpath={data-dir}/reader/ipc --port=30305 --http --http.api=eth,net,web3 --http.port=8547 --http.addr=0.0.0.0 --http.vhosts=* --firehose-enabled --firehose-genesis-file=./reader/genesis.json --authrpc.port=8552 ... ``` -------------------------------- ### Fixing Polygon Index Data - Shell Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md This specific shell command uses the `fireeth tools fix-polygon-index` utility to address a data issue found in Polygon/Mumbai blocks where system transactions had incorrect indices. Operators running these chains with older data may need to use this tool to fix corrupted blocks. ```shell fireeth tools fix-polygon-index /your/merged/blocks /temporary/destination 0 48200000 ``` -------------------------------- ### Run Release Script - Shell Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/README.md Executes the project's release script, automating the process of creating a new release. Runs in dry-run mode by default. Requires execution permissions on the script. ```Shell ./bin/release.sh ``` -------------------------------- ### Developer Certificate of Origin Text - Plaintext Source: https://github.com/streamingfast/firehose-ethereum/blob/develop/CONTRIBUTING.md This snippet contains the full text of the Developer Certificate of Origin (DCO) version 1.1. By contributing, developers certify that they have the right to submit their work under the project's open source license, ensuring legal clarity and proper provenance for contributions. ```Plaintext Developer Certificate of Origin Version 1.1 Copyright (C) 2004, 2006 The Linux Foundation and its contributors. 1 Letterman Drive Suite D4700 San Francisco, CA, 94129 Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.