Firehose Core

https://github.com/streamingfast/firehose-core
Admin
Firehose Core provides the base components for the Firehose software, enabling it to run for...
Tokens:3,457
Snippets:36
Trust Score:8
License:Apache-2.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Installing Dummy Blockchain for Development (Go Shell)

Source: https://github.com/streamingfast/firehose-core/blob/develop/README.md

Uses the Go programming language's installation command to fetch and install the dummy-blockchain binary, which is a prerequisite for setting up a local development environment for Firehose Core.

```Shell
go install github.com/streamingfast/dummy-blockchain@latest
```

--------------------------------

### Example Firehose Core Configuration (YAML)

Source: https://github.com/streamingfast/firehose-core/blob/develop/README.md

A sample YAML configuration file (`firehose.yaml`) demonstrating how to specify which Firehose components to start (`reader-node`, `merger`) and how to provide specific flags for a component, such as the reader node's executable path and arguments.

```YAML
start:
  args:
  - reader-node
  - merger
  flags:
    reader-node-path: "/usr/local/bin/firesol"
    reader-node-args: ["fetch", "rpc", "http://localhost:8545", "0"]
```

--------------------------------

### Installing Firehose Core via Homebrew (Shell)

Source: https://github.com/streamingfast/firehose-core/blob/develop/README.md

Provides the command to install the Firehose Core binary on MacOS and Linux systems using the Homebrew package manager.

```Shell
brew install tap/streamingfast/firehose-core
```

--------------------------------

### Running Firehose Core with a Configuration File (Shell)

Source: https://github.com/streamingfast/firehose-core/blob/develop/README.md

Executes the Firehose Core application, instructing it to start components and apply settings defined in a specified YAML configuration file.

```Shell
firecore start --config-file=./firehose.yaml
```

--------------------------------

### Installing Firehose Core via Docker Alias (Shell)

Source: https://github.com/streamingfast/firehose-core/blob/develop/README.md

Creates a shell alias named 'firecore' to run the Firehose Core application using its Docker image. This command requires Docker to be installed and utilizes an environment variable for an API token.

```Shell
alias firecore='docker run --rm -it -e="SF_API_TOKEN=${SF_API_TOKEN}" ghcr.io/streamingfast/firehose-core'
```

--------------------------------

### Running Development Start Script (Shell)

Source: https://github.com/streamingfast/firehose-core/blob/develop/README.md

Executes the provided shell script `./devel/standard/start.sh` within the repository, which launches a fully configured Firehose Core instance utilizing the dummy blockchain for development purposes. The `-c` flag cleans previous data.

```Shell
./devel/standard/start.sh -c
```

--------------------------------

### Running Firehose Core with Default Configuration (Shell)

Source: https://github.com/streamingfast/firehose-core/blob/develop/README.md

Starts the Firehose Core application using its built-in default configuration values. This is achieved by providing an empty value to the configuration file flag.

```Shell
firecore start --config-file=
```

--------------------------------

### Defining ReaderNodeBootstrapper Factory Before v0.2.0 Go

Source: https://github.com/streamingfast/firehose-core/blob/develop/UPDATE.md

Shows the previous function signature for the `ReaderNodeBootstrapperFactory` before the v0.2.0 update. It accepted a `cobra.Command` and a `nodeDataDir` string to return an `operator.Bootstrapper`.

```Go
func newReaderNodeBootstrapper(cmd *cobra.Command, nodeDataDir string) (operator.Bootstrapper, error) {
  ...
}
```

--------------------------------

### Defining ReaderNodeBootstrapper Factory After v0.2.0 Go

Source: https://github.com/streamingfast/firehose-core/blob/develop/UPDATE.md

Shows the updated function signature for the `ReaderNodeBootstrapperFactory` in v0.2.0. It now receives `context.Context`, `zap.Logger`, `cobra.Command`, `resolvedNodeArguments`, and a `ReaderNodeArgumentResolver`. It demonstrates how to retrieve the `nodeDataDir` using the new resolver.

```Go
func newReaderNodeBootstrapper(ctx context.Context,
  logger *zap.Logger,
  cmd *cobra.Command,
  resolvedNodeArguments []string,
  resolver firecore.ReaderNodeArgumentResolver,
) (operator.Bootstrapper, error) {
  nodeDataDir := resolver("{node-data-dir}")

  ...
}
```

--------------------------------

### Configuring TransformFlags Before v0.2.0 Go

Source: https://github.com/streamingfast/firehose-core/blob/develop/UPDATE.md

Illustrates the configuration structure for `firecore.ToolsConfig#TransformFlags` used before v0.2.0. It was defined as a map where each key was the flag name and the value was a `firecore.TransformFlag` struct containing a description and a single `Parser` function.

```Go
TransformFlags: map[string]*firecore.TransformFlag{
  "receipt-account-filters": {
    Description: "Comma-separated accounts to use as filter/index. If it contains a colon (:), it will be interpreted as <prefix>:<suffix> (each of which can be empty, ex: 'hello:' or ':world')",
    Parser:      parseReceiptAccountFilters,
  },
},
```

--------------------------------

### Example Node Manager Resume with Extra Env (Shell)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This `curl` command demonstrates how to use the Node Manager's `/v1/resume` HTTP endpoint to restart a node. It sends a POST request, includes `sync=true` to make the call synchronous, and passes the environment variable `NODE_DEBUG=true` using the `extra-env` query parameter for the subsequent node process.

```Shell
curl -XPOST "http://localhost:10011/v1/resume?sync=true&extra-env=NODE_DEBUG=true"
```

--------------------------------

### Parsing Receipt Account Filters Before v0.2.0 Go

Source: https://github.com/streamingfast/firehose-core/blob/develop/UPDATE.md

Displays the previous function signature for a flag parser function used in `TransformFlags` before v0.2.0. It only accepted a single input string representing the flag's value and returned a single `*anypb.Any` and an error.

```Go
func parseReceiptAccountFilters(in string) (*anypb.Any, error) {
  ...

  return anypb.New(filters)
}
```

--------------------------------

### Parsing Receipt Account Filters After v0.2.0 Go

Source: https://github.com/streamingfast/firehose-core/blob/develop/UPDATE.md

Presents the updated function signature for a flag parser used in the new `TransformFlags` structure in v0.2.0. It now receives the `cobra.Command` and `zap.Logger`, allowing access to multiple flags and context. It returns a slice of `*anypb.Any` and an error.

```Go
func parseReceiptAccountFilters(cmd *cobra.Command, logger *zap.Logger) ([]*anypb.Any, error) {
	in := sflags.MustGetString(cmd, "receipt-filter-accounts")

  ...

  any, err := anypb.New(filters)
  // Deal with error

  return []*anypb.Any{any}, err
}
```

--------------------------------

### Implementing GetFirehoseBlockVersion Method Go

Source: https://github.com/streamingfast/firehose-core/blob/develop/UPDATE.md

Demonstrates the required implementation of the `GetFirehoseBlockVersion` method for types implementing the `firecore.Block` interface in v0.2.0. This method should return the block's protocol version as an `int32`, hard-coded to the value used in the chain's config.

```Go
func (b *Block) GetFirehoseBlockVersion() int32 {
  return N
}
```

--------------------------------

### Supported Block ID Encoding Forms (Configuration Values)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

These are examples of the valid values that can be used with the `--advertise-block-id-encoding` configuration flag. The shorter forms (`hex`, `base64`) are the preferred options for new configurations, while the older longer form (`BLOCK_ID_ENCODING_HEX`) remains supported for backward compatibility.

```Configuration
hex
```

```Configuration
base64
```

```Configuration
BLOCK_ID_ENCODING_HEX
```

--------------------------------

### Configuring TransformFlags After v0.2.0 Go

Source: https://github.com/streamingfast/firehose-core/blob/develop/UPDATE.md

Shows the updated configuration structure for `firecore.ToolsConfig#TransformFlags` in v0.2.0. It is now a struct with `Register` and `Parse` fields, allowing flag registration via a function receiving a `pflag.FlagSet` and a single `Parse` function to handle retrieving and processing all registered flags.

```Go
TransformFlags: &firecore.TransformFlags{
  Register: func(flags *pflag.FlagSet) {
    flags.String("receipt-account-filters", "", "Comma-separated accounts to use as filter/index. If it contains a colon (:), it will be interpreted as <prefix>:<suffix> (each of which can be empty, ex: 'hello:' or ':world')")
  },

  Parse: parseReceiptAccountFilters,
},
```

--------------------------------

### Generating Protobuf Definitions (Bash)

Source: https://github.com/streamingfast/firehose-core/blob/develop/proto/README.md

This Bash command triggers the Go code generation process for the Protobuf definitions located in the `./protoregistry` directory. It relies on `go generate` directives within that directory to pull definitions from the configured Buf Registry paths and generate corresponding Go code. Ensure you run this command from the project root or adjust the path accordingly.

```bash
go generate ./protoregistry
```

--------------------------------

### Substreams Development Workflow - Bash

Source: https://github.com/streamingfast/firehose-core/blob/develop/test/substreams_acme/README.md

This snippet shows the standard command-line workflow for working with a Substream project: `build` compiles the modules, `auth` authenticates with a remote endpoint (often for deployment), and `gui` launches a local graphical interface to inspect the stream's output.

```Bash
substreams build
substreams auth
substreams gui
```

--------------------------------

### Initialize Substreams Project (Shell Command)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This command is part of the substreams command-line tooling used to initialize a new substreams project structure. The `v1.10.0` release included enhancements to the `substreams init` process.

```Shell
substreams init
```

--------------------------------

### Node Manager Resume Extra Environment Variables (API Query Parameter)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This query parameter format is used with the Node Manager's `/v1/resume` endpoint to inject temporary environment variables into the node process during its restart. Multiple key-value pairs can be provided by repeating the `extra-env` parameter in the URL query string. These overrides are only applied for the next restart triggered by this call.

```API
extra-env=<key>=<value>&extra-env=<keyN>=<valueN>
```

--------------------------------

### Configure GRPC Listen Addresses (Configuration)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

These configuration flags specify the network addresses and ports where the firehose-grpc and substreams-tier1 services listen for incoming gRPC connections. They now support comma-separated values, enabling the service to listen on multiple interfaces or ports simultaneously, including plaintext and SSL configurations.

```Configuration
firehose-grpc-listen-addr
```

```Configuration
substreams-tier1-grpc-listen-addr
```

--------------------------------

### Extracting Changelog Content Between Versions (Bash)

Source: https://github.com/streamingfast/firehose-core/blob/develop/README.md

A bash command leveraging `awk` and `grep` to selectively extract log entries from the `CHANGELOG.md` file that fall between two specified version headers. This is a utility for maintainers preparing release notes.

```Bash
awk '/## v0.1.11/,/## v0.1.8/' CHANGELOG.md | grep -v '## v0.1.8'
```

--------------------------------

### Node Manager Resume Endpoint (API)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This HTTP endpoint exposed by the Node Manager allows triggering a resume operation for a managed node. It accepts POST requests and supports query parameters like `sync=true` to wait for completion and `extra-env=<key>=<value>` to temporarily override environment variables for the restarted process.

```API
/v1/resume
```

--------------------------------

### Configure Advertised Block ID Encoding (Configuration)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This configuration flag determines the encoding format used when advertising block identifiers. It now supports shorter, more convenient forms like `hex` or `base64` in addition to the older, longer forms like `BLOCK_ID_ENCODING_HEX`, which are still supported but recommended for migration to the shorter versions.

```Configuration
--advertise-block-id-encoding
```

--------------------------------

### Configure Source Channel Max Capacity (Environment Variable)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This environment variable controls the maximum capacity of the internal channel used by the substreams source to buffer data. Adjusting `SOURCE_CHAN_SIZE` can impact performance and memory usage depending on the workload. The default value is 100.

```Shell
SOURCE_CHAN_SIZE
```

--------------------------------

### Substreams Endpoint Info API (API)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This is a new API endpoint introduced in substreams v1.10.0. The `EndpointInfo/Info` endpoint provides information about the capabilities and status of the substreams endpoint it is called on.

```API
EndpointInfo/Info
```

--------------------------------

### Deprecated Metering Metrics (Prometheus)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

These Prometheus metrics were previously used for metering byte usage in substreams. They are now deprecated and will be removed in a future release. Users are instructed to migrate to the new, more detailed metering metrics provided.

```Prometheus
bytes_read
```

```Prometheus
bytes_written
```

--------------------------------

### Disable Substreams Tier1 Compression Enforcement (Configuration)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This configuration flag disables the enforcement of GZIP compression for incoming Substreams connections specifically on the tier1 component. Setting it to `false` allows clients without GZIP support to connect, maintaining older behavior. Logs showing `compressed: false` for incoming requests can help identify clients that need to be updated to support compression.

```Configuration
--substreams-tier1-enforce-compression=false
```

--------------------------------

### Substreams Tier1 Rejected Request Metric (Prometheus)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This Prometheus metric tracks the total number of requests rejected by the substreams-tier1 component. It is labeled by the gRPC/ConnectRPC status code returned for the request, providing detailed reasons for rejection. Requests successfully processed or canceled gracefully are not included in this counter.

```Prometheus
substreams_tier1_rejected_request_counter
```

--------------------------------

### Disable Firehose Compression Enforcement (Configuration)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This configuration flag disables the enforcement of GZIP or ZSTD compression for incoming Firehose connections. Setting it to `false` preserves compatibility with clients that do not support compression. Users are advised to check logs for instances where `compressed: false` is reported to identify clients needing updates.

```Configuration
--firehose-enforce-compression=false
```

--------------------------------

### Substreams Tier2 Rejected Request Metric (Prometheus)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This Prometheus metric tracks the total number of requests rejected by the substreams-tier2 component. Like the tier1 metric, it is labeled by the gRPC/ConnectRPC status code returned for the request, helping diagnose rejection causes. Requests with 'ok' or 'canceled' codes are excluded from this counter.

```Prometheus
substreams_tier2_rejected_request_counter
```

--------------------------------

### Configure Substreams Tier1 Soft Request Limit (Configuration)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This configuration flag sets a soft limit on the number of active client requests a substreams-tier1 instance will accept. Reaching this limit causes the instance to report as 'unready' in health checks, facilitating load balancing by directing new connections to other instances. A value of 0 or less disables the soft limit.

```Configuration
--substreams-tier1-active-requests-soft-limit
```

--------------------------------

### Checking Merged Blocks Batch Consistency - Shell

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This command checks the consistency of a batch of merged block files within a specified range. It helps identify corrupted files with out-of-order or duplicate blocks.

```shell
tools check merged-blocks-batch <merged-blocks-store> <start> <stop>
```

--------------------------------

### Fixing Corrupted Merged Blocks - Shell

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This command attempts to fix merged block files that contain duplicates or blocks outside their intended range within a specified start/stop range. It requires an output store to write the corrected files.

```shell
tools fix-bloated-merged-blocks <merged-blocks-store> <output-store> <start>:<stop>
```

--------------------------------

### Firehose Metering Metric Values (Metering)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

These values represent the reported names for the Firehose block request metering metric. `sf.firehose.v2.Firehose/Blocks` is the correct metric name used since v1.6.5. `sf.firehose.v2.Firehose/Block` was incorrectly reported as the metric name between versions v1.6.1 and v1.6.4 inclusive.

```Metering
sf.firehose.v2.Firehose/Blocks
```

```Metering
sf.firehose.v2.Firehose/Block
```

--------------------------------

### Configure Substreams Tier1 Hard Request Limit (Configuration)

Source: https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md

This configuration flag sets a hard limit on the number of active client requests a substreams-tier1 instance will accept. Exceeding this limit results in incoming gRPC requests being rejected with an 'Unavailable' code, preventing the instance from becoming overloaded. Clients typically auto-reconnect on 'Unavailable', finding another instance. A value of 0 or less disables the hard limit.

```Configuration
--substreams-tier1-active-requests-hard-limit
```
Tokens:
Raw