### Nix Installation and Build Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Use Nix for the most reliable setup. This command enters a Nix shell and then builds the project in release mode. ```bash nix-shell cargo build --release ``` -------------------------------- ### Linux (Arch) Non-Nix Setup Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Installs Clang 16, LLVM 16, and other necessary libraries. Sets environment variables for Clang, libclang, and library paths. Consider using yay for gcc13 if not on system. ```bash sudo pacman -S clang16 llvm16 zlib openssl libtool yay -S gcc13 # or use system gcc # Environment variables (add to ~/.bashrc) export CC=clang-16 export CXX=clang++-16 export LIBCLANG_PATH=/usr/lib/llvm16/lib/libclang.so export LD_LIBRARY_PATH=/usr/lib/llvm16/lib:$LD_LIBRARY_PATH ``` -------------------------------- ### Jetstreamer Plugin on_load Implementation Example Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Example implementation of the `on_load` callback. This snippet demonstrates how to create a table if it doesn't exist using a ClickHouse client. ```rust fn on_load(&self, db: Option>) -> PluginFuture<'_> { async move { if let Some(client) = db { client.query("CREATE TABLE IF NOT EXISTS events (...)").execute().await?; } Ok(()) }.boxed() } ``` -------------------------------- ### Example: Register a Plugin Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Shows how to register a custom plugin instance with the PluginRunner. ```rust runner.register(Box::new(MyPlugin)); ``` -------------------------------- ### Example: Instantiate PluginRunner Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Demonstrates how to create a PluginRunner with a ClickHouse DSN, number of threads, and streaming options. ```rust let runner = PluginRunner::new("http://localhost:8123", 4, false, false, None); ``` -------------------------------- ### Linux (Ubuntu/Debian) Non-Nix Setup Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Installs Clang 16, essential build tools, and configures environment variables for Clang and libclang. Ensure Clang 16 is used, not 17, due to RocksDB dependencies. ```bash # Install Clang 16 wget -qO- https://apt.llvm.org/llvm.sh | sudo bash -s -- 16 sudo apt update && sudo apt install -y gcc-13 g++-13 zlib1g-dev libssl-dev libtool # Set as default sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-16 100 sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-16 100 # Environment variables (add to ~/.bashrc) export CC=clang export CXX=clang++ export LIBCLANG_PATH=/usr/lib/llvm16/lib/libclang.so ``` -------------------------------- ### Run Epoch with Default Configuration Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/START_HERE.md Execute an epoch using the default configuration. This is the quickest way to start. ```bash cargo run --release -- 800 ``` -------------------------------- ### macOS Non-Nix Setup Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Installs LLVM 16 and related libraries using Homebrew. Configures environment variables for Clang, libclang, linker, and preprocessor paths. Ensure correct paths for your Homebrew installation. ```bash brew install llvm@16 zlib openssl libtool # Environment variables (add to ~/.zshrc) export CC=/opt/homebrew/opt/llvm@16/bin/clang export CXX=/opt/homebrew/opt/llvm@16/bin/clang++ export LIBCLANG_PATH=/opt/homebrew/opt/llvm@16/lib/libclang.dylib export LDFLAGS="-L/opt/homebrew/opt/llvm@16/lib" export CPPFLAGS="-I/opt/homebrew/opt/llvm@16/include" ``` -------------------------------- ### Example: Execute PluginRunner Run Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Demonstrates how to asynchronously run the PluginRunner over a slot range, enabling ClickHouse. ```rust use std::sync::Arc; let runner = Arc::new(runner); runner.clone().run(0..1000000, true).await?; ``` -------------------------------- ### Configure Jetstreamer with Environment Variables Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/START_HERE.md Set runtime behavior using environment variables. This example shows how to configure threads and sequential processing. ```bash # Environment JETSTREAMER_THREADS=8 JETSTREAMER_SEQUENTIAL=1 ``` -------------------------------- ### CLI Usage Examples for Jetstreamer Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/README.md Demonstrates various command-line interface commands for Jetstreamer, including replaying epochs, using custom plugins, and setting reverse chronological order. ```bash # Replay epoch 800 with default plugin cargo run --release -- 800 # Custom threads and plugins cargo run --release -- 800 \ --with-plugin instruction-tracking \ --clickhouse-dsn http://remote.example.com:8123 # Reverse chronological (newest to oldest) cargo run --release -- 800:810 --reverse # Sequential mode for predictable ordering JETSTREAMER_SEQUENTIAL=1 cargo run --release -- 800 ``` -------------------------------- ### Run Epoch with Custom Configuration Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/START_HERE.md Execute an epoch with custom settings, including environment variables and specific plugins. Useful for advanced setups. ```bash JETSTREAMER_THREADS=4 \ cargo run --release -- 800 \ --with-plugin program-tracking \ --clickhouse-dsn http://remote.example.com:8123 ``` -------------------------------- ### on_load Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Invoked once before the firehose starts streaming events. Use this for initializing tables, creating indexes, or setting up thread-local state. ```APIDOC ## on_load ### Description Invoked once before the firehose starts streaming events. ### Parameters #### Path Parameters - **db** (Option>) - Optional - ClickHouse client if enabled. ### Returns `PluginFuture<'_>` ### Default No-op ### Use Case Initialize tables, create indexes, set up thread-local state. ### Example ```rust fn on_load(&self, db: Option>) -> PluginFuture<'_> { async move { if let Some(client) = db { client.query("CREATE TABLE IF NOT EXISTS events (...)").execute().await?; } Ok(()) }.boxed() } ``` ``` -------------------------------- ### Full Jetstreamer Runner Configuration Example Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md This Rust code demonstrates how to initialize and configure the JetstreamerRunner with custom settings, including log level, plugins, thread count, slot range, and ClickHouse DSN. It also includes a custom plugin for logging transactions. ```rust use std::sync::Arc; use clickhouse::Client; use jetstreamer::{ JetstreamerRunner, firehose::epochs, plugin::{Plugin, PluginFuture}, }; struct LoggingPlugin; impl Plugin for LoggingPlugin { fn name(&self) -> &'static str { "logging" } fn on_transaction<'a>( &'a self, _thread_id: usize, _db: Option>, tx: &'a jetstreamer::firehose::TransactionData, ) -> PluginFuture<'a> { async move { println!("tx {} in slot {}", tx.signature, tx.slot); Ok(()) }.boxed() } } let (start_slot, end_inclusive) = epochs::epoch_to_slot_range(800); JetstreamerRunner::new() .with_log_level("info") .with_plugin(Box::new(LoggingPlugin)) .with_threads(8) .with_slot_range_bounds(start_slot, end_inclusive + 1) .with_clickhouse_dsn("http://localhost:8123") .run()?; ``` -------------------------------- ### Example: Streaming Epoch 800 with Firehose Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/firehose.md Demonstrates how to use the `firehose` function to stream blocks and transactions for a specific epoch (epoch 800). It configures the number of threads, the slot range, and provides closures for handling blocks and transactions. ```rust use futures_util::FutureExt; use jetstreamer_firehose:: epochs, firehose::{self, BlockData, TransactionData, Stats}; async fn stream_epoch_800() -> Result<(), Box> { let (start, _) = epochs::epoch_to_slot_range(800); let (_, end_inclusive) = epochs::epoch_to_slot_range(800); firehose( 4, // 4 threads false, // not sequential false, // not reverse None, // default buffer window start..(end_inclusive + 1), // epoch 800 Some(|_thread_id, block: BlockData| { async move { println!("block slot {}", block.slot()); Ok(()) } .boxed() }), Some(|_thread_id, tx: TransactionData| { async move { println!("tx {}", tx.signature); Ok(()) } .boxed() }), None, // no entry handler None, // no reward handler None, // no error handler None, // no stats None, // no shutdown signal ) .await .map_err(|(err, slot)| format!("failed at slot {}: {}", slot, err))?; Ok(()) } ``` -------------------------------- ### S3 URI Examples for Jetstreamer Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Examples of S3 URIs that can be used for configuring Jetstreamer's storage backends. These URIs specify the bucket and optional prefixes for storing CAR files and indexes. ```plaintext s3://my-bucket/ledger/ s3://my-bucket/data/prefix/ s3://backup-cluster.s3.example.com/jetstreamer/ ``` -------------------------------- ### Implement Plugin Error Handling Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/errors.md Shows how to implement the `on_error` hook for a plugin to handle `FirehoseErrorContext`. This example logs the error details including slot and epoch. ```rust impl Plugin for MyPlugin { fn on_error<'a>( &'a self, thread_id: usize, db: Option>, error: &'a FirehoseErrorContext, ) -> PluginFuture<'a> { async move { eprintln!( "Thread {} failed at slot {} (epoch {}): {}", thread_id, error.slot, error.epoch, error.error ); // Log to database, update metrics, etc. Ok(()) }.boxed() } } ``` -------------------------------- ### Replay Slot Ranges Across Epochs with Explicit Threads Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Replay a specific range of slots, which can span multiple epochs. This example also explicitly sets the number of worker threads. ```bash JETSTREAMER_THREADS=8 cargo run --release -- 358560000:367631999 ``` -------------------------------- ### Implement a Logging Plugin for Jetstreamer Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Implement the `Plugin` trait to observe and log transaction and block events. This example demonstrates reacting to both types of events and printing details to the console. Ensure downstream services are designed to handle potentially arbitrary write orders due to parallel processing. ```rust use std::sync::Arc; use clickhouse::Client; use jetstreamer::{ JetstreamerRunner, firehose::{BlockData, TransactionData}, firehose::epochs, plugin::{Plugin, PluginFuture}, }; struct LoggingPlugin; impl Plugin for LoggingPlugin { fn name(&self) -> &'static str { "logging" } fn on_transaction<'a>( &'a self, _thread_id: usize, _db: Option>, tx: &'a TransactionData, ) -> PluginFuture<'a> { Box::pin(async move { println!("tx {} landed in slot {}", tx.signature, tx.slot); Ok(()) }) } fn on_block<'a>( &'a self, _thread_id: usize, _db: Option>, block: &'a BlockData, ) -> PluginFuture<'a> { Box::pin(async move { if block.was_skipped() { println!("slot {} was skipped", block.slot()); } else { println!("processed block at slot {}", block.slot()); } Ok(()) }) } } let (start_slot, end_inclusive) = epochs::epoch_to_slot_range(800); JetstreamerRunner::new() .with_plugin(Box::new(LoggingPlugin)) .with_threads(4) .with_slot_range_bounds(start_slot, end_inclusive + 1) .with_clickhouse_dsn("https://clickhouse.example.com") .run() .expect("runner completed"); ``` -------------------------------- ### Implement a Custom Plugin Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/START_HERE.md Define custom observers for Firehose events by implementing the Plugin trait. This example shows basic trait method signatures. ```rust impl Plugin for MyPlugin { fn name(&self) -> &'static str { "my-plugin" } fn on_transaction<'a>(...) -> PluginFuture<'a> { ... } fn on_block<'a>(...) -> PluginFuture<'a> { ... } } ``` -------------------------------- ### Replay a Single Epoch Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/README.md Replays a single epoch by specifying the epoch number. This example demonstrates basic epoch replay functionality. ```rust use jetstreamer::JetstreamerRunner; use jetstreamer::firehose::epochs; let (start, end_inclusive) = epochs::epoch_to_slot_range(800); JetstreamerRunner::new() .with_threads(4) .with_slot_range_bounds(start, end_inclusive + 1) .run()?; ``` -------------------------------- ### Get Configured Compact Index Location Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/system-and-utils.md Retrieves the configured location for compact indexes, prioritizing JETSTREAMER_COMPACT_INDEX_BASE_URL, falling back to JETSTREAMER_ARCHIVE_BASE, and defaulting to 'https://files.old-faithful.net'. ```rust pub fn index_location() -> &'static Location ``` -------------------------------- ### Set Slot Range Bounds Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Configures the slot range using explicit start (inclusive) and end (exclusive) bounds. Panics if start_slot is greater than or equal to end_slot. ```rust let (start, end_inclusive) = jetstreamer::firehose::epochs::epoch_to_slot_range(800); runner.with_slot_range_bounds(start, end_inclusive + 1) ``` -------------------------------- ### Get Compact Index Base URL Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/parse-and-index.md Retrieves the base URL for compact index files. It prioritizes the JETSTREAMER_COMPACT_INDEX_BASE_URL environment variable, falls back to JETSTREAMER_ARCHIVE_BASE, and defaults to 'https://files.old-faithful.net'. ```rust pub fn get_index_base_url() -> Result ``` -------------------------------- ### NodeReader::read_next_block Example Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/parse-and-index.md Iterates through blocks in a CAR stream, printing the CID and parsing the data for each block. Ensure `read_header` is called first. Handles stream end and decode errors. ```rust let mut reader = NodeReader::new(file_reader); reader.read_header().await?; while let Some((cid, data)) = reader.read_next_block().await? { println!("Block CID: {}", cid); let node = parse_any_from_cbordata(data)?; } ``` -------------------------------- ### No Plugins and Custom ClickHouse Endpoint Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Run Jetstreamer without any built-in plugins and override the ClickHouse DSN to point to a custom endpoint. This is useful for isolated testing or specific infrastructure setups. ```bash # No plugins, custom ClickHouse endpoint cargo run --release -- 800 --no-plugins \ --clickhouse-dsn http://clickhouse.example.com:8123 ``` -------------------------------- ### epoch_to_slot_range Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/epochs.md Converts a given epoch number into its corresponding inclusive start and end slot numbers. ```APIDOC ## epoch_to_slot_range ### Description Converts an epoch number to the inclusive start and end slot numbers. ### Function Signature `pub fn epoch_to_slot_range(epoch: u64) -> (u64, u64)` ### Parameters #### Path Parameters - **epoch** (u64) - Required - Epoch number ### Returns - `(start_slot, end_slot_inclusive)` tuple ### Example ```rust let (start, end_inclusive) = epoch_to_slot_range(800); println!("Epoch 800: slots {} to {}", start, end_inclusive); // Output: Epoch 800: slots 345600000 to 346199999 let slot_range = start..(end_inclusive + 1); // Half-open for Range ``` ``` -------------------------------- ### Set Slot Range Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Restricts execution to a specific half-open slot range [start, end). ```rust runner.with_slot_range(start_slot..end_slot) ``` -------------------------------- ### Sequential Replay Configuration Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Set up sequential replay by enabling the `JETSTREAMER_SEQUENTIAL` flag and defining the buffer window size. ```bash JETSTREAMER_SEQUENTIAL=1 \ JETSTREAMER_BUFFER_WINDOW=8GiB \ cargo run --release -- 800:805 ``` -------------------------------- ### JetstreamerRunner::with_slot_range Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Restricts the runner's execution to a specific half-open slot range [start, end). ```APIDOC ## JetstreamerRunner::with_slot_range ### Description Restricts execution to a specific slot range. ### Parameters #### Path Parameters - **slot_range** (`Range`) - Required - Half-open slot range [start, end) ### Returns Self for chaining ``` -------------------------------- ### Configure and Run JetstreamerRunner Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/START_HERE.md Use JetstreamerRunner for the main entry point. Configure threads, plugins, and slot ranges using a builder pattern. ```rust JetstreamerRunner::new() .with_threads(8) .with_plugin(Box::new(MyPlugin)) .with_slot_range_bounds(start, end) .run()?; ``` -------------------------------- ### slot_to_offset Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/parse-and-index.md A convenience function that uses the global singleton index to get the byte offset for a given slot. ```APIDOC ## slot_to_offset ### Description Convenience function using the global singleton index to resolve a slot to its byte offset. ### Method `async fn slot_to_offset(slot: u64) -> Result` ### Parameters #### Path Parameters - **slot** (`u64`) - Required - Slot number ### Response #### Success Response (200) - **u64** (`u64`) - Byte offset in CAR file. - **SlotOffsetIndexError** (`SlotOffsetIndexError`) - An error if the slot is not found or another issue occurs. ### Example ```rust let offset = slot_to_offset(345600000).await?; ``` ``` -------------------------------- ### Get Slot Timestamp Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/epochs.md Retrieves the Unix timestamp for a given slot number using an HTTP client. ```APIDOC ## get_slot_timestamp ### Description Retrieves the Unix timestamp for a given slot. ### Parameters #### Path Parameters - **slot** (u64) - Required - Slot number - **client** (&Client) - Required - HTTP client for queries ### Response #### Success Response (200) - **timestamp** (i64) - Unix timestamp #### Error Response - **SlotTimestampError::NotFound** - Slot has no timestamp (leader skipped) - **SlotTimestampError::RequestFailed** - Network or parsing error ### Request Example ```rust match get_slot_timestamp(345600000, &client).await { Ok(timestamp) => println!("Slot 345600000: {}", timestamp), Err(e) => println!("Error: {}", e), } ``` ``` -------------------------------- ### List Built-in Plugins Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Display a list of available built-in plugins and exit without running a replay. This helps in discovering available plugins. ```bash # List built-in plugins cargo run --release -- --list-plugins ``` -------------------------------- ### Sequential Replay with Buffer Window Override Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Run in sequential mode and explicitly set the ripget buffer window size. This controls the amount of data buffered during download. ```bash JETSTREAMER_SEQUENTIAL=1 cargo run --release -- 800 --buffer-window 4GiB ``` -------------------------------- ### run Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Builds the plugin runtime and streams blocks through every registered plugin, respecting environment variables. ```APIDOC ## run ### Description Builds the plugin runtime and streams blocks through every registered plugin. Respects all environment variables documented in the crate-level docs. ### Method Rust Function ### Returns - `Result<(), PluginRunnerError>` ### Errors - `PluginRunnerError::PluginLifecycle`: Plugin `on_load` or `on_exit` failed, or ClickHouse startup failed. - `PluginRunnerError::Firehose`: Firehose streaming encountered an error at a specific slot. ### Example ```rust JetstreamerRunner::new() .with_threads(4) .with_slot_range_bounds(0, 1000000) .run()?; ``` ``` -------------------------------- ### JetstreamerRunner::with_slot_range_bounds Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Configures the slot range using explicit start (inclusive) and end (exclusive) bounds. Panics if start_slot is greater than or equal to end_slot. ```APIDOC ## JetstreamerRunner::with_slot_range_bounds ### Description Configures the slot range using explicit start (inclusive) and end (exclusive) bounds. ### Parameters #### Path Parameters - **start_slot** (`u64`) - Required - Starting slot (inclusive) - **end_slot** (`u64`) - Required - Ending slot (exclusive) ### Returns Self for chaining ### Panics If `start_slot >= end_slot` ### Example ```rust let (start, end_inclusive) = jetstreamer::firehose::epochs::epoch_to_slot_range(800); runner.with_slot_range_bounds(start, end_inclusive + 1) ``` ``` -------------------------------- ### JetstreamerRunner::new Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Creates a new JetstreamerRunner instance with default configurations. Default settings are influenced by environment variables for thread count and ClickHouse DSN. ```APIDOC ## JetstreamerRunner::new ### Description Creates a new runner with default configuration. The runner reads environment variables to establish thread count, ClickHouse DSN, and other runtime parameters. ### Returns `JetstreamerRunner` with defaults: - Threads: auto-detected via `jetstreamer_firehose::system::optimal_firehose_thread_count()` - ClickHouse DSN: from `JETSTREAMER_CLICKHOUSE_DSN` env var or `http://localhost:8123` - Sequential mode: disabled - Reverse mode: disabled ### Example ```rust let runner = JetstreamerRunner::new(); ``` ``` -------------------------------- ### System and Utility Functions Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/README.md Offers heuristic functions like `optimal_firehose_thread_count()` and `default_firehose_buffer_window_bytes()`, along with utilities for buffer size parsing, HTTP client creation, and network operations. ```APIDOC ## System and Utility Functions ### Description This section covers various system-level and utility functions designed to optimize Jetstreamer's performance and manage network interactions. It includes heuristic functions such as `optimal_firehose_thread_count()` to determine the ideal number of threads for processing, and `default_firehose_buffer_window_bytes()` to calculate default buffer sizes. Utilities for parsing and formatting buffer sizes, creating HTTP clients, and performing general network operations are also provided. ### Method - `optimal_firehose_thread_count()`: Calculates the optimal number of threads. - `default_firehose_buffer_window_bytes()`: Calculates the default buffer window size in bytes. ### Endpoint N/A (These are utility functions) ### Parameters - `optimal_firehose_thread_count()`: May take system-specific parameters. - `default_firehose_buffer_window_bytes()`: May take system-specific parameters. - Buffer size parsing and formatting utilities. - HTTP client creation utilities. - Network utilities (e.g., epoch queries). ### Request Example N/A ### Response N/A ``` -------------------------------- ### Sequential Mode with Custom Buffer Window Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Enable sequential mode and specify a custom buffer window size for ripget downloads. This is useful for fine-tuning memory usage. ```bash # Replay in sequential mode with custom buffer window JETSTREAMER_SEQUENTIAL=1 cargo run --release -- 800 --buffer-window 2GiB ``` -------------------------------- ### Get Slot Byte Offset Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/parse-and-index.md Retrieves the byte offset for a specific slot in a CAR archive using the SlotOffsetIndex. Returns SlotOffsetIndexError::SlotNotFound for unindexed slots. ```rust pub async fn get(&self, slot: u64) -> Result ``` ```rust let offset = index.get(345600000).await?; println!("Slot 345600000 is at byte offset {}", offset); ``` -------------------------------- ### Regenerate and Open Documentation Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Command to regenerate the project's documentation and automatically open it in a web browser. ```bash cargo doc --workspace --open ``` -------------------------------- ### Get Configured CAR Snapshot Location Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/system-and-utils.md Retrieves the configured location for CAR snapshots, prioritizing JETSTREAMER_HTTP_BASE_URL, falling back to JETSTREAMER_ARCHIVE_BASE, and defaulting to 'https://files.old-faithful.net'. ```rust pub fn car_location() -> &'static Location ``` -------------------------------- ### Enable Instruction Tracking Plugin Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Run the Jetstreamer Runner for a specific epoch and enable the 'instruction-tracking' plugin instead of the default plugins. ```bash cargo run --release -- 800 --with-plugin instruction-tracking ``` -------------------------------- ### Create and Format CID Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/block-and-entry.md Demonstrates creating a CID from bytes and converting it to its string representation (Base32). Uses the `cid` crate. ```rust let cid_bytes = vec![...]; let cid = Cid::try_from(cid_bytes.as_slice())?; println!("CID: {}", cid.to_string()); ``` -------------------------------- ### Replay Slot Range with Multiple Plugins Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Replay a range of slots and enable multiple built-in plugins simultaneously. This allows for comprehensive data analysis. ```bash # Replay slot range 358560000 to 367631999 with both plugins cargo run --release -- 358560000:367631999 \ --with-plugin program-tracking \ --with-plugin instruction-tracking ``` -------------------------------- ### Convert Epoch to Slot Range Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/epochs.md Converts a given epoch number into its corresponding inclusive start and end slot numbers. This is useful for defining slot ranges for processing or querying. ```rust pub fn epoch_to_slot_range(epoch: u64) -> (u64, u64) let (start, end_inclusive) = epoch_to_slot_range(800); println!("Epoch 800: slots {} to {}", start, end_inclusive); // Output: Epoch 800: slots 345600000 to 346199999 let slot_range = start..(end_inclusive + 1); // Half-open for Range ``` -------------------------------- ### Fetch Epoch Stream with Options Function Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/epochs.md Opens a streaming connection with custom configuration, allowing overrides for base URL, compression format, and request timeout. ```rust pub async fn fetch_epoch_stream_with_options( epoch: u64, client: &Client, options: FetchEpochStreamOptions, ) -> EpochStream ``` ```rust let options = FetchEpochStreamOptions { base_url: Some("https://mirror.example.com".to_string()), compression: CompressionFormat::Zstd, }; let stream = fetch_epoch_stream_with_options(800, &client, options).await; ``` -------------------------------- ### Get Slot Timestamp (Rust) Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/epochs.md Retrieves the Unix timestamp for a specific slot using an HTTP client. Handles cases where a slot might have been skipped or if a network/parsing error occurs. ```rust pub async fn get_slot_timestamp( slot: u64, client: &Client, ) -> Result ``` ```rust match get_slot_timestamp(345600000, &client).await { Ok(timestamp) => println!("Slot 345600000: {}", timestamp), Err(e) => println!("Error: {}", e), } ``` -------------------------------- ### Create SlotOffsetIndex Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/parse-and-index.md Initializes a SlotOffsetIndex for a given network. This client downloads and caches compact index files from Old Faithful. ```rust pub async fn new(network: &str) -> Result ``` ```rust let index = SlotOffsetIndex::new("mainnet").await?; ``` -------------------------------- ### Format and Lint Code Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Use these commands to format and lint the codebase. Ensure code adheres to project standards. ```bash cargo fmt --all ``` ```bash cargo clippy --workspace ``` -------------------------------- ### High-Throughput Backfill Configuration Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Configure for high-throughput backfilling by setting network capacity and ClickHouse mode. Requires specifying the ClickHouse DSN. ```bash JETSTREAMER_NETWORK_CAPACITY_MB=30000 \ JETSTREAMER_CLICKHOUSE_MODE=remote \ cargo run --release -- 800 \ --with-plugin program-tracking \ --clickhouse-dsn http://remote-ch.example.com:8123 ``` -------------------------------- ### on_load Event Handler Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Callback invoked when the plugin is loaded. Use this for any initialization logic, such as setting up database connections or resources. ```rust fn on_load(&self, db: Option>) -> PluginFuture<'_> ``` -------------------------------- ### PluginRunner::new Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Creates a new PluginRunner instance. It configures the connection to ClickHouse, the number of worker threads, and streaming behavior. ```APIDOC ## PluginRunner::new ### Description Creates a new runner instance, configuring ClickHouse connection, worker threads, and streaming options. ### Signature ```rust pub fn new( clickhouse_dsn: impl Display, num_threads: usize, sequential: bool, reverse: bool, buffer_window_bytes: Option, ) -> Self ``` ### Parameters #### Path Parameters * None #### Query Parameters * None #### Request Body * None ### Parameters Table | Parameter | Type | Description | |-----------|------|-------------| | clickhouse_dsn | `impl Display` | HTTP(S) DSN for ClickHouse (e.g., `http://localhost:8123`) | | num_threads | `usize` | Number of firehose worker threads | | sequential | `bool` | Enable single-worker sequential streaming | | reverse | `bool` | Stream epochs highest-to-lowest (implies sequential) | | buffer_window_bytes | `Option` | Sequential mode window size; None uses default | ### Returns `PluginRunner` ### Example ```rust let runner = PluginRunner::new("http://localhost:8123", 4, false, false, None); ``` ``` -------------------------------- ### Reverse Backfill Mode Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Perform a backfill operation starting from the newest epochs and moving towards older ones. Reverse mode implies sequential mode and processes epochs in descending order. ```bash cargo run --release -- 345600000:1296000000 --reverse ``` -------------------------------- ### Create New PluginRunner Instance Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Creates a new PluginRunner instance with specified configuration for ClickHouse DSN, threading, streaming mode, and buffer window. ```rust pub fn new( clickhouse_dsn: impl Display, num_threads: usize, sequential: bool, reverse: bool, buffer_window_bytes: Option, ) -> Self ``` -------------------------------- ### Replay with Specific Plugin Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Replay a specific epoch while specifying a different built-in plugin, such as instruction-tracking. ```bash # Replay epoch 800 with instruction-tracking instead cargo run --release -- 800 --with-plugin instruction-tracking ``` -------------------------------- ### Get Optimal Firehose Thread Count Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/system-and-utils.md Calculates the recommended number of worker threads based on system resources like CPU, RAM, and network capacity. This is useful for optimizing performance by matching the thread pool size to available resources. ```rust pub fn optimal_firehose_thread_count() -> usize ``` ```rust let recommended = jetstreamer_firehose::system::optimal_firehose_thread_count(); println!("Recommended threads: {}", recommended); ``` -------------------------------- ### Custom Archive Mirror Configuration Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Specify custom URLs for the archive and index base using environment variables. This is useful for using a private mirror. ```bash JETSTREAMER_HTTP_BASE_URL=https://mirror.company.com/old-faithful \ JETSTREAMER_COMPACT_INDEX_BASE_URL=https://mirror.company.com/indexes \ cargo run --release -- 800 ``` -------------------------------- ### Basic Epoch Replay Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Replay a specific epoch using the default program-tracking plugin. This is the simplest form of replay. ```bash # Replay epoch 800 with default plugin (program-tracking) cargo run --release -- 800 ``` -------------------------------- ### Get Default Firehose Buffer Window Size Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/system-and-utils.md Determines the default buffer window size in bytes for sequential mode, calculated based on available RAM. This helps in setting appropriate buffer sizes to avoid memory exhaustion while accommodating typical workloads. ```rust pub fn default_firehose_buffer_window_bytes() -> u64 ``` ```rust let window = jetstreamer_firehose::system::default_firehose_buffer_window_bytes(); println!("Default window: {} bytes", format_byte_size(window)); ``` -------------------------------- ### Configure Jetstreamer with Builder Methods Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/START_HERE.md Adjust runtime settings like threads and sequential processing directly within the code using builder methods. ```rust # Builder .with_threads(4).with_sequential(true) ``` -------------------------------- ### Run Jetstreamer with Plugins Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Builds the plugin runtime and streams blocks through registered plugins. This method respects environment variables and can encounter errors during plugin lifecycle or firehose streaming. ```rust pub fn run(self) -> Result<(), PluginRunnerError> ``` ```rust JetstreamerRunner::new() .with_threads(4) .with_slot_range_bounds(0, 1000000) .run()?; ``` -------------------------------- ### fetch_epoch_stream_with_options Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/epochs.md Opens a streaming connection to an epoch's CAR archive with custom configuration options. This allows for overriding the base URL, specifying compression format, and setting a request timeout. ```APIDOC ## fetch_epoch_stream_with_options ### Description Opens a streaming connection to an epoch's CAR archive with custom configuration. ### Method `async fn` ### Parameters #### Path Parameters - **epoch** (u64) - Required - Epoch to stream - **client** (&Client) - Required - HTTP client - **options** (FetchEpochStreamOptions) - Required - Custom fetch options ### Returns - **EpochStream** - An EpochStream struct containing the epoch number, the underlying HTTP response, and a reader for the decompressed stream. ``` -------------------------------- ### JetstreamerRunner Builder Methods Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Programmatically configure the Jetstreamer runner using builder methods. These methods allow setting log level, plugins, threads, sequential mode, buffer window, slot ranges, and ClickHouse DSN. ```rust pub struct JetstreamerRunner { pub fn new() -> Self pub fn with_log_level(mut self, level: impl Into) -> Self pub fn with_plugin(mut self, plugin: Box) -> Self pub fn with_threads(mut self, threads: usize) -> Self pub fn with_sequential(mut self, sequential: bool) -> Self pub fn with_reverse(mut self, reverse: bool) -> Self pub fn with_buffer_window_bytes(mut self, bytes: Option) -> Self pub fn with_slot_range(mut self, range: Range) -> Self pub fn with_slot_range_bounds(mut self, start: u64, end: u64) -> Self pub fn with_clickhouse_dsn(mut self, dsn: impl Into) -> Self } ``` -------------------------------- ### Reverse Chronological Replay Configuration Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Enable reverse chronological replay by setting the `JETSTREAMER_REVERSE` flag. Specify the epoch range for replay. ```bash JETSTREAMER_REVERSE=1 \ cargo run --release -- 345600000:346799999 ``` -------------------------------- ### Low-level Streaming with Firehose Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/START_HERE.md Utilize the Firehose for direct access to block, transaction, and entry data. Configure threading, buffering, and event handlers. ```rust firehose( 4, false, false, None, // threads, sequential, reverse, buffer slot_range, Some(block_handler), Some(tx_handler), Some(entry_handler), Some(reward_handler), None, Some(stats), None ).await?; ``` -------------------------------- ### Configure Jetstreamer with CLI Arguments Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/START_HERE.md Control Jetstreamer's execution via command-line interface arguments, including thread count and plugin usage. ```bash # CLI cargo run -- 800 --with-plugin instruction-tracking ``` -------------------------------- ### Create New JetstreamerRunner Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Creates a new runner with default configuration. The runner reads environment variables for thread count, ClickHouse DSN, and other parameters. ```rust let runner = JetstreamerRunner::new(); ``` -------------------------------- ### create_http_client Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/system-and-utils.md Creates a configured reqwest::Client for Old Faithful operations with connection pooling, timeouts, gzip compression, and a User-Agent header. ```APIDOC ## create_http_client ### Description Creates a configured `reqwest::Client` for Old Faithful operations. ### Returns `reqwest::Client` with: - Connection pooling enabled - Reasonable timeout values - gzip compression support - User-Agent header set ### Example ```rust let client = create_http_client(); let response = client.get("https://files.old-faithful.net/...") .send() .await?; ``` ``` -------------------------------- ### Set Ripget Sequential Download Window Size Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Sets the ripget sequential download window size in bytes. This is only used when sequential mode is enabled. Use None to use the default. ```rust runner.with_buffer_window_bytes(Some(1024 * 1024 * 4)) ``` -------------------------------- ### Enable S3 Backend Feature Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Enable the 's3-backend' Cargo feature when running or depending on Jetstreamer to support S3-compatible archives. ```bash cargo run --features s3-backend -- 800 ``` -------------------------------- ### Skip ClickHouse for Connection Issues Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/errors.md During 'on_load', if ClickHouse is unavailable due to connectivity or authentication failures, you can skip its integration using the '--no-plugins' flag. Verify connectivity separately. ```bash # Verify connectivity curl http://localhost:8123/ping ``` ```bash # Or use --no-plugins to skip ClickHouse cargo run --release -- 800 --no-plugins ``` -------------------------------- ### Manage ClickHouse Server and Client Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Use these Cargo aliases to manage the bundled ClickHouse server and client. `cargo clickhouse-server` launches the server, and `cargo clickhouse-client` connects to the local instance for inspection. ```bash cargo clickhouse-server ``` ```bash cargo clickhouse-client ``` -------------------------------- ### PluginRunner::run Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Executes the firehose stream across a specified slot range, with an option to enable ClickHouse integration. ```APIDOC ## PluginRunner::run ### Description Runs the firehose across the specified slot range, with an option to enable ClickHouse client for plugins. ### Signature ```rust pub async fn run( self: Arc, slot_range: Range, clickhouse_enabled: bool, ) -> Result<(), PluginRunnerError> ``` ### Parameters #### Path Parameters * None #### Query Parameters * None #### Request Body * None ### Parameters Table | Parameter | Type | Description | |-----------|------|-------------| | slot_range | `Range` | Half-open slot range [start, end) | | clickhouse_enabled | `bool` | Whether to pass ClickHouse client to plugins | ### Returns `Result<(), PluginRunnerError>` ### Errors | Error | Condition | |-------|-----------| | `PluginRunnerError::PluginLifecycle` | `on_load` or `on_exit` hook failed | | `PluginRunnerError::Firehose` | Firehose streaming failed at a slot | ### Example ```rust use std::sync::Arc; let runner = Arc::new(runner); runner.clone().run(0..1000000, true).await?; ``` ``` -------------------------------- ### Testing Without Database Configuration Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Configure Jetstreamer to run tests without a database by setting `JETSTREAMER_CLICKHOUSE_MODE` to `off` and disabling plugins. ```bash JETSTREAMER_CLICKHOUSE_MODE=off \ cargo run --release -- 800 --no-plugins ``` -------------------------------- ### Calculate Default Firehose Buffer Window Bytes Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Determines the default buffer window size for sequential mode, capped at 4 GiB or 15% of available RAM. Override with JETSTREAMER_BUFFER_WINDOW or --buffer-window CLI flag. ```python min(4 GiB, available_ram * 0.15) ``` -------------------------------- ### SlotOffsetIndex::new Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/parse-and-index.md Creates a new SlotOffsetIndex client for the specified network. This function downloads and caches compact index files from Old Faithful. ```APIDOC ## SlotOffsetIndex::new ### Description Creates a new index client for the specified network. ### Method `async fn new(network: &str) -> Result` ### Parameters #### Path Parameters - **network** (`&str`) - Required - Network suffix (e.g., "mainnet", "testnet") ### Response #### Success Response (200) - **SlotOffsetIndex** (`SlotOffsetIndex`) - A new instance of the index. - **SlotOffsetIndexError** (`SlotOffsetIndexError`) - An error during index creation. ### Behavior Downloads and caches compact index files from Old Faithful. ### Example ```rust let index = SlotOffsetIndex::new("mainnet").await?; ``` ``` -------------------------------- ### Plugin Version Implementation Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Implement the `version` method to specify the semantic version for the plugin. Defaults to `1` if not overridden. ```rust fn version(&self) -> u16 { 1 } ``` -------------------------------- ### JetstreamerRunner::with_buffer_window_bytes Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-runner.md Sets the ripget sequential download window size in bytes. This is applicable only when sequential mode is enabled. ```APIDOC ## JetstreamerRunner::with_buffer_window_bytes ### Description Sets the ripget sequential download window size in bytes. Only used when sequential mode is enabled. ### Parameters #### Path Parameters - **buffer_window_bytes** (`Option`) - Required - Window size in bytes; None uses default (min 4GiB, 15% RAM) ### Returns Self for chaining ``` -------------------------------- ### Location struct Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/system-and-utils.md Configures archive backend (HTTP or S3) with a base URL and optional S3 configuration. ```APIDOC ## Location ### Description Configures archive backend (HTTP or S3). ### Fields - **base_url** (String) - HTTP base URL or `s3://` URI - **s3_config** (Option) - S3 credentials and settings (if S3 backend active) ``` -------------------------------- ### Plugin Name Implementation Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Implement the `name` method to return a human-friendly plugin name used in logs and persisted metadata. This is a required method for all plugins. ```rust fn name(&self) -> &'static str { "my-plugin" } ``` -------------------------------- ### FetchEpochStreamOptions Struct Definition Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/epochs.md Defines the configuration options for epoch streaming, including an optional base URL, compression format, and request timeout. ```rust pub struct FetchEpochStreamOptions { pub base_url: Option, pub compression: CompressionFormat, pub timeout: Option, } ``` -------------------------------- ### Run Workspace Tests Source: https://github.com/anza-xyz/jetstreamer/blob/main/README.md Execute all tests within the project's workspace to ensure code correctness. ```bash cargo test --workspace ``` -------------------------------- ### S3-backed Archive Configuration Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/configuration.md Configure Jetstreamer to use an S3 bucket for archives. Requires enabling the `s3-backend` feature and specifying bucket and region. ```bash JETSTREAMER_ARCHIVE_BACKEND=s3 \ JETSTREAMER_S3_BUCKET=my-archive \ JETSTREAMER_S3_REGION=us-west-2 \ cargo run --release --features s3-backend -- 800 ``` -------------------------------- ### Create HTTP Client for Old Faithful Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/system-and-utils.md Creates a configured reqwest::Client for Old Faithful operations with connection pooling, timeouts, gzip compression, and a User-Agent header. ```rust pub fn create_http_client() -> Client ``` ```rust let client = create_http_client(); let response = client.get("https://files.old-faithful.net/...") .send() .await?; ``` -------------------------------- ### Run PluginRunner Firehose Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/api-reference/jetstreamer-plugin.md Executes the firehose process across a specified slot range, with an option to enable ClickHouse integration. Returns a Result indicating success or a PluginRunnerError. ```rust pub async fn run( self: Arc, slot_range: Range, clickhouse_enabled: bool, ) -> Result<(), PluginRunnerError> ``` -------------------------------- ### Jetstreamer File Structure Source: https://github.com/anza-xyz/jetstreamer/blob/main/_autodocs/README.md This snippet outlines the directory structure of the Jetstreamer project, detailing the purpose of each file and subdirectory. ```text /workspace/home/output/ ├── README.md (this file) ├── types.md (all data type definitions) ├── configuration.md (env vars, CLI, builder methods) ├── errors.md (error types and recovery) └── api-reference/ ├── jetstreamer-runner.md (JetstreamerRunner) ├── firehose.md (core streaming) ├── jetstreamer-plugin.md (plugin framework) ├── epochs.md (epoch utilities) ├── block-and-entry.md (block/entry types) ├── parse-and-index.md (parsing and indexing) └── system-and-utils.md (utilities) ```