### Start Raft Node with gRPC Server Source: https://github.com/databendlabs/openraft/blob/main/examples/raft-kv-memstore-grpc/README.md Launch individual Raft nodes by executing the compiled binary. Each node requires a unique ID and network address. This setup is for starting a cluster with multiple nodes. ```shell ./raft-key-value --id 1 --addr 127.0.0.1:21001 ``` ```shell ./raft-key-value --id 2 --addr 127.0.0.1:21002 ``` ```shell ./raft-key-value --id 3 --addr 127.0.0.1:21003 ``` -------------------------------- ### Sled Store Example Source: https://github.com/databendlabs/openraft/blob/main/change-log.md An example demonstrating the use of the sled key-value store with Openraft, adapted from the rocksdb example. ```rust Added sled store example based on rocks example ``` -------------------------------- ### Handle Incoming Snapshot Request with Chunked Receiver Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/upgrade_guide/upgrade-v08-v09.md Example of handling an `InstallSnapshotRequest` on the receiving end. It uses `Chunked::receive_snapshot` for default chunk-based data reception and then installs the snapshot. ```rust async fn handle_install_snapshot_request( &self, req: InstallSnapshotRequest, ) -> Result<_, _> where C::SnapshotData: AsyncRead + AsyncWrite + AsyncSeek + Unpin, { let my_vote = self.with_raft_state(|state| *state.vote_ref()).await?; let resp = InstallSnapshotResponse { vote: my_vote }; let finished_snapshot = { use crate::network::snapshot_transport::Chunked; use crate::network::snapshot_transport::SnapshotTransport; let mut streaming = self.snapshot.lock().await; Chunked::receive_snapshot(&mut *streaming, self, req).await? }; if let Some(snapshot) = finished_snapshot { let resp = self.install_full_snapshot(req_vote, snapshot).await?; return Ok(resp.into()); } Ok(resp) } ``` -------------------------------- ### Start Raft Key-Value Node Source: https://github.com/databendlabs/openraft/blob/main/examples/raft-kv-memstore/README.md Builds the Raft key-value binary and starts a single node with specified IDs and network addresses. This is a manual step before interacting with the cluster. ```shell ./target/debug/raft-key-value --id 1 --api-addr 127.0.0.1:21001 --raft-addr 127.0.0.1:22001 ``` -------------------------------- ### Log I/O Progress Example Table Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/data/log_io_progress.md Illustrates the progression of log entries through Accepted, Submitted, and Flushed states over time on a leader node. ```text Time | Accepted | Submitted | Flushed -----|----------------|----------------|---------------- t1 | (L1, 1-1) | None | None t2 | (L1, 1-2) | (L1, 1-1) | None t3 | (L1, 1-3) | (L1, 1-2) | (L1, 1-1) t4 | (L1, 1-3) | (L1, 1-3) | (L1, 1-2) t5 | (L1, 1-3) | (L1, 1-3) | (L1, 1-3) ``` -------------------------------- ### LeaderId Partial Order Examples Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/data/leader_id.md Illustrates the partial ordering behavior of LeaderId in standard mode with various combinations of terms and voted_for values. ```ignore LeaderId(3, None) > LeaderId(2, None): true LeaderId(3, None) > LeaderId(2, Some(y)): true LeaderId(3, None) == LeaderId(3, None): true LeaderId(3, Some(x)) > LeaderId(2, Some(y)): true LeaderId(3, Some(x)) > LeaderId(3, None): true LeaderId(3, Some(x)) == LeaderId(3, Some(x)): true LeaderId(3, Some(x)) > LeaderId(3, Some(y)): false ``` -------------------------------- ### Fast Commit Example 1: Membership Change Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/obsolete/fast_commit.md Illustrates a scenario where entry 4 changes membership from 'a' to 'abc', leading to the commit of entries 2 and 3. ```text 1 2 3 4 5 6 a x x a y y b c ``` -------------------------------- ### Type Alias Example for SnapshotData Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/feature_flags/feature-flags.md Demonstrates the usage of type shortcuts provided by the `type-alias` feature flag for `SnapshotData`. This allows for more concise type definitions. ```rust use openraft::alias::SnapshotDataOf; struct MyTypeConfig; impl RaftTypeconfig For MyTypeConfig { /*...*/ } // The following two lines are equivalent: let snapshot_data: SnapshotDataOf; let snapshot_data: ::SnapshotData; ``` -------------------------------- ### Leader Distributing Snapshot Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/faq/04-storage/03-snapshot-synchronization.md Example of a leader node building a snapshot using `trigger().snapshot()` and then distributing the snapshot data to followers via custom RPC calls. ```rust use std::io; use openraft::Raft; // Leader builds and distributes snapshot async fn distribute_snapshot( leader: &Raft, followers: Vec ) -> Result<(), io::Error> { // Build snapshot on leader let snapshot = leader.trigger().snapshot().await? .expect("snapshot built"); // Send to followers via custom RPC for follower_id in followers { let snapshot_data = read_snapshot_file(&snapshot).await?; send_snapshot_rpc(follower_id, snapshot_data).await?; } Ok(()) } ``` -------------------------------- ### Initialize Cluster with Single Node (Incremental) Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/faq/01-getting-started/01-initialize-cluster.md Start by initializing a cluster with only the current node. This is useful for testing or when the cluster will be expanded incrementally. ```rust n1.initialize(btreeset! {1}) ``` -------------------------------- ### Run Raft RPC Server and Application Server Source: https://github.com/databendlabs/openraft/blob/main/examples/network-v2-http/README.md Start the Raft RPC server on the internal Raft address and the application server on the public API address. Both servers run concurrently. ```rust let raft_server = network_v2_http::Server::new(raft.clone()).run(raft_addr); let app_server = run_application_server(api_addr); tokio::try_join!(raft_server, app_server)?; ``` -------------------------------- ### Problematic Timeline Example Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/protocol/commit.md Illustrates a scenario where out-of-order RPC delivery leads to an inconsistent state on a node, highlighting the challenge of distinguishing log entries from different terms. ```text N1 | T1 E11 N2 | T2 E21 N3 | E11 T3 E11 E32 N4 | T1 T2 T3 E11 E32 N5 | T1 T2 T3 E11 E32 ------+------+--+-----+----+-----+--------------------------> time t1 t2 t3 t4 t5 t6 Timeline: t1: L1 replicates E11 to N1 t2: N2 becomes leader L2 (term T2), proposes E21 instead t3: L2 sends Append(T2, E21) to N1 (inflight) t4: N3 becomes leader L3 (term T3) t5: L3 replicates E11 to quorum (N3,N4,N5), E11 is cluster-committed L3 sends Append(T3, E11) and Commit(T3, E32) to N1 (inflight) t6: RPCs arrive at N1 in this order: 1. Commit(T3, E32) arrives → N1 learns E11 is cluster-committed - N1's log: [E11 from T1] - Problem: E11 from T1 ≠ E11 from T3 (different leaders) - Cannot apply yet 2. Append(T2, E21) arrives → truncates E11, appends E21 - N1's log: [E21 from T2] - cluster_committed still points to E11 (inconsistent) 3. Append(T3, E11) arrives → truncates E21, re-appends E11 - N1's log: [E11 from T3] - Now safe to apply ``` -------------------------------- ### Initialize Cluster with All Nodes (Single-Step) Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/faq/01-getting-started/01-initialize-cluster.md Call `Raft::initialize()` on one node with the complete set of node IDs. This method is suitable for setting up a cluster with a known set of members from the start. ```rust n1.initialize(btreeset! {1,2,3}) ``` -------------------------------- ### Example Server Implementation for AppendEntries Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/getting_started/getting-started.md A basic Rust implementation for handling AppendEntries RPCs on the server side. It forwards the request to the local Raft instance and returns the response. ```rust async fn handle_append_entries( raft: Arc>, req: AppendEntriesRequest, ) -> Result { let resp = raft.append_entries(req).await?; Ok(resp) } ``` -------------------------------- ### AppendEntries RPC Handler (Unsafe Implementation) Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/protocol/log_replication.md A pseudocode example demonstrating an unsafe approach to handling AppendEntries RPCs. This method can lead to committed log loss if not implemented carefully. ```rust fn handle_append_entries(req) { if store.has(req.prev_log_id) { store.delete_logs(req.prev_log_id.index..) store.append_logs(req.entries) } } ``` -------------------------------- ### Fast Commit Example 2: Membership Change Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/obsolete/fast_commit.md Illustrates a scenario where entry 4 changes membership from 'abc' to 'a', leading to the commit of entries 2, 3, 4, 5, and 6. ```text 1 2 3 4 5 6 a x x a y y b c ``` -------------------------------- ### Handle Snapshot Installation with Purged Logs Source: https://github.com/databendlabs/openraft/blob/main/change-log.md Fixes an issue where handling append-entries with a purged prev_log_id could incorrectly delete logs. Uses `last_applied` instead of `committed` to ensure correctness after restarts. ```rust when handling append-entries, if prev_log_id is purged, it should not delete any logs.; by 张炎泼; 2022-08-14 When handling append-entries, if the local log at `prev_log_id.index` is purged, a follower should not believe it is a **conflict** and should not delete all logs. It will get committed log lost. To fix this issue, use `last_applied` instead of `committed`: `last_applied` is always the committed log id, while `committed` is not persisted and may be smaller than the actually applied, when a follower is restarted. ``` -------------------------------- ### Build FAQ Documentation Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/faq/README.md Run the 'make' command to generate the consolidated 'faq.md' and 'faq-toc.md' files from individual FAQ entries. ```bash make ``` -------------------------------- ### Initialize Application HTTP Server with Openraft Routes Source: https://github.com/databendlabs/openraft/blob/main/examples/app-http/README.md Sets up an application HTTP server, adding Openraft's specific routes and custom read routes. The server is then run on the specified API address. ```rust let app_server = app_http::Server::new(app) .add_openraft_routes() .post("/read", api::read) .run(api_addr); ``` -------------------------------- ### Initialize and Use HTTP Client Source: https://github.com/databendlabs/openraft/blob/main/examples/app-http/README.md Demonstrates how to create, initialize, write, and read data using the application-specific HTTP client. This client is designed for key-value examples and updates its target when an OpenRaft error indicates a known leader. ```rust let client = app_http::Client::new(1, "127.0.0.1:21001".to_string()); client.init().await??; client.write(&request).await??; let value = client.read(&"foo".to_string()).await?; ``` -------------------------------- ### Handle Snapshot Installation Conflicts Source: https://github.com/databendlabs/openraft/blob/main/change-log.md Fixes potential inconsistencies when installing a snapshot by ensuring conflicting logs before the snapshot's last log ID are deleted. This prevents data loss if a node crashes after snapshot installation. ```rust potential inconsistency when installing snapshot; by 张炎泼; 2022-09-22 The conflicting logs that are before `snapshot_meta.last_log_id` should be deleted before installing a snapshot. Otherwise there is chance the snapshot is installed but conflicting logs are left in the store, when a node crashes. ``` -------------------------------- ### Install Snapshot Response Logic Source: https://github.com/databendlabs/openraft/blob/main/change-log.md Ensures that a follower does not install a smaller snapshot by replying with its last applied log ID if it already has a higher one. This prevents the applied state from reverting. ```rust InstallSnapshotResponse: replies the last applied log id; Do not install a smaller snapshot; by 张炎泼; 2022-09-22 A snapshot may not be installed by a follower if it already has a higher `last_applied` log id locally. In such a case, it just ignores the snapshot and respond with its local `last_applied` log id. This way the applied state(i.e., `last_applied`) will never revert back. ``` -------------------------------- ### Fix Race Condition in Snapshot Installation Source: https://github.com/databendlabs/openraft/blob/main/change-log.md Addresses a race condition between concurrent snapshot installation and state machine application that could lead to incorrect `last_applied` values. The solution serializes state machine modifications. ```text RaftCore: -. install-snapshot, .-> replicate_to_sm_handle.next(), | update last_applied=5 | | | v | task: apply 2------------------------' --------------------------------------------------------------------> time ``` -------------------------------- ### Build Change Log Script Source: https://github.com/databendlabs/openraft/blob/main/scripts/README.md Generates a versioned change log from git commit messages. It can build the log since the last git tag or a specified commit. Ensure Python dependencies are installed via 'pip install -r requirements.txt'. ```bash ./scripts/build_change_log.py ``` ```bash ./scripts/build_change_log.py ``` -------------------------------- ### Fix Race Condition in Snapshot Installation and Apply Source: https://github.com/databendlabs/openraft/blob/main/change-log/v0.6.2.md Addresses a race condition where concurrent snapshot installation and state machine application could lead to incorrect `last_applied` values. This fix serializes state machine modifications to prevent data corruption. ```text RaftCore: -. install-snapshot, .-> replicate_to_sm_handle.next(), | update last_applied=5 | update last_applied=2 | | v | task: apply 2------------------------' --------------------------------------------------------------------> time ``` -------------------------------- ### Run OpenRaft Benchmark with Default Settings Source: https://github.com/databendlabs/openraft/blob/main/benchmarks/minimal/README.md Execute the benchmark with its default configuration. This is useful for a quick performance baseline. ```sh cargo run --release --bin bench ``` -------------------------------- ### Register Leader Change Callbacks in OpenRaft Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/faq/05-monitoring/05-leader-change-services.md Use `Raft::on_leader_change()` to register start and stop callbacks. The start callback is invoked when the node becomes the leader, and the stop callback is invoked when it is no longer the leader. Ensure your callbacks are idempotent and handle asynchronous operations correctly. ```rust let service_handle = Arc::new(Mutex::new(None)); let service_handle_clone = service_handle.clone(); let mut watch_handle = raft.on_leader_change( // start: called when this node becomes leader move |_leader_id| { let service_handle = service_handle_clone.clone(); async move { let mut handle = service_handle.lock().unwrap(); if handle.is_none() { *handle = Some(start_cron_service().await); } } }, // stop: called when this node is no longer leader move |_old_leader_id| { let service_handle = service_handle.clone(); async move { let mut handle = service_handle.lock().unwrap(); if let Some(h) = handle.take() { h.shutdown().await; } } }, ); // Later, when shutting down: watch_handle.close().await; ``` -------------------------------- ### Project Structure for FAQs Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/faq/README.md Organize FAQ content into numbered category directories. Each category contains a README.md for the section title and individual FAQ entries as NN-*.md files. ```markdown NN-category-name/ ├── README.md # Section name (e.g., "Getting Started") └── NN-*.md # FAQ entries ``` -------------------------------- ### Finalize Snapshot Installation Refactor Source: https://github.com/databendlabs/openraft/blob/main/change-log/v0.6.2-alpha.5.md Highlights the removal of the redundant 'delete_through' parameter from the finalize_snapshot_installation function. ```rust Refactor: remove redundant param `delete_through` from `finalize_snapshot_installation`. ``` -------------------------------- ### Add FAQ Entry Example Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/faq/README.md To add a new FAQ entry, create a markdown file (e.g., NN-my-faq.md) within the appropriate category directory. Include a '###' header for the title, content with link references, and link definitions at the bottom. ```markdown ### My FAQ Title Content with [`SomeType`][] references. [`SomeType`]: `crate::path::SomeType` ``` -------------------------------- ### Build RaftStorage Instance with Adapter Source: https://github.com/databendlabs/openraft/blob/main/change-log/v0.8.4.md Use an adapter to wrap `RaftStorage` when separating log storage and state machine. This is the new pattern for initializing Raft with distinct log and state machine components. ```rust // Before: let store = MyRaftStorage::new(); Raft::new(..., store); // After: let store = MyRaftStorage::new(); let (log_store, sm) = Adaptoer::new(store); Raft::new(..., log_store, sm); ``` -------------------------------- ### Concrete Raft Type Implementation Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/getting_started/getting-started.md An example of a concrete implementation of `RaftTypeConfig` for `TypeConfig`, showing default and custom type definitions. ```rust pub struct TypeConfig {} impl openraft::RaftTypeConfig for TypeConfig { type D = Request; type R = Response; // Following are absent in `declare_raft_types` and filled with default values: type NodeId = u64; type Node = openraft::impls::BasicNode; type Entry = openraft::impls::Entry; type Responder = openraft::impls::OneshotResponder where T: openraft::OptionalSend + 'static; type AsyncRuntime = openraft::impls::TokioRuntime; type SnapshotData = Cursor>; } ``` -------------------------------- ### Pull Request Template in Openraft Source: https://github.com/databendlabs/openraft/blob/main/change-log.md Includes a pull request template to standardize contributions and guide developers submitting changes to the project. ```markdown a pull request template. ``` -------------------------------- ### Create RocksDB Persistent Store Source: https://github.com/databendlabs/openraft/blob/main/examples/rocksstore/README.md Instantiate a new RocksStore for persistent storage. Ensure the provided path is valid for RocksDB. ```rust use openraft_rocksstore::RocksStore; // Create a persistent store let store = RocksStore::new(path)?; ``` -------------------------------- ### Define Key-Value Store Request Source: https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/getting_started/getting-started.md Define the structure for client requests to modify the Raft state machine. This example shows a simple key-based request. ```rust #[derive(Clone, Debug, Serialize, Deserialize)] pub struct Request {key: String} ``` -------------------------------- ### Build the Project with Cargo Source: https://github.com/databendlabs/openraft/blob/main/examples/raft-kv-memstore-grpc/README.md Use `cargo build` to compile the project. This command is essential for preparing the executable for running the distributed key-value store. ```shell cargo build ``` -------------------------------- ### Implement GroupRouter and Use GroupNetworkAdapter Source: https://github.com/databendlabs/openraft/blob/main/multiraft/README.md Implement the `GroupRouter` trait on your shared router or connection pool. Then, use `GroupNetworkAdapter` to automatically get a `RaftNetworkV2` implementation. ```rust use openraft_multiraft::{GroupRouter, GroupNetworkAdapter, GroupNetworkFactory}; // Your Router implements GroupRouter impl GroupRouter for Router { // ... } // Create factory for a specific group let factory = GroupNetworkFactory::new(router, group_id); // Factory creates adapters that implement RaftNetworkV2 impl RaftNetworkFactory for NetworkFactory { type Network = GroupNetworkAdapter; async fn new_client(&mut self, target: NodeId, _node: &Node) -> Self::Network { GroupNetworkAdapter::new(self.factory.clone(), target, self.group_id.clone()) } } ``` -------------------------------- ### Run Test Cluster Script Source: https://github.com/databendlabs/openraft/blob/main/examples/raft-kv-memstore/README.md Executes a script to start a three-node Raft cluster and drive it with raw HTTP requests. Useful for observing the HTTP exchange. ```shell ./test-cluster.sh ```