### Next Method Example Implementation Pattern Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/broadcaster-interface.md An example pattern for implementing the Next method, demonstrating how to handle context cancellation and channel reception. ```go func (b *MyBroadcaster) Next(ctx context.Context) ([]byte, error) { select { case <-ctx.Done(): return nil, crdt.ErrNoMoreBroadcast case msg := <-b.receiveChan: if msg == nil { return nil, crdt.ErrNoMoreBroadcast } return msg.Data, nil } } ``` -------------------------------- ### Configuration Options Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Shows how to get default options and create a PubSubBroadcaster. ```go // Configuration opts := crdt.DefaultOptions() broadcaster, err := crdt.NewPubSubBroadcaster(ctx, ps, topic) ``` -------------------------------- ### Broadcast Method Example Implementation Pattern Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/broadcaster-interface.md An example pattern for implementing the Broadcast method, showing how to handle context cancellation and network publishing. ```go func (b *MyBroadcaster) Broadcast(ctx context.Context, data []byte) error { select { case <-ctx.Done(): return ctx.Err() default: } // Send to network err := b.publishToNetwork(data) if err != nil { return fmt.Errorf("broadcast failed: %w", err) } return nil } ``` -------------------------------- ### Development/Testing Configuration Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/README.md Example configuration for development or testing environments. Uses default options. ```go opts := crdt.DefaultOptions() ``` -------------------------------- ### Minimal PubSub Setup Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/pubsub-broadcaster.md Sets up a libp2p host and GossipSub for basic PubSub communication. This is the foundational step before creating a broadcaster. ```go package main import ( "context" "github.com/libp2p/go-libp2p" pubsub "github.com/libp2p/go-libp2p-pubsub" crdt "github.com/ipfs/go-ds-crdt" ) func setupNetwork(ctx context.Context) (*pubsub.PubSub, error) { // Create libp2p host h, err := libp2p.New() if err != nil { return nil, err } // Create GossipSub ps, err := pubsub.NewGossipSub(ctx, h) if err != nil { return nil, err } return ps, nil } func main() { ctx := context.Background() ps, _ := setupNetwork(ctx) broadcaster, _ := crdt.NewPubSubBroadcaster(ctx, ps, "crdt-demo") // Use broadcaster with CRDT datastore... } ``` -------------------------------- ### Instantiate PubSubBroadcaster Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/broadcaster-interface.md Example of how to create a new PubSubBroadcaster instance using libp2p GossipSub. ```go broadcaster, err := crdt.NewPubSubBroadcaster(ctx, pubsub, "topic-name") ``` -------------------------------- ### Custom Delta Implementation Example Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/delta-interface.md An example of a custom Delta implementation allowing for alternative serialization formats. It requires implementing methods like GetElements, GetTombstones, Marshal, and Unmarshal. ```go type CustomDelta struct { elements []*pb.Element tombstones []*pb.Element priority uint64 dagName string } func (d *CustomDelta) GetElements() ([]*pb.Element, error) { return d.elements, nil } func (d *CustomDelta) GetTombstones() ([]*pb.Element, error) { return d.tombstones, nil } // ... implement other methods ... func (d *CustomDelta) Marshal() ([]byte, error) { // Custom serialization logic return customSerialize(d) } func (d *CustomDelta) Unmarshal(data []byte) error { // Custom deserialization logic return customDeserialize(data, d) } ``` ```go internalOpts := &crdt.MerkleCRDTOptions{ DeltaFactory: func() crdt.Delta { return &CustomDelta{} }, } mcrdt, _ := crdt.NewMerkleCRDT(..., internalOpts) ``` -------------------------------- ### Handling Key-Value Operation Errors Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/errors.md This example illustrates how to handle errors from key-value operations like Get(). It specifically checks for ds.ErrNotFound and logs other potential errors. ```Go val, err := crdtStore.Get(ctx, ds.NewKey("mykey")) if err == ds.ErrNotFound { log.Info("Key does not exist") } else if err != nil { log.Errorf("Get failed: %v", err) } ``` -------------------------------- ### Build the GlobalDB CLI Source: https://github.com/ipfs/go-ds-crdt/blob/master/examples/globaldb/README.md Clone the repository and build the GlobalDB CLI binary using Go. Ensure Go is installed and configured. ```bash git clone https://github.com/ipfs/go-ds-crdt cd examples/globaldb go build -o globaldb ``` -------------------------------- ### Interact with GlobalDB: Put and Get Source: https://github.com/ipfs/go-ds-crdt/blob/master/examples/globaldb/README.md Use the 'put' command to store a key-value pair and 'get' to retrieve it. Shows basic data manipulation. ```plaintext > put exampleKey exampleValue > get exampleKey [exampleKey] -> exampleValue ``` -------------------------------- ### Run GlobalDB CLI with Data Directory Source: https://github.com/ipfs/go-ds-crdt/blob/master/examples/globaldb/README.md Start the GlobalDB CLI, specifying a directory for local database and key storage. This is a basic startup command. ```bash ./globaldb -datadir /path/to/data ``` -------------------------------- ### Custom Broadcaster Implementation Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/broadcaster-interface.md A template for implementing a custom Broadcaster, demonstrating the Broadcast, Next, and Close methods. This example uses channels for internal communication. ```go package main import ( "context" "fmt" crdt "github.com/ipfs/go-ds-crdt" ) // CustomBroadcaster implements crdt.Broadcaster using custom logic type CustomBroadcaster struct { sendCh chan []byte receiveCh chan []byte closeCh chan struct{} // ... other fields (connections, buffers, etc.) } // Broadcast sends data to other replicas func (cb *CustomBroadcaster) Broadcast(ctx context.Context, data []byte) error { select { case <-ctx.Done(): return ctx.Err() case <-cb.closeCh: return fmt.Errorf("broadcaster closed") case cb.sendCh <- data: return nil } } // Next blocks until data is received from peers func (cb *CustomBroadcaster) Next(ctx context.Context) ([]byte, error) { select { case <-ctx.Done(): return nil, crdt.ErrNoMoreBroadcast case <-cb.closeCh: return nil, crdt.ErrNoMoreBroadcast case data := <-cb.receiveCh: if data == nil { return nil, crdt.ErrNoMoreBroadcast } return data, nil } } // Close shuts down the broadcaster func (cb *CustomBroadcaster) Close() error { close(cb.closeCh) return nil } // Usage with CRDT func example() { broadcaster := &CustomBroadcaster{ sendCh: make(chan []byte, 100), receiveCh: make(chan []byte, 100), closeCh: make(chan struct{}), } ctx := context.Background() store, _ := crdt.New( underlyingStore, namespace, dagService, broadcaster, // Pass custom broadcaster nil, ) defer store.Close() // ... use store ... } ``` -------------------------------- ### Production Configuration for go-ds-crdt Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/configuration.md Example of a comprehensive production configuration for go-ds-crdt, including logging, rebroadcasting, hooks, and throughput tuning. ```go package main import ( time "time" crdt "github.com/ipfs/go-ds-crdt" "github.com/ipfs/go-log/v2" ) func createProductionOptions() *crdt.Options { opts := crdt.DefaultOptions() // Configure logging opts.Logger = logging.Logger("myapp.datastore") // Periodic rebroadcast for reliable propagation opts.RebroadcastInterval = 2 * time.Minute // Hook for updating application indices opts.PutHook = func(k ds.Key, v []byte) { indexUpdateChan <- IndexUpdate{Key: k.String(), Value: v} } // Hook for removing from cache opts.DeleteHook = func(k ds.Key) { cacheInvalidateChan <- k.String() } // Tune for expected throughput opts.NumWorkers = 8 opts.DAGSyncerTimeout = 3 * time.Minute opts.MaxBatchDeltaSize = 2 * 1024 * 1024 // 2MB // Recovery opts.RepairInterval = 30 * time.Minute // Throughput optimization opts.MultiHeadProcessing = true opts.BroadcastBatchDelay = 500 * time.Millisecond return opts } ``` -------------------------------- ### CRDT Receive Loop Example Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/broadcaster-interface.md Illustrates how the CRDT Datastore uses the Next method in a loop to process incoming updates. ```go for { data, err := broadcaster.Next(ctx) if err == crdt.ErrNoMoreBroadcast { return // Shutdown complete } if err != nil { logger.Error(err) continue // Transient error, try again } // Process received data (heads broadcast from peer) receivedHeads, err := store.decodeBroadcast(ctx, data) // ... handle update ... } ``` -------------------------------- ### Error Handling Example Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/README.md Demonstrates how to handle errors returned by public methods. Includes checks for specific error types like ErrNoMoreBroadcast, context.Canceled, and ds.ErrNotFound. ```go if err != nil { // Handle based on error type if err == crdt.ErrNoMoreBroadcast { // Broadcaster shutdown } else if err == context.Canceled { // Cancellation } else if err == ds.ErrNotFound { // Key not found } else { // Other errors log.Errorf("Operation failed: %v", err) } } ``` -------------------------------- ### Standard Datastore Usage Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/README.md Demonstrates the basic usage of the go-ds-crdt Datastore, including creation, putting, getting, and deleting key-value pairs. Ensure to close the store when done. ```go store, _ := crdt.New(...) defer store.Close() store.Put(ctx, key, value) value, _ := store.Get(ctx, key) store.Delete(ctx, key) ``` -------------------------------- ### Basic Key-Value Store Workflow Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Demonstrates setting up and using the CRDT datastore as a basic key-value store. This includes setting up underlying storage, IPFS for block storage, pubsub for distribution, and then performing basic Put and Get operations. ```go package main import ( "context" "log" ds "github.com/ipfs/go-datastore" crdt "github.com/ipfs/go-ds-crdt" pebble "github.com/ipfs/go-ds-pebble" pubsub "github.com/libp2p/go-libp2p-pubsub" ipfslite "github.com/hsanjuan/ipfs-lite" ) func main() { ctx := context.Background() // 1. Setup underlying storage pebbleStore, _ := pebble.NewDatastore("./data") defer pebbleStore.Close() // 2. Setup IPFS for block storage h, dht, _ := ipfslite.SetupLibp2p(ctx, ...) defer h.Close() lite, _ := ipfslite.New(ctx, pebbleStore, h, dht, ...) // 3. Setup pubsub for distribution ps, _ := pubsub.NewGossipSub(ctx, h) broadcaster, _ := crdt.NewPubSubBroadcaster(ctx, ps, "my-topic") // 4. Create CRDT datastore store, _ := crdt.New( pebbleStore, ds.NewKey("/crdt"), lite, broadcaster, crdt.DefaultOptions(), ) defer store.Close() // 5. Use like a normal datastore store.Put(ctx, ds.NewKey("greeting"), []byte("hello")) val, _ := store.Get(ctx, ds.NewKey("greeting")) log.Printf("Retrieved: %s", string(val)) } ``` -------------------------------- ### CRDT Shutdown Sequence Example Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/broadcaster-interface.md Demonstrates the correct order for shutting down the CRDT and its broadcaster. The broadcaster should be signaled to shut down before the main datastore. ```go ctx, cancel := context.WithCancel(context.Background()) broadcaster, _ := crdt.NewPubSubBroadcaster(ctx, ps, "topic") store, _ := crdt.New(ds, ns, dag, broadcaster, opts) // ... use store ... // Shutdown: broadcaster first, then datastore cancel() // Signal broadcaster to shutdown time.Sleep(100 * time.Millisecond) // Allow graceful shutdown store.Close() // Shutdown datastore ``` -------------------------------- ### Example PubSubBroadcaster Topic Names Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/pubsub-broadcaster.md Illustrates various naming conventions for pubsub topics to uniquely identify CRDT instances, avoid conflicts, and aid debugging. Versioning is recommended for migration scenarios. ```text "crdt-events" "crdt-app-v1" "distributed-db/replica-1" "blockchain-state-mainnet" ``` -------------------------------- ### Custom Broadcaster Test Example Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/broadcaster-interface.md A unit test for a custom Broadcaster implementation. It verifies the Broadcast and Next methods, as well as the shutdown behavior by checking for ErrNoMoreBroadcast. ```go func TestCustomBroadcaster(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() b := &CustomBroadcaster{} // Test Broadcast err := b.Broadcast(ctx, []byte("test")) if err != nil { t.Fatal(err) } // Test Next (should receive what was broadcast) data, err := b.Next(ctx) if err != nil { t.Fatal(err) } if string(data) != "test" { t.Fatalf("got %s, want test", string(data)) } // Test shutdown signal cancel() _, err = b.Next(context.Background()) if err != crdt.ErrNoMoreBroadcast { t.Fatalf("expected ErrNoMoreBroadcast, got %v", err) } } ``` -------------------------------- ### Get Operation Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/datastore.md Retrieves the value associated with a given key from the Datastore. If the key does not exist, it returns an error. ```APIDOC ## Get ### Description Retrieves the value associated with a given key from the Datastore. If the key does not exist, it returns an error. ### Method GET (conceptual, as this is an SDK method) ### Endpoint N/A (SDK method) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters Table | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | ctx | context.Context | Yes | Context for cancellation. | | key | ds.Key | Yes | The key to retrieve. | ### Return Type `([]byte, error)` Returns the value associated with the key. Returns an error (typically from the underlying datastore) if the key does not exist. ### Example ```go val, err := crdtStore.Get(ctx, ds.NewKey("mykey")) if err != nil { // Key not found or other error log.Fatal(err) } fmt.Println(string(val)) ``` ``` -------------------------------- ### Handling New() Constructor Errors Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/errors.md This example shows how to check for errors when creating a new CRDT datastore instance using the New() constructor. It logs a fatal error if the constructor fails. ```Go store, err := crdt.New(ds, ns, dag, broadcaster, opts) if err != nil { log.Fatalf("Failed to create CRDT datastore: %v", err) } ``` -------------------------------- ### Size Method Usage Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/delta-interface.md Shows how to get the serialized size of a Delta object in bytes. This is useful for determining when to auto-commit batches based on size limits. ```go delta := ... // build delta if delta.Size() > 1024*1024 { // > 1MB fmt.Println("Delta is large") } ``` -------------------------------- ### Diagnostics Workflow Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Provides examples of diagnostic operations for the CRDT store, including checking for dirty state, repairing the store, retrieving internal statistics, and exporting the DAG for visualization. ```go // Check store health if store.IsDirty(ctx) { log.Warn("Store requires repair") err := store.Repair(ctx) if err != nil { log.Fatal(err) } } // Get internal stats stats := store.InternalStats(ctx) log.Printf("Heads: %d, Max Height: %d", len(stats.Heads), stats.MaxHeight) // Visualize DAG (small DAGs only) store.PrintDAG(ctx) // Export for graphviz (large DAGs) f, _ := os.Create("dag.dot") store.DotDAG(ctx, f) f.Close() ``` -------------------------------- ### Registering Topic Validators Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/pubsub-broadcaster.md Register validators before creating the broadcaster to reject malformed messages. This example shows how to reject messages with empty data or exceeding a 10MB size limit. ```go ctx := context.Background() ps, _ := pubsub.NewGossipSub(ctx, h) // Register validator to reject malformed broadcasts ps.RegisterTopicValidator("my-topic", func(ctx context.Context, peer peer.ID, msg *pubsub.Message) pubsub.ValidationResult { // Validate message format if len(msg.Data) == 0 { return pubsub.ValidationReject } // Check message size if len(msg.Data) > 10*1024*1024 { // Max 10MB return pubsub.ValidationReject } return pubsub.ValidationAccept }) // Create broadcaster after validators are registered broadcaster, _ := crdt.NewPubSubBroadcaster(ctx, ps, "my-topic") ``` -------------------------------- ### Run GlobalDB CLI in Daemon Mode Source: https://github.com/ipfs/go-ds-crdt/blob/master/examples/globaldb/README.md Start the GlobalDB CLI in daemon mode for continuous operation, specifying the data directory. The CLI will periodically report network status. ```bash ./globaldb -daemon -datadir /path/to/data ``` -------------------------------- ### GetPriority and SetPriority Usage Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/delta-interface.md Shows how to get and set the priority for conflict resolution on a Delta object. Priority is a uint64 value where higher numbers indicate higher precedence. ```go // Get priority p := delta.GetPriority() fmt.Printf("Delta priority: %d\n", p) // Set priority (usually automatic) delta.SetPriority(42) ``` -------------------------------- ### Get Default CRDT Options Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/configuration.md Returns an Options struct with sensible defaults suitable for most use cases. This is the starting point for custom configurations. ```go func DefaultOptions() *Options { return &Options{ Logger: logging.Logger("crdt"), RebroadcastInterval: 1 * time.Minute, PutHook: nil, DeleteHook: nil, NumWorkers: 5, DAGSyncerTimeout: 5 * time.Minute, MaxBatchDeltaSize: 1 * 1024 * 1024, // 1 MB RepairInterval: 1 * time.Hour, MultiHeadProcessing: false, BroadcastBatchDelay: 0, // internal crdtOpts: MerkleCRDTOptions{ DeltaFactory: func() Delta { return &pbDelta{Delta: &pb.Delta{}} }, Namespaces: InternalNamespaces{ Heads: "h", DAGHeads: "a", Set: "s", ProcessedBlocks: "b", DirtyBitKey: "d", BadShutdownKey: "bs", VersionKey: "crdt_version", }, }, } } ``` -------------------------------- ### Initialize CRDT Datastore Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/datastore.md Demonstrates how to set up and initialize the CRDT Datastore with underlying storage, DAG service, and a broadcaster. Ensure all components are properly configured before creating the Datastore. Remember to close the Datastore and its dependencies when done. ```go package main import ( "context" ds "github.com/ipfs/go-datastore" crdt "github.com/ipfs/go-ds-crdt" pebble "github.com/ipfs/go-ds-pebble" pubsub "github.com/libp2p/go-libp2p-pubsub" ipfslite "github.com/hsanjuan/ipfs-lite" ) func main() { ctx := context.Background() // Setup underlying storage store, _ := pebble.NewDatastore("./data") defer store.Close() // Setup IPFS-Lite for DAG operations h, dht, _ := ipfslite.SetupLibp2p(ctx, ...) defer h.Close() lite, _ := ipfslite.New(ctx, store, h, dht, ...) // Setup pubsub broadcaster ps, _ := pubsub.NewGossipSub(ctx, h) broadcaster, _ := crdt.NewPubSubBroadcaster(ctx, ps, "my-topic") // Create CRDT datastore crdtStore, _ := crdt.New( store, ds.NewKey("/crdt"), lite, broadcaster, crdt.DefaultOptions(), ) defer crdtStore.Close() // Use like a standard datastore crdtStore.Put(ctx, ds.NewKey("key1"), []byte("value1")) val, _ := crdtStore.Get(ctx, ds.NewKey("key1")) } ``` -------------------------------- ### Traverse Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/merkle-crdt.md Performs a depth-first traversal of the Merkle-DAG starting from specified CIDs. ```APIDOC ## Traverse ### Description Performs a depth-first traversal of the Merkle-DAG starting from the given CIDs. The visit callback is called with each IPLD node, allowing inspection of deltas and structure. Skips duplicate nodes and automatically handles multiple roots by creating a temporary synthetic root. ### Method `Traverse(ctx context.Context, from []cid.Cid, visit func(ipld.Node) error) error` ### Parameters #### Path Parameters - **ctx** (context.Context) - Required - Context for cancellation. - **from** ([]cid.Cid) - Required - Starting CIDs to traverse from (typically heads). Must not be empty. - **visit** (func(ipld.Node) error) - Required - Callback function invoked for each node. Return error to abort. ### Response #### Success Response - **error**: An error if the operation fails. ### Example ```go heads, _, _ := mcrdt.Heads().List(ctx) headCids := make([]cid.Cid, len(heads)) for i, h := range heads { headCids[i] = h.Cid } err := mcrdt.Traverse(ctx, headCids, func(node ipld.Node) error { fmt.Printf("Visiting node: %s\n", node.Cid()) // Extract delta from node protoNode := node.(*dag.ProtoNode) deltaBytes := protoNode.Data() // Process delta delta := mcrdt.newDelta() if err := delta.Unmarshal(deltaBytes); err != nil { return err } fmt.Printf(" Priority: %d\n", delta.GetPriority()) return nil }) if err != nil { log.Fatal(err) } ``` ``` -------------------------------- ### Create a new PubSubBroadcaster Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/pubsub-broadcaster.md Initializes a new PubSubBroadcaster. Requires an active context, a libp2p GossipSub instance, and a topic name. Cancelling the context will shut down the broadcaster. ```go package main import ( "context" crdt "github.com/ipfs/go-ds-crdt" pubsub "github.com/libp2p/go-libp2p-pubsub" "github.com/libp2p/go-libp2p" ) func main() { ctx := context.Background() // Create a libp2p host h, _ := libp2p.New() defer h.Close() // Create GossipSub ps, _ := pubsub.NewGossipSub(ctx, h) // Create broadcaster on topic "crdt-updates" broadcaster, _ := crdt.NewPubSubBroadcaster(ctx, ps, "crdt-updates") defer func() { // Shutdown: cancel the context cancel := ctx.Done() <-cancel // wait for completion }() // Use broadcaster with CRDT datastore // store, _ := crdt.New(..., broadcaster, ...) } ``` -------------------------------- ### Has Operation Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/datastore.md Checks if a key exists in the Datastore without retrieving its value. This is more efficient than `Get` when only existence needs to be verified. ```APIDOC ## Has ### Description Checks if a key exists in the Datastore without retrieving its value. This is more efficient than `Get` when only existence needs to be verified. ### Method GET (conceptual, as this is an SDK method) ### Endpoint N/A (SDK method) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters Table | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | ctx | context.Context | Yes | Context for cancellation. | | key | ds.Key | Yes | The key to check. | ### Return Type `(bool, error)` Returns whether the key exists in the datastore without retrieving the value. More efficient than Get when only checking existence. ### Example ```go exists, err := crdtStore.Has(ctx, ds.NewKey("mykey")) if exists { fmt.Println("Key found") } ``` ``` -------------------------------- ### GetDagName and SetDagName Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/delta-interface.md Gets or sets the DAG name for multi-DAG scenarios. When empty string, refers to the default DAG. ```APIDOC ## GetDagName and SetDagName ### Description Gets or sets the DAG name for multi-DAG scenarios. When empty string, refers to the default DAG. ### Method - `func (d Delta) GetDagName() string` - `func (d Delta) SetDagName(name string)` ### Parameters - `SetDagName`: `name string` - The name of the DAG. ### Return Value - `GetDagName`: `string` - The name of the DAG. ### Usage - Organize data into separate DAG branches per name - Enable selective purging: `PurgeDAG(ctx, "dagName")` - Useful for data sharding or multi-tenant scenarios ### Example ```go // Set DAG name before publishing delta := mcrdt.newDelta() delta.SetDagName("archive-2024") delta.SetElements([]*pb.Element{ {Key: "item1", Value: []byte("archive data")}, }) head, _ := mcrdt.Publish(ctx, delta) // Later, purge entire archive DAG removed, _ := mcrdt.PurgeDAG(ctx, "archive-2024") ``` ### Default Empty string ("") ``` -------------------------------- ### New Datastore Constructor Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/datastore.md Initializes a new Datastore instance. This is the main entry point for creating a distributed, replicated key-value store. It requires an underlying datastore, a namespace, a DAG service, a broadcaster, and optional configuration options. ```APIDOC ## New ### Description Initializes a new Datastore instance. This is the main entry point for creating a distributed, replicated key-value store. It requires an underlying datastore, a namespace, a DAG service, a broadcaster, and optional configuration options. ### Function Signature ```go func New( store ds.Datastore, namespace ds.Key, dagSyncer ipld.DAGService, bcast Broadcaster, opts *Options, ) (*Datastore, error) ``` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters Table | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | store | ds.Datastore | Yes | — | Underlying datastore for persistent storage. Must be thread-safe. | | namespace | ds.Key | Yes | — | Key prefix under which all CRDT-Datastore contents are stored. | | dagSyncer | ipld.DAGService | Yes | — | DAG service for publishing and retrieving IPLD nodes (deltas). | | bcast | Broadcaster | Yes | — | Broadcaster for distributing updates to peers. | | opts | *Options | No | DefaultOptions() | Configuration options. If nil, defaults are used. | ### Return Type `(*Datastore, error)` Returns an initialized Datastore or an error if initialization fails. The returned Datastore spawns internal worker goroutines that must be shut down with Close(). ### Example ```go package main import ( "context" ds "github.com/ipfs/go-datastore" crdt "github.com/ipfs/go-ds-crdt" pebble "github.com/ipfs/go-ds-pebble" pubsub "github.com/libp2p/go-libp2p-pubsub" ipfslite "github.com/hsanjuan/ipfs-lite" ) func main() { ctx := context.Background() // Setup underlying storage store, _ := pebble.NewDatastore("./data") defer store.Close() // Setup IPFS-Lite for DAG operations h, dht, _ := ipfslite.SetupLibp2p(ctx, ...) defer h.Close() lite, _ := ipfslite.New(ctx, store, h, dht, ...) // Setup pubsub broadcaster ps, _ := pubsub.NewGossipSub(ctx, h) broadcaster, _ := crdt.NewPubSubBroadcaster(ctx, ps, "my-topic") // Create CRDT datastore crdtStore, _ := crdt.New( store, ds.NewKey("/crdt"), lite, broadcaster, crdt.DefaultOptions(), ) defer crdtStore.Close() // Use like a standard datastore crdtStore.Put(ctx, ds.NewKey("key1"), []byte("value1")) val, _ := crdtStore.Get(ctx, ds.NewKey("key1")) } ``` ``` -------------------------------- ### Organize Data into Multi-DAGs Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/merkle-crdt.md Demonstrates creating and publishing separate DAGs for different data types (users and posts). Includes functionality to purge a specific DAG and verify the existence of elements from other DAGs. ```go // Create deltas for different DAGs usersDelta := mcrdt.newDelta() usersDelta.SetDagName("users") usersDelta.SetElements([]*pb.Element{ {Key: "user:1", Value: []byte("Alice")}, {Key: "user:2", Value: []byte("Bob")}, }) postsDelta := mcrdt.newDelta() postsDelta.SetDagName("posts") postsDelta.SetElements([]*pb.Element{ {Key: "post:101", Value: []byte("Hello World")}, }) // Publish separately mcrdt.Publish(ctx, usersDelta) mcrdt.Publish(ctx, postsDelta) // Later, purge posts if needed removed, _ := mcrdt.PurgeDAG(ctx, "posts") fmt.Printf("Removed %d nodes from posts DAG\n", removed) // But users remain exists, _ := mcrdt.Has(ctx, ds.NewKey("user:1")) fmt.Printf("User still exists: %v\n", exists) ``` -------------------------------- ### Datastore Creation Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Illustrates how to create a new CRDT datastore instance. ```go // Datastore creation store, err := crdt.New(ds, ns, dag, bc, opts) mcrdt, err := crdt.NewMerkleCRDT(ds, ns, dag, bc, opts, intOpts) ``` -------------------------------- ### GetPriority and SetPriority Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/delta-interface.md Gets or sets the priority/height for conflict resolution. When multiple deltas update the same key, the one with higher priority wins. ```APIDOC ## GetPriority and SetPriority ### Description Gets or sets the priority/height for conflict resolution. When multiple deltas update the same key, the one with higher priority wins. If priorities are equal, IDs are compared alphabetically. ### Method - `func (d Delta) GetPriority() uint64` - `func (d Delta) SetPriority(p uint64)` ### Parameters - `SetPriority`: `p uint64` - Priority value (higher wins in conflicts) ### Return Value - `GetPriority`: `uint64` - The current priority value. ### Semantics - **Priority = 0:** Used internally for invalid/uninitialized deltas - **Priority = 1+:** Valid deltas; higher values take precedence - **CRDT Semantics:** Priority equals the height of the node in the DAG tree (distance from leaves) ### Usage ```go // Get priority p := delta.GetPriority() fmt.Printf("Delta priority: %d\n", p) // Set priority (usually automatic) delta.SetPriority(42) ``` ``` -------------------------------- ### Performing Batch Operations Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/README.md Illustrates how to perform batch operations on the datastore for efficiency. This involves creating a batch, adding multiple Put operations, and then committing the batch. ```go batch, _ := store.Batch(ctx) for i := 0; i < 1000; i++ { batch.Put(ctx, key(i), value(i)) } batch.Commit(ctx) ``` -------------------------------- ### PubSubBroadcaster Implementation Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/INDEX.md Documentation for the PubSubBroadcaster implementation, including its constructor and broadcasting methods. ```APIDOC ## PubSubBroadcaster ### Description An implementation of the Broadcaster interface using a publish-subscribe mechanism. ### Constructor - **NewPubSubBroadcaster()**: Initializes and returns a new PubSubBroadcaster instance. ### Broadcasting Methods - **Broadcast(delta Delta)**: Broadcasts a delta message to subscribers. - **Next() (*Delta, error)**: Retrieves the next delta message from the pub/sub channel. ``` -------------------------------- ### Get Internal Datastore Statistics Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/datastore.md Returns internal datastore statistics including current heads, maximum height, and number of queued jobs. ```go stats := crdtStore.InternalStats(ctx) fmt.Printf("Heads: %d, Max height: %d, Queued jobs: %d\n", len(stats.Heads), stats.MaxHeight, stats.QueuedJobs, ) ``` -------------------------------- ### Monitoring and Hooks Workflow Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Illustrates how to set up Put and Delete hooks to monitor and react to data changes. These hooks can be used for tasks like updating indices or sending notifications. ```go opts := crdt.DefaultOptions() opts.PutHook = func(k ds.Key, v []byte) { log.Infof("Added/Updated: %s = %s", k, string(v)) // Update indices, send notifications, etc. } opts.DeleteHook = func(k ds.Key) { log.Infof("Removed: %s", k) // Clean up caches, indices, etc. } store, _ := crdt.New(..., opts) ``` -------------------------------- ### Get the size of a value in CRDT Datastore Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/datastore.md Returns the size in bytes of the value associated with a key. Uses the default implementation which retrieves the value and measures it. ```go size, err := crdtStore.GetSize(ctx, ds.NewKey("mykey")) if err != nil { log.Fatal(err) } fmt.Printf("Value size: %d bytes\n", size) ``` -------------------------------- ### Check Key Existence in Datastore Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/datastore.md Checks if a key exists in the CRDT Datastore without retrieving its value. This is more efficient than using Get() for existence checks. ```go exists, err := crdtStore.Has(ctx, ds.NewKey("mykey")) if exists { fmt.Println("Key found") } ``` -------------------------------- ### Handling NewPubSubBroadcaster() Errors Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/errors.md This snippet demonstrates error checking when initializing a PubSubBroadcaster. It logs a fatal error if the broadcaster cannot be created. ```Go broadcaster, err := crdt.NewPubSubBroadcaster(ctx, ps, "crdt-topic") if err != nil { log.Fatalf("Failed to create broadcaster: %v", err) } ``` -------------------------------- ### Advanced Delta Control Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/README.md Demonstrates advanced control over deltas using MerkleCRDT. This involves creating a delta, setting its elements, and publishing it to get a head. ```go mcrdt, _ := crdt.NewMerkleCRDT(...) delta := mcrdt.newDelta() delta.SetElements([...]) head, _ := mcrdt.Publish(ctx, delta) ``` -------------------------------- ### Interact with GlobalDB: List and Connect Source: https://github.com/ipfs/go-ds-crdt/blob/master/examples/globaldb/README.md Use the 'list' command to view all stored items and 'connect' to join a peer using its multiaddress. ```plaintext > list [exampleKey] -> exampleValue > connect /ip4/192.168.1.3/tcp/33123/p2p/12D3KooWEkgRTTXGsmFLBembMHxVPDcidJyqFcrqbm9iBE1xhdXq ``` -------------------------------- ### Create CRDT Datastore Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Instantiate the main CRDT Datastore. Requires an underlying persistent store, a namespace, an IPLD DAG service, and a broadcaster for peer communication. Optional configuration can be provided. ```go store, err := crdt.New( underlyingStore, // Required: persistent storage namespace, // Required: key prefix dagService, // Required: IPLD block store broadcaster, // Required: peer communication options, // Optional: configuration ) ``` -------------------------------- ### Batch Operations Workflow Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Demonstrates performing batch operations for efficiency. Changes are accumulated in a batch and then committed together, reducing the overhead of individual operations. ```go batch, _ := store.Batch(ctx) // Accumulate changes for i := 0; i < 1000; i++ { key := ds.NewKey(fmt.Sprintf("item/%d", i)) batch.Put(ctx, key, []byte(fmt.Sprintf("value-%d", i))) } // Publish once batch.Commit(ctx) ``` -------------------------------- ### Traverse DAG from Heads Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/merkle-crdt.md Iterates through all nodes in a DAG starting from the specified heads. Extracts and validates delta information from each node, checking for consistency and empty deltas. ```go heads := mcrdt.Heads() headList, _, _ := heads.List(ctx) headCids := make([]cid.Cid, len(headList)) for i, h := range headList { headCids[i] = h.Cid } nodeCount := 0 err := mcrdt.Traverse(ctx, headCids, func(node ipld.Node) error { nodeCount++ protoNode := node.(*dag.ProtoNode) deltaBytes := protoNode.Data() delta := mcrdt.newDelta() if err := delta.Unmarshal(deltaBytes); err != nil { return fmt.Errorf("corrupted node: %w", err) } if delta.IsEmpty() { fmt.Printf("Warning: empty delta in node %s\n", node.Cid()) } return nil }) if err != nil { log.Fatalf("Traversal failed: %v", err) } fmt.Printf("Traversed %d nodes\n", nodeCount) ``` -------------------------------- ### Persistence Operations Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Shows methods related to datastore persistence and closing. ```go // Persistence err := store.Sync(ctx, prefix) err := store.Close() ``` -------------------------------- ### Handle MerkleCRDT Traverse Errors Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/merkle-crdt.md Shows how to handle errors during MerkleCRDT traversal. The example illustrates stopping traversal on error, but also how to log and continue if a specific node is invalid. ```go err := mcrdt.Traverse(ctx, heads, func(node ipld.Node) error { // If this returns error, traversal stops if !isValidNode(node) { // Log but don't stop - replace error log.Warn("Skipping invalid node: %s", node.Cid()) return nil // Continue traversal } return nil }) if err != nil { // Traversal stopped due to error log.Fatalf("Traversal stopped: %v", err) } ``` -------------------------------- ### Manual Delta Creation Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md For advanced use cases, deltas can be manually created and published. ```go delta := mcrdt.newDelta() delta.SetElements([]*pb.Element{ {Key: "k1", Value: []byte("v1")}, }) head, err := mcrdt.Publish(ctx, delta) ``` -------------------------------- ### Connecting to Bootstrap Peers Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/pubsub-broadcaster.md Connects the libp2p host to a list of bootstrap peers. This is essential for discovering and connecting to other nodes in the network. ```go import ( "github.com/multiformats/go-multiaddr" "github.com/libp2p/go-libp2p/core/peer" ) func bootstrapNetwork(h host.Host, bootstrapPeers []string) error { for _, peerAddr := range bootstrapPeers { // Parse multiaddr: /ip4/x.x.x.x/tcp/port/p2p/QmXXX ma, _ := multiaddr.NewMultiaddr(peerAddr) pi, _ := peer.AddrInfoFromP2pAddr(ma) // Connect to bootstrap peer if err := h.Connect(context.Background(), *pi); err != nil { return err } } return nil } ``` -------------------------------- ### Diagnostic Operations Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Provides methods for datastore diagnostics and visualization. ```go // Diagnostics stats := store.InternalStats(ctx) err := store.PrintDAG(ctx) err := store.DotDAG(ctx, writer) ``` -------------------------------- ### Create Multiple PubSubBroadcasters for Different Topics Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/pubsub-broadcaster.md Instantiate separate PubSubBroadcasters for distinct CRDT instances or data types, each associated with a unique topic. ```go broadcaster1, _ := crdt.NewPubSubBroadcaster(ctx, ps, "data-v1") broadcaster2, _ := crdt.NewPubSubBroadcaster(ctx, ps, "data-v2") store1, _ := crdt.New(ds1, ns1, dag1, broadcaster1, opts) store2, _ := crdt.New(ds2, ns2, dag2, broadcaster2, opts) ``` -------------------------------- ### Inspect MerkleCRDT state Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/merkle-crdt.md Demonstrates how to inspect the CRDT state by retrieving current heads, checking if a specific CID is processed, and accessing the internal set to get element values. ```go mcrdt, _ := crdt.NewMerkleCRDT(...) // Get current heads heads := mcrdt.Heads() headList, maxHeight, _ := heads.List(ctx) fmt.Printf("Current heads: %d\n", len(headList)) fmt.Printf("Max height: %d\n", maxHeight) // Check specific CID c := headList[0].Cid processed, _ := mcrdt.IsProcessed(ctx, c) fmt.Printf("Processed: %v\n", processed) // Get internal set set := mcrdt.Set() value, _ := set.Element(ctx, "mykey") fmt.Printf("Value: %s\n", string(value)) ``` -------------------------------- ### GetElements Method Usage Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/delta-interface.md Demonstrates how to retrieve and iterate over elements (additions/updates) from a Delta object. Assumes a Delta object has been obtained. ```go delta := ... // obtain delta elems, err := delta.GetElements() if err != nil { log.Fatal(err) } for _, elem := range elems { fmt.Printf("Add: %s = %s\n", elem.Key, string(elem.Value)) } ``` -------------------------------- ### Default pbDelta Implementation Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/delta-interface.md Shows the default protobuf-based Delta implementation (pbDelta) which wraps a protobuf message. It is created via the default DeltaFactory. ```go type pbDelta struct { *pb.Delta // wrapped protobuf message } ``` ```go func() Delta { return &pbDelta{Delta: &pb.Delta{}} } ``` -------------------------------- ### Traverse a Merkle-DAG Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/merkle-crdt.md Use `Traverse` to perform a depth-first traversal starting from specified CIDs. The `visit` callback is invoked for each node, allowing processing of deltas and structure. Handles multiple roots and skips duplicates. ```go heads, _, _ := mcrdt.Heads().List(ctx) headCids := make([]cid.Cid, len(heads)) for i, h := range heads { headCids[i] = h.Cid } err := mcrdt.Traverse(ctx, headCids, func(node ipld.Node) error { fmt.Printf("Visiting node: %s\n", node.Cid()) // Extract delta from node protoNode := node.(*dag.ProtoNode) deltaBytes := protoNode.Data() // Process delta delta := mcrdt.newDelta() if err := delta.Unmarshal(deltaBytes); err != nil { return err } fmt.Printf(" Priority: %d\n", delta.GetPriority()) return nil }) if err != nil { log.Fatal(err) } ``` -------------------------------- ### SetElements and SetTombstones Usage Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/delta-interface.md Illustrates how to populate a Delta object with new elements (additions/updates) and mark keys for deletion using tombstones. ```go delta := mcrdt.newDelta() // Add multiple elements delta.SetElements([]*pb.Element{ {Key: "key1", Value: []byte("value1")}, {Key: "key2", Value: []byte("value2")}, }) // Mark keys for deletion delta.SetTombstones([]*pb.Element{ {Key: "old_key"}, }) head, _ := mcrdt.Publish(ctx, delta) ``` -------------------------------- ### Monitoring Datastore Changes Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/README.md Shows how to monitor additions and deletions in the datastore using PutHook and DeleteHook options. These hooks are executed when data is added or removed. ```go opts := crdt.DefaultOptions() opts.PutHook = func(k ds.Key, v []byte) { /* handle add */ } opts.DeleteHook = func(k ds.Key) { /* handle delete */ } store, _ := crdt.New(..., opts) ``` -------------------------------- ### Options Struct Definition Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/types.md Holds all configurable parameters for the CRDT Datastore. Use DefaultOptions() for recommended defaults. ```go type Options struct { Logger logging.StandardLogger RebroadcastInterval time.Duration PutHook func(k ds.Key, v []byte) DeleteHook func(k ds.Key) NumWorkers int DAGSyncerTimeout time.Duration MaxBatchDeltaSize int RepairInterval time.Duration MultiHeadProcessing bool BroadcastBatchDelay time.Duration crdtOpts MerkleCRDTOptions // internal } ``` -------------------------------- ### Implement Cross-Network Bridging with Kafka Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/pubsub-broadcaster.md Wrap PubSubBroadcaster to integrate with different networks, such as Kafka, by broadcasting messages to both systems. ```go type BridgedBroadcaster struct { pubsub *PubSubBroadcaster kafka KafkaProducer } func (b *BridgedBroadcaster) Broadcast(ctx context.Context, data []byte) error { // Send to both networks _ = b.pubsub.Broadcast(ctx, data) return b.kafka.Send(data) } ``` -------------------------------- ### Low Latency Configuration Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/README.md Configuration options focused on achieving low latency. Primarily by setting broadcast batch delay to zero. ```go opts := crdt.DefaultOptions() opts.RebroadcastInterval = 10 * time.Second opts.BroadcastBatchDelay = 0 ``` -------------------------------- ### Key-Value Operations Source: https://github.com/ipfs/go-ds-crdt/blob/master/_autodocs/api-reference/overview.md Lists standard go-datastore Key-Value operations. ```go // KV Operations (standard go-datastore) err := store.Put(ctx, key, value) value, err := store.Get(ctx, key) exists, err := store.Has(ctx, key) err := store.Delete(ctx, key) size, err := store.GetSize(ctx, key) results, err := store.Query(ctx, q) ```