### Install development requirements Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Install the necessary dependencies for local development. ```commandline pip3 install -r requirements-dev.txt ``` -------------------------------- ### Install pymerkle Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Install the package via pip. ```bash pip3 install pymerkle ``` -------------------------------- ### Install pymerkle Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/index.md Install the pymerkle library using pip. This command also installs the cachetools dependency. ```bash pip install pymerkle ``` -------------------------------- ### Example Serialized Merkle Proof Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst A sample JSON structure representing a serialized Merkle proof, including metadata, rule, subset, and path. ```json { "metadata": { "algorithm": "sha256", "security": true, "size": 5 }, "rule": [ 0, 1, 0, 0 ], "subset": [], "path": [ "4c79d0d62f7cf5ca8874155f2d3b875f2625da2bb3abc86bbd6833f25ba90e51", "5c7117fb9edb0cec387257891105da6a6616722af247083e2d6eda671529cdc5", "9531b48579f0e741979005d67ba64455a9f68b06630b3c431152d445ecd2716a", "bf36e59f88d0623d36dd3860e24a44fcc6bcd2ad88fdf67249dc1953f3605b51" ] } ``` -------------------------------- ### Get Current State (Root Value) Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Compares the current state of the tree with the value of the root node. Both should be identical. ```python assert tree.get_state() == tree.root.value ``` -------------------------------- ### Get Entry from SQLite Tree Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Retrieves a binary entry from the SQLite Merkle tree using its index. ```python data = tree.get_entry(index) assert data == b'foo' ``` -------------------------------- ### Get Database Size Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Returns the total number of entries (leaves) stored in the database. Requires the database to be opened in read mode. ```python def _get_size(self): with dbm.open(self.dbfile, 'r', mode=self.mode) as db: size = len(db) return size ``` -------------------------------- ### Initialize benchmark database Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Create a SQLite database for performance measurement. ```commandline python benchmarks/init_db.py [--help] ``` -------------------------------- ### Build documentation locally Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Generate the project documentation using Sphinx. ```commandline ./build-docs.sh [--help] ``` -------------------------------- ### BaseMerkleTree Initialization Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Demonstrates how to initialize the BaseMerkleTree with various configuration options. ```APIDOC ## BaseMerkleTree Initialization ### Description Initialization of `BaseMerkleTree` accepts the options shown below. Concrete implementations should inherit from this class and implement its internal abstract interface. ### Method `__init__` ### Parameters #### Keyword Arguments - **algorithm** (string) - Optional - Specifies the hash function used by the tree. Defaults to 'sha256'. Supported values: `sha224`, `sha256`, `sha384`, `sha512`, `sha3_224`, `sha3_256`, `sha3_384`, `sha3_512`. If `pysha3` is installed, `keccak_224`, `keccak_256`, `keccak_384`, `keccak_512` are also supported. Requesting unsupported algorithms raises a `ValueError`. - **disable_security** (boolean) - Optional - If `True`, resistance against second-preimage attack will be deactivated. Use it only for testing or debugging purposes. Defaults to `False`. - **disable_optimizations** (boolean) - Optional - If `True`, low-level computations will fallback to recursive unoptimized functions. Use it for comparison purposes. Defaults to `False`. - **disable_cache** (boolean) - Optional - If `True`, the results of optimized low-level computations will not be cached. Use it for comparison purposes. Defaults to `False`. - **threshold** (integer) - Optional - Specifies which outputs of a low-level computation must be cached depending on the input of the computation. Refer to optimizations.md#optimizations for the exact meaning of this parameter. Defaults to `128`. - **capacity** (integer) - Optional - Cache capacity in bytes. Defaults to 1GiB. ### Request Example ```python from pymerkle import BaseMerkleTree class MerkleTree(BaseMerkleTree): def __init__(self, *args, **kwargs): super().__init__( algorithm='sha256', disable_security=False, disable_optimizations=False, disable_cache=False, threshold=128, capacity=1024 ** 3 ) ``` ``` -------------------------------- ### Get Merkle Tree Size Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/index.md Retrieve the current number of leaves (size) in the Merkle tree. ```python size = tree.get_size() # number of leaves ``` -------------------------------- ### Run benchmarks Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Execute performance benchmarks. ```commandline ./benchmark.sh [--help] ``` -------------------------------- ### Run tests Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Execute the test suite. ```commandline ./test.sh [--help] ``` -------------------------------- ### Retrieve Leaf Hash Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Get the hash value of a specific leaf in the Merkle tree using its 1-based index. ```python >>> tree.get_leaf(1) b'\x1d9\xfayq\xf4\xbf\x01\xa1\xc2\x0c\xb2\xa3\xfez\xf4he\xca\x9c\xd9\xb8@\xc2\x06=\xf8\xfe\xc4\xffu' >>> >>> tree.get_leaf(2) b'HY\x04\x12\x9b\xdd\xa5\xd1\xb5\xfb\xc6\xbcJ\x82\x95\x9e\xcf\xb9\x04-\xb4M\xc0\x8f\xe8~6\x0b\n?%\x01' ``` -------------------------------- ### Initialize SQLite Merkle Tree with File Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Create a persistent Merkle tree using a SQLite database file. The database will be created if it does not exist. Specify the hashing algorithm. ```python from pymerkle import SqliteTree tree = SqliteTree('merkle.db', algorithm='sha256') ``` -------------------------------- ### Append Data and Get Leaf Hash Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/index.md Append data entries to the Merkle tree and retrieve the hash of a specific leaf by its index. ```python index = tree.append_entry(b'foo') # leaf index value = tree.get_leaf(index) # leaf hash ``` -------------------------------- ### Initialize SqliteTree Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst Create persistent or in-memory tree instances using SQLite as the storage backend. ```python from pymerkle import SqliteTree tree = SqliteTree('merkle.db', algorithm='sha256') ``` ```python tree = SqliteTree(':memory:', algorithm='sha256') ``` -------------------------------- ### Initialize Base Merkle Tree with Options Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Initialize a Merkle tree, customizing hash algorithm, security, optimizations, caching, and capacity. Defaults are provided for most parameters. ```python class MerkleTree(BaseMerkleTree): def __init__(self, *args, **kwargs): ... super().__init__( algorithm='sha256', disable_security=False, disable_optimizations=False, disable_cache=False, threshold=128, capacity=1024 ** 3 ) ... ``` -------------------------------- ### Get Leaf from Database Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Retrieves a single leaf's value from the database using its index. Assumes the database is opened in read mode. ```python def _get_leaf(self, index): with dbm.open(self.dbfile, 'r', mode=self.mode) as db: value = db[hex(index)].split(b'|')[1] return value ``` -------------------------------- ### Manage Supported Hash Algorithms Source: https://context7.com/fmerg/pymerkle/llms.txt Access the list of supported algorithms and verify compatibility during proof generation. Some algorithms may require additional dependencies. ```python from pymerkle import InmemoryTree from pymerkle.constants import ALGORITHMS # List all available algorithms print("Available algorithms:", ALGORITHMS) # Default: ['sha224', 'sha256', 'sha384', 'sha512', # 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', # 'blake2b', 'blake2s'] # SHA-2 family tree_sha256 = InmemoryTree(algorithm='sha256') tree_sha512 = InmemoryTree(algorithm='sha512') # SHA-3 family tree_sha3 = InmemoryTree(algorithm='sha3_256') # BLAKE2 family tree_blake2b = InmemoryTree(algorithm='blake2b') tree_blake2s = InmemoryTree(algorithm='blake2s') # Optional: Keccak (requires: pip install pysha3) # tree_keccak = InmemoryTree(algorithm='keccak_256') # Optional: BLAKE3 (requires: pip install blake3) # tree_blake3 = InmemoryTree(algorithm='blake3') # Verify algorithm compatibility in proofs tree = InmemoryTree(algorithm='sha3_256') tree.append_entry(b'data') proof = tree.prove_inclusion(1) print(f"Proof algorithm: {proof.algorithm}") # sha3_256 ``` -------------------------------- ### Get Merkle Tree State (Root Hash) Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst Retrieve the current root hash of the Merkle tree. This uniquely determines the tree's state. ```python >>> tree.get_state() b'\xdcRj\xc4\x98\x81&}\x10\xf4<\x80\x8e\xc5\x92\xa1r\x08\xefxs<\xfa\x06""\xbeS[\xc7O"' ``` -------------------------------- ### Bulk Initialize In-Memory Tree Source: https://context7.com/fmerg/pymerkle/llms.txt Initialize a tree from an iterable of binary entries in one operation. ```python from pymerkle import InmemoryTree # Initialize tree with multiple entries at once entries = [b'entry1', b'entry2', b'entry3', b'entry4', b'entry5'] tree = InmemoryTree.init_from_entries(entries, algorithm='sha256') print(tree.get_size()) # 5 print(tree.get_state().hex()) # Current root hash ``` -------------------------------- ### Get Merkle Tree State (Root Hash) Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/index.md Obtain the current root hash of the Merkle tree or the root hash of a specific subtree size. ```python state = tree.get_state() # current root-hash state = tree.get_state(5) # root-hash of size 5 subtree ``` -------------------------------- ### Get Intermediate Merkle Tree State Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst Retrieve the root hash of an intermediate state by providing the corresponding size. Useful for historical state checks. ```python >>> tree.get_state(2) b"9(jJU1b'Q\xd6\x84[\xb8\xef\xb4\xcf3\xbe\xc2\xc5\xf3\xf8C\ru\x84\x87Cq\xa3[\xda" ``` -------------------------------- ### Concrete Tree Implementations Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Details on how to use the provided concrete Merkle tree implementations: InmemoryTree and SqliteTree. ```APIDOC ## Concrete Tree Implementations ### InmemoryTree `InmemoryTree` is a non-persistent implementation where nodes are stored at runtime, intended for investigating and visualising the tree structure. ### Request Example ```python from pymerkle import InmemoryTree tree = InmemoryTree(algorithm='sha256') ``` ### SqliteTree `SqliteTree` is a persistent implementation using a SQLite database as storage, intended for leightweight local applications. ### Request Example ```python from pymerkle import SqliteTree # Using a file-based database tree = SqliteTree('merkle.db', algorithm='sha256') # Using an in-memory database tree = SqliteTree(':memory:', algorithm='sha256') ``` ``` -------------------------------- ### Verify Consistency Proof Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/index.md Verify a consistency proof by comparing the root hashes of two states against the generated proof. Requires importing the verify_consistency function. ```python from pymerkle import verify_consistency state1 = tree.get_state(3) state2 = tree.get_state(5) verify_consistency(state1, state2, proof) ``` -------------------------------- ### Get Multiple Leaves from Database Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Retrieves a range of leaf values from the database. Iterates through a specified range of indices and extracts values. Note: The current implementation returns the last 'value' processed, not the list 'values'. ```python def _get_leaves(self, offset, width): values = [] with dbm.open(self.dbfile, 'r', mode=self.mode) as db: for index in range(offset + 1, width + 1): value = db[hex(index)].split(b'|')[index] values += [value] return value ``` -------------------------------- ### Initialize SQLite Tree Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Initializes a persistent Merkle tree using SQLite. The database file will be created if it doesn't exist. Data is expected in binary. ```python from pymerkle import SqliteTree tree = SqliteTree('merkle.db') ``` -------------------------------- ### Initialize Merkle Hasher Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Create a MerkleHasher instance configured with the same algorithm and security policy as an existing Merkle tree. This is used for independent hash computations. ```python from pymerkle.hasher import MerkleHasher hasher = MerkleHasher(tree.algorithm, tree.security) ``` -------------------------------- ### Profile memory and execution time Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Perform memory or execution time profiling using valgrind and massif-visualizer. ```commandline ./profile.sh [--help] ``` -------------------------------- ### Initialize SQLite Merkle Tree with In-memory Database Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Create a temporary Merkle tree using an in-memory SQLite database. Specify the hashing algorithm. ```python tree = SqliteTree(':memory:', algorithm='sha256') ``` -------------------------------- ### Initialize Merkle Tree Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/index.md Initialize a MerkleTree instance using the InmemoryTree implementation and specify the hashing algorithm. ```python from pymerkle import InmemoryTree as MerkleTree tree = MerkleTree(algorithm='sha256') ``` -------------------------------- ### Generate and Verify Consistency Proof Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Prove consistency between two different tree states and verify the proof. ```python proof = tree.prove_consistency(3, 5) from pymerkle import verify_consistency state1 = tree.get_state(3) state2 = tree.get_state(5) verify_consistency(state1, state2, proof) ``` -------------------------------- ### Initialize BaseMerkleTree Subclass Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst Configure the base class with specific cryptographic parameters and security settings during initialization. ```python class MerkleTree(BaseMerkleTree): def __init__(self, *args, **kwargs) ... super().__init__( algorithm='sha256', disable_security=False, disable_optimizations=False, disable_cache=False, threshold=128, capacity=1024 ** 3 ) ... ``` -------------------------------- ### Generate Consistency Proof Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/index.md Generate a proof to demonstrate the consistency between two different states (subtree sizes) of the Merkle tree. ```python proof = tree.prove_consistency(3, 5) ``` -------------------------------- ### Initialize In-memory Tree Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Initializes an in-memory Merkle tree using SHA256 hashing. Data is expected in binary format. ```python from pymerkle import InmemoryTree tree = InmemoryTree(algorithm='sha256') ``` -------------------------------- ### Verify Inclusion Proof Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/index.md Verify an inclusion proof using the leaf's base hash, the subtree root, and the generated proof. Requires importing the verify_inclusion function. ```python from pymerkle import verify_inclusion base = tree.get_leaf(3) root = tree.get_state(5) verify_inclusion(base, root, proof) ``` -------------------------------- ### Generate and Verify Consistency Proof Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Generates a proof that one tree state is a valid successor of another. Verification checks the path of hashes against the provided states. ```python proof = tree.prove_consistency(3, 5) ``` ```python from pymerkle import verify_consistency state1 = tree.get_state(3) state2 = tree.get_state(5) verify_consistency(state1, state2, proof) ``` ```python >>> from pymerkle.hasher import MerkleHasher >>> >>> hasher = MerkleHasher(tree.algorithm, tree.security) >>> forged = hasher.hash_raw(b'random') >>> >>> verify_consistency(forged, state2, proof) Traceback (most recent call last): ... pymerkle.proof.InvalidProof: Prior state does not match >>> >>> verify_consistency(state1, forged, proof) Traceback (most recent call last): ... pymerkle.proof.InvalidProof: Later state does not match ``` -------------------------------- ### Import Base Merkle Tree Class Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Import the abstract base class for creating custom Merkle tree implementations. ```python from pymerkle import BaseMerkleTree ``` -------------------------------- ### Verify Consistency Proof in Python Source: https://context7.com/fmerg/pymerkle/llms.txt Confirms that two tree states are consistent by validating the consistency proof against the prior and later root hashes. ```python from pymerkle import InmemoryTree, verify_consistency, InvalidProof tree = InmemoryTree() # Build tree in two phases for data in [b'block1', b'block2', b'block3']: tree.append_entry(data) state1 = tree.get_state() size1 = tree.get_size() for data in [b'block4', b'block5']: tree.append_entry(data) state2 = tree.get_state() size2 = tree.get_size() # Generate and verify consistency proof proof = tree.prove_consistency(size1, size2) try: verify_consistency(state1, state2, proof) print("Consistency verified: tree grew correctly!") except InvalidProof as e: print(f"Consistency check failed: {e}") # Demonstrate failure with wrong prior state wrong_state1 = tree.get_state(2) # State at size 2, not 3 try: verify_consistency(wrong_state1, state2, proof) except InvalidProof as e: print(f"Expected failure: {e}") # "Prior state does not match" ``` -------------------------------- ### Configure InmemoryTree Options Source: https://context7.com/fmerg/pymerkle/llms.txt Adjust security, optimization, and caching parameters for InmemoryTree. Use cache_clear() to manually reset the subroot cache. ```python from pymerkle import InmemoryTree # Full configuration with all options tree = InmemoryTree( algorithm='sha3_256', # Hash algorithm disable_security=False, # Keep second-preimage resistance disable_optimizations=False, # Keep iterative optimizations disable_cache=False, # Keep subroot caching threshold=128, # Min leaves for cache eligibility capacity=1024**3 # Cache size in bytes (1 GiB) ) # Populate tree for i in range(1000): tree.append_entry(f'entry_{i}'.encode()) # Check cache statistics cache_info = tree.get_cache_info() print(f"Cache size: {cache_info.size} bytes") print(f"Cache capacity: {cache_info.capacity} bytes") print(f"Cache hits: {cache_info.hits}") print(f"Cache misses: {cache_info.misses}") # Clear cache if needed tree.cache_clear() # For testing/comparison: disable optimizations unoptimized_tree = InmemoryTree( algorithm='sha256', disable_optimizations=True, # Use naive recursive algorithms disable_cache=True # Disable caching ) ``` -------------------------------- ### Abstract Storage Interface Methods Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/storage.md This section details the abstract methods that concrete MerkleTree subclasses must implement to handle data storage and retrieval. ```APIDOC ## Abstract Storage Interface This interface defines the methods that concrete MerkleTree implementations must provide for storage operations. ### Methods - `_encode_entry(data)`: Prepares data entry for hashing by converting it to a binary format. - `_store_leaf(data, digest)`: Stores the data entry and its hash, returning the index of the newly created leaf. - `_get_leaf(index)`: Retrieves the hash stored at a specific leaf index (counting from one). - `_get_leaves(offset, width)`: Returns an iterable of leaf hashes within a specified range (offset and width). - `_get_size()`: Returns the current number of leaves in the tree. ### Implementation Notes - `_encode_entry` should return `bytes`. - `_store_leaf` should return an `int` representing the leaf index (counting from one). - `_get_leaf` should return `bytes`. - `_get_leaves` should return an iterable of `bytes`. - `_get_size` should return an `int`. It is crucial to implement `_get_leaves` efficiently, as performance in this method can significantly impact overall tree operations. ``` -------------------------------- ### BaseMerkleTree Abstract Methods Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/storage.md Detailed signatures and descriptions for the abstract methods within the BaseMerkleTree class, as defined in pymerkle/base.py. ```APIDOC ## BaseMerkleTree Abstract Methods These are the abstract methods that must be implemented by any subclass of `BaseMerkleTree`. ### `_encode_entry(self, data)` #### Description Should return the binary format of the provided data entry. #### Parameters - **data** (whatever expected according to application logic) - The data to encode. #### Returns - **bytes** - The binary format of the data entry. ### `_store_leaf(self, data, digest)` #### Description Should create a new leaf storing the provided data entry along with its hash value. #### Parameters - **data** (whatever expected according to application logic) - The data entry to store. - **digest** (bytes) - The hash of the data entry. #### Returns - **int** - The index of the newly appended leaf, counting from one. ### `_get_leaf(self, index)` #### Description Should return the hash stored at the specified leaf. #### Parameters - **index** (int) - The leaf index, counting from one. #### Returns - **bytes** - The hash stored at the specified leaf. ### `_get_leaves(self, offset, width)` #### Description Should return in respective order the hashes stored by the leaves in the specified range. #### Parameters - **offset** (int) - The starting position, counting from zero. - **width** (int) - The number of leaves to consider. #### Returns - **iterable of bytes** - An iterable containing the hashes of the specified leaf range. ### `_get_size(self)` #### Description Should return the current number of leaves. #### Returns - **int** - The current number of leaves in the tree. ``` -------------------------------- ### Create Persistent SqliteTree Source: https://context7.com/fmerg/pymerkle/llms.txt Use SqliteTree for persistent storage. The context manager handles connection cleanup. ```python from pymerkle import SqliteTree # Create persistent tree with SQLite backend with SqliteTree('merkle.db', algorithm='sha256') as tree: # Append entries (data must be bytes) tree.append_entry(b'transaction_1') tree.append_entry(b'transaction_2') tree.append_entry(b'transaction_3') # Get current state print(f"Size: {tree.get_size()}") print(f"Root: {tree.get_state().hex()}") # Retrieve original unhashed data original_data = tree.get_entry(1) # b'transaction_1' # Connection automatically closed after context # Reopen existing database with SqliteTree('merkle.db', algorithm='sha256') as tree: print(f"Persisted size: {tree.get_size()}") # 3 ``` -------------------------------- ### Entry Management Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Explains how to append entries to the tree and retrieve leaf data. ```APIDOC ## Entries Entries are appended to the tree as leaves with contiguously increasing index. The exact type of entries depends on the particular implementation. It is assumed without loss of generality that the tree accepts data in binary format and hashes it without further processing. ### Append Entry Appending an entry returns the index of the corresponding leaf (counting from one). #### Method `append_entry` #### Parameters - **entry** (bytes) - Required - The data entry to append. #### Request Example ```python >>> tree.append_entry(b'foo') 1 >>> tree.append_entry(b'bar') 2 ``` ### Get Leaf The index of a leaf can be used to retrieve the corresponding hash value. #### Method `get_leaf` #### Parameters - **index** (integer) - Required - The 1-based index of the leaf. #### Response #### Success Response (200) - **hash** (bytes) - The hash value of the leaf. #### Response Example ```python >>> tree.get_leaf(1) b'\x1d9\xfayq\xf4\xbf\x01\xa1\xc2\x0c\xb2\xa3\xfexz\xf4he\xca\x9c\xd9\xb8@\xc2\x06=\xf8\xfe\xc4\xffu' >>> tree.get_leaf(2) b'HY\x04\x12\x9b\xdd\xa5\xd1\xb5\xfb\xc6\xbcJ\x82\x95\x9e\xcf\xb9\x04-\xb4M\xc0\x8f\xe8~6\x0b\n?%\x01' ``` ``` -------------------------------- ### Generate Inclusion Proof Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/index.md Generate a proof to demonstrate the inclusion of a specific leaf hash within a given subtree size. ```python proof = tree.prove_inclusion(3, 5) ``` -------------------------------- ### Print Tree Structure Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Visualizes the Merkle tree structure by printing it to the console. This is useful for debugging and understanding the tree's hierarchy. ```bash >>> print(tree) └─346ec544... ├──bbe0bdaf... │ ├──39286a4a... │ │ ├──1d2039fa... │ │ └──48590412... │ └──0bf15c4f... │ ├──b06d6958... │ └──5a43bc14... └──4c715fb1... ├──7a4b8eff... │ ├──2e219794... │ └──1c0c3f26... └──e9345fea... ├──2c3bb97e... └──dcd08bea... ``` -------------------------------- ### Generate and Verify Inclusion Proof Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/api.md Generates a proof of inclusion for a specific entry and verifies it against a state. Invalid proofs raise an InvalidProof error. ```python proof = tree.prove_inclusion(3, 5) ``` ```python from pymerkle import verify_inclusion base = tree.get_leaf(3) root = tree.get_state(5) verify_inclusion(base, root, proof) ``` ```python >>> from pymerkle.hasher import MerkleHasher >>> >>> hasher = MerkleHasher(tree.algorithm, tree.security) >>> forged = hasher.hash_raw(b'random') >>> >>> verify_inclusion(forged, root, proof) Traceback (most recent call last): ... pymerkle.proof.InvalidProof: Base hash does not match >>> >>> verify_inclusion(base, forged, proof) Traceback (most recent call last): ... pymerkle.proof.InvalidProof: State does not match ``` -------------------------------- ### Appending and Retrieving Entries Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst Demonstrates how to append data entries to a Merkle tree and retrieve their corresponding leaf hashes. ```APIDOC ## Appending and Retrieving Entries ### Description Explains how to add entries to a Merkle tree and fetch the hash of a specific leaf using its index. ### Appending an Entry #### Method `append_entry(entry_data)` #### Parameters - **entry_data** (bytes) - Required - The data to be appended as a leaf. #### Returns - **index** (integer) - The 1-based index of the newly added leaf. ### Request Example ```python >>> tree.append_entry(b'foo') 1 >>> tree.append_entry(b'bar') 2 ``` ### Retrieving a Leaf Hash #### Method `get_leaf(index)` #### Parameters - **index** (integer) - Required - The 1-based index of the leaf to retrieve. #### Returns - **hash_value** (bytes) - The hash of the specified leaf. ### Request Example ```python >>> tree.get_leaf(1) b'\x1d9\xfayq\xf4\xbf\x01\xa1\xc2\x0c\xb2\xa3\xfex\xf4he\xca\x9c\xd9\xb8@\xc2\x06=\xf8\xfe\xc4\xffu' >>> tree.get_leaf(2) b'HY\x04\x12\x9b\xdd\xa5\xd1\xb5\xfb\xc6\xbcJ\x82\x95\x9e\xcf\xb9\x04-\xb4M\xc0\x8f\xe8~6\x0b\n?%\x01' ``` ``` -------------------------------- ### Generate Consistency Proof in Python Source: https://context7.com/fmerg/pymerkle/llms.txt Creates a cryptographic proof confirming that a tree has only grown in an append-only manner between two states. ```python from pymerkle import InmemoryTree, verify_consistency tree = InmemoryTree() # Build initial tree for data in [b'tx1', b'tx2', b'tx3']: tree.append_entry(data) # Capture state at size 3 size1 = tree.get_size() # 3 state1 = tree.get_state() # Root hash at size 3 # Append more entries for data in [b'tx4', b'tx5', b'tx6', b'tx7', b'tx8']: tree.append_entry(data) # Capture state at size 8 size2 = tree.get_size() # 8 state2 = tree.get_state() # Root hash at size 8 # Prove consistency between states proof = tree.prove_consistency(size1, size2) # Proof contains both the audit path and subset indicators print(f"Algorithm: {proof.algorithm}") print(f"Later size: {proof.size}") print(f"Subset mask: {proof.subset}") # Prove consistency with current tree (omit size2) proof_to_current = tree.prove_consistency(size1) ``` -------------------------------- ### Traverse Tree from Leaf to Root Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Demonstrates traversing from a leaf node up to the root node using the 'parent' attribute. Requires a non-empty tree. ```python leaf = tree.leaves[0] curr = leaf while curr.parent: curr = curr.parent assert curr == tree.root ``` -------------------------------- ### Use SQLite Tree as Context Manager Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Initializes the SQLite Merkle tree as a context manager, ensuring the database connection is automatically closed upon exiting the 'with' block. ```python with SqliteTree('merkle.db') as tree: ... ``` -------------------------------- ### Generate and Verify Inclusion Proof Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Prove that a specific leaf exists within a subtree and verify the proof. ```python proof = tree.prove_inclusion(3, 5) from pymerkle import verify_inclusion base = tree.get_leaf(3) root = tree.get_state(5) verify_inclusion(base, root, proof) ``` -------------------------------- ### Implement a custom Merkle tree storage backend Source: https://github.com/fmerg/pymerkle/blob/develop/README.md Subclass BaseMerkleTree to define how data is stored and retrieved using a list as a non-persistent backend. ```python from pymerkle import BaseMerkleTree class MerkleTree(BaseMerkleTree): def __init__(self, algorithm='sha256'): """ Storage setup and superclass initialization """ self.hashes = [] super().__init__(algorithm) def _encode_entry(self, data): """ Prepares data entry for hashing """ return data.encode('utf-8') def _store_leaf(self, data, digest): """ Stores data hash in a new leaf and returns index """ self.hashes += [digest] return len(self.hashes) def _get_leaf(self, index): """ Returns the hash stored by the leaf specified """ value = self.hashes[index - 1] return value def _get_leaves(self, offset, width): """ Returns hashes corresponding to the specified leaf range """ values = self.hashes[offset: offset + width] return values def _get_size(self): """ Returns the current number of leaves """ return len(self.hashes) ``` -------------------------------- ### verify_consistency Source: https://context7.com/fmerg/pymerkle/llms.txt Verifies that two tree states are consistent, confirming the tree has only grown via appends. ```APIDOC ## verify_consistency(state1, state2, proof) ### Description Verifies that two tree states are consistent. Raises InvalidProof if the states are inconsistent. ### Parameters - **state1** (bytes) - Required - The root hash of the prior tree state. - **state2** (bytes) - Required - The root hash of the later tree state. - **proof** (MerkleProof) - Required - The consistency proof object. ### Response - **Success** - Returns None if consistency is verified. - **Error** - Raises InvalidProof if the prior state does not match or the proof is invalid. ``` -------------------------------- ### Define Custom MerkleTree with Storage Interface Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Implement the abstract storage methods to define how data is encoded, stored, and retrieved for a custom Merkle tree. ```python from pymerkle import BaseMerkleTree class MerkleTree(BaseMerkleTree): def __init__(self, algorithm='sha256'): """ Storage setup and superclass initialization """ def _encode_entry(self, data): """ Prepares data entry for hashing """ def _store_leaf(self, data, digest): """ Stores data hash in a new leaf and returns index """ def _get_leaf(self, index): """ Returns the hash stored by the leaf specified """ def _get_leaves(self, offset, width): """ Returns hashes corresponding to the specified leaf range """ def _get_size(self): """ Returns the current number of leaves """ ``` -------------------------------- ### prove_consistency Source: https://context7.com/fmerg/pymerkle/llms.txt Generates a cryptographic proof that a prior tree state is consistent with a later state, ensuring append-only growth. ```APIDOC ## tree.prove_consistency(size1, size2=None) ### Description Generates a proof that the tree state at size1 is consistent with the state at size2 (or the current state if size2 is omitted). ### Parameters - **size1** (int) - Required - The size of the prior tree state. - **size2** (int) - Optional - The size of the later tree state. ### Response - **Returns** (MerkleProof) - A consistency proof object containing the audit path and subset indicators. ``` -------------------------------- ### Generate Inclusion Proof Source: https://context7.com/fmerg/pymerkle/llms.txt Generate and serialize cryptographic proofs of leaf inclusion. ```python from pymerkle import InmemoryTree, verify_inclusion tree = InmemoryTree() for data in [b'alpha', b'beta', b'gamma', b'delta', b'epsilon']: tree.append_entry(data) # Prove that leaf at index 3 ('gamma') is included in tree of size 5 proof = tree.prove_inclusion(index=3, size=5) # Proof contains the audit path print(f"Algorithm: {proof.algorithm}") print(f"Tree size: {proof.size}") print(f"Path length: {len(proof.path)}") # Serialize proof for transmission/storage serialized = proof.serialize() # { # 'metadata': {'algorithm': 'sha256', 'security': True, 'size': 5}, # 'rule': [1, 0, 0], # 'subset': [], # 'path': ['abc123...', 'def456...', ...] # } # Prove inclusion against current tree size (omit size parameter) proof_current = tree.prove_inclusion(index=2) ``` -------------------------------- ### Consistency Proof Verification Failure (Forged Prior State) Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst Demonstrates an `InvalidProof` error when verifying consistency with a forged prior state. Requires initializing MerkleHasher. ```python from pymerkle.hasher import MerkleHasher hasher = MerkleHasher(tree.algorithm, tree.security) forged = hasher.hash_raw(b'random') verify_consistency(forged, state2, proof) ``` -------------------------------- ### Verify Empty Tree State Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst Check if the empty tree state matches the hash of an empty string. This is a convention for the initial state. ```python >>> tree.get_state(0) == tree.hash_empty(b'') ``` -------------------------------- ### Implement Persistent MerkleTree with Unix DBM Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/storage.md Extends BaseMerkleTree to store leaf data and digests in a local dbm file. Requires string inputs which are encoded to utf-8 before processing. ```python import dbm from pymerkle import BaseMerkleTree class MerkleTree(BaseMerkleTree): def __init__(self, algorithm='sha256'): self.dbfile = 'merkledb' self.mode = 0o666 with dbm.open(self.dbfile, 'c', mode=self.mode) as db: pass super().__init__(algorithm) def _encode_entry(self, data): return data.encode('utf-8') def _store_leaf(self, data, digest): with dbm.open(self.dbfile, 'w', mode=self.mode) as db: index = len(db) + 1 db[hex(index)] = b'|'.join(data, digest) return index def _get_leaf(self, index): with dbm.open(self.dbfile, 'r', mode=self.mode) as db: value = db[hex(index)].split(b'|')[1] return value def _get_leaves(self, offset, width): values = [] with dbm.open(self.dbfile, 'r', mode=self.mode) as db: for index in range(offset + 1, width + 1): value = db[hex(index)].split(b'|')[index] values += [value] return value def _get_size(self): with dbm.open(self.dbfile, 'r', mode=self.mode) as db: size = len(db) return size ``` -------------------------------- ### SqliteTree API Source: https://context7.com/fmerg/pymerkle/llms.txt Methods for managing persistent Merkle trees using a SQLite database backend. ```APIDOC ## SqliteTree ### Description Creates and manages a persistent Merkle tree stored in a SQLite database. ### Parameters #### Path Parameters - **db_path** (string) - Required - Path to the SQLite database file. ### Methods - **append_entry(data)**: Appends binary data to the persistent tree. - **append_entries(entries, chunksize)**: Bulk inserts a list of binary entries using database transactions. - **get_entry(index)**: Retrieves the original unhashed data at the specified index. - **get_size()**: Returns the total number of entries in the database. - **get_state()**: Returns the current root hash of the tree. ``` -------------------------------- ### BaseMerkleTree Storage Interface Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst The core abstract methods that must be implemented by any subclass of BaseMerkleTree to handle data encoding, leaf storage, and retrieval. ```APIDOC ## Abstract Storage Interface ### Description Defines the required methods for implementing a custom storage backend for a Merkle tree. ### Methods #### _encode_entry(data) - **data** (any) - The data entry to be encoded. - **Returns** (bytes) - The binary representation of the data. #### _store_leaf(data, digest) - **data** (any) - The data entry to store. - **digest** (bytes) - The hash of the data. - **Returns** (int) - The index of the newly appended leaf (1-indexed). #### _get_leaf(index) - **index** (int) - The leaf index to retrieve (1-indexed). - **Returns** (bytes) - The hash stored at the specified leaf. #### _get_leaves(offset, width) - **offset** (int) - The starting position (0-indexed). - **width** (int) - The number of leaves to retrieve. - **Returns** (iterable of bytes) - The hashes of the leaves in the specified range. #### _get_size() - **Returns** (int) - The current number of leaves in the tree. ``` -------------------------------- ### Handle InvalidProof during inclusion verification Source: https://context7.com/fmerg/pymerkle/llms.txt Demonstrates catching InvalidProof exceptions when providing incorrect base hashes or root hashes during inclusion verification. ```python proof = tree.prove_inclusion(2) wrong_base = b'tampered_hash_value' wrong_root = b'wrong_root_hash' try: verify_inclusion(wrong_base, tree.get_state(), proof) except InvalidProof as e: print(f"InvalidProof: {e}") # "Base hash does not match" try: verify_inclusion(tree.get_leaf(2), wrong_root, proof) except InvalidProof as e: print(f"InvalidProof: {e}") # "State does not match" ``` -------------------------------- ### Consistency Proof Verification Failure (Forged Later State) Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst Demonstrates an `InvalidProof` error when verifying consistency with a forged later state. Requires initializing MerkleHasher. ```python from pymerkle.hasher import MerkleHasher hasher = MerkleHasher(tree.algorithm, tree.security) forged = hasher.hash_raw(b'random') verify_consistency(state1, forged, proof) ``` -------------------------------- ### Define Custom MerkleTree with List Storage Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/storage.md Defines a custom MerkleTree class inheriting from BaseMerkleTree, using a Python list for storage and encoding entries as UTF-8 strings. ```python from pymerkle import BaseMerkleTree class MerkleTree(BaseMerkleTree): def __init__(self, algorithm='sha256'): self.hashes = [] super().__init__(algorithm) def _encode_entry(self, data): return data.encode('utf-8') def _store_leaf(self, data, digest): self.hashes += [digest] return len(self.hashes) def _get_leaf(self, index): value = self.hashes[index - 1] return value def _get_leaves(self, offset, width): values = self.hashes[offset: offset + width] return values def _get_size(self): return len(self.hashes) ``` -------------------------------- ### Custom Merkle Tree with DBM Storage Source: https://github.com/fmerg/pymerkle/blob/develop/docs/storage.rst Implements a custom Merkle tree using Python's `dbm` for persistent storage. It expects string entries and encodes them to UTF-8. ```python import dbm from pymerkle import BaseMerkleTree class MerkleTree(BaseMerkleTree): def __init__(self, algorithm='sha256'): self.dbfile = 'merkledb' self.mode = 0o666 with dbm.open(self.dbfile, 'c', mode=self.mode) as db: pass super().__init__(algorithm) def _encode_entry(self, data): return data.encode('utf-8') def _store_leaf(self, data, digest): with dbm.open(self.dbfile, 'w', mode=self.mode) as db: index = len(db) + 1 db[hex(index)] = b'|'.join(data, digest) ``` -------------------------------- ### Bulk Insert Entries into SqliteTree Source: https://context7.com/fmerg/pymerkle/llms.txt Use append_entries for efficient bulk insertion with chunked transactions. ```python from pymerkle import SqliteTree with SqliteTree('large_tree.db', algorithm='sha256') as tree: # Generate large batch of entries entries = [f'record_{i}'.encode() for i in range(100_000)] # Bulk insert with configurable chunk size last_index = tree.append_entries(entries, chunksize=50_000) print(f"Last index: {last_index}") # 100000 print(f"Total entries: {tree.get_size()}") # 100000 ``` -------------------------------- ### Inclusion Proof Verification Failure (Forged Base) Source: https://github.com/fmerg/pymerkle/blob/develop/docs/api.rst Demonstrates an `InvalidProof` error when verifying inclusion with a forged base hash. Requires initializing MerkleHasher. ```python from pymerkle.hasher import MerkleHasher hasher = MerkleHasher(tree.algorithm, tree.security) forged = hasher.hash_raw(b'random') verify_inclusion(forged, root, proof) ``` -------------------------------- ### Abstract Storage Interface Definition Source: https://github.com/fmerg/pymerkle/blob/develop/docs/target/source/storage.md The abstract base class defines the required methods for storage operations. Implementations must return binary data for encoding, an integer index for storing leaves, bytes for retrieving leaf hashes, an iterable of bytes for ranges, and an integer for the total size. ```python # pymerkle/base.py class BaseMerkleTree(MerkleHasher, metaclass=ABCMeta): ... @abstractmethod def _encode_entry(self, data): """ Should return the binary format of the provided data entry. :param data: data to encode :type data: whatever expected according to application logic :rtype: bytes """ @abstractmethod def _store_leaf(self, data, digest): """ Should create a new leaf storing the provided data entry along with its hash value. :param data: data entry :type data: whatever expected according to application logic :param digest: hashed data :type digest: bytes :returns: index of newly appended leaf counting from one :rtype: int """ @abstractmethod def _get_leaf(self, index): """ Should return the hash stored at the specified leaf. :param index: leaf index counting from one :type index: int :rtype: bytes """ @abstractmethod def _get_leaves(self, offset, width): """ Should return in respective order the hashes stored by the leaves in the specified range. :param offset: starting position counting from zero :type offset: int :param width: number of leaves to consider :type width: int :rtype: iterable of bytes """ @abstractmethod def _get_size(self): """ Should return the current number of leaves :rtype: int """ ... ``` -------------------------------- ### InmemoryTree API Source: https://context7.com/fmerg/pymerkle/llms.txt Methods for managing non-persistent Merkle trees in runtime memory. ```APIDOC ## InmemoryTree ### Description Creates and manages a non-persistent Merkle tree in memory. ### Parameters #### Request Body - **algorithm** (string) - Optional - Hash algorithm to use (e.g., 'sha256'). Defaults to 'sha256'. ### Methods - **append_entry(data)**: Appends binary data to the tree and returns the leaf index. - **get_size()**: Returns the current number of leaves in the tree. - **get_leaf(index)**: Returns the hash of the leaf at the specified 1-based index. - **get_state(size)**: Returns the root hash of the tree. If size is provided, returns the historical root hash for that tree size. ```