### SparseGraph Initialization and Usage Example Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/graph.html Demonstrates how to initialize a SparseGraph object and read graph data from an edge list file. ```python >>> from pecanpy.graph import SparseGraph >>> >>> # initialize SparseGraph object >>> g = SparseGraph() >>> >>> # read graph from edgelist >>> g.read_edg(path_to_edg_file, weighted=True, directed=False) >>> >>> # save the csr graph as npz file to be used later >>> g.save(npz_outpath) ``` -------------------------------- ### Alias Method Setup (alias_setup) Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Constructs the alias lookup table for efficient O(1) sampling from a discrete random distribution. This function is a Numba-jitted implementation based on the alias method algorithm. ```python @njit(nogil=True) def alias_setup(probs): """Construct alias lookup table. This code is modified from the blog post here: https://lips.cs.princeton.edu/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ , where you can find more details about how the method works. In general, the alias method improves the time complexity of sampling from a discrete random distribution to O(1) if the alias table is setup in advance. Args: probs (list(float32)): normalized transition probabilities array, could be in either list or NDArray, of float32 values. """ k = probs.size q = np.zeros(k, dtype=np.float32) j = np.zeros(k, dtype=np.uint32) smaller = np.zeros(k, dtype=np.uint32) larger = np.zeros(k, dtype=np.uint32) smaller_ptr = 0 larger_ptr = 0 for kk in range(k): q[kk] = k * probs[kk] if q[kk] < 1.0: smaller[smaller_ptr] = kk smaller_ptr += 1 else: larger[larger_ptr] = kk larger_ptr += 1 while (smaller_ptr > 0) & (larger_ptr > 0): smaller_ptr -= 1 small = smaller[smaller_ptr] larger_ptr -= 1 large = larger[larger_ptr] j[small] = large q[large] = q[large] + q[small] - 1.0 if q[large] < 1.0: smaller[smaller_ptr] = large smaller_ptr += 1 else: larger[larger_ptr] = large larger_ptr += 1 return j, q ``` -------------------------------- ### Setup Normalized Transition Probabilities Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Configures the transition probability computation based on whether node2vec+ extension is used. It returns the appropriate probability function and associated noise thresholds, or None if standard node2vec is employed. ```python def setup_get_normalized_probs(self): """Transition probability computation setup. This function performs necessary preprocessing of computing the average edge weights array, which is used later by the transition probability computation function ``get_extended_normalized_probs``, if node2vec+ is used. Otherwise, returns the normal transition function ``get_noramlized_probs`` with a trivial placeholder for average edge weights array ``noise_thresholds``. """ if self.extend: # use n2v+ get_normalized_probs = self.get_extended_normalized_probs noise_thresholds = self.get_noise_thresholds() else: # use normal n2v get_normalized_probs = self.get_normalized_probs noise_thresholds = None return get_normalized_probs, noise_thresholds ``` -------------------------------- ### Initialize DenseGraph Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/graph.html Initializes an empty DenseGraph object. This is the starting point for creating or loading dense graph data. ```python def __init__(self): super().__init__() self._data: Optional[AdjMat] = None ``` -------------------------------- ### simulate_walks Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Generates random walks starting from each node in the graph. ```APIDOC ## simulate_walks ### Description Generate walks starting from each nodes ``num_walks`` time. Note: This is the master process that spawns worker processes, where the worker function ``node2vec_walks`` genearte a single random walk starting from a vertex of the graph. ### Parameters - **num_walks** (int) - number of walks starting from each node. - **walk_length** (int) - length of walk. ``` -------------------------------- ### Get Move Forward Function (SparseOTF) Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Returns a Numba-compiled function for moving forward in a graph. This function calculates transition probabilities on-the-fly and is used by the simulate_walks method. ```python def get_move_forward(self): """Wrap ``move_forward``. This function returns a ``numba.njit`` compiled function that takes current vertex index (and the previous vertex index if available) and returns the next vertex index by sampling from a discrete random distribution based on the transition probabilities that are calculated on-the-fly. Note: The returned function is used by the ``simulate_walks`` method. """ data = self.data indices = self.indices indptr = self.indptr p = self.p ``` -------------------------------- ### Get All Graph Edges Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/graph.html Returns a list of all edges in the graph, represented as (head, tail, weight) tuples. ```python self.edges ``` -------------------------------- ### simulate_walks Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Generates node2vec random walks starting from each node a specified number of times and length. ```APIDOC ## simulate_walks ### Description Generate walks starting from each nodes `num_walks` time. ### Parameters * **num_walks** (int) - Number of walks starting from each node. * **walk_length** (int) - Length of walk. ``` -------------------------------- ### Simulate Random Walks Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Generates a matrix of random walks for graph embedding. It initializes walk parameters, shuffles starting nodes for balanced workload, and uses a progress bar to display simulation progress. The function returns a matrix where each row represents a walk. ```python self._preprocess_transition_probs() nodes = np.array(range(self.num_nodes), dtype=np.uint32) start_node_idx_ary = np.concatenate([nodes] * num_walks) tot_num_jobs = start_node_idx_ary.size random_state = self.random_state np.random.seed(random_state) np.random.shuffle(start_node_idx_ary) # for balanced work load move_forward = self.get_move_forward() has_nbrs = self.get_has_nbrs() verbose = self.verbose # Acquire numba progress proxy for displaying the progress bar with ProgressBar(total=tot_num_jobs, disable=not verbose) as progress: walk_idx_mat = self._random_walks( tot_num_jobs, walk_length, random_state, start_node_idx_ary, has_nbrs, move_forward, progress, ) # Map node index back to node ID walks = [self._map_walk(walk_idx_ary) for walk_idx_ary in walk_idx_mat] return walks ``` -------------------------------- ### Simulate Random Walks with PecanPy Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Generates random walks starting from each node in the graph multiple times. This method orchestrates parallel worker processes for walk generation. ```python def simulate_walks( self, num_walks: int, walk_length: int, ) -> List[List[str]]: """Generate walks starting from each nodes ``num_walks`` time. Note: This is the master process that spawns worker processes, where the worker function ``node2vec_walks`` genearte a single random walk starting from a vertex of the graph. Args: num_walks (int): number of walks starting from each node. walks_length (int): length of walk. """ ``` -------------------------------- ### setup_get_normalized_probs Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Sets up transition probability computation, handling preprocessing for node2vec+ or returning normal transition functions. ```APIDOC ## setup_get_normalized_probs ### Description Transition probability computation setup. This function performs necessary preprocessing of computing the average edge weights array, which is used later by the transition probability computation function `get_extended_normalized_probs`, if node2vec+ is used. Otherwise, returns the normal transition function `get_noramlized_probs` with a trivial placeholder for average edge weights array `noise_thresholds`. ``` -------------------------------- ### Get Number of Edges Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/graph.html Returns the total count of edges present in the graph. ```python self.num_edges ``` -------------------------------- ### alias_setup Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Constructs an alias lookup table for efficient O(1) sampling from a discrete random distribution. ```APIDOC ## Function: alias_setup ### Description Construct alias lookup table. This code is modified from the blog post here: https://lips.cs.princeton.edu/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ , where you can find more details about how the method works. In general, the alias method improves the time complexity of sampling from a discrete random distribution to O(1) if the alias table is setup in advance. ### Parameters - **probs** (_list_ _(__float32_ _)_) – normalized transition probabilities array, could be in either list or NDArray, of float32 values. ``` -------------------------------- ### Display PecanPy Help Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Execute this command to view the full list of available parameters for the PecanPy command-line utility. ```bash $ pecanpy --help ``` -------------------------------- ### PreCompFirstOrder Initialization Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Initializes the PreCompFirstOrder class, designed for precomputing transition probabilities for first-order random walks. ```APIDOC ## PreCompFirstOrder.__init__ ### Description Initializes the PreCompFirstOrder class for precomputing transition probabilities for first-order random walks. ### Parameters - ***args: Variable length argument list. - ****kwargs: Arbitrary keyword arguments. ``` -------------------------------- ### Get Node Index or Create Node Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/graph.html Retrieves the index for a given node ID. If the node does not exist, it is added to the graph first. ```python self.get_node_idx(node_id: str) -> int: """Get index of the node and create new node when necessary.""" self.add_node(node_id) return self._node_idmap[node_id] ``` -------------------------------- ### PreComp Initialization Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Initializes the PreComp class, which precomputes second-order transition probabilities for random walks. It uses the SparseRWGraph structure. ```APIDOC ## PreComp.__init__ ### Description Initializes the PreComp class for precomputing second-order transition probabilities. This class uses ``SparseRWGraph`` and requires calling ``preprocess_transition_probs()`` before generating walks. ### Parameters - ***args: Variable length argument list. - ****kwargs: Arbitrary keyword arguments. ``` -------------------------------- ### Dense Graph Transition On-the-Fly (move_forward) Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Implements a 'move_forward' function for dense graphs that calculates transition probabilities on-the-fly during random walks. It uses Numba for JIT compilation and alias sampling for efficient selection of the next node. ```python q = self.q get_normalized_probs, noise_thresholds = self.setup_get_normalized_probs() @njit(nogil=True) def move_forward(cur_idx, prev_idx=None): """Move to next node.""" normalized_probs = get_normalized_probs( data, indices, indptr, p, q, cur_idx, prev_idx, noise_thresholds, ) cdf = np.cumsum(normalized_probs) choice = np.searchsorted(cdf, np.random.random()) return indices[indptr[cur_idx] + choice] ``` -------------------------------- ### Sparse Graph Transition On-the-Fly (PecanPy) Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Implements sparse graph transitions by calculating probabilities on-the-fly during random walks, rather than precomputing them. Uses SparseRWGraph. ```python class SparseOTF(Base, SparseRWGraph): """Sparse graph transition on the fly. This implementation does *NOT* precompute transition probabilities in advance but instead calculates them on-the-fly during the process of random walk. The graph type used is ``SparseRWGraph``. """ def __init__(self, *args, **kwargs): Base.__init__(self, *args, **kwargs) ``` -------------------------------- ### alias_setup Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Constructs the alias lookup table for efficient O(1) sampling from a discrete random distribution. This is a utility function used internally for probability sampling. ```APIDOC ## alias_setup ### Description Constructs the alias lookup table for efficient O(1) sampling from a discrete random distribution. This is a utility function used internally for probability sampling. ### Method Signature `alias_setup(probs)` ### Parameters #### Arguments - **probs** (list(float32) or NDArray) - Normalized transition probabilities array. ### Returns - **j** (numpy.ndarray) - Alias table. - **q** (numpy.ndarray) - Probability distribution table. ``` -------------------------------- ### FirstOrderUnweighted Class Initialization Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Initializes the FirstOrderUnweighted class for direct edge sampling for first-order random walks. ```APIDOC ## FirstOrderUnweighted Class ### Description Directly sample edges for first order random walks. ``` -------------------------------- ### Run PecanPy in Command Line Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Use this command to run PecanPy in the command line with the PreComp mode for embedding a network. Specify input, output, and mode. ```bash $ pecanpy --input demo/karate.edg --ouptut demo/karate.emb --mode PreComp ``` -------------------------------- ### PreCompFirstOrder.preprocess_transition_probs Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Precomputes and stores the first-order transition probabilities using the alias method for efficient sampling during random walks. ```APIDOC ## PreCompFirstOrder.preprocess_transition_probs ### Description Precomputes and stores the first-order transition probabilities for all nodes in the graph. This method utilizes the alias method for efficient sampling during random walk generation. ### Return - **alias_j** (numpy.ndarray): Alias table for indices. - **alias_q** (numpy.ndarray): Alias table for probabilities. ``` -------------------------------- ### Create and Read SparseGraph from Edg File Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Initialize a SparseGraph object and read graph data from an edge list file. The graph can be weighted and directed. The saved graph can be used later. ```python from pecanpy.graph import SparseGraph # initialize SparseGraph object g = SparseGraph() # read graph from edgelist g.read_edg(path_to_edg_file, weighted=True, directed=False) # save the csr graph as npz file to be used later g.save(npz_outpath) ``` -------------------------------- ### pecanpy.cli.main() Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html The main pipeline for representational learning for all nodes in a graph. ```APIDOC ## pecanpy.cli.main() ### Description Pipeline for representational learning for all nodes in a graph. ### Method (Not specified in source, likely a Python function call) ### Endpoint (Not applicable, this is a Python function) ### Parameters (Parameters not explicitly detailed in the source) ### Request Example (Not applicable) ### Response (Response details not explicitly detailed in the source) ``` -------------------------------- ### Initialize and Use PecanPy for Node2vec Embeddings Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Demonstrates initializing a PecanPy node2vec object, loading a graph, preprocessing transition probabilities, and generating random walks or embeddings directly. Use this for generating node2vec embeddings from an edgelist file. ```python from pecanpy import pecanpy as node2vec # initialize node2vec object, similarly for SparseOTF and DenseOTF g = node2vec.PreComp(p=0.5, q=1, workers=4, verbose=True) # alternatively, can specify ``extend=True`` for using node2vec+ # load graph from edgelist file g.read_edg(path_to_edg_file, weighted=True, directed=False) # precompute and save 2nd order transition probs (for PreComp only) g.preprocess_transition_probs() # generate random walks, which could then be used to train w2v walks = g.simulate_walks(num_walks=10, walk_length=80) # alternatively, generate the embeddings directly using ``embed`` emd = g.embed() ``` -------------------------------- ### Base Class Initialization Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Initializes the base node2vec object with parameters controlling walk behavior and parallelization. ```APIDOC ## __init__ ### Description Initializes the base node2vec object with parameters controlling walk behavior and parallelization. ### Parameters - **p** (float) - Optional - return parameter, value less than 1 encourages returning back to previous vertex, and discourage for value grater than 1 (default: 1). - **q** (float) - Optional - in-out parameter, value less than 1 encourages walks to go "outward", and value greater than 1 encourage walking within a localized neighborhood (default: 1). - **workers** (int) - Optional - number of threads to be spawned for running node2vec including walk generation and word2vec embedding (default: 1). - **verbose** (bool) - Optional - show progress bar for walk generation. - **extend** (bool) - Optional - use node2vec+ extension if set to :obj:`True` (default: :obj:`False`). - **gamma** (float) - Optional - Multiplication factor for the std term of edge weights added to the average edge weights as the noisy edge threshold, only used by node2vec+ (default: 0). - **random_state** (int, optional) - Optional - Random seed for generating random walks. Note that to fully ensure reproducibility, use single thread (i.e., workers=1), and potentially need to set the Python environment variable ``PYTHONHASHSEED`` to match the random_state (default: :obj:`None`). ``` -------------------------------- ### PreCompFirstOrder Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Implements precomputation of transition probabilities for first-order random walks. It inherits from Base and SparseRWGraph. ```APIDOC ## Class: PreCompFirstOrder ### Description Precompute transition probabilities for first-order random walks. ### Methods - `get_move_forward()`: Wraps `move_forward`. - `preprocess_transition_probs()`: Precomputes and stores first-order transition probabilities. ``` -------------------------------- ### pecanpy.cli.check_mode() Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Checks and recommends the appropriate PecanPy mode based on graph size and density. ```APIDOC ## pecanpy.cli.check_mode(_g_, _args_) ### Description Check mode selection. Give recommendation to user for pecanpy mode based on graph size and density. ### Method (Not specified in source, likely a Python function call) ### Endpoint (Not applicable, this is a Python function) ### Parameters - **_g_** (type not specified) - Description: Graph object. - **_args_** (type not specified) - Description: Parsed arguments. ### Request Example (Not applicable) ### Response (Response details not explicitly detailed in the source) ``` -------------------------------- ### FirstOrderUnweighted Initialization Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Initializes the FirstOrderUnweighted class, which is used for directly sampling edges for first-order random walks. ```APIDOC ## FirstOrderUnweighted.__init__ ### Description Initializes the FirstOrderUnweighted class for direct edge sampling in first-order random walks. ### Parameters - ***args: Variable length argument list. - ****kwargs: Arbitrary keyword arguments. ``` -------------------------------- ### Create DenseGraph from Edg File Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Initialize a DenseGraph object and read graph data from an edge list file. The graph can be weighted and directed. The saved graph can be used later. ```python from pecanpy.graph import DenseGraph # initialize DenseGraph object g = DenseGraph() # read graph from edgelist g.read_edg(path_to_edg_file, weighted=True, directed=False) # save the dense graph as npz file to be used later g.save(npz_outpath) ``` -------------------------------- ### PreComp Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Precomputes and stores second-order transition probabilities for random walks on sparse graphs. Requires `preprocess_transition_probs()` to be called first. ```APIDOC ## Class: PreComp ### Description Precompute transition probabilities. This implementation precomputes and stores 2nd order transition probabilities first and uses read off transition probabilities during the process of random walk. The graph type used is `SparseRWGraph`. ### Note Need to call `preprocess_transition_probs()` first before generating walks. ### Methods - `get_move_forward()`: Wraps `move_forward`. Returns a `numba.njit` compiled function for sampling the next vertex. - `preprocess_transition_probs()`: Precomputes and stores 2nd order transition probabilities. Uses uint64 for `alias_indptr` to prevent overflow. ``` -------------------------------- ### Create DenseGraph from Npz File Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Initialize a DenseGraph object and read graph data from a .npz file. The graph can be weighted and directed. ```python from pecanpy.graph import DenseGraph g = DenseGraph() # initialize DenseGraph object g.read_npz(paht_to_npz_file, weighted=True, directed=False) ``` -------------------------------- ### Preprocess Transition Probabilities Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/cli.html Precomputes and stores transition probabilities for the graph, which is a necessary step before simulating random walks. This function is timed for performance monitoring. ```python @Timer("pre-compute transition probabilities") def preprocess(g): """Preprocessing transition probabilities with timer.""" g.preprocess_transition_probs() ``` -------------------------------- ### Precompute Second-Order Transition Probabilities (PecanPy) Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Precomputes and stores second-order transition probabilities for efficient random walks. It sets up alias tables for sampling next nodes based on the current and previous nodes. ```python def preprocess_transition_probs(self): """Precompute and store 2nd order transition probabilities. Each node contains n ** 2 number of 2nd order transition probabilities, where n is the number of neighbors of that specific node, since one can pick any one of its neighbors as the previous node and / or the next node. For each second order transition probability of a node, set up the alias draw table to be used during random walk. Note: Uses uint64 instead of uint32 for tracking alias_indptr to prevent overflowing since the 2nd order transition probs grows much faster than the first order transition probs, which is the same as the total number of edges in the graph. """ data = self.data indices = self.indices indptr = self.indptr p = self.p q = self.q # Retrieve transition probability computation callback function get_normalized_probs, noise_thresholds = self.setup_get_normalized_probs() # Determine the dimensionality of the 2nd order transition probs n_nodes = self.indptr.size - 1 # number of nodes n = self.indptr[1:] - self.indptr[:-1] # number of nbrs per node n2 = np.power(n, 2) # number of 2nd order trans probs per node # Set the dimensionality of alias probability table self.alias_dim = alias_dim = n self.alias_indptr = alias_indptr = np.zeros(self.indptr.size, dtype=np.uint64) alias_indptr[1:] = np.cumsum(n2) n_probs = alias_indptr[-1] # total number of 2nd order transition probs @njit(parallel=True, nogil=True) def compute_all_transition_probs(): alias_j = np.zeros(n_probs, dtype=np.uint32) alias_q = np.zeros(n_probs, dtype=np.float32) for idx in range(n_nodes): offset = alias_indptr[idx] dim = alias_dim[idx] nbrs = indices[indptr[idx] : indptr[idx + 1]] for nbr_idx in prange(n[idx]): nbr = nbrs[nbr_idx] probs = get_normalized_probs( data, indices, indptr, p, q, idx, nbr, noise_thresholds, ) start = offset + dim * nbr_idx end = start + dim alias_j[start:end], alias_q[start:end] = alias_setup(probs) return alias_j, alias_q self.alias_j, self.alias_q = compute_all_transition_probs() ``` -------------------------------- ### Base Class Initialization Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Initializes the base node2vec object with various parameters controlling walk generation and embedding process. ```APIDOC ## Base Class ### Description Base node2vec object that provides the skeleton for the node2vec walk algorithm. ### Parameters * **p** (float) - Return parameter, value less than 1 encourages returning back to previous vertex, and discourage for value greater than 1 (default: 1). * **q** (float) - In-out parameter, value less than 1 encourages walks to go “outward”, and value greater than 1 encourage walking within a localized neighborhood (default: 1). * **workers** (int) - Number of threads to be spawned for running node2vec including walk generation and word2vec embedding (default: 1). * **verbose** (bool) - Show progress bar for walk generation. * **extend** (bool) - Use node2vec+ extension if set to `True` (default: `False`). * **gamma** (float) - Multiplication factor for the std term of edge weights added to the average edge weights as the noisy edge threshold, only used by node2vec+ (default: 0). * **random_state** (int | None) - Random seed for generating random walks. Note that to fully ensure reproducibility, use single thread (i.e., workers=1), and potentially need to set the Python environment variable `PYTHONHASHSEED` to match the random_state (default: `None`). ``` -------------------------------- ### preprocess_transition_probs Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Default null preprocess method for transition probabilities. ```APIDOC ## preprocess_transition_probs ### Description Null default preprocess method. ``` -------------------------------- ### Default Preprocess Transition Probabilities Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html A null default method for preprocessing transition probabilities. It serves as a placeholder and can be overridden by subclasses for specific preprocessing logic. ```python def preprocess_transition_probs(self): """Null default preprocess method.""" pass ``` -------------------------------- ### Initialize and Read Edglist File with AdjlstGraph Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Use AdjlstGraph to initialize a graph object and read data from an edgelist file. Supports weighted and directed graphs. The graph can then be converted to CSR format or a dense adjacency matrix. ```python from pecanpy.graph import AdjlstGraph # initialize SparseGraph object g = AdjlstGraph() # read graph from edgelist g.read(path_to_edg_file, weighted=True, directed=False) indptr, indices, data = g.to_csr() # convert to csr dense_mat = g.to_dense() # convert to dense adjacency matrix g.save(edg_outpath) # save the graph to an edge list file ``` -------------------------------- ### pecanpy.cli.parse_args() Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Parses command-line arguments for the Node2vec functionality. ```APIDOC ## pecanpy.cli.parse_args() ### Description Parse node2vec arguments. ### Method (Not specified in source, likely a Python function call) ### Endpoint (Not applicable, this is a Python function) ### Parameters (Parameters not explicitly detailed in the source) ### Request Example (Not applicable) ### Response (Response details not explicitly detailed in the source) ``` -------------------------------- ### Precompute First Order Transition Probabilities Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Precomputes and stores first-order transition probabilities using alias sampling. This function is decorated with numba.njit for parallel execution. ```python @njit(parallel=True, nogil=True) def compute_all_transition_probs(): alias_j = np.zeros(n_probs, dtype=np.uint32) alias_q = np.zeros(n_probs, dtype=np.float32) for idx in range(n_nodes): start, end = indptr[idx], indptr[idx + 1] probs = get_normalized_probs(data, indices, indptr, idx) alias_j[start:end], alias_q[start:end] = alias_setup(probs) return alias_j, alias_q self.alias_j, self.alias_q = compute_all_transition_probs() ``` -------------------------------- ### PreCompFirstOrder.get_move_forward Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Returns a compiled function for moving forward using precomputed transition probabilities. This function samples the next node based on alias method sampling. ```APIDOC ## PreCompFirstOrder.get_move_forward ### Description Wraps the internal ``move_forward`` function, providing a compiled method for sampling the next node in a first-order random walk using precomputed transition probabilities via the alias method. ### Return - **move_forward** (function): A numba.njit compiled function that takes the current node index and returns the next node index. ``` -------------------------------- ### SparseGraph.save Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Save AdjLst as an .edg edge list file. ```APIDOC ## SparseGraph.save ### Description Save AdjLst as an .edg edge list file. ### Method `save(_path : str_, _unweighted : bool = False_, _delimiter : str = '\t'_) ### Parameters #### Path Parameters - **_path** (str) - Required - Path to save the file. - **_unweighted** (bool) - Optional - If set to True, only write two columns, corresponding to the head and tail nodes of the edges, and ignore the edge weights (default: `False`). - **_delimiter** (str) - Optional - Delimiter for separating fields (default: '\t'). ``` -------------------------------- ### DenseOTF Class: get_move_forward Method Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Provides a Numba-compiled 'move_forward' function for the DenseOTF class, calculating transition probabilities on-the-fly. This function is utilized by the 'simulate_walks' method. ```python def get_move_forward(self): """Wrap ``move_forward``. This function returns a ``numba.njit`` compiled function that takes current vertex index (and the previous vertex index if available) and returns the next vertex index by sampling from a discrete random distribution based on the transition probabilities that are calculated on-the-fly. Note: The returned function is used by the ``simulate_walks`` method. """ data = self.data nonzero = self.nonzero p = self.p q = self.q get_normalized_probs, noise_thresholds = self.setup_get_normalized_probs() @njit(nogil=True) def move_forward(cur_idx, prev_idx=None): """Move to next node.""" normalized_probs = get_normalized_probs( data, nonzero, p, q, cur_idx, prev_idx, noise_thresholds, ) cdf = np.cumsum(normalized_probs) choice = np.searchsorted(cdf, np.random.random()) nbrs = np.where(nonzero[cur_idx])[0] return nbrs[choice] return move_forward ``` -------------------------------- ### Check Graph Mode and Density Recommendations Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/cli.html Validates the selected mode against graph density and size, issuing warnings for potentially suboptimal choices. This function helps users select the most efficient mode for their graph characteristics. ```python if mode == "PreCompFirstOrder": if not p == q == 1: raise ValueError( f"PreCompFirstOrder only works when p = q = 1, got {p=}, {q=}", ) return if mode != "PreCompFirstOrder" and p == 1 == q: warnings.warn( "When p = 1 and q = 1, it is highly recommended to use " f"PreCompFirstOrder over {mode} (current selection). The runtime " "could be improved greatly with low memory usage.", stacklevel=2, ) return # Check network density and recommend appropriate mode g_size = g.num_nodes g_dens = g.density if (g_dens >= 0.2) & (mode != "DenseOTF"): warnings.warn( f"Network density = {g_dens:.3f} (> 0.2), it is recommended to use " f"DenseOTF over {mode} (current selection)", stacklevel=2, ) if (g_dens < 0.001) & (g_size < 10000) & (mode != "PreComp"): warnings.warn( f"Network density = {g_dens:.2e} (< 0.001) with {g_size} nodes " f"(< 10000), it is recommended to use PreComp over {mode} (current " "selection)", stacklevel=2, ) if (g_dens >= 0.001) & (g_dens < 0.2) & (mode != "SparseOTF"): warnings.warn( f"Network density = {g_dens:.3f}, it is recommended to use " f"SparseOTF over {mode} (current selection)", stacklevel=2, ) if (g_dens < 0.001) & (g_size >= 10000) & (mode != "SparseOTF"): warnings.warn( f"Network density = {g_dens:.3f} (< 0.001) with {g_size} nodes " f"(>= 10000), it is recommended to use SparseOTF over {mode} " "(current selection)", stacklevel=2, ) ``` -------------------------------- ### DenseGraph.read_edg Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Read .edg files and create DenseGraph object using read_edg. ```APIDOC ## DenseGraph.read_edg ### Description Read `.edg` files and create `DenseGraph` object using `read_edg`. ### Method `read_edg(path_to_edg_file, weighted=True, directed=False)` ``` -------------------------------- ### Simulate Random Walks Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/cli.html Generates random walks on the graph to be used as input for the embedding learning process. The simulation is timed for performance analysis. ```python @Timer("generate walks") def simulate_walks(args, g): """Simulate random walks with timer.""" return g.simulate_walks(args.num_walks, args.walk_length) ``` -------------------------------- ### SparseOTF Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Calculates transition probabilities on-the-fly during random walks for sparse graphs. Inherits from Base and SparseRWGraph. ```APIDOC ## Class: SparseOTF ### Description Sparse graph transition on the fly. This implementation does _NOT_ precompute transition probabilities in advance but instead calculates them on-the-fly during the process of random walk. The graph type used is `SparseRWGraph`. ### Methods - `get_move_forward()`: Wraps `move_forward`. Returns a `numba.njit` compiled function for sampling the next vertex based on on-the-fly calculated probabilities. ``` -------------------------------- ### Pecanpy Main Pipeline Execution Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/cli.html Orchestrates the entire representational learning pipeline for graph nodes. It parses arguments, reads the graph, preprocesses it, simulates walks, and learns embeddings. ```python def main(): """Pipeline for representational learning for all nodes in a graph.""" args = parse_args() if args.workers == 0: args.workers = numba.config.NUMBA_DEFAULT_NUM_THREADS numba.set_num_threads(args.workers) g = read_graph(args) preprocess(g) walks = simulate_walks(args, g) learn_embeddings(args, walks) if __name__ == "__main__": main() ``` -------------------------------- ### Numba Parallel Random Walks Simulation Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Efficiently simulates random walks using Numba for parallel execution. It seeds the random number generator for each thread and iteratively determines the next node in the walk based on neighbor availability and transition probabilities. The effective walk length is tracked. ```python @staticmethod @njit(parallel=True, nogil=True) def _random_walks( tot_num_jobs: int, walk_length: int, random_state: Optional[int], start_node_idx_ary: Uint32Array, has_nbrs: HasNbrs, move_forward: MoveForward, progress_proxy: ProgressBar, ) -> Uint32Array: """Simulate a random walk starting from start node.""" # Seed the random number generator if random_state is not None: np.random.seed(random_state + get_thread_id()) # use the last entry of each walk index array to keep track of the # effective walk length walk_idx_mat: Uint32Array = np.zeros( (tot_num_jobs, walk_length + 2), dtype=np.uint32, ) walk_idx_mat[:, 0] = start_node_idx_ary # initialize seeds walk_idx_mat[:, -1] = walk_length + 1 # set to full walk length by default for i in prange(tot_num_jobs): # initialize first step as normal random walk start_node_idx = walk_idx_mat[i, 0] if has_nbrs(start_node_idx): walk_idx_mat[i, 1] = move_forward(start_node_idx) else: walk_idx_mat[i, -1] = 1 continue # start bias random walk for j in range(2, walk_length + 1): cur_idx = walk_idx_mat[i, j - 1] if has_nbrs(cur_idx): prev_idx = walk_idx_mat[i, j - 2] walk_idx_mat[i, j] = move_forward(cur_idx, prev_idx) else: walk_idx_mat[i, -1] = j break progress_proxy.update(1) return walk_idx_mat ``` -------------------------------- ### PreCompFirstOrder Random Walk Move Forward Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Implements the 'move_forward' function for precomputed first-order random walks using alias sampling. This function is decorated with numba.njit for performance. ```python @njit(nogil=True) def move_forward(cur_idx, prev_idx=None): start, end = indptr[cur_idx], indptr[cur_idx + 1] choice = alias_draw(alias_j[start:end], alias_q[start:end]) return indices[indptr[cur_idx] + choice] return move_forward ``` -------------------------------- ### save Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Saves the current dense graph to a .dense.npz file. ```APIDOC ## save ### Description Save dense graph as a .dense.npz file. ### Parameters #### Path Parameters - **path** (str) - Required - The path where the graph will be saved. ``` -------------------------------- ### DenseOTF.get_move_forward Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Returns a Numba-compiled function that simulates a single step in a random walk. This function calculates transition probabilities on-the-fly and samples the next node based on these probabilities. ```APIDOC ## DenseOTF.get_move_forward ### Description Returns a Numba-compiled function that simulates a single step in a random walk. This function calculates transition probabilities on-the-fly and samples the next node based on these probabilities. ### Method Signature `get_move_forward()` ### Returns A Numba-compiled function (`move_forward`) that takes `cur_idx` (current node index) and optionally `prev_idx` (previous node index) and returns the index of the next node. ``` -------------------------------- ### from_mat Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html A class method to construct a dense graph using an adjacency matrix and a list of node IDs. ```APIDOC ## from_mat ### Description Construct dense graph using adjacency matrix and node IDs. ### Parameters #### Path Parameters - **adj_mat** (NDArray) - Required - 2D numpy array of adjacency matrix. - **node_ids** (list of str) - Required - Node ID list. - **kwargs** - Optional - Additional keyword arguments. ``` -------------------------------- ### PreComp.get_move_forward Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Returns a compiled function for moving forward in the graph using precomputed second-order transition probabilities. This function is used internally by the simulate_walks method. ```APIDOC ## PreComp.get_move_forward ### Description Wraps the internal ``move_forward`` function, providing a compiled method for sampling the next node in a second-order random walk using precomputed transition probabilities. This function is utilized by the ``simulate_walks`` method. ### Return - **move_forward** (function): A numba.njit compiled function that takes the current node index and the previous node index, returning the next node index. ``` -------------------------------- ### FirstOrderUnweighted Random Walk Move Forward Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Implements the 'move_forward' function for first-order unweighted random walks. This function is decorated with numba.njit for performance. ```python @njit(nogil=True) def move_forward(cur_idx, prev_idx=None): start, end = indptr[cur_idx], indptr[cur_idx + 1] return indices[np.random.randint(start, end)] return move_forward ``` -------------------------------- ### BaseGraph Initialization Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/graph.html Initializes a BaseGraph object, setting up internal structures for node IDs and their mappings. This class serves as a foundation for derived graph types. ```python class BaseGraph: """Base Graph object. Handles node id and provides general properties including num_nodes, and density. The num_edges property is to be specified by the derived graph objects. """ def __init__(self): self._node_ids: List[str] = [] self._node_idmap: Dict[str, int] = {} # id -> index ``` -------------------------------- ### Move Forward in Graph Traversal (PecanPy) Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Calculates the next node index during a random walk. It handles both the initial step (without a previous node) and subsequent steps (using the previous node to determine transition probabilities). ```python def move_forward(cur_idx, prev_idx=None): """Move to next node based on transition probabilities.""" if prev_idx is None: normalized_probs = get_normalized_probs( data, indices, indptr, p, q, cur_idx, None, None, ) cdf = np.cumsum(normalized_probs) choice = np.searchsorted(cdf, np.random.random()) else: # Find index of neighbor (previous node) for reading alias start = indptr[cur_idx] end = indptr[cur_idx + 1] nbr_idx = np.searchsorted(indices[start:end], prev_idx) if indices[start + nbr_idx] != prev_idx: print("FATAL ERROR! Neighbor not found.") dim = alias_dim[cur_idx] start = alias_indptr[cur_idx] + dim * nbr_idx end = start + dim choice = alias_draw(alias_j[start:end], alias_q[start:end]) return indices[indptr[cur_idx] + choice] ``` -------------------------------- ### DenseOTF Source: https://pecanpy.readthedocs.io/en/latest/pecanpy.html Calculates transition probabilities on-the-fly during random walks for dense graphs. Inherits from Base and DenseRWGraph. ```APIDOC ## Class: DenseOTF ### Description Dense graph transition on the fly. This implementation does _NOT_ precompute transition probabilities in advance but instead calculates them on-the-fly during the process of random walk. The graph type used is `DenseRWGraph`. ### Methods - `get_move_forward()`: Wraps `move_forward`. Returns a `numba.njit` compiled function for sampling the next vertex based on on-the-fly calculated probabilities. ``` -------------------------------- ### Create Graph from Adjacency Matrix - `from_mat` Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/graph.html Constructs a graph object from a given adjacency matrix and a list of node IDs. ```python @classmethod def from_mat(cls, adj_mat: AdjMat, node_ids: List[str], **kwargs): """Construct graph using adjacency matrix and node IDs. Args: adj_mat(NDArray): 2D numpy array of adjacency matrix node_ids(:obj:`list` of str): node ID list Return: An adjacency graph object representing the adjacency matrix. """ g = cls(**kwargs) # Setup node idmap in the order of node_ids for node_id in node_ids: g.add_node(node_id) # Fill in edge data for idx1, idx2 in zip(*np.where(adj_mat != 0)): g._add_edge_from_idx(idx1, idx2, adj_mat[idx1, idx2]) return g ``` -------------------------------- ### PreComp Second Order Random Walk Move Forward Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/pecanpy.html Implements the 'move_forward' function for precomputed second-order random walks using alias sampling. This function is decorated with numba.njit for performance. ```python @njit(nogil=True) ``` -------------------------------- ### Graph Class Methods Source: https://pecanpy.readthedocs.io/en/latest/_modules/pecanpy/graph.html This section details the methods available for interacting with graph objects, including adding nodes and edges, and reading graph data from files. ```APIDOC ## get_node_idx(node_id: str) -> int ### Description Get index of the node and create new node when necessary. ### Method get_node_idx ### Parameters #### Path Parameters - **node_id** (str) - Description: The ID of the node to retrieve the index for. ### Returns - **int**: The index of the node. ``` ```APIDOC ## add_node(node_id: str) ### Description Create a new node. Add a new node to the graph if not already existing, by updating the ID list, ID map, and the adjacency list data. Otherwise pass through without further actions. Note: Does not raise error even if the node alrealy exists. ### Method add_node ### Parameters #### Path Parameters - **node_id** (str) - Description: The ID of the node to add. ``` ```APIDOC ## add_edge(id1: str, id2: str, weight: float = 1.0, directed: bool = False) ### Description Add an edge to the graph. Non-positive edges are ignored. ### Method add_edge ### Parameters #### Path Parameters - **id1** (str) - Description: First node id. - **id2** (str) - Description: Second node id. - **weight** (float) - Optional - Description: The edge weight, default is 1.0. - **directed** (bool) - Optional - Description: Whether the edge is directed or not. ``` ```APIDOC ## read(path: str, weighted: bool, directed: bool, delimiter: str = "\t") ### Description Read an edgelist file and create sparse graph. Implicitly discard zero weighted edges; if the same edge is defined multiple times with different edge weights, then the last specified weight will be used (warning for such behavior will be printed). ### Method read ### Parameters #### Path Parameters - **path** (str) - Description: Path to edgelist file, where the file is tab separated and contains 2 or 3 columns depending on whether the input graph is weighted, where the the first column contains the source nodes and the second column contains the - **weighted** (bool) - Description: Indicates if the graph is weighted. - **directed** (bool) - Description: Indicates if the graph is directed. - **delimiter** (str) - Optional - Description: The delimiter used in the edgelist file, defaults to tab ('\t'). ```