### Basic Graph Traversal Example

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

A simple example demonstrating how to retrieve all outgoing edges from a specific vertex in the graph.

```python
g.V(vids).out().toArray()
```

--------------------------------

### Subgraph Creation and Traversal

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Demonstrates creating a subgraph using `subgraph_coo` and performing graph traversals with `pybitgraph` and `pygremlinxx`. It includes examples of vertex and edge operations.

```python
from pybitgraph import BitGraph
import torch

graph = BitGraph(
    'uint64',
    'uint64',
    'DEVICE',
    'MANAGED',
    'DEVICE',
)

src = torch.tensor([5, 4, 1, 0, 2, 3, 5, 1, 2, 0], dtype=torch.uint64)
dst = torch.tensor([1, 3, 2, 5, 1, 5, 4, 4, 4, 1], dtype=torch.uint64)

graph.add_vertices(6)
graph.add_edges(src, dst, 'e')

g = graph.traversal()

print(g.E().toArray())

print(graph.subgraph_coo(torch.tensor([0, 2, 4], dtype=torch.uint64)))

print(g.V(2).bothE().toArray())

from pygremlinxx import GraphTraversal
__ = lambda : GraphTraversal()

g.V([0, ]).bothE().dedup()._as('h0').inV().bothE().dedup()._union([__().select('h0'), __().identity()]).dedup().toArray()
```

--------------------------------

### Install BitGraph

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/README.md

Installs BitGraph by building its C++ libraries and Python extensions. Requires Maelstrom and Gremlin++ to be cloned in the same directory.

```shell
#!/bin/bash

# Clone Maelstrom and Gremlin++ into the same directory as BitGraph
# Example: /opt/code/bitgraph, /opt/code/maelstrom, /opt/code/gremlin++

# Build BitGraph (assuming build.sh handles C++ libraries and Python extensions)
./build.sh
```

--------------------------------

### GRetriever Model Initialization

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Initializes a `GRetriever` model, which combines a Graph Neural Network (GNN) with a Large Language Model (LLM). This setup is suitable for tasks requiring both graph structure understanding and natural language processing.

```python
from torch_geometric.nn import GRetriever, GAT
from torch_geometric.nn.nlp import LLM

llm = LLM(
    model_name='TinyLlama/TinyLlama-1.1B-Chat-v0.1',
    num_params=1,
)

gnn = GAT(
    in_channels=300,
    hidden_channels=256,
    out_channels=300,
    num_layers=4,
    heads=4,
)

model = GRetriever(llm=llm, gnn=gnn, mlp_out_channels=2048)
```

--------------------------------

### Named Entity Recognition (NER) Setup

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Initializes a tokenizer and a token classification model for Named Entity Recognition using the transformers library. The `extract` function is defined to process the NER output.

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

def extract(entsList):
    words = []
    for ents in entsList:
        row = []
        for ent in ents:
            row.append(ent['word'])
        words.append(row)
    return words

tokenizer = AutoTokenizer.from_pretrained("dslim/bert-large-NER")
model = AutoModelForTokenClassification.from_pretrained("dslim/bert-large-NER")

ner = pipeline("ner", model=model, tokenizer=tokenizer, device=0, aggregation_strategy="max")
```

--------------------------------

### Displaying DataFrame

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Prints the entire content of the pandas DataFrame 'df'.

```python
df
```

--------------------------------

### Graph Query Parameters

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Defines a dictionary of parameters for controlling graph query behavior, including limits for vertex matching and hops.

```python
qp = {"question_vertex_match_limit": 1, "hop_1_outgoing_limit": 8, "hop_1_incoming_limit": 8, "hop_0_outgoing_limit": 2, "hop_0_incoming_limit": 2, "entity_vertex_match_limit": 2}
```

--------------------------------

### Graph Traversal with Specific Entity and Decoding

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Performs a graph traversal starting from the entity 'Shameless' with a limit of 4, and then decodes the results.

```python
vids = g.V().like('emb', [getem('Shameless')], 4).toArray()
decode(vids)
```

--------------------------------

### Querying Data

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Executes a query using a 'query' function with a specific question from the DataFrame.

```python
query(truth_df.question.iloc[167453])
```

--------------------------------

### Get Word2Vec Embeddings

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Generates embeddings for a given text using a pre-trained Word2Vec model.

```python
def getem_w2v(model, text):
    return model(text)
```

--------------------------------

### Initialize RMM with PyTorch and CuPy Allocators

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

This code snippet demonstrates the initialization of the RMM (RAPIDS Memory Manager) library. It configures RMM to use specific allocators for PyTorch and CuPy, enabling efficient GPU memory management for deep learning and data science tasks. This setup is crucial for projects like Bitgraph that heavily rely on GPU acceleration.

```python
import argparse
import sys, os
import re
import warnings

import numpy as np

import rmm
from rmm.allocators.torch import rmm_torch_allocator
from rmm.allocators.cupy import rmm_cupy_allocator

# Example of how you might initialize RMM (actual initialization might be elsewhere in the project)
# rmm.reinitialize(memory_pool_size=None, # Use default or specify
#                    allocator=rmm_torch_allocator,
#                    device_buffer_allocator=rmm_cupy_allocator)

# print("RMM initialized with PyTorch and CuPy allocators.")
```

--------------------------------

### Graph Traversal Initialization

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Initializes the GraphTraversal object using a lambda function for concise graph traversal initiation.

```python
from pygremlinxx import GraphTraversal
__ = lambda : GraphTraversal()
```

--------------------------------

### Get RoBERTa Embeddings

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Generates embeddings for a given text using a RoBERTa model and tokenizer. It handles tokenization and ensures the input sequence length does not exceed 512 tokens by truncating if necessary.

```python
from transformers import AutoModel, AutoTokenizer

def getem_roberta(model, tokenizer, text):
    t = tokenizer(text, return_tensors='pt')
    while t.input_ids.shape[1] > 512:
        a = a[:-10] # Assuming 'a' is a variable holding the text, this line might need context
        t = tokenizer(a, return_tensors='pt')
    return model(t.input_ids, t.attention_mask)
```

--------------------------------

### Displaying Named Entities

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Prints the identified named entities from a processed question.

```python
ents
```

--------------------------------

### Data Loading and Initial Inspection

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Loads a JSON file into a pandas DataFrame and displays the DataFrame. This is typically the first step in analyzing the dataset.

```python
import pandas
truth_df = pandas.read_json('/mnt/data/train.json')
truth_df
```

--------------------------------

### Graph Traversal - Similarity Search and Ordering

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Finds nodes similar to 'Pumpin \' Up The Party', traverses incoming edges, orders them, and returns the results.

```gremlin
v = g.V().like('emb', [getem("Pumpin \' Up The Party")], 2).inE().order().toArray().get()
v
#titles.iloc[v]
```

--------------------------------

### Install spaCy Language Model

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag.ipynb

Downloads the 'en_core_web_md' language model for spaCy, which is a medium-sized English model containing word vectors.

```bash
!python3 -m spacy download en_core_web_md
```

--------------------------------

### Graph Construction and Embedding Loading

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Sets up the BitGraph, adds vertices and edges from processed Wikipedia data, and loads embeddings based on the specified type ('w2v' or 'roberta'). It also configures the embedding retrieval function (`getem`) accordingly.

```python
import torch
import warnings
import gensim
from pybitgraph import BitGraph
from preprocess import Sentence_Transformer, Word2Vec_Transformer

# Assuming eix, titles, sentences are already defined from read_wiki_data
# Assuming args dictionary is defined with necessary paths and configurations

# Example usage (assuming args and read_wiki_data are available):
# eix, titles, sentences = read_wiki_data(args['fname'], args['skip_empty_vertices'])

graph = BitGraph(
    'int64',
    'int64',
    'DEVICE',
    'DEVICE',
    args['property_storage'].upper(),
)

graph.add_vertices(eix.max() + 1)
graph.add_edges(eix[0], eix[1], 'link')

read_embeddings(
    graph,
    args['embeddings_dir'],
    td=300 if args['embedding_type'] == 'w2v' else 1024,
)
print('read embeddings into graph')

g = graph.traversal()
print('constructed graph')

if args['embedding_type'] == 'w2v':
    warnings.warn("Word2Vec encoder is for testing/debugging purposes only!")
    module = Word2Vec_Transformer(
        gensim.models.KeyedVectors.load_word2vec_format(args['w2v_path'], binary=True),
        dim=300,
    )
    getem = lambda t : getem_w2v(module, t)
elif args['embedding_type'] == 'roberta':
    model = AutoModel.from_pretrained('sentence-transformers/all-roberta-large-v1')
    tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-roberta-large-v1')
    
    mod = Sentence_Transformer(model).cuda()
    import torch._dynamo
    torch._dynamo.reset()

    module = torch.compile(mod, fullgraph=True)
    getem = lambda t : getem_roberta(module, tokenizer, t)
else:
    raise ValueError("Expected 'w2v' or 'roberta' for embedding type")
```

--------------------------------

### Formatting Context Data

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Joins context sentences into a single string, making it easier to read and analyze.

```python
'\n'.join([' '.join(z[1]) for z in truth_df.context.iloc[2]])
```

--------------------------------

### Loading JSON Data with Pandas

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Loads data from a JSON file located at '/mnt/data/train.json' into a pandas DataFrame.

```python
import pandas
df = pandas.read_json('/mnt/data/train.json')
```

--------------------------------

### Getting Entity Embedding

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Retrieves the embedding for a given entity name, 'Shameless'.

```python
getem('Shameless')
```

--------------------------------

### Graph-based Question Answering Pipeline

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Implements a pipeline for question answering using graph traversal and a GRetriever model. It involves entity recognition, graph neighborhood expansion, and model inference. Includes performance timing for graph queries.

```python
import torch
import numpy as np
from time import perf_counter

# Assuming 'g', 'truth_df', 'ner', 'getem', 'graph', 'coo_to_data', 'model' are defined elsewhere

ent_match_limit = 4
que_match_limit = 4

out_limit_h0 = 4
out_limit_h1 = 4
in_limit_h0 = 4
in_limit_h1 = 4

for i in range(3):
    question = truth_df.question.iloc[i]
    answer = truth_df.answer.iloc[i]
    emb_q = getem(question)

    vids_q = np.concatenate(
        [
            g.V().like('emb', [getem(ent['word'])], ent_match_limit).toArray()
            for ent in ner(question)
        ] + [
            g.V().like('emb', [emb_q], que_match_limit).toArray()
        ]
    )
    
    # TODO control hops
    start_time = perf_counter()
    eids = g.V(vids_q)._union([
        __().outE().order().by(__().inV().similarity('emb', [emb_q])).limit(4)._as('h0').inV(),
        __().inE().order().by(__().outV().similarity('emb', [emb_q])).limit(4)._as('h0').outV(),
    ])._union([
        __().outE().order().by(__().inV().similarity('emb', [emb_q])).limit(4)._as('h1').inV(),
        __().inE().order().by(__().outV().similarity('emb', [emb_q])).limit(4)._as('h1').outV(),
    ])._union([__().select('h0'), __().select('h1')]).dedup().toArray()
    end_time = perf_counter()

    print('query time:', end_time - start_time)

    out = graph.subgraph_coo(
        eids
    )

    data = coo_to_data(out)
    print(data)

    loss = model(
        question=[f'question: {question}\nanswer:'],
        x=data.x,
        edge_index=data.edge_index,
        batch=data.batch,
        label=[answer],
        edge_attr=None, # edge features
        additional_text_context=None # additional context
    )
    print(loss)
```

--------------------------------

### Graph Traversal - Sentence Retrieval

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Retrieves sentences based on a similarity search for 'Pumpin \' Up The Party', calculating an index based on titles and sentences.

```gremlin
sentences.iloc[g.V().like('emb', [getem("Pumpin \' Up The Party")], 1)._in().toArray().get() - len(titles)]
```

--------------------------------

### Accessing Specific Data Points

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Demonstrates how to access a specific element from a pandas Series, useful for inspecting individual data entries.

```python
truth_df.question.iloc[2]
```

--------------------------------

### Graph Traversal with 'like' and 'toArray'

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Performs a graph traversal using the 'like' predicate with an embedding and query parameters, then converts the result to an array.

```python
g.V().like('emb', [emb_q], qp['question_vertex_match_limit']).toArray()
```

--------------------------------

### Accessing Query Parameter

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Retrieves the value of 'question_vertex_match_limit' from the query parameters dictionary.

```python
qp['question_vertex_match_limit']
```

--------------------------------

### Extracting Supporting Facts

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Extracts and formats supporting facts from a DataFrame column, likely for further processing or display.

```python
[z[0] for z in truth_df.supporting_facts.iloc[2]]
```

--------------------------------

### Retrieving Sentence Values

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Fetches sentence values from specific indices derived from 'src' and converts them to a list.

```python
sentences.iloc[src[-5:]].sentences.values_host.tolist()
```

--------------------------------

### Accessing Question by Index

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Retrieves the question at index 30 from the 'question' column of the DataFrame 'df'.

```python
df.question[30]
```

--------------------------------

### Accessing Specific Title

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Retrieves a specific title from the 'titles' Series using its index.

```python
titles.iloc[2902052]
```

--------------------------------

### Accessing DataFrame Rows

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Demonstrates accessing specific rows of a DataFrame using iloc and calculating indices based on DataFrame length.

```python
sentences.iloc[13560798-len(df)]
```

--------------------------------

### JSON Lines Data Loading with CuDF

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Loads data from a JSON Lines file using `cudf`. This is typically used for large datasets that can be processed on the GPU.

```python
import cudf
df = cudf.read_json('/mnt/para_with_hyperlink.jsonl', lines=True)
```

--------------------------------

### Accessing Title by Index

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Retrieves a title from the 'titles' Series using a specific index.

```python
titles.iloc[5956065]
```

--------------------------------

### Query Graph with Embeddings

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Performs a graph query using an embedding. It retrieves vertices similar to the query embedding using the `like` method and returns the results as an array.

```python
# Assuming g and getem are defined from previous steps

def query(search_query, lim=4):
    qe = getem(search_query)
    vids = g.V().like('emb', [qe], lim).toArray()
```

--------------------------------

### Complex Graph Traversal and Decoding

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Constructs a complex graph traversal involving multiple hops, ordering by similarity, and applying limits. It then decodes the resulting video IDs.

```python
vids_q = cupy.concatenate(
    [
        g.V().like('emb', [getem(ent['word'])], qp['entity_vertex_match_limit']).toArray()
        for ent in ents
    ] + [
        g.V().like('emb', [emb_q], qp['question_vertex_match_limit']).toArray()
    ]
)
decode(vids_q)
```

--------------------------------

### Accessing Last Elements

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Retrieves the last five elements from a variable named 'src'.

```python
src[-5:]
```

--------------------------------

### Graph Traversal - Batch Similarity Search

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Performs a similarity search on a batch of vertex IDs against 'Move (1970 film)' embeddings.

```gremlin
g.V([5013434, 374345]).similarity('emb', [getem('Move (1970 film)')]).toArray()
```

--------------------------------

### Load Wikipedia Data into Graph

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Reads Wikipedia data from a JSONL file, processes mentions and sentence lengths, and constructs graph edges (sources and destinations) for both mentions and sentences. It handles potential empty vertices and prepares data for graph construction.

```python
import cudf
import cupy
import torch

def read_wiki_data(fname, skip_empty=True):
    df = cudf.read_json('/mnt/para_with_hyperlink.jsonl', lines=True)

    mentions = df.mentions.explode()
    mentions = mentions[~mentions.struct.field('sent_idx').isna()]
    mentions = mentions[~mentions.struct.field('ref_ids').isna()]

    slens = df.sentences.list.len().astype('int64')
    slens[(slens==0)] = 1

    df['sentence_offsets'] = cupy.concatenate([
        cupy.array([0]),
        slens.cumsum().values[:-1]
    ])

    mix = torch.as_tensor(
        mentions.struct.field('ref_ids').list.get(0).astype('int64').values,
        device='cuda'
    )
    ids = torch.as_tensor(df.id.astype('int64').values, device='cuda')
    vals, inds = torch.sort(ids)

    destinations_m = inds[torch.searchsorted(vals, mix)]
    sources_m = torch.as_tensor(
        mentions.struct.field('sent_idx').values + df.sentence_offsets[mentions.index].values + len(df),
        device='cuda'
    )

    if skip_empty:
        # Does not add vertices/edges for vertices with no embedding
        f = destinations_m < len(df)
        destinations_m = destinations_m[f]
        sources_m = sources_m[f]
        del f

    eim = torch.stack([
        torch.as_tensor(sources_m, device='cuda'),
        torch.as_tensor(destinations_m, device='cuda'),
    ])

    sentences = df.sentences.explode().reset_index().rename({"index": 'article'},axis=1)

    sources_s = sentences.index.values + len(df)
    destinations_s = sentences.article.values
    eis = torch.stack([
        torch.as_tensor(sources_s, device='cuda'),
        torch.as_tensor(destinations_s, device='cuda'),
    ])

    eix = torch.concatenate([eim,eis],axis=1)
    del eis
    del eim

    return eix, df.title.to_pandas(), sentences.sentences.to_pandas()

```

--------------------------------

### Graph Traversal - Advanced Similarity Search

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Performs a graph traversal to find nodes similar to 'Pumpin \' Up The Party' with a higher similarity threshold.

```gremlin
g.V().like('emb', [getem("Pumpin \' Up The Party")], 2).toArray()
```

--------------------------------

### Calculating Index Offset

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Calculates an index offset by adding a numerical value to the length of a DataFrame.

```python
7600544+len(df)
```

--------------------------------

### Accessing Sentences by Calculated Index

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Retrieves sentences from the 'sentences' DataFrame using an index calculated by subtracting the length of 'titles' from 'vids_q'.

```python
sentences.iloc[vids_q.get() - len(titles)]
```

--------------------------------

### Accessing Struct Fields

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Retrieves specific fields ('ref_url' and 'ref_ids') from the last five elements of a 'mentions' structure.

```python
mentions[-5:].struct.field('ref_url')
```

```python
mentions[-5:].struct.field('ref_ids')
```

--------------------------------

### Named Entity Recognition and Embedding

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Processes a natural language question using Named Entity Recognition (NER) and retrieves an embedding for the question.

```python
#question = "What is the date of birth of the director of film Rathimanmadhan?"
#question = "What is the place of birth of the director of film Discord (Film)?"
#question = "Did the movies Torkaman (Film) and Shameless (2008 Film), originate from the same country?"
question = "Do both directors of films The Big Bang (1989 Film) and Tender Fictions share the same nationality?"

ents = ner(question)
emb_q = getem(question)
ents
```

--------------------------------

### Filtering DataFrame by Article

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Filters a DataFrame to select rows where the 'article' column matches a specific ID.

```python
sentences[sentences.article==1954484]
```

--------------------------------

### NER and Graph Traversal Integration

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Uses the NER pipeline to extract entities from a question, then performs graph traversals based on these entities to find related articles and sentences.

```python
import numpy as np
vids = np.concatenate([
    g.V().like('emb', [getem(ent['word'])], 4).toArray()
    for ent in ner(truth_df.question.iloc[167453])
])

print(vids)

f = (vids < len(titles))
print('articles:', titles.iloc[vids[f].get()])
print('sentences:', sentences.iloc[vids[~f].get() - len(titles)])
```

--------------------------------

### Graph Traversal - Similarity Search and Count

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Finds nodes similar to 'Miley Cyrus', traverses incoming edges, and counts the results.

```gremlin
g.V().like('emb', [getem("Miley Cyrus")], 1)._in().count().toArray()
```

--------------------------------

### Graph Traversal - Basic Similarity Search

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Performs a graph traversal to find nodes similar to 'Miley Cyrus' based on an 'emb' embedding, returning an array of results.

```gremlin
g.V().like('emb', [getem("Miley Cyrus")], 1).toArray()
```

--------------------------------

### Advanced Graph Traversal with Unions and Aliases

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Executes a sophisticated graph traversal using Gremlin-like syntax, employing unions to combine results from outgoing and incoming edges at different hop levels, aliasing intermediate results, and deduplicating the final output before decoding.

```python
from pygremlinxx import GraphTraversal
__ = lambda : GraphTraversal()

vids = g.V(vids_q)._union([
    __().out().order().by(__().similarity('emb', [emb_q])).limit(qp['hop_0_outgoing_limit'])._as('h0'),
    __()._in().order().by(__().similarity('emb', [emb_q])).limit(qp['hop_0_incoming_limit'])._as('h0'),
])._union([
    __().out().order().by(__().similarity('emb', [emb_q])).limit(qp['hop_1_outgoing_limit'])._as('h1'),
    __()._in().order().by(__().similarity('emb', [emb_q])).limit(qp['hop_1_incoming_limit'])._as('h1'),
])._union([__().select('h0'), __().select('h1')]).dedup().toArray()

decode(vids)
```

--------------------------------

### Load Embeddings into Graph

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Loads pre-computed embeddings from specified directories ('titles', 'sentences') into the BitGraph. It sorts files by a custom key derived from filenames and sets vertex embeddings using `graph.set_vertex_embeddings`.

```python
import torch
import os
import re

def read_embeddings(graph, directory, td):
    ex = re.compile(r'part_([0-9]+)_([0-9]+).pt')
    def fname_to_key(s):
        m = ex.match(s)
        return int(m[1]), int(m[2])

    ix = 0

    for emb_type in ['titles', 'sentences']:
        path = os.path.join(directory, emb_type)
        files = os.listdir(path)

        files = sorted(files, key=fname_to_key)
        for f in files:
            e = torch.load(os.path.join(path, f), weights_only=True, map_location='cuda').reshape((-1, td))

            print(ix, e.shape)
            graph.set_vertex_embeddings('emb', ix, ix + e.shape[0] - 1, e)
            
            ix += e.shape[0]
            del e
```

--------------------------------

### Creating Source Indices for Graph Edges

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Generates source indices for graph edges based on sentence indices and article offsets. It adds an offset based on the total number of articles in the DataFrame.

```python
import torch
import cudf

# Assuming 'df' and 'mentions' are processed as above
src = torch.as_tensor(
    mentions.struct.field('sent_idx').values + df.sentence_offsets[mentions.index].values,
    device='cuda'
) + len(df)
print(src[(destinations_m == 4111782)] == 13590393).sum()
```

--------------------------------

### Custom Data Decoding Function

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

A function to decode video IDs, separating them based on whether they are less than the length of 'titles' and printing corresponding article or sentence information.

```python
def decode(vids):
    f = (vids < len(titles))
    print('articles:', titles.iloc[vids[f].get()])
    print('sentences:', sentences.iloc[vids[~f].get() - len(titles)])
```

--------------------------------

### Process Text with spaCy and Get Vectors

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag.ipynb

Processes text using the loaded spaCy model and extracts word vectors for comparison.

```python
t1 = nlp('brown fox')

m = df.title.map(lambda x : nlp(x).vector)
m
```

--------------------------------

### COO to PyTorch Geometric Data Conversion

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Converts a graph in COO format to a `torch_geometric.data.Data` object. This function prepares graph data for use with PyTorch Geometric models, including setting edge indices, node features, and batch information.

```python
from torch_geometric.data import Data
import torch

def coo_to_data(coo):
    data = Data()
    data.edge_index = torch.stack([
        torch.as_tensor(coo['dst'].astype('int64'), device='cuda'),
        torch.as_tensor(coo['src'].astype('int64'), device='cuda'),
    ])
    data.x = torch.as_tensor(
        g.V(coo['vid']).encode('emb').toArray(),
        device='cuda'
    ).reshape((-1, 300))
    data.batch = torch.zeros((data.x.shape[0],), dtype=torch.int64, device='cuda')

    return data
```

--------------------------------

### Processing Mentions and Sentence Indices

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Processes mentions from a DataFrame, filtering for valid sentence indices and reference IDs. It then maps these mentions to corresponding vertex destinations using sorted indices.

```python
import torch
import cudf

# Assuming 'df' is loaded with cudf
mentions = df.mentions.explode()
mentions = mentions[~mentions.struct.field('sent_idx').isna()]
mentions = mentions[~mentions.struct.field('ref_ids').isna()]

mix = torch.as_tensor(
    mentions.struct.field('ref_ids').list.get(0).astype('int64').values,
    device='cuda'
)
ids = torch.as_tensor(df.id.astype('int64').values, device='cuda')
vals, inds = torch.sort(ids)


destinations_m = inds[torch.searchsorted(vals, mix)]
print(destinations_m)
```

--------------------------------

### Initialize RMM and Set Allocators

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Initializes the RMM shared allocator and sets CuPy and PyTorch allocators to use RMM for memory management. This is crucial for efficient GPU memory usage in deep learning and graph processing tasks.

```python
import rmm

# Initialize shared allocator to prevent fragmentation
rmm.reinitialize(devices=0, pool_allocator=False, managed_memory=False)

import cupy
cupy.cuda.set_allocator(rmm_cupy_allocator)

import torch
torch.cuda.change_current_allocator(rmm_torch_allocator)
```

--------------------------------

### Processing Sentences and Article Mapping

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Explodes the sentences column and maps sentence indices to article indices. It cleans the data by removing rows with missing information and resetting the index.

```python
import cudf

# Assuming 'df' is loaded with cudf
sentences = df.sentences.explode().reset_index().rename({"index": 'article'},axis=1)
sentences.dropna(inplace=True)
sentences.reset_index(drop=True, inplace=True)

destinations_s = sentences.index.values + len(df)
sources_s = sentences.article.values

print(destinations_s[sources_s==1954484])
```

--------------------------------

### Sentence Length Calculation and Offset Mapping

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag_demo.ipynb

Calculates sentence lengths from a DataFrame and computes cumulative offsets. This is used to map sentence indices to global offsets within the dataset.

```python
import cupy
import cudf

# Assuming 'df' is loaded with cudf
slens = df.sentences.list.len().astype('int64')
slens[(slens==0)] = 1

df['sentence_offsets'] = cupy.concatenate([
    cupy.array([0]),
    slens.cumsum().values[:-1]
])
print(df)
```

--------------------------------

### Nanobind Python Module Setup

Source: https://github.com/bgamer50/bitgraph/blob/main/CMakeLists.txt

Configures the build to find and use the nanobind library for creating Python bindings. It then adds a Python module named 'pybitgraph' using a specified C++ source file and links it with the project's libraries.

```cmake
if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
  set(CMAKE_BUILD_TYPE Release CACHE STRING "Choose the type of build." FORCE)
  set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug" "Release" "MinSizeRel" "RelWithDebInfo")
endif()

# Detect the installed nanobind package and import it into CMake
execute_process(
  COMMAND "${Python_EXECUTABLE}" -m nanobind --cmake_dir
  OUTPUT_STRIP_TRAILING_WHITESPACE OUTPUT_VARIABLE NB_DIR)
list(APPEND CMAKE_PREFIX_PATH "${NB_DIR}")
find_package(nanobind CONFIG REQUIRED)

nanobind_add_module(pybitgraph bindings/PyBitGraph.cpp)
target_link_libraries(
    pybitgraph 
    PRIVATE  
        bitgraph 
        gremlinxx 
        maelstrom
        faiss
)
target_link_directories(
       pybitgraph
       PRIVATE
         "../maelstrom/" 
         "../gremlin++/")
```

--------------------------------

### Calculate Sentence Offsets with CuPy

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/advanced_rag.ipynb

Calculates the starting offsets for each sentence within the DataFrame using CuPy. This is crucial for correctly mapping sentence indices to graph vertices.

```python
import cupy
df['sentence_offsets'] = cupy.concatenate([
    cupy.array([0]),
    df.sentences.list.len().cumsum().values[:-1]
])
df
```

--------------------------------

### Initialize BitGraph and Add Paths

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag.ipynb

Appends necessary directories to the Python path for importing custom modules and initializes the BitGraph library.

```python
import sys
sys.path.append('/mnt/bitgraph')
sys.path.append('/mnt/gremlin++')
from pybitgraph import BitGraph
```

--------------------------------

### Initialize and Populate BitGraph

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/advanced_rag.ipynb

Initializes a BitGraph instance with specified data types and device settings. It then adds vertices and edges to the graph using the pre-calculated edge indices.

```python
import cupy

graph = BitGraph(
    "int64",
    "int64",
    "DEVICE",
    "DEVICE",
    "PINNED",
)

src, dst = eix
graph.add_vertices(eix.max() + 1)
graph.add_edges(src, dst, 'link')
```

--------------------------------

### Get Maximum and Minimum Source Indices

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/advanced_rag.ipynb

Retrieves the maximum and minimum source indices from both the sentence ('eis') and mention ('eim') edge index tensors. This is used to determine the total number of vertices.

```python
eis[0].max(), eim[0].max()
eis[0].min(), eim[0].min()
```

--------------------------------

### Initialize BitGraph and Add Edges

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/similarity.ipynb

Demonstrates how to initialize a BitGraph with specified vertex and edge types, add vertices, and add edges between them. It also shows how to set vertex embeddings and initiate a graph traversal.

```python
import sys
sys.path.append('/mnt/bitgraph')
sys.path.append('/mnt/gremlin++')
from pybitgraph import BitGraph
import numpy as np

src = np.array([0, 1, 2, 3, 4, 5, 6])
dst = np.array([5, 4, 3, 2, 1, 6, 0])
emb = np.array([[1.1, 2.1], [2.2, 4.4], [3.3, 5.5], [4.4, 1.6], [5.5, 4.1], [6.6, 3.0], [-7.7, 9.9]], dtype='float32')

graph = BitGraph(
    "int64",
    "int64",
    "DEVICE",
    "PINNED",
    "DEVICE",
)

graph.add_vertices(7)
graph.add_edges(src, dst, 'link')
graph.set_vertex_embeddings('emb', np.array([],dtype='int64'), emb)
g = graph.traversal()
```

--------------------------------

### Import BitGraph and Set Path

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/advanced_rag.ipynb

Appends necessary directories to the system path and imports the BitGraph class from the pybitgraph library. This prepares the environment for graph operations.

```python
import sys
sys.path.append('/mnt/bitgraph')
sys.path.append('/mnt/gremlin++')
from pybitgraph import BitGraph
```

--------------------------------

### Run QRAG Benchmarks Script

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/README.md

Shell script to execute the main QRAG benchmarks. Assumes necessary data and embeddings are prepared.

```bash
#!/bin/bash

# Ensure preprocess.py has been run to generate embeddings
# Ensure data files (para_with_hyperlink.jsonl, train.json, test.json) are available

# Run the main QRAG benchmarks using construct.py
python construct.py \
  --fname data_ids_april7/para_with_hyperlink.jsonl \
  --truth_fname data_ids_april7/train.json \
  --embedding_path ./embeddings \
  --output_path ./results

# To run with visualization model:
# python construct.py \
#   --fname data_ids_april7/para_with_hyperlink.jsonl \
#   --truth_fname data_ids_april7/train.json \
#   --output_subgraphs
```

--------------------------------

### Get Vector Shape from spaCy Processed Document

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag.ipynb

Iterates through processed documents and prints the shape of the entity vectors.

```python
for doc in nlp.pipe(['What is the capital of Afghanistan?']):
    print(doc.ents[0].vector.shape)
```

--------------------------------

### Initialize Graph Traversal

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/advanced_rag.ipynb

Initializes the graph traversal object from the constructed BitGraph instance. This prepares the graph for querying and analysis.

```python
g = graph.traversal()
```

--------------------------------

### Using BitGraph as a Header Library

Source: https://github.com/bgamer50/bitgraph/blob/main/README.md

BitGraph is a header library. To use it in your project, you simply need to set it as an include directory. This approach simplifies integration without requiring complex build configurations.

```bash
g++ your_project.cpp -I/path/to/bitgraph/include -o your_executable
```

--------------------------------

### BitGraph Release History and Features

Source: https://github.com/bgamer50/bitgraph/blob/main/README.md

This section details the evolution of BitGraph, highlighting key features introduced in each version. Notable updates include Python bindings, Gremlin++ enhancements, FAISS integration, and performance optimizations.

```text
Version 1.1.0 (December 2024):
- Added Python bindings.
- Introduced new features in Gremlin++ (traversal strategies, optimizations).
- Added FAISS integration.
- New examples demonstrating QRAG (query-based GraphRAG).

Version 1.0.0 (September 26, 2023):
- Major refactor using _Maelstrom_ for operations (sparse matrix, vector, hash tables).
- Increased delegation to Gremlin++ and its steps.
- Significant performance and memory improvements.

Version 0.6.1 (October 11, 2022):
- Updates to match Gremlin++ v0.6.1 semantic changes.
- Makefile updated to build with g++11 and nvcc.
- Improved builds planned (likely via conda).

Version 0.6.0 (June 11, 2022):
- Introduced Hybrid backend (GPU graph structure, CPU properties).
- Licensed under Apache 2.0.

Version 0.1.0 (October 18, 2019):
- Deprecated OpenCL acceleration.
- Supports Linux platform.
- Licensed under Apache 2.0.
```

--------------------------------

### Run QRAG Benchmarks with construct.py

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/README.md

Executes QRAG benchmarks, including comparisons to G-Retriever and a non-RAG solution. Can also output subgraphs for debugging.

```python
import construct

# Example benchmark run
construct.main(
    fname='path/to/para_with_hyperlink.jsonl',
    truth_fname='path/to/train.json',
    embedding_path='path/to/embeddings',
    output_path='path/to/output'
)

# Example for visualization model
construct.main(
    fname='path/to/para_with_hyperlink.jsonl',
    truth_fname='path/to/train.json',
    output_subgraphs=True
)
```

--------------------------------

### Perform Approximate Nearest Neighbor Search (LIKE)

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/similarity.ipynb

Demonstrates the 'like' traversal step, which performs an approximate nearest neighbor search based on a given embedding and a similarity threshold.

```python
g.V().like('emb', [np.array([1.1, 2.1],dtype='float32')], 0.90).toArray()
```

--------------------------------

### Test Executable Target Creation Function

Source: https://github.com/bgamer50/bitgraph/blob/main/CMakeLists.txt

A CMake function to define and configure test executable targets. Similar to the execution target function, it sets the runtime output directory for tests and links the required libraries and directories.

```cmake
function(AddBitGraphTest EXEC_NAME)
    
    add_executable(${EXEC_NAME} ${ARGN})

    set_target_properties(
        ${EXEC_NAME}
        PROPERTIES
            RUNTIME_OUTPUT_DIRECTORY "test/bin"
    )

    target_link_directories(
       ${EXEC_NAME}
       PRIVATE
         "../maelstrom/" 
         "../gremlin++/")

    target_link_libraries(
        ${EXEC_NAME}
        PRIVATE
            bitgraph
            gremlinxx
            maelstrom
            faiss
    )

endfunction()

AddBitGraphTest(
    "test_basic.exe" "test/test_basic.cpp"
)
```

--------------------------------

### Load spaCy Language Model

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag.ipynb

Loads the 'en_core_web_md' language model from spaCy for natural language processing tasks.

```python
import spacy
nlp = spacy.load('en_core_web_md')
```

--------------------------------

### Executable Target Creation Function

Source: https://github.com/bgamer50/bitgraph/blob/main/CMakeLists.txt

A CMake function to define and configure executable targets. It sets the runtime output directory and links necessary libraries and directories, including the core 'bitgraph' library and external dependencies.

```cmake
function(AddBitGraphExec EXEC_NAME)
    
    add_executable(${EXEC_NAME} ${ARGN})

    set_target_properties(
        ${EXEC_NAME}
        PROPERTIES
            RUNTIME_OUTPUT_DIRECTORY "bin"
    )

    target_link_directories(
        ${EXEC_NAME}
        PRIVATE
          "../maelstrom/" 
          "../gremlin++/")

    target_link_libraries(
        ${EXEC_NAME}
        PRIVATE
            bitgraph   
            gremlinxx
            maelstrom
            faiss
    )

endfunction()

AddBitGraphExec(
    "components.exe" "examples/components.cu"
)

AddBitGraphExec(
    "edge_query.exe" "examples/edge_query.cu"
)

AddBitGraphExec(
    "shortest_path.exe" "examples/shortest_path.cu"
)
```

--------------------------------

### Display First 10 Titles

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag.ipynb

Retrieves and displays the first 10 titles from the DataFrame.

```python
df.title[:10]
```

--------------------------------

### Generate Embeddings with preprocess.py

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/README.md

Generates word2vec (w2v) or roberta embeddings required for QRAG. This script needs to be run twice: once for sentences and once for articles.

```python
import preprocess

# Example usage for sentences
preprocess.main(fname_in='path/to/para_with_hyperlink.jsonl', embedding_type='roberta', stage='sentences')

# Example usage for articles
preprocess.main(fname_in='path/to/para_with_hyperlink.jsonl', embedding_type='roberta', stage='articles')
```

--------------------------------

### Load Embeddings from NumPy File

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag.ipynb

Loads embeddings from a .npy file using NumPy.

```python
import numpy as np
emb = np.load('/mnt/bitgraph/data/rag/emb.npy')
```

--------------------------------

### Construct Edge Index for Mentions with PyTorch

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/advanced_rag.ipynb

Constructs the edge index tensor for mentions using PyTorch. It combines source and destination node indices derived from the DataFrame and mention data.

```python
import torch
destinations_m = mentions.struct.field('ref_ids').list.get(0).astype('int64').values
sources_m = mentions.struct.field('sent_idx').values + df.sentence_offsets[mentions.index].values + len(df)
eim = torch.stack([
    torch.as_tensor(sources_m, device='cuda'),
    torch.as_tensor(destinations_m, device='cuda'),
])
eim,eim.shape
```

--------------------------------

### Load Article and Edgelist Data

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/rag.ipynb

Loads article and edgelist data from Parquet files into pandas DataFrames.

```python
import pandas




























































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































```

--------------------------------

### Perform Similarity Search

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/similarity.ipynb

Illustrates how to perform a similarity search on the graph using vertex embeddings. This operation finds vertices similar to a given embedding vector.

```python
g.V().similarity('emb', [np.array([1.1, 2.1], dtype='float32')]).toArray()
```

--------------------------------

### Construct Edge Index for Sentences with PyTorch

Source: https://github.com/bgamer50/bitgraph/blob/main/examples/advanced_rag.ipynb

Constructs the edge index tensor for sentences using PyTorch. It maps sentence indices to article indices to represent sentence-to-article relationships.

```python
sources_s = sentences.index.values + len(df)
destinations_s = sentences.article.values
eis = torch.stack([
    torch.as_tensor(sources_s, device='cuda'),
    torch.as_tensor(destinations_s, device='cuda'),
])
eis,eis.shape
```