### Start CosmosAIGraph Microservices with Docker Compose Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/local_execution.md This command initiates the CosmosAIGraph application, comprising both web and graph microservices, by leveraging Docker Compose. It reads the service definitions from the specified docker-compose.yml file and brings up all defined containers. Ensure Docker Desktop is running before executing this command. ```Shell docker compose -f docker/docker-compose.yml up ``` -------------------------------- ### Configure Azure OpenAI Environment Variables Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/initial_paas_provisioning.md This snippet details the environment variables required to connect the CosmosAIGraph application to an Azure OpenAI resource. It includes variables for the service endpoint, API key, and deployment names for completion and embedding models. ```APIDOC Azure OpenAI Configuration Variables: CAIG_AZURE_OPENAI_URL - Description: Endpoint URL for the Azure OpenAI resource. - Source: Azure Portal -> Resource Management -> Keys and Endpoint. CAIG_AZURE_OPENAI_KEY - Description: API Key (Key 1) for the Azure OpenAI resource. - Source: Azure Portal -> Resource Management -> Keys and Endpoint. CAIG_AZURE_OPENAI_COMPLETIONS_DEP - Description: Deployment name for the GPT-4 completions model. - Source: Azure Portal -> Azure OpenAI -> Model deployments. CAIG_AZURE_OPENAI_EMBEDDINGS_DEP - Description: Deployment name for the text-embedding-ada-002 model. - Source: Azure Portal -> Azure OpenAI -> Model deployments. ``` -------------------------------- ### Activate Python Virtual Environment in PowerShell Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/developer_workstation.md This command activates the Python virtual environment created by the venv.ps1 script. It sets up the necessary environment variables and modifies the PATH to use the Python packages installed within the virtual environment for the current session. ```PowerShell .\venv\Scripts\Activate.ps1 ``` -------------------------------- ### Set up Python Virtual Environment for CosmosAIGraph Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_nosql.md Instructions for initializing and activating a Python virtual environment. This ensures that project dependencies are managed in an isolated environment, preventing conflicts with system-wide Python installations. ```PowerShell .\venv.ps1 .\venv\Scripts\Activate.ps1 ``` -------------------------------- ### Start Java Graph Service Locally Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/graph_app/readme.md Various methods to run the Java graph service locally for development or testing. Options include using Gradle's `bootRun` command, dedicated service scripts (`websvc.ps1`, `websvc.sh`), or Docker Compose for a containerized deployment, which requires editing the `docker-compose-graph-only.yml` file. ```Gradle gradle bootRun ``` ```PowerShell websvc.ps1 ``` ```Shell websvc.sh ``` ```Docker docker compose -f docker-compose-graph-only.yml up ``` -------------------------------- ### Configure Azure Cosmos DB for NoSQL Environment Variables Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/initial_paas_provisioning.md This snippet specifies the environment variables needed to establish a connection to an Azure Cosmos DB for NoSQL account. It covers the account name, authentication mechanism, and the primary key for secure access. ```APIDOC Azure Cosmos DB for NoSQL Configuration Variables: CAIG_COSMOSDB_NOSQL_ACCT - Description: Name of the Azure Cosmos DB for NoSQL account. CAIG_COSMOSDB_NOSQL_AUTH_MECHANISM - Description: Authentication mechanism for Cosmos DB. - Value: "key" CAIG_COSMOSDB_NOSQL_KEY - Description: Primary Key for the Azure Cosmos DB for NoSQL account. - Source: Azure Portal -> Cosmos DB account -> Settings -> Keys. ``` -------------------------------- ### Check Java and Gradle Versions Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/graph_app/readme.md Commands to verify the installed versions of Java (OpenJDK 21) and Gradle (8.11 or 8.12) on a system. These checks are crucial to ensure compatibility with the project's build requirements. ```Shell java --version ``` ```Shell gradle --version ``` -------------------------------- ### Azure Container Apps Replica Scale Configuration Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/faq.md Example Bicep configuration snippet demonstrating how to set minimum and maximum replica counts for an Azure Container Apps workload profile. This ensures high availability and scalability for the graph service by maintaining at least two instances. ```Bicep scale: { maxReplicas: 2 minReplicas: 2 } ``` -------------------------------- ### Example RDF Triples for m26 Python Library Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/faq.md Illustrates actual RDF triples used in the reference implementation graph, demonstrating how subjects, predicates, and objects are expressed as URIs to define relationships and properties for the 'm26' Python library. These verbose values become efficient data structures within the database. ```RDF . "mit" . . . . "m26" . . "pace_per_mile runwalkcalculator seconds_per_mile agecalculator calculates" . "pypi" . ``` -------------------------------- ### JavaScript Form Handling and UI Interaction Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/views/gen_sparql_console.html This JavaScript code manages user interface interactions for a SPARQL console. It includes functionality to cycle through example natural language queries on double-click, disable buttons and submit forms upon 'Generate' or 'Execute' button clicks, and dynamically resize a textarea to fit its content. ```javascript const nl_form_field = document.getElementById("natural_language"); const generate_form = document.getElementById("generate_form"); const generate_button = document.getElementById("generate_button"); const execute_form = document.getElementById("execute_form"); const execute_button = document.getElementById("execute_button"); var nl_query_index = -1; var nl_queries = [ "What is the most connected node?", "What entity has the most outgoing connections?", "How many nodes are there?", "How many connections are there?", "What are the dependencies of the flask library?", "What are the dependencies of the pandas library?", "What are the dependencies of the openai library?", "what 5 libraries have the most dependencies ?" ]; nl_form_field.addEventListener('dblclick', function(event) { nl_query_index++; if (nl_query_index >= nl_queries.length) { nl_query_index = 0; } $("#natural_language").val(nl_queries[nl_query_index]); } ); generate_button.addEventListener('click', function(event) { event.preventDefault(); generate_button.disabled = true; generate_button.textContent = "Processing..."; execute_button.disabled = true; generate_form.submit(); } ); execute_button.addEventListener('click', function(event) { event.preventDefault(); execute_button.disabled = true; execute_button.textContent = "Processing..."; generate_button.disabled = true; execute_form.submit(); } ); function autoResize(textarea) { textarea.style.height = 'auto'; // Reset height textarea.style.height = textarea.scrollHeight + 'px'; // Set to scroll height } // Optional: auto-resize on page load window.addEventListener('DOMContentLoaded', function () { const textarea = document.getElementById('sparql'); if (textarea) autoResize(textarea); }); ``` -------------------------------- ### FastAPI Microservice Liveness HTTP GET Endpoint Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/understanding_the_code.md Each microservice within the architecture implements a `/liveness` HTTP GET endpoint. This endpoint serves as a health check, allowing container orchestration systems like Azure Container Apps (ACA) or Azure Kubernetes Service (AKS) to verify the process's operational status and initiate restarts if it's unresponsive. ```APIDOC GET /liveness - Description: Health check endpoint to ensure the microservice process is alive. - Purpose: Used by container orchestration systems (e.g., ACA, AKS) for process monitoring. - Returns: HTTP 200 OK on success, indicating the service is running. ``` -------------------------------- ### Example JSON for Edge Signatures Metadata Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/code_generation.md This JSON snippet illustrates the structure of the edge signatures metadata file. It defines relationships between different entity types (e.g., Movie has_principal Person). The numeric values are informational, indicating the observed frequency of the relationship. ```JSON { "Movie|has_principal|Person": 10000, "Person|in_movie|Movie": 10000 } ``` -------------------------------- ### Example JSON Document Structure for Python Library Data Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/cosmos_design_modeling.md This JSON snippet illustrates a partial document structure for Python library data within Cosmos DB, as used by CosmosAIGraph. It demonstrates how 'developers' and 'dependency_ids' fields are populated to represent relationships, which are then used to construct the in-memory RDF graph. ```JSON ... "developers": [ "contact@palletsprojects.com" ], ... "dependency_ids": [ "pypi_asgiref", "pypi_blinker", "pypi_click", "pypi_importlib_metadata", "pypi_itsdangerous", "pypi_jinja2", "pypi_python_dotenv", "pypi_werkzeug" ], ... ``` -------------------------------- ### Example JSON for Vertex Signatures Metadata Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/code_generation.md This JSON snippet demonstrates the structure of the vertex signatures metadata file. It defines attributes for entities, including their name and data type (e.g., Person has born attribute of type integer). The numeric values are informational counts. ```JSON { "Movie|title|": 10000, "Movie|year|": 10000, "Movie|rating|": 10000, "Movie|genre|": 10000, "Person|name|": 10000, "Person|born|": 10000, "Person|died|": 10000 } ``` -------------------------------- ### SPARQL Query for Library Usage Analysis in Cosmos DB Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/sparql/top_libraries.txt This SPARQL query retrieves libraries of a specific type (`{{libtype}}`) from a graph database, counts how many times each library is used, and then orders them by their usage count in descending order, limiting the results. It's designed to analyze library dependencies or popularity within a graph, potentially for AI-related applications in Cosmos DB. ```SPARQL PREFIX c: SELECT ?lib ?libname (COUNT(?lib) as ?use_count) WHERE { ?lib c:used_by_lib ?user . ?lib c:lt '{{libtype}}' . ?lib c:ln ?libname . } GROUP BY ?lib ORDER BY DESC(?use_count) LIMIT {{limit}} ``` -------------------------------- ### Querying User Libraries in Azure Cosmos DB AI Graph Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/sparql/users_of_library_v2.txt This SPARQL query retrieves the user, library name, and library type associated with a given ID in an Azure Cosmos DB AI Graph. It uses a prefix 'c' for the 'http://cosmosdb.com/caig#' namespace to simplify property references. The query filters results based on a provided ID and limits the number of returned records. ```SPARQL PREFIX c: SELECT ?user ?libname ?libtype WHERE { c:used_by_lib ?user . ?user c:ln ?libname . ?user c:lt ?libtype . } LIMIT {{limit}} ``` -------------------------------- ### Retrieve Developer by ID from Cosmos DB Graph (SPARQL) Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/sparql/developers_of_library.txt This SPARQL query retrieves the entity linked by the 'developed_by' predicate for a specific ID within an Azure Cosmos DB graph. It defines a custom prefix 'c' for the 'http://cosmosdb.com/caig#' namespace and allows limiting the number of results returned. ```SPARQL PREFIX c: SELECT ?o WHERE { c:developed_by ?o . } LIMIT {{limit}} ``` -------------------------------- ### Generate Configuration and Documentation Files using main_common.py Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/faq.md These Python command-line functions, implemented in `main_common.py`, allow users to generate various configuration artifacts and documentation. They can create PowerShell environment variable scripts, Bicep file fragments, Markdown documentation for environment variables, or generate all these artifacts simultaneously based on the environment variables defined in the `ConfigService` class. ```python python main_common.py gen_ps1_env_var_script python main_common.py gen_bicep_file_fragments python main_common.py gen_environment_variables_md python main_common.py gen_all ``` -------------------------------- ### Clone CosmosAIGraph Repository and Navigate Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/readme.md This snippet provides the commands to clone the CosmosAIGraph GitHub repository and navigate into the project directory. It also includes a command to verify the current location, which is needed for setting the CAIG_HOME environment variable. ```PowerShell git clone https://github.com/AzureCosmosDB/CosmosAIGraph.git cd CosmosAIGraph Get-Location ``` -------------------------------- ### FastAPI Application Initialization and Configuration Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/understanding_the_code.md This Python snippet demonstrates the standard initialization process for a FastAPI application. It includes loading environment variables using `dotenv`, configuring basic logging, logging defined environment variables for debugging, and instantiating the FastAPI application object along with an AI service. ```Python # standard initialization load_dotenv(override=True) logging.basicConfig( format="%(asctime)s - %(message)s", level=LoggingLevelService.get_level() ) ConfigService.log_defined_env_vars() app = FastAPI() ai_svc = AiService() ``` -------------------------------- ### Build and Package Java Application Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/graph_app/readme.md Instructions for compiling and packaging the Java application into a deployable artifact. The process can be initiated using either a PowerShell script (`build.ps1`) or a generic shell script (`build.sh`), automating the build steps. ```PowerShell build.ps1 ``` ```Shell build.sh ``` -------------------------------- ### Create Azure Cosmos DB vCore Collections and Indexes via Python Script Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md Provides PowerShell commands to set up the Python virtual environment and execute the `main_vcore.py` script, which automates the creation of necessary Cosmos DB Mongo vCore collections and their default indexes. ```PowerShell .\venv.ps1 .\venv\Scripts\Activate.ps1 python main_vcore.py create_vcore_collections_and_indexes ``` -------------------------------- ### Initialize In-Memory Graph on Application Startup Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/understanding_the_code.md This Java code snippet demonstrates the application's startup logic, specifically how the singleton instance of the in-memory graph (AppGraph) is initialized. It utilizes the AppGraphBuilder to construct the graph and then sets it as the global singleton for the application, ensuring the graph is ready for queries upon startup. ```Java AppGraph g = AppGraphBuilder.build(null); AppGraph.setSingleton(g); ``` -------------------------------- ### Executing CosmosAIGraph Code Generators Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/code_generation.md These shell commands demonstrate how to activate the Python virtual environment and then execute the `main_console.py` script. The first command generates an OWL ontology file, and the second generates an `RdflibTriplesBuilder` class, both using the provided metadata files. ```Shell > .\venv\Scripts\Activate.ps1 > python main_console.py generate_owl meta/vertex_signatures_imdb.json meta/edge_signatures_imdb.json http://cosmosdb.com/imdb ... file written: tmp/generated.owl > python main_console.py generate_rdflib_triples_builder meta/vertex_signatures_imdb.json ... file written: tmp/rdflib_triples_builder.py ``` -------------------------------- ### Visualize Dependencies with D3.js Force-Directed Graph Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/views/sparql_console.html This extensive JavaScript block initializes a D3.js force-directed graph to visualize software library dependencies. It parses an embedded JSON Bill-of-Materials (BOM), transforms it into D3-compatible nodes and edges, and sets up interactive features like zoom, node clicking, and mouse events. The graph renders with custom markers and styling, simulating forces to arrange nodes and links dynamically. ```javascript $(document).ready(function() { var method = "{{ method }}"; var libtype = "{{ libtype }}"; var libtype_prefix_idx = libtype.length + 1; var bom_json = document.getElementById("inline_bom_json").innerHTML.trim().replaceAll("'", "\""); var bom_obj = JSON.parse(bom_json); if (bom_obj.hasOwnProperty('libs')) { var bom_libs = bom_obj['libs']; var nodes_set = new Set(); var lib_keys = Object.keys(bom_libs); var nodes = Array(); var edges = Array(); for (let i = 0; i < lib_keys.length; i++) { var lib_key = lib_keys[i]; var lib_key_tokens = lib_key.split("#"); var lib_key_no_prefix = lib_key_tokens[1]; nodes_set.add(lib_key_no_prefix); dep_libs = bom_libs[lib_key]["dependencies"]; if (Array.isArray(dep_libs)) { for (let d = 0; d < dep_libs.length; d++) { var dep_lib_uri = dep_libs[d]; var dep_lib_uri_tokens = dep_lib_uri.split("#"); var dep_lib_no_prefix = dep_lib_uri_tokens[1]; nodes_set.add(dep_lib_no_prefix); edge = {}; edge["source"] = lib_key_no_prefix; edge["target"] = dep_lib_no_prefix; edge["weight"] = 1.0; edges.push(edge); } } } nodes_set.forEach(s => { n = {}; n["name"] = s; n["type"] = "vertex"; n["adjCount"] = 0; nodes.push(n); }); console.log('nodes count: ' + nodes.length); console.log('edges count: ' + edges.length); function nodeClicked(e, d) { console.log("nodeClicked: " + d.name); } function nodeDblClicked(e, d) { console.log("nodeDblClicked: " + d.name); } function nodeMouseOver(e, d) { } function nodeMouseOut(e, d) { } function initZoom() { d3.select('svg').call(zoom); } function handleZoom(e) { console.log('handle zoom'); d3.select('svg g').attr('transform', e.transform); } let zoom = d3.zoom().on('zoom', handleZoom); var typeScale = d3.scaleOrdinal() .domain(["library", "author", "maintainer"]) .range(["#75739F", "#41A368", "#FE9922"]); generateGraphViz(nodes, edges); function generateGraphViz(nodes, edges) { console.log("generateGraphViz"); var marker = d3.select("svg").append('defs') .append('marker') .attr("id", "Triangle") .attr("refX", 12) .attr("refY", 6) .attr("markerUnits", 'userSpaceOnUse') .attr("markerWidth", 12) .attr("markerHeight", 18) .attr("orient", 'auto') .append('path') .attr("d", 'M 0 0 12 6 0 12 3 6'); var nodeHash = {}; nodes.forEach(n => { nodeHash[n.name] = n; }); edges.forEach(edge => { edge.weight = parseInt(edge.weight); edge.source = nodeHash[edge.source]; edge.target = nodeHash[edge.target]; }); var linkForce = d3.forceLink(edges); var simulation = d3.forceSimulation() .force("charge", d3.forceManyBody().strength(-2000)) .force("center", d3.forceCenter().x(800).y(500)) .force("link", linkForce) .nodes(nodes) .on("tick", forceTick); simulation.force("link").links(edges); d3.select("svg g").selectAll("line.link") .data(edges, d => `${d.source.id}-${d.target.id}`) .enter() .append("line") .attr("class", "link") .style("opacity", 1.0) .style("stroke", "red") .style("stroke-width", d => d.weight); d3.selectAll("line").attr("marker-end", "url(#Triangle)"); var nodeEnter = d3.select("svg g").selectAll("g.node") .data(nodes, d => d.name) .enter() .append("g") .attr("class", "node"); nodeEnter.append("circle") .attr("r", 5) .style("fill", d => typeScale(d.type)); nodeEnter.append("text") .style("text-anchor", "middle") .attr("y", 15) .text(d => d.name); d3.selectAll("g.node").on("click", function(e, d) { nodeClicked(e, d); }); d3.selectAll("g.node").on("dblclick", function(e, d) { nodeDblClicked(e, d); }); d3.selectAll("g.node").on("mouseover", function(e, d) { nodeMouseOver(e, d); }); d3.selectAll("g.node").on("mouseout", function(e, d) { nodeMouseOut(e, d); }); function forceTick() { d3.selectAll("line.link") .attr("x1", d => d.source.x) .attr("x2", d => d.target.x) .attr("y1", d => d.source.y) .attr("y2", d => d.target.y); d3.selectAll("g.node") .attr("transform", d => `translate(${d.x},${d.y})`); } initZoom(); } } }); ``` -------------------------------- ### Generate Bicep Parameters with Python Script Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/aca_deployment.md This command executes a Python script (`main_common.py`) to generate Bicep parameter files (`generated.bicepparam`) and other configuration files. It automates the creation of deployment parameters based on environment variables, simplifying the Bicep deployment process for Azure Container Apps. ```Python python main_common.py gen_all ``` -------------------------------- ### Load Library Data into Azure Cosmos DB vCore via Python Script Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md Provides PowerShell commands to activate the Python virtual environment and execute specific `main_vcore.py` functions. These functions are responsible for loading library documents into the 'libraries' container and a single configuration document into the 'config' container. ```PowerShell .\venv\Scripts\Activate.ps1 python main_vcore.py load_vcore_with_library_documents python main_vcore.py persist_entities ``` -------------------------------- ### Jinja2 Template for Application Metadata Display Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/views/about.html This snippet illustrates a Jinja2 template defining standard HTML page blocks (`title`, `content`, `js`). The `content` block specifically includes placeholders for dynamically displaying application version, graph source, database, and container information, which are typically passed from a backend context. ```Jinja2 {% extends "layout.html" %} {% block title %} About {% endblock %} {% block content %} ![...](static/omnirag-pattern-v3.png) ![...](static/deployment-architecture.png) version: {{ application_version }} source: {{ graph_source }} db: {{ graph_source_db }} container: {{ graph_source_container }} {% endblock %} {% block js %} {% endblock %} ``` -------------------------------- ### Configure Azure Cosmos DB for NoSQL Environment Variables Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_nosql.md Lists the essential environment variables required to connect CosmosAIGraph to an Azure Cosmos DB for NoSQL instance. These variables define the source type, connection credentials (URI, key, authentication mechanism), and the names of the database and various containers used by the application. ```bash CAIG_GRAPH_SOURCE_TYPE <-- must be set to 'cosmos_nosql' CAIG_COSMOSDB_NOSQL_URI <-- this value is unique to your Azure deployment CAIG_COSMOSDB_NOSQL_KEY <-- Read/Write key value CAIG_COSMOSDB_NOSQL_AUTH_MECHANISM <-- Authentication mechanism - key or RBAC (Entra ID) CAIG_GRAPH_SOURCE_DB <-- defaults to 'caig' CAIG_GRAPH_SOURCE_CONTAINER <-- defaults to 'libraries' CAIG_CONFIG_CONTAINER <-- defaults to 'config' CAIG_CONVERSATIONS_CONTAINER <-- defaults to 'conversations' CAIG_FEEDBACK_CONTAINER <-- defaults to 'feedback' ``` -------------------------------- ### Vis.js Ontology Graph Visualization Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/views/gen_sparql_console.html This JavaScript code uses the Vis.js library to create and display a network graph representing an ontology. It defines nodes (entities) and edges (relationships) with specific properties, configures graph options such as arrow styles and physics, and initializes the network visualization within a specified HTML container. ```javascript // create an array with the nodes (i.e. - "entities") var nodes = new vis.DataSet([ { id: "Dev", label: "Dev" }, { id: "Doc", label: "Doc" }, { id: "Lib", label: "Lib" } ]); // create an array with the edges (i.e. - "relationships") var edges = new vis.DataSet([ { from: "Dev", to: "Lib", title: "developer_of" }, { from: "Lib", to: "Dev", title: "developed_by" }, { from: "Lib", to: "Lib", title: "used_by_lib, uses_lib" } ]); var html_container = document.getElementById("ontology_viz"); var graph_data = { nodes: nodes, edges: edges }; var graph_options = { edges: { arrows: { to: { enabled: true, scaleFactor: 0.2, type: "arrow" } }, color: '#A9A9A9', font: '12px arial #A9A9A9', scaling: { label: true, }, shadow: false, smooth: true, }, physics:{ enabled: true, repulsion: { centralGravity: 0.2, springLength: 200, springConstant: 0.05, nodeDistance: 200, damping: 0.09 } } }; var network = new vis.Network(html_container, graph_data, graph_options); ``` -------------------------------- ### Verify Data Load with Document Queries in Azure Cosmos DB vCore Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md MongoDB shell commands to query documents from the 'libraries' container. These commands demonstrate how to retrieve a single arbitrary document and how to find specific documents by name, such as the 'flask' document. ```MongoDB Shell db.libraries.findOne() db.libraries.find({name: "flask"}) ``` -------------------------------- ### Generate OWL Ontology RDF/XML with Jinja2 Template Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/templates/owl.txt This Jinja2 template constructs a complete OWL RDF/XML ontology. It iterates over provided data structures (classes, class_attributes, class_relationships) to define OWL classes and object properties, including their labels, comments, domains, and ranges. The template uses placeholders like `{{ ns }}` for namespace and `{{ spacer }}` for formatting, and conditional logic (`{% if 'http' in c -%}`) for resource references. ```Jinja2/XML Custom Ontology File Custom Ontology File {{ spacer }} {%- for c in classes %} {{ c['id'] }} {% endfor %} {{ spacer }} {%- for ca in class_attributes %} {{ ca['aname'] }} {% for c in ca.classes -%} {% if 'http' in c -%} {% else -%} {% endif -%} {% endfor -%} {% endfor %} {{ spacer }} {%- for cr in class_relationships %} {{ cr['rel_name'] }} {% for d in cr.domain_list -%} {% endfor -%} {% for r in cr.range_list -%} {% endfor -%} {% endfor %} ``` -------------------------------- ### Configure Environment Variables for CosmosAIGraph Azure Cosmos DB vCore Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md Lists the essential environment variables required for configuring the CosmosAIGraph project with Azure Cosmos DB vCore, including the source type, connection string, and default container names for various data types. ```Shell CAIG_GRAPH_SOURCE_TYPE <-- must be set to 'cosmos_vcore' CAIG_AZURE_MONGO_VCORE_CONN_STR <-- this value is unique to your Azure deployment CAIG_GRAPH_SOURCE_DB <-- defaults to 'caig' CAIG_GRAPH_SOURCE_CONTAINER <-- defaults to 'libraries' CAIG_CONFIG_CONTAINER <-- defaults to 'config' CAIG_CONVERSATIONS_CONTAINER <-- defaults to 'conversations' CAIG_FEEDBACK_CONTAINER <-- defaults to 'feedback' ``` -------------------------------- ### Application Configuration via Environment Variables Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/understanding_the_code.md The application's configuration is primarily managed through environment variables, following a common pattern for containerized applications. Variables prefixed with CAIG_ are used to control various aspects, including graph sourcing and dumping. While application.properties is used by Spring Boot, AppConfig centralizes environment variable-based configuration. ```APIDOC CAIG_ environment variables: - CAIG_GRAPH_SOURCE_TYPE: Specifies the source for graph data. - Values: "json_docs_file", "rdf_file", "cosmos_nosql" - Description: Determines how the in-memory graph is populated. - CAIG_GRAPH_SOURCE_RDF_FILENAME: (Conditional) Path to RDF file if CAIG_GRAPH_SOURCE_TYPE is "rdf_file". - CAIG_GRAPH_DUMP_UPON_BUILD: (Optional) Boolean, true to dump the graph after build. - CAIG_GRAPH_DUMP_OUTFILE: (Optional) Path to file for dumping the graph. ``` -------------------------------- ### Verify Indexes for 'libraries' Container in Azure Cosmos DB vCore Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md MongoDB shell command to display all indexes defined on the 'libraries' container. This confirms the presence of standard indexes (`_id_`, `id_1`, `name_1`, `libtype_1`) and the crucial `vectorSearchIndex`. ```MongoDB Shell db.libraries.getIndexes() ``` -------------------------------- ### Define a Basic RDF/OWL Ontology with Jinja2 Templating Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/templates/test_owl.txt This XML snippet provides a foundational RDF/OWL ontology structure. It leverages Jinja2 templating to dynamically insert values for the namespace, comment, and label, making it adaptable for various ontology definitions or unit testing scenarios. The template adheres to standard RDF and OWL specifications. ```XML {{ comment }} {{ label }} ``` -------------------------------- ### Load Libraries Dataset into Cosmos DB Libraries Container Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_nosql.md Executes a Python script to populate the 'libraries' container within the 'caig' database with the main dataset. This command specifies the database, container, and an optional maximum number of documents to load, facilitating the bulk import of library information. ```python python main_nosql.py load_libraries caig libraries 999999 ``` -------------------------------- ### Execute Vector Search Command with Python Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_nosql.md This command line snippet demonstrates how to initiate a vector search operation using a Python script. It passes a phrase to the script, which then generates an embedding and performs a semantic search against a Cosmos DB container, returning relevant library documents. ```shell python main_nosql.py vector_search_words asynchronous web framework with pydantic ``` -------------------------------- ### Azure Cosmos DB for NoSQL Container Configuration Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_nosql.md Provides detailed configuration specifications for creating the necessary containers within the 'caig' database for CosmosAIGraph. It specifies the unique indexing requirement for the 'libraries' container (DiskANN vector index) and outlines partition keys and throughput settings for 'config', 'conversations', and 'feedback' containers, along with their respective purposes. ```APIDOC Libraries Container: - Purpose: Stores domain data embeddings. - Indexing: Requires a DiskANN vector index. Other Containers (in 'caig' database): - config: - Partition Key Attribute: /pk - Request Units: 4000 autoscale - Purpose: Contains a document identifying system entities, used by EntitiesService and StrategyBuilder. - conversations: - Partition Key Attribute: /pk - Request Units: 10000 autoscale - Purpose: Records the evolving state of AIConversations. - feedback: - Partition Key Attribute: /conversation_id - Request Units: 4000 autoscale - Purpose: Records feedback for AIConversations, useful for analysis and optimization. ``` -------------------------------- ### Test Graph Service HTTP Endpoints Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/graph_app/readme.md Commands to invoke and test the locally running Java graph service's HTTP endpoints. The service listens on port 8001, and testing can be performed using either a PowerShell client script (`http_client.ps1`) or a generic shell client script (`http_client.sh`). ```PowerShell http_client.ps1 ``` ```Shell http_client.sh ``` -------------------------------- ### Terminate CosmosAIGraph Microservices with Docker Compose Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/local_execution.md This command is used to gracefully shut down and remove the containers, networks, and volumes created by a 'docker compose up' command. It effectively stops the running CosmosAIGraph microservices and cleans up the associated Docker resources. ```Shell docker compose -f docker/docker-compose.yml down ``` -------------------------------- ### Verify Indexes for 'config' Container in Azure Cosmos DB vCore Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md MongoDB shell command to display all indexes defined on the 'config' container. This confirms the presence of the default `_id_` index and the custom `id_1` index. ```MongoDB Shell db.config.getIndexes() ``` -------------------------------- ### Verify Cosmos DB Container Names using MongoDB Shell Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md MongoDB shell command to retrieve a list of all collection names in the current database. This is used to confirm that the expected Cosmos DB containers ('libraries', 'cache', 'config', 'conversations') have been successfully created. ```MongoDB Shell db.getCollectionNames() ``` -------------------------------- ### Manually Create Azure Cosmos DB vCore Vector Search Index Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md MongoDB shell command to manually create a vector search index on the 'libraries' collection within the 'caig' database. This index is crucial for vector similarity searches and is configured with specific `cosmosSearchOptions`. ```MongoDB Shell use caig db.runCommand({ createIndexes: 'libraries', indexes: [ { name: 'vectorSearchIndex', key: { "embedding": "cosmosSearch" }, cosmosSearchOptions: { kind: 'vector-ivf', numLists: 20, similarity: 'COS', dimensions: 1536 } } ] }); ``` -------------------------------- ### Verify Indexes for 'conversations' Container in Azure Cosmos DB vCore Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md MongoDB shell command to display all indexes defined on the 'conversations' container. This verifies the presence of `_id_`, `conversation_id_1`, `created_date_1`, and `created_at_1` indexes. ```MongoDB Shell db.conversations.getIndexes() ``` -------------------------------- ### Verify Indexes for 'cache' Container in Azure Cosmos DB vCore Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/docs/load_cosmos_vcore.md MongoDB shell command to display all indexes defined on the 'cache' container. This helps confirm the presence of the default `_id_` index and the custom `cache_key_1` index. ```MongoDB Shell db.cache.getIndexes() ``` -------------------------------- ### Displaying Conversational AI History with Jinja2 Source: https://github.com/azurecosmosdb/cosmosaigraph/blob/main/impl/web_app/views/conv_ai_console.html This Jinja2 template code is responsible for rendering the conversation history on the web page. It iterates through a list of 'completions' to display each user's query, the AI's response (handling multi-line content), and detailed usage statistics such as prompt tokens, completion tokens, total tokens, and the RAG (Retrieval Augmented Generation) strategy used. ```Jinja2
Conversation ID: {{ conversation_id }}
{% for comp in conv["completions"] %} {% set content_lines = comp["content"].split("\\n") %} {{ comp["user_text"] }} {% for line in content_lines %} {{ line }} {% endfor %} prompt tokens: {{ comp["usage"]["prompt_tokens"] }}, completion tokens: {{ comp["usage"]["completion_tokens"] }}, total tokens: {{ comp["usage"]["total_tokens"] }}, RAG strategy: {{ comp["rag_strategy"] }} {% endfor %} ```