### Install Plotting Libraries

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/text_embedding/nvidia_ai_endpoints.ipynb

Install matplotlib and scikit-learn for visualizing the similarity matrix. These libraries are required for the plotting example.

```python
%pip install --upgrade --quiet  matplotlib scikit-learn
```

--------------------------------

### Install Dependencies with Poetry

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Install project dependencies using Poetry. Ensure Poetry is installed and configured for local virtual environments.

```bash
cd libs/ai-endpoints
pip install poetry                         # if not already installed
poetry config virtualenvs.in-project true --local
poetry install --with test
```

--------------------------------

### Install langchain-nvidia-ai-endpoints

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/providers/nvidia.md

Install the necessary package using pip.

```python
pip install -U --quiet langchain-nvidia-ai-endpoints
```

--------------------------------

### Install langchain-nvidia-ai-endpoints

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/llms/nvidia_ai_endpoints.ipynb

Install the necessary package for NVIDIA NIM integration.

```python
#%pip install -qU langchain-nvidia-ai-endpoints
```

--------------------------------

### Install FAISS package

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/retrievers/nvidia_rerank.ipynb

Installs the FAISS library for creating vector stores. Choose the appropriate command based on your hardware (GPU or CPU).

```python
# Use the following install command that matches your hardware

# %pip install --upgrade --quiet faiss-gpu
%pip install --upgrade --quiet faiss-cpu
```

--------------------------------

### Quick Start with StateGraph and OptimizationConfig

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/langgraph/README.md

Demonstrates replacing LangGraph's StateGraph with langchain_nvidia_langgraph's version and compiling with optional parallel optimization.

```python
from langchain_nvidia_langgraph.graph import StateGraph, OptimizationConfig

graph = StateGraph(ResearchState)
graph.add_node("parse_query", parse_query)
graph.add_edge("parse_query", "classify_intent")
# ... add nodes and edges as usual

# Baseline (vanilla LangGraph)
compiled = graph.compile()

# With parallel optimization
compiled = graph.compile(optimization=OptimizationConfig(enable_parallel=True))
```

--------------------------------

### Install necessary packages

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/retrievers/nvidia_rerank.ipynb

Installs the required Python packages for BM25 retrieval and web scraping.

```python
%pip install --upgrade --quiet langchain-community duckduckgo-search beautifulsoup4 rank_bm25
```

--------------------------------

### Install langchain-nvidia-ai-endpoints

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/text_embedding/nvidia_ai_endpoints.ipynb

Install the necessary package for NVIDIA AI endpoints integration. This should be run before any other code.

```python
%pip install --upgrade --quiet  langchain-nvidia-ai-endpoints
```

--------------------------------

### Install langchain-nvidia-ai-endpoints

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/retrievers/nvidia_rag_retriever.ipynb

Install the necessary package for using NVIDIA AI endpoints.

```python
%pip install --upgrade --quiet langchain-nvidia-ai-endpoints
```

--------------------------------

### Install DDGS package

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/retrievers/nvidia_rerank.ipynb

Installs the DuckDuckGo Search package.

```python
%pip install --upgrade --quiet ddgs
```

--------------------------------

### Install Dependencies

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/structured_report_generation_elastic/structured_report_generation_elastic.ipynb

Installs necessary Python packages for LangGraph, LangChain, Elasticsearch, and NVIDIA AI endpoints. The %%capture magic command suppresses output during installation.

```python
%pip install --quiet -U langgraph langchain_community langchain_core elasticsearch langchain_nvidia_ai_endpoints
```

--------------------------------

### LLM Invoke Example

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/llms/nvidia_ai_endpoints.ipynb

Example of invoking the LLM with a specific prompt related to machine learning.

```python
response = llm.invoke(
    "X_train, y_train, X_test, y_test = train_test_split(X, y, test_size=0.1) #Train a logistic regression model, predict the labels on the test set and compute the accuracy score"
)
print(response)
```

--------------------------------

### Setting up ChatNVIDIA with RunnableWithMessageHistory

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

Integrate ChatNVIDIA with RunnableWithMessageHistory for managing conversational context. This example shows how to set up memory storage and retrieve chat history for a given session ID.

```python
%pip install --upgrade --quiet langchain
```

```python
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory

# store is a dictionary that maps session IDs to their corresponding chat histories.
store = {}  # memory is maintained outside the chain


# A function that returns the chat history for a given session ID.
def get_session_history(session_id: str) -> InMemoryChatMessageHistory:
    if session_id not in store:
        store[session_id] = InMemoryChatMessageHistory()
    return store[session_id]


chat = ChatNVIDIA(
    model="mistralai/mixtral-8x22b-instruct-v0.1",
    temperature=0.1,
    max_tokens=100,
    top_p=1.0,
)

#  Define a RunnableConfig object, with a `configurable` key. session_id determines thread
config = {"configurable": {"session_id": "1"}}

conversation = RunnableWithMessageHistory(
    chat,
    get_session_history,
)

conversation.invoke(
    "Hi I'm Srijan Dubey.",  # input or query
    config=config,
)
```

```python
conversation.invoke(
    "I'm doing well! Just having a conversation with an AI.",
    config=config,
)
```

```python
conversation.invoke(
    "Tell me about yourself.",
    config=config,
)
```

--------------------------------

### GPU Warmup

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_inference_priority.ipynb

Performs 20 concurrent 'ainvoke' calls to warm up the GPU before starting the main test sequence.

```python
print("Warming up GPU with 20 concurrent calls...")
warmup_tasks = [test_llm.ainvoke("Warm up the GPU.", osl=500) for _ in range(20)]
await asyncio.gather(*warmup_tasks)
print("Warmup done.\n")
```

--------------------------------

### Install langchain-nvidia-trt Package

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/trt/docs/llms.ipynb

Install the necessary package to enable Langchain's integration with Triton and TRT-LLM.

```python
# install package
%pip install -U langchain-nvidia-trt
```

--------------------------------

### Install Dependencies

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/structured_report_generation.ipynb

Installs necessary Python libraries including LangGraph, LangChain community, LangChain core, Tavily, and LangChain NVIDIA AI endpoints. The `--quiet` flag suppresses verbose output.

```python
%%capture --no-stderr
%pip install --quiet -U langgraph langchain_community langchain_core tavily-python langchain_nvidia_ai_endpoints
```

--------------------------------

### Batch Invocation with ChatNVIDIA

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/README.md

Demonstrates batch invocation for handling multiple requests concurrently. This example shows the synchronous batch method.

```python
# Batch example
print(llm.batch(["What's 2*3?", "What's 2*6?"]))
```

--------------------------------

### Install Dependencies

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/langgraph_rag_agent_llama3_nvidia_nim.ipynb

Installs necessary Python packages for LangChain, NVIDIA AI endpoints, LangGraph, and other utilities.

```python
%pip install -U langchain-nvidia-ai-endpoints langchain-community langchain langgraph tavily-python beautifulsoup4 lxml
```

--------------------------------

### Install Dependencies

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/nvidia_nim_agents_llama3.1.ipynb

Installs necessary Python packages for LangChain, NVIDIA AI Endpoints, community integrations, OpenAI compatibility, Tavily Search, and geocoding.

```python
%pip install -U langchain langgraph langchain-nvidia-ai-endpoints langchain-community langchain-openai langchain-tavily geocoder
```

--------------------------------

### Async Batch Invocation with ChatNVIDIA

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/README.md

Demonstrates asynchronous batch invocation for handling multiple requests concurrently. This example shows the asynchronous batch method.

```python
# Batch example (async)
await llm.abatch(["What's 2*3?", "What's 2*6?"])
```

--------------------------------

### Priority Scheduling Under Sustained Load Setup

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_inference_priority.ipynb

Sets up a test environment to demonstrate fine-grained, monotonic priority ordering under sustained GPU contention. This involves configuring background workers and defining test tiers with varying priorities.

```python
import asyncio
import statistics
import time
from dataclasses import dataclass
from typing import Any

import aiohttp
from langchain_nvidia_ai_endpoints import ChatNVIDIADynamo, inference_priority
from data.queries import QUERIES

# ── Test configuration ─────────────────────────────────────────────────────────
# Fixed for ALL requests — only priority changes between tiers.
TEST_MAX_TOKENS = 2048
OSL = 2048
LATENCY_SENSITIVITY = 1.0
IAT = 250

# Background load
BACKGROUND_WORKERS = 100
BACKGROUND_PRIORITY = 10
RAMP_UP_SECONDS = 5

# Test tiers — 95 requests injected per run
TEST_TIERS: list[dict[str, Any]] = [
    {"priority": 1,  "label": "CRITICAL",    "count": 5},
    {"priority": 3,  "label": "HIGH",        "count": 10},
    {"priority": 5,  "label": "MEDIUM",      "count": 20},
    {"priority": 7,  "label": "LOW",         "count": 40},
    {"priority": 10, "label": "BACKGROUND",  "count": 20},
]

# ── Single LLM instance — priority is set by decorator, not by the instance ───
test_llm = ChatNVIDIADynamo(
    base_url=NIM_BASE_URL,
    model=MODEL,
    max_completion_tokens=TEST_MAX_TOKENS,
)

# Override the default aiohttp session factory to allow longer request times.
# Under sustained load, low-priority requests can take 10+ minutes — well beyond

```

--------------------------------

### Register Jupyter Kernel

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Install ipykernel and register a Jupyter kernel for the project's virtual environment.

```bash
.venv/bin/pip install ipykernel
.venv/bin/python -m ipykernel install --user --name langchain-nvidia --display-name "langchain-nvidia"
```

--------------------------------

### Structured Output with Pydantic

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

Demonstrates how to use `with_structured_output` with a Pydantic model to get structured responses from ChatNVIDIA. Ensure the selected model supports structured output.

```python
from pydantic import BaseModel, Field

class Person(BaseModel):
    first_name: str = Field(..., description="The person's first name.")
    last_name: str = Field(..., description="The person's last name.")

llm = ChatNVIDIA(model=structured_models[0].id).with_structured_output(Person)
response = llm.invoke("Who is Michael Jeffrey Jordon?")
response
```

--------------------------------

### NVIDIA Ranking Model

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/README.md

Utilize NVIDIARerank for document ranking tasks. This example demonstrates how to compress documents based on a given query.

```python
from langchain_nvidia_ai_endpoints import NVIDIARerank
from langchain_core.documents import Document

query = "What is the GPU memory bandwidth of H100 SXM?"
passages = [
    "The Hopper GPU is paired with the Grace CPU using NVIDIA's ultra-fast chip-to-chip interconnect, delivering 900GB/s of bandwidth, 7X faster than PCIe Gen5. This innovative design will deliver up to 30X higher aggregate system memory bandwidth to the GPU compared to today's fastest servers and up to 10X higher performance for applications running terabytes of data.",
    "A100 provides up to 20X higher performance over the prior generation and can be partitioned into seven GPU instances to dynamically adjust to shifting demands. The A100 80GB debuts the world's fastest memory bandwidth at over 2 terabytes per second (TB/s) to run the largest models and datasets.",
    "Accelerated servers with H100 deliver the compute power—along with 3 terabytes per second (TB/s) of memory bandwidth per GPU and scalability with NVLink and NVSwitch™.",
]

client = NVIDIARerank(model="nvidia/llama-3.2-nv-rerankqa-1b-v1")

response = client.compress_documents(
  query=query,
  documents=[Document(page_content=passage) for passage in passages]
)

print(f"Most relevant: {response[0].page_content}\nLeast relevant: {response[-1].page_content}")
```

--------------------------------

### Completions API with NVIDIA

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/README.md

Utilize models supporting the Completions API by providing a prompt instead of messages. This example shows how to retrieve available models and generate text.

```python
from langchain_nvidia_ai_endpoints import NVIDIA

completions_llm = NVIDIA().bind(max_tokens=512)
[model.id for model in completions_llm.get_available_models()]

# [
#   ...
#   'bigcode/starcoder2-7b',
#   'bigcode/starcoder2-15b',
#   ...
# ]
```

```python
prompt = "# Function that does quicksort written in Rust without comments:"
for chunk in completions_llm.stream(prompt):
    print(chunk, end="", flush=True)
```

--------------------------------

### Invoke LLM with a Prompt

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/llms/nvidia_ai_endpoints.ipynb

Send a prompt to the NVIDIA LLM and get a completion. This is the basic synchronous invocation.

```python
prompt = "# Function that does quicksort written in Rust without comments:"
print(llm.invoke(prompt))
```

--------------------------------

### Document Grader Setup

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/langgraph_rag_agent_llama3_nvidia_nim.ipynb

Defines the instructions and prompt for a document grader, which assesses the relevance of retrieved documents to a user's question. It uses a structured output format for binary scoring ('yes' or 'no').

```python
doc_grader_instructions = """You are a grader assessing relevance of a retrieved document to a user question.

If the document contains keyword(s) or semantic meaning related to the question, grade it as relevant."""

# Grader prompt
doc_grader_prompt = """Here is the retrieved document: \n\n {document} \n\n Here is the user question: \n\n {question}. 

This carefully and objectively assess whether the document contains at least some information that is relevant to the question.

Return structured output with single key, binary_score, that is 'yes' or 'no' score to indicate whether the document contains at least some information that is relevant to the question."""

# Test
question = "What is Chain of thought prompting?"
docs = retriever.invoke(question)
doc_txt = docs[1].page_content
doc_grader_prompt_formatted = doc_grader_prompt.format(
    document=doc_txt, question=question
)
result = structured_llm_grader.invoke(
    [SystemMessage(content=doc_grader_instructions)]
    + [HumanMessage(content=doc_grader_prompt_formatted)]
)
result
```

--------------------------------

### Prompt for Report Planner Instructions

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/structured_report_generation.ipynb

System instructions for an LLM acting as an expert technical writer to generate a report outline. It guides the LLM to create sections with name, description, research flag, and content (initially blank), considering the provided topic, structure, and research context.

```python
report_planner_instructions="""You are an expert technical writer, helping to plan a report.

Your goal is to generate the outline of the sections of the report. 

The overall topic of the report is:

{topic}

The report should follow this organization: 

{report_organization}

You should reflect on this information to plan the sections of the report: 

{context}

Now, generate the sections of the report. Each section should have the following fields:

- Name - Name for this section of the report.
- Description - Brief overview of the main topics and concepts to be covered in this section.
- Research - Whether to perform web research for this section of the report.
- Content - The content of the section, which you will leave blank for now.

Consider which sections require web research. For example, introduction and conclusion will not require research because they will distill information from other parts of the report."""
```

--------------------------------

### Set NVIDIA API Key

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/llms/nvidia_ai_endpoints.ipynb

Configure the NVIDIA API key from environment variables or prompt the user for input. Ensure the key starts with 'nvapi-'.

```python
import os
from getpass import getpass

# del os.environ['NVIDIA_API_KEY']  ## delete key and reset
if os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
    print("Valid NVIDIA_API_KEY already in environment. Delete to reset")
else:
    candidate_api_key = getpass("NVAPI Key (starts with nvapi-): ")
    assert candidate_api_key.startswith("nvapi-"), f"{candidate_api_key[:5]}... is not a valid key"
    os.environ["NVIDIA_API_KEY"] = candidate_api_key
```

--------------------------------

### Set NVIDIA API Key

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/text_embedding/nvidia_ai_endpoints.ipynb

Configure your NVIDIA API key by either setting the NVIDIA_API_KEY environment variable or by entering it when prompted. The key must start with 'nvapi-'.

```python
import getpass
import os

if os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
    print("Valid NVIDIA_API_KEY already in environment. Delete to reset")
else:
    nvapi_key = getpass.getpass("NVAPI Key (starts with nvapi-): ")
    assert nvapi_key.startswith(
        "nvapi-" 
    ), f"{nvapi_key[:5]}... is not a valid key"
    os.environ["NVIDIA_API_KEY"] = nvapi_key
```

--------------------------------

### Structured Output with Enum

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

Shows how to use `with_structured_output` with an Enum for generating responses that conform to a predefined set of choices. This example uses a specific model ID from the structured_models list.

```python
from enum import Enum

class Choices(Enum):
    A = "A"
    B = "B"
    C = "C"

llm = ChatNVIDIA(model=structured_models[2].id).with_structured_output(Choices)
response = llm.invoke("""
        What does 1+1 equal?
            A. -100
            B. 2
            C. doorstop
        """
)
response
```

--------------------------------

### Initialize BM25 Retriever

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/retrievers/nvidia_rerank.ipynb

Sets up the BM25 retriever by defining a search tool, a text splitter, and then building the documents from web search results. This might take a few minutes to run.

```python
# This might take a few minutes to run.

bm25_tool = lambda query: DuckDuckGoSearchAPIWrapper().results(query, max_results=100, source="text")
bm25_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200, length_function=len)
bm25_retriever = BM25Retriever.from_documents(build_documents(query, bm25_tool, bm25_splitter, "DuckDuckGo Text"))
```

--------------------------------

### Configure ChatNVIDIADynamo with Default Hints

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Instantiate ChatNVIDIADynamo with specific Dynamo hints for latency-critical, high-priority requests with short expected outputs.

```python
# High-priority: short responses, latency-critical
llm_critical = ChatNVIDIADynamo(
    base_url=NIM_BASE_URL,
    model=MODEL,
    max_completion_tokens=MAX_TOKENS,
    osl=20,
    priority=10,
    latency_sensitivity=1.0,
)
```

--------------------------------

### RAG Generation Prompt and Formatting

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/langgraph_rag_agent_llama3_nvidia_nim.ipynb

Sets up the prompt for the RAG generation step and defines a helper function to format retrieved documents for inclusion in the prompt. This ensures the LLM receives context in a usable format.

```python
# Prompt
rag_prompt = """You are an assistant for question-answering tasks. 

Here is the context to use to answer the question:

{context} 

Think carefully about the above context. 

Now, review the user question:

{question}

Provide an answer to this questions using only the above context. 

Use three sentences maximum and keep the answer concise.

Answer:"""

# Post-processing
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# Test
docs = retriever.invoke(question)
docs_txt = format_docs(docs)
rag_prompt_formatted = rag_prompt.format(context=docs_txt, question=question)
generation = llm.invoke([HumanMessage(content=rag_prompt_formatted)])
print(generation.content)
```

--------------------------------

### Initialize NVIDIA LLM

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/llms/nvidia_ai_endpoints.ipynb

Import and initialize the NVIDIA LLM class for text completion. You can bind parameters like `max_tokens` during initialization.

```python
from langchain_nvidia_ai_endpoints import NVIDIA
```

```python
llm = NVIDIA().bind(max_tokens=256)
llm
```

--------------------------------

### Connect to Local NVIDIA NIM Microservices

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/text_embedding/nvidia_ai_endpoints.ipynb

Demonstrates how to connect to locally hosted NVIDIA NIM microservices for chat, embeddings, and reranking by specifying their respective base URLs and model names.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings, NVIDIARerank

# connect to an chat NIM running at localhost:8000, specifying a specific model
llm = ChatNVIDIA(base_url="http://localhost:8000/v1", model="meta/llama3-8b-instruct")

# connect to an embedding NIM running at localhost:8080
embedder = NVIDIAEmbeddings(base_url="http://localhost:8080/v1")

# connect to a reranking NIM running at localhost:2016
ranker = NVIDIARerank(base_url="http://localhost:2016/v1")
```

--------------------------------

### Start Background Workers

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_inference_priority.ipynb

Initializes and starts the specified number of background workers, each running at a defined priority, and waits for the ramp-up period to complete.

```python
stop_event = asyncio.Event()
bg_counter: list[int] = [0]
bg_tasks = [
    asyncio.create_task(background_worker(i, stop_event, bg_counter))
    for i in range(BACKGROUND_WORKERS)
]
print(f"Started {BACKGROUND_WORKERS} background workers (priority={BACKGROUND_PRIORITY})")
print(f"Ramping up for {RAMP_UP_SECONDS}s to fill queue...")
await asyncio.sleep(RAMP_UP_SECONDS)
print(f"Ramp-up done. Background requests completed so far: {bg_counter[0]}\n")
```

--------------------------------

### End-to-End Tool Invocation Process

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/nvidia_nim_agents_llama3.1.ipynb

Demonstrates the complete process of invoking a tool with an LLM, handling the tool call, and printing the result.

```python
ai_response = llm_with_tools.invoke("What's the weather in San Francisco?")
if ai_response.tool_calls:
    tool_call = ai_response.tool_calls[0]
    tools_by_name = {t.name: t for t in tools}
    result = tools_by_name[tool_call["name"]].invoke(tool_call["args"])
    print(f"Tool: {tool_call['name']}")
    print(f"Result: {result}")
```

--------------------------------

### Get Supported Models with Langchain NVIDIA

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/README.md

Query available_models to get a list of models accessible with your API credentials. Ensure you have the necessary API credentials configured.

```python
[model.id for model in llm.available_models if model.model_type]
```

--------------------------------

### Connect to Local NVIDIA NIM Microservices

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

Configure `ChatNVIDIA`, `NVIDIAEmbeddings`, and `NVIDIARerank` to connect to locally hosted NIM microservices by specifying the `base_url` for each service.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings, NVIDIARerank

# connect to a chat NIM running at localhost:8000, specifying a specific model
llm = ChatNVIDIA(base_url="http://localhost:8000/v1", model="meta/llama3-8b-instruct")

# connect to an embedding NIM running at localhost:8080
embedder = NVIDIAEmbeddings(base_url="http://localhost:8080/v1")

# connect to a reranking NIM running at localhost:2016
ranker = NVIDIARerank(base_url="http://localhost:2016/v1")
```

--------------------------------

### Get Available Models

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/llms/nvidia_ai_endpoints.ipynb

Retrieve a list of all models available through your NVIDIA API credentials.

```python
NVIDIA.get_available_models()
# llm.get_available_models()
```

--------------------------------

### Initialize ToolNode with Custom and Built-in Tools

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/nvidia_nim_agents_llama3.1.ipynb

This code initializes a `ToolNode` with a list of tools, including the custom `get_current_location` tool and the Tavily search tool. It also binds these tools to the LLM for agent use.

```python
from langgraph.prebuilt import ToolNode

# Declare two tools: Tavily and custom get_current_location tool.
tools = [TavilySearch(max_results=1), get_current_location]
tool_node = ToolNode(tools)

# be sure to bind the updated tools to the LLM!
llm_with_tools = llm.bind_tools(tools)
```

--------------------------------

### Configure ChatNVIDIADynamo with Priority and Latency Sensitivity

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Instantiate ChatNVIDIADynamo with specific `osl`, `priority`, and `latency_sensitivity` settings to tune model behavior for different use cases like critical or background tasks.

```python
llm_critical = ChatNVIDIADynamo(
    base_url=NIM_BASE_URL,
    model=MODEL,
    max_completion_tokens=MAX_TOKENS,
    osl=512,
    priority=10,
    latency_sensitivity=1.0,
)

# Low-priority: long responses, latency-tolerant
llm_background = ChatNVIDIADynamo(
    base_url=NIM_BASE_URL,
    model=MODEL,
    max_completion_tokens=MAX_TOKENS,
    osl=512,
    priority=1,
    latency_sensitivity=0.1,
)

print(f"Critical:   osl={llm_critical.osl}, priority={llm_critical.priority}, "
      f"latency_sensitivity={llm_critical.latency_sensitivity}")
print(f"Background: osl={llm_background.osl}, priority={llm_background.priority}, "
      f"latency_sensitivity={llm_background.latency_sensitivity}")
```

--------------------------------

### Basic ChatNVIDIA and ChatNVIDIADynamo Usage

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Instantiate and invoke both standard ChatNVIDIA and ChatNVIDIADynamo models. ChatNVIDIADynamo automatically injects agent hints for Dynamo.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA, ChatNVIDIADynamo

# Standard ChatNVIDIA — no Dynamo hints
llm_standard = ChatNVIDIA(base_url=NIM_BASE_URL, model=MODEL, max_completion_tokens=MAX_TOKENS)

# ChatNVIDIADynamo — identical interface, automatically injects agent_hints
llm = ChatNVIDIADynamo(base_url=NIM_BASE_URL, model=MODEL, max_completion_tokens=MAX_TOKENS)

result = llm.invoke("What is KV cache optimization?")
print(result.content)
```

--------------------------------

### Initialize Multimodal Chat Model

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

Initialize the ChatNVIDIA model for multimodal capabilities, specifying the model name.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA

llm = ChatNVIDIA(model="nvidia/neva-22b")
```

--------------------------------

### Configure NIM Deployment Parameters

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Set the base URL and model name for the local NIM deployment, along with the maximum completion tokens.

```python
NIM_BASE_URL = "http://localhost:8099/v1"
MODEL = "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16"
MAX_TOKENS = 4096
```

--------------------------------

### Get Available ChatNVIDIA Models

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

Retrieve a list of all models available through your NVIDIA API credentials using the static `get_available_models` method on the `ChatNVIDIA` class.

```python
ChatNVIDIA.get_available_models()
```

--------------------------------

### Initialize Elasticsearch Clients

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/structured_report_generation_elastic/structured_report_generation_elastic.ipynb

Initializes both synchronous and asynchronous Elasticsearch clients using credentials from environment variables. This allows for both blocking and non-blocking interactions with Elasticsearch.

```python
from elasticsearch import Elasticsearch, AsyncElasticsearch
es_client = Elasticsearch(
    hosts=[f"http://{os.environ['ELASTIC_HOST']}:{os.environ['ELASTIC_PORT']}"],
)
es_async_client = AsyncElasticsearch(
    hosts=[f"http://{os.environ['ELASTIC_HOST']}:{os.environ['ELASTIC_PORT']}"],
)
```

--------------------------------

### Verify NVIDIA API Key

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/providers/nvidia.md

Verify your NVIDIA API Key by checking the environment variable or prompting the user for input. Asserts that the key starts with 'nvapi-'.

```python
import getpass
import os

if os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
    print("Valid NVIDIA_API_KEY already in environment. Delete to reset")
else:
    nvapi_key = getpass.getpass("NVAPI Key (starts with nvapi-): ")
    assert nvapi_key.startswith(
        "nvapi-"
    ), f"{nvapi_key[:5]}... is not a valid key"
    os.environ["NVIDIA_API_KEY"] = nvapi_key
```

--------------------------------

### LangChain Chain Integration with ChatNVIDIADynamo

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Integrate ChatNVIDIADynamo into LangChain chains for complex workflows. This example shows a classifier chain that uses the model to determine user intent.

```python
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "Classify the user's intent into exactly one category: "
               "billing, technical_support, general_inquiry, or complaint. "
               "Respond with only the category name."),
    ("user", "{input}"),
])

classifier_chain = (
    prompt
    | ChatNVIDIADynamo(
        base_url=NIM_BASE_URL,
        model=MODEL,
        max_completion_tokens=MAX_TOKENS,
        osl=5,
        priority=10,
        latency_sensitivity=1.0,
    )
    | StrOutputParser()
)

intent = classifier_chain.invoke({"input": "My invoice has the wrong amount"})
print(f"Detected intent: {intent}")
```

--------------------------------

### Configure LangChain Tracing

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/langgraph_rag_agent_llama3_nvidia_nim.ipynb

Sets up environment variables for LangChain tracing, enabling observability of agent execution in LangSmith.

```python
import getpass
import os

os.environ['LANGCHAIN_TRACING_V2'] = 'true'
os.environ['LANGCHAIN_ENDPOINT'] = 'https://api.smith.langchain.com'
os.environ['LANGCHAIN_API_KEY'] = '<your-api-key>'
```

--------------------------------

### General Chat with Langchain NVIDIA

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/README.md

Utilize ChatNVIDIA for general chat messages with models like Nemotron or Mixtral. This example sets up a prompt template and a chain for streaming responses.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful AI assistant named Fred."),
        ("user", "{input}")
    ]
)
chain = (
    prompt
    | ChatNVIDIA(model="nvidia/nemotron-3-super-120b-a12b")
    | StrOutputParser()
)

for txt in chain.stream({"input": "What's your name?"}):
    print(txt, end="")
```

--------------------------------

### Structured Output with Enum and Specific Model ID

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

This example demonstrates using `with_structured_output` with an Enum, explicitly specifying the model ID. It prints the model ID before invoking the LLM.

```python
model = structured_models[3].id
llm = ChatNVIDIA(model=model).with_structured_output(Choices)
print(model)
response = llm.invoke("""
        What does 1+1 equal?
            A. -100
            B. 2
            C. doorstop
        """
)
response
```

--------------------------------

### Code Generation with NVIDIA Models

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

Utilize models like CodeLlama for code generation tasks. The setup is similar to general chat, but the system prompt should emphasize code-only responses.

```python
prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an expert coding AI. Respond only in valid python; no narration whatsoever.",
        ),
        ("user", "{input}"),
    ]
)
chain = prompt | ChatNVIDIA(model="meta/codellama-70b") | StrOutputParser()

for txt in chain.stream({"input": "How do I solve this fizz buzz problem?"}):
    print(txt, end="")
```

--------------------------------

### Implementing Tool Calling with ChatNVIDIA

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

Demonstrates how to bind tools to a ChatNVIDIA model for tool calling. Ensure you select a model that is trained for tool calling and define your tools using Pydantic and LangChain's `@tool` decorator.

```python
from pydantic import Field
from langchain_core.tools import tool

@tool
def get_current_weather(
    location: str = Field(..., description="The location to get the weather for.")
):
    """Get the current weather for a location."""
    ...
```

```python
llm = ChatNVIDIA(model=tool_models[0].id).bind_tools(tools=[get_current_weather])
response = llm.invoke("What is the weather in Boston?")
response.tool_calls
```

--------------------------------

### Initialize Tavily Clients

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/structured_report_generation.ipynb

Initializes both synchronous and asynchronous Tavily clients. These clients are used for performing web searches.

```python
from tavily import TavilyClient, AsyncTavilyClient
tavily_client = TavilyClient()
tavily_async_client = AsyncTavilyClient()
```

--------------------------------

### Code Generation with Langchain NVIDIA

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/README.md

Employ ChatNVIDIA with code generation models like Codellama or Codegemma for tasks involving code. This example configures the model to respond only in valid Python.

```python
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are an expert coding AI. Respond only in valid python; no narration whatsoever."),
        ("user", "{input}")
    ]
)
chain = (
    prompt
    | ChatNVIDIA(model="meta/codellama-70b", max_tokens=419)
    | StrOutputParser()
)

for txt in chain.stream({"input": "How do I solve this fizz buzz problem?"}):
    print(txt, end="")
```

--------------------------------

### Connect to Locally Hosted NVIDIA NIM Microservices

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/providers/nvidia.md

Connect to local NIM microservices for chat, embeddings, and reranking. Specify the base URL and model for each service.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings, NVIDIARerank

# connect to an chat NIM running at localhost:8000, specifyig a specific model
llm = ChatNVIDIA(base_url="http://localhost:8000/v1", model="meta/llama3-8b-instruct")

# connect to an embedding NIM running at localhost:8080
embedder = NVIDIAEmbeddings(base_url="http://localhost:8080/v1")

# connect to a reranking NIM running at localhost:2016
ranker = NVIDIARerank(base_url="http://localhost:2016/v1")
```

--------------------------------

### Hallucination Grader Setup

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/langgraph_rag_agent_llama3_nvidia_nim.ipynb

Configures a grader to detect hallucinations in the generated answer by comparing it against the provided facts. It uses a Pydantic model for structured output and defines specific grading instructions.

```python
# Data model
class GradeHallucinations(BaseModel):
    """Binary score for hallucination present in generation answer."""

    binary_score: str = Field(
        description="Answer is grounded in the facts, 'yes' or 'no'"
    )

# LLM with structured output
structured_llm_grader = llm.with_structured_output(GradeDocuments)

# Hallucination grader instructions
hallucination_grader_instructions = """

You are a teacher grading a quiz. 

You will be given FACTS and a STUDENT ANSWER. 

Here is the grade criteria to follow:

(1) Ensure the STUDENT ANSWER is grounded in the FACTS. 

(2) Ensure the STUDENT ANSWER does not contain "hallucinated" information outside the scope of the FACTS.

Score:

A score of yes means that the student's answer meets all of the criteria. This is the highest (best) score. 

A score of no means that the student's answer does not meet all of the criteria. This is the lowest possible score you can give.

Explain your reasoning in a step-by-step manner to ensure your reasoning and conclusion are correct. 

Avoid simply stating the correct answer at the outset."""

# Grader prompt
hallucination_grader_prompt = """FACTS: \n\n {documents} \n\n STUDENT ANSWER: {generation}. 

Return structured output with two two keys, binary_score is 'yes' or 'no' score to indicate whether the STUDENT ANSWER is grounded in the FACTS."""

# Test using documents and generation from above
hallucination_grader_prompt_formatted = hallucination_grader_prompt.format(
    documents=docs_txt, generation=generation.content
)
result = structured_llm_grader.invoke(
    [SystemMessage(content=hallucination_grader_instructions)]
    + [HumanMessage(content=hallucination_grader_prompt_formatted)]
)
result
```

--------------------------------

### Get Models Supporting Structured Output

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints.ipynb

Retrieve a list of available NVIDIA AI models that support structured output features. This is useful for selecting appropriate models for structured data generation.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA
structured_models = [model for model in ChatNVIDIA.get_available_models() if model.supports_structured_output]
structured_models
```

--------------------------------

### Adding Optimization to Existing LangGraph Graphs

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/langgraph/README.md

Demonstrates how to apply optimization to an existing StateGraph built with standard LangGraph using the with_app_compile utility.

```python
from langchain_nvidia_langgraph.graph import with_app_compile, OptimizationConfig

graph = create_baseline_graph()  # returns standard StateGraph
compilable = with_app_compile(graph)
compiled = compilable.compile(optimization=OptimizationConfig(enable_parallel=True))
```

--------------------------------

### Low-priority Request Payload for NVIDIA Dynamo

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Example JSON payload for a low-priority request, suitable for parallel background analysis. Features agent hints that indicate lower priority and higher latency tolerance.

```json
{
  "model": "meta/llama-3.1-8b-instruct",
  "messages": [{"role": "user", "content": "..."}],
  "nvext": {
    "agent_hints": {
      "prefix_id": "langchain-dynamo-f6e5d4c3b2a1",
      "osl": 500,
      "iat": 250,
      "latency_sensitivity": 0.1,
      "priority": 1
    }
  }
}
```

--------------------------------

### High-priority Request Payload for NVIDIA Dynamo

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Example JSON payload for a high-priority request, typically used for critical path operations like classification or review. Includes agent hints for latency sensitivity and priority.

```json
{
  "model": "meta/llama-3.1-8b-instruct",
  "messages": [{"role": "user", "content": "..."}],
  "nvext": {
    "agent_hints": {
      "prefix_id": "langchain-dynamo-a1b2c3d4e5f6",
      "osl": 10,
      "iat": 250,
      "latency_sensitivity": 1.0,
      "priority": 10
    }
  }
}
```

--------------------------------

### Streaming Responses with Dynamo Hints

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/docs/chat/nvidia_ai_endpoints_dynamo.ipynb

Utilize the streaming capability of ChatNVIDIADynamo to receive tokens as they are generated. Dynamo hints are included in the initial request to help the Smart Router select the optimal worker.

```python
for chunk in llm_critical.stream("Give a one-sentence summary of GPU computing."):
    print(chunk.content, end="", flush=True)
print()  # newline
```

--------------------------------

### Set NVIDIA API Key Environment Variable

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/structured_report_generation.ipynb

Prompts the user to enter their NVIDIA API key and asserts it starts with 'nvapi-'. The key is then stored as an environment variable 'NVIDIA_API_KEY'. This is crucial for authenticating with NVIDIA NIM endpoints.

```python
import getpass
import os

if not os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
    nvapi_key = getpass.getpass("Enter your NVIDIA API key: ")
    assert nvapi_key.startswith("nvapi-"), f"{nvapi_key[:5]}... is not a valid key"
    os.environ["NVIDIA_API_KEY"] = nvapi_key
```

--------------------------------

### Define a Custom Tool for Location

Source: https://github.com/langchain-ai/langchain-nvidia/blob/main/cookbook/nvidia_nim_agents_llama3.1.ipynb

This snippet defines a custom tool using `@tool` decorator to get the user's current location based on IP address. It requires the `geocoder` library and returns latitude and longitude.

```python
import geocoder
from langchain.tools import tool
from typing import Tuple

@tool
def get_current_location() -> list:
    """Return the current location of the user based on IP address"""
    loc = geocoder.ip('me')
    return loc.latlng
```