### Install LlamaIndex Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/llamaindex/llama_index_quickstart.ipynb

Install necessary libraries for LlamaIndex, TruLens, and OpenAI integration. This is a prerequisite for running the quickstart.

```python
# !pip install trulens trulens-apps-llamaindex trulens-providers-openai llama_index openai
```

--------------------------------

### Install Frontend Dependencies

Source: https://github.com/truera/trulens/blob/main/src/dashboard/react_components/record_viewer/README.md

Install all necessary frontend packages for the record viewer.

```bash
npm i
```

--------------------------------

### Setup LlamaIndex and TruLens Environment

Source: https://github.com/truera/trulens/blob/main/examples/experimental/llamaindex_async_stream.ipynb

Initializes the LlamaIndex environment, downloads necessary data, and sets up the TruLens session. This code is a prerequisite for all subsequent examples.

```python
import os
from pathlib import Path
import sys

import dotenv
from llama_index.core import Settings
from llama_index.core import SimpleDirectoryReader
from llama_index.core import VectorStoreIndex
from llama_index.llms.openai import OpenAI
from trulens.apps.llamaindex import TruLlama
from trulens.core.session import TruSession

dotenv.load_dotenv()

repo = Path.cwd().parent.parent
if str(repo) not in sys.path:
    sys.path.append(str(repo))

if not (Path("data") / "paul_graham_essay.txt").exists():
    os.system(
        "wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt -P data/"
    )

message = "What did the author do growing up?"

Settings.llm = OpenAI(model="gpt-3.5-turbo", temperature=0.0)
Settings.num_output = 64

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)

session = TruSession()
session.reset_database()
session.run_dashboard()
```

--------------------------------

### Install Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/expositional/models/google/gemini_multi_modal_evaluation.ipynb

Installs the required libraries for TruLens, Google Gemini provider, and Google GenAI. Use this at the beginning of your project setup.

```python
!
```

--------------------------------

### Generate Test Set with Few-Shot Examples

Source: https://github.com/truera/trulens/blob/main/docs/component_guides/evaluation/generate_test_cases.md

Provide a list of example prompts to guide the LLM in generating test cases that align with specific question types or desired outputs.

```python
examples = [
  "What is sensory memory?",
  "How much information can be stored in short term memory?"
]

fewshot_test_set = test.generate_test_set(
  test_breadth = 3,
  test_depth = 2,
  examples = examples
)
fewshot_test_set
```

--------------------------------

### Instrument MCP Client with TruLens

Source: https://github.com/truera/trulens/blob/main/docs/component_guides/instrumentation/mcp.md

This example shows a complete setup of an MCP client instrumented with TruLens. It includes defining the client, its tools, and the instrumentation decorators for tracking tool calls and responses. The TruLens setup configures metrics for evaluating tool selection, calling, and quality.

```python
import json
from trulens.core import TruSession, Metric, Selector
from trulens.apps.app import TruApp
from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes
from trulens.providers.openai import OpenAI

# --- App definition ---

class WeatherMCPClient:
    """Simulates an MCP client that calls a weather tool."""

    TOOLS = {
        "get_weather": {
            "description": "Get current weather for a city",
            "input_schema": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "units": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city"],
            },
        }
    }

    @instrument(
        span_type=SpanAttributes.SpanType.MCP,
        attributes={
            SpanAttributes.MCP.TOOL_NAME: "tool_name",
            SpanAttributes.MCP.TOOL_DESCRIPTION: lambda ret, exc, *args, **kwargs: (
                WeatherMCPClient.TOOLS.get(kwargs.get("tool_name", ""), {}).get(
                    "description", ""
                )
            ),
            SpanAttributes.MCP.INPUT_ARGUMENTS: lambda ret, exc, *args, **kwargs: (
                json.dumps(kwargs.get("arguments", {}))
            ),
            SpanAttributes.MCP.OUTPUT_CONTENT: "return",
            SpanAttributes.MCP.OUTPUT_IS_ERROR: lambda ret, exc, *args, **kwargs: (
                exc is not None
            ),
        },
    )
    def call_tool(self, tool_name: str, arguments: dict) -> str:
        """Call an MCP tool."""
        if tool_name == "get_weather":
            city = arguments.get("city", "Unknown")
            units = arguments.get("units", "celsius")
            # Simulated tool response
            return json.dumps({
                "city": city,
                "temperature": 22 if units == "celsius" else 72,
                "units": units,
                "condition": "Partly cloudy",
            })
        raise ValueError(f"Unknown tool: {tool_name}")

    @instrument(
        span_type=SpanAttributes.SpanType.RECORD_ROOT,
        attributes={
            SpanAttributes.RECORD_ROOT.INPUT: "query",
            SpanAttributes.RECORD_ROOT.OUTPUT: "return",
        },
    )
    def answer(self, query: str) -> str:
        """Answer a user query by calling MCP tools."""
        result = self.call_tool("get_weather", {"city": "San Francisco", "units": "celsius"})
        data = json.loads(result)
        return (
            f"The weather in {data['city']} is {data['temperature']}°C "
            f"and {data['condition']}."
        )


# --- TruLens setup ---

session = TruSession()
provider = OpenAI()

metrics = [
    Metric(
        implementation=provider.tool_selection_with_cot_reasons,
        name="Tool Selection",
        selectors={"trace": Selector(trace_level=True)},
    ),
    Metric(
        implementation=provider.tool_calling_with_cot_reasons,
        name="Tool Calling",
        selectors={"trace": Selector(trace_level=True)},
    ),
    Metric(
        implementation=provider.tool_quality_with_cot_reasons,
        name="Tool Quality",
        selectors={"trace": Selector(trace_level=True)},
    ),
]

mcp_client = WeatherMCPClient()

tru_app = TruApp(
    mcp_client,
    app_name="WeatherMCPClient",
    app_version="v1",
    metrics=metrics,
)

# --- Run and evaluate ---

with tru_app as recording:
    response = mcp_client.answer("What's the weather in San Francisco?")

print(response)

session.get_leaderboard()

```

--------------------------------

### Setup Python Environment and Imports

Source: https://github.com/truera/trulens/blob/main/examples/experimental/end2end_apps/trubot/trubot_example.ipynb

Configures the Python environment by adjusting the system path for local development and optionally enables detailed logging. Imports necessary modules for the TruBot example.

```python
%load_ext autoreload
%autoreload 2
from pathlib import Path
import sys

# If running from github repo, can use this:
sys.path.append(str(Path().cwd().parent.parent.parent.parent.resolve()))

# Uncomment for more debugging printouts.
"""
import logging
root = logging.getLogger()
root.setLevel(logging.DEBUG)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
root.addHandler(handler)
"""
None
```

--------------------------------

### Install Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/openai_agent_sdk/openai_agent_sdk_snowflake_tools/README.md

Synchronize Python dependencies using uv and install Node.js dependencies for the frontend.

```bash
uv sync
cd frontend && npm install
```

--------------------------------

### Install Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/expositional/models/anthropic/claude3_quickstart.ipynb

Install necessary libraries for TruLens, LiteLLM providers, and ChromaDB.

```python
# !pip install trulens trulens-providers-litellm chromadb openai
```

--------------------------------

### Install Dependencies

Source: https://github.com/truera/trulens/blob/main/src/benchmark/trulens/benchmark/benchmark_frameworks/experiments/competitive_analysis.ipynb

Installs necessary libraries for TruLens, RAGAS, and MLflow. Run this cell to set up your environment.

```python
# ! pip install trulens-core trulens-providers-openai ragas mlflow
```

--------------------------------

### Install Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/langchain/langgraph_functional.ipynb

Installs necessary libraries for TruLens, LangGraph, and Langchain. Use this to set up your environment.

```python
# !pip install trulens trulens-apps-langchain trulens-providers-openai langgraph langchain langchain-openai -q
```

--------------------------------

### Install TruLens Basic

Source: https://github.com/truera/trulens/blob/main/docs/contributing/optional.md

Installs the core, feedback, and dashboard packages.

```bash
pip install trulens
```

--------------------------------

### Start Development Server

Source: https://github.com/truera/trulens/blob/main/src/dashboard/react_components/record_viewer/README.md

Launch the frontend development server for the record viewer.

```bash
npm run dev
```

--------------------------------

### DummyApp Configuration and Usage Example

Source: https://github.com/truera/trulens/blob/main/examples/dev/dummy_app/dummy_app_example.ipynb

Demonstrates synchronous and streaming usage of DummyApp with and without tracing. It iterates through different configurations of `use_parallel`, `use_app`, and `use_recorder` to show their impact.

```python
# This cell executes the app in the various configuration options describe
# above. The final app output should be identical in all cases.

for use_parallel in [False, True]:
    for use_app in [False, True]:
        for use_recorder in [
            False
        ]:  # , True]: # currently not working with nested recorder
            app_version = f"(use_parralel={use_parallel} use_app={use_app} use_recorder={use_recorder})"

            # Create custom app:
            ca = DummyApp(
                use_parallel=use_parallel,
                comp_kwargs={
                    DummyAgent: {
                        "use_app": use_app,
                        "use_recorder": use_recorder,
                    }
                },
            )

            # Create trulens wrapper:
            ta = TruApp(
                ca,
                app_name="customapp",
                app_version=app_version,
            )

            # Sync usage
            print("  sync")
            print("    non-streaming")
            print(
                "      untraced\t",
                ca.respond_to_query(query="What is the capital of Indonesia?"),
            )
            with ta as recorder:
                print(
                    "      traced\t",
                    ca.respond_to_query(
                        query="What is the capital of Indonesia?"
                    ),
                )

            # Sync streaming usage
            print("    streaming")
            print("      untraced\t ", end="")
            for chunk in ca.stream_respond_to_query(
                query="What is the capital of Indonesia?"
            ):
                print(chunk, end="")
            print()
            with ta as recorder:
                print("      traced\t ", end="")
                for chunk in ca.stream_respond_to_query(
                    query="What is the capital of Indonesia?"
                ):
                    print(chunk, end="")
                print()

            # TOFIX: Async usage
            # res1 = await ca.arespond_to_query(query="What is the capital of Indonesia?")
            # with ta as recorder:
            #    resp2 = await ca.arespond_to_query(query="What is the capital of Indonesia?")

            # TOFIX: Async streaming usage...

```

--------------------------------

### Install LlamaIndex and TruLens Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/llamaindex/llama_index_queryplanning.ipynb

Install necessary libraries for LlamaIndex, TruLens, and OpenAI integration. This is a prerequisite for running the example.

```python
# !pip install trulens trulens-apps-llamaindex trulens-providers-openai llama_index llama-index-readers-web llama-index-question-gen-openai
```

--------------------------------

### Install TruLens and Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/experimental/random_evaluation.ipynb

Installs the trulens library along with specific versions of chromadb and openai. Run this before proceeding with other examples.

```python
# !pip install trulens chromadb==0.4.18 openai==1.3.7
```

--------------------------------

### Install Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/expositional/models/google/google_quickstart.ipynb

Installs the necessary TruLens libraries and Google provider packages. Run this in your environment before proceeding.

```python
# !pip install trulens trulens-providers-google chromadb
```

--------------------------------

### Install Make and Configure PATH

Source: https://github.com/truera/trulens/blob/main/DEVELOPMENT.md

Installs the 'make' utility using Homebrew and adds its binary path to the Zsh configuration for accessibility.

```bash
brew install make

```

```bash
echo 'PATH="$HOMEBREW_PREFIX/opt/make/libexec/gnubin:$PATH"' >> ~/.zshrc
```

--------------------------------

### Install Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/llamaindex/llama_index_hybrid_retriever.ipynb

Installs necessary libraries including trulens, llama_index, and related components. Ensure you have the required packages for your setup.

```python
# !pip install trulens llama_index llama-index-readers-file llama-index-llms-openai llama-index-retrievers-bm25 openai pypdf torch sentence-transformers
```

--------------------------------

### Setup VirtualApp with Components

Source: https://github.com/truera/trulens/blob/main/examples/experimental/virtual_example.ipynb

Configures a virtual application environment using `VirtualApp` or a dictionary. This allows for defining and referencing components like LLMs and templates for evaluation purposes.

```python
from trulens.apps.virtual import VirtualApp
from trulens.core import Select

# VirtualApp setup. You can store any information you would like by passing in a
# VirtualApp or a plain dictionary to TruVirtual (later). This may involve an
# index of components or versions, or anything else. You can refer to these
# values for evaluating feedback.

virtual_app = dict(
    llm=dict(modelname="some llm component model name"),
    template="information about the template I used in my app",
    debug="all of these fields are completely optional",
)

# (Optional) If you use the `VirtualApp` class instead of a plain dictionary,
# you can use selectors to position the virtual app components and their
# properties.

virtual_app = VirtualApp(virtual_app)  # can start with the prior dictionary
virtual_app[Select.RecordCalls.llm.maxtokens] = 1024

# Using Selectors here lets you use reuse the setup you use to define feedback
# functions (later in the notebook). We will use `retriever_component`
# exemplified below place information about retrieved context in a virtual
# record that will match the information about the retriever component in the
# virtual app. While this is not necessary, laying out the virtual app and
# virtual records in a mirrored fashion as would be the same for real apps may
# aid interpretability.

retriever_component = Select.RecordCalls.retriever
virtual_app[retriever_component] = "this is the retriever component"
```

--------------------------------

### Initialize Relevance Metric with Fewshot Examples

Source: https://github.com/truera/trulens/blob/main/examples/experimental/feedback_with_fewshot_examples.ipynb

Initializes the OpenAI relevance metric with a list of few-shot examples to guide its scoring.

```python
f_answer_relevance_with_examples = Metric(
    implementation=provider.relevance,
    name="Answer Relevance",
    examples=fewshot_relevance_examples_list,
)
```

--------------------------------

### Configure Dummy Providers and Run Dashboard

Source: https://github.com/truera/trulens/blob/main/examples/experimental/dummy_example.ipynb

Sets up dummy providers for Hugging Face and LLM interactions with specific configurations for request handling. Initializes TruSession, resets the database, and runs the dashboard interface.

```python
from trulens.core import Metric
from trulens.core import Selector
from trulens.core import TruSession
from trulens.core.utils.threading import TP
from trulens.dashboard.run import run_dashboard
from trulens.feedback.dummy.provider import DummyProvider as DummyLLM
from trulens.providers.huggingface.provider import Dummy as DummyHugs

from examples.dev.dummy_app.app import DummyApp

tp = TP()

provider_hugs = DummyHugs(
    loading_prob=0.0,
    freeze_prob=0.0,  # we expect requests to have their own timeouts so freeze should never happen
    error_prob=0.0,
    overloaded_prob=0.0,
    rpm=10000,
    alloc=0,  # how much fake data to allocate during requests
    delay=0.1,
)

provider_llm = DummyLLM(
    loading_prob=0.0,
    freeze_prob=0.0,  # we expect requests to have their own timeouts so freeze should never happen
    error_prob=0.0,
    overloaded_prob=0.0,
    rpm=10000,
    alloc=0,  # how much fake data to allocate during requests
    delay=0.1,
)

session = TruSession()

session.reset_database()

run_dashboard(session, force=True, _dev=base_dir, port=8080)
```

--------------------------------

### Install and Setup Trulens for Development

Source: https://github.com/truera/trulens/blob/main/examples/experimental/dev_notebook.ipynb

Installs trulens from a git repository and sets up the Python path for local development. This is useful for debugging or developing new features.

```python
# pip uninstall -y trulens
# pip install git+https://github.com/truera/trulens@piotrm/azure_bugfixes#subdirectory=trulens

# trulens notebook dev

# %load_ext autoreload
# %autoreload 2
from pathlib import Path
import sys

base = Path().cwd()
while not (base / "trulens").exists():
    base = base.parent


# import os
# if os.path.exists("default.sqlite"):
#    os.unlink("default.sqlite")

print(base)

# import shutil
# shutil.copy(base / "release_dbs" / "0.19.0" / "default.sqlite", "default.sqlite")

# If running from github repo, can use this:
sys.path.append(str(base))
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/truera/trulens/blob/main/DEVELOPMENT.md

Builds and serves the project's documentation locally. This command is useful for previewing documentation changes before deployment.

```bash
make docs-serve
```

--------------------------------

### Quick Start with TruGEPA and run_evolution

Source: https://github.com/truera/trulens/blob/main/src/apps/gepa/README.md

Demonstrates how to use TruGEPA to wrap a feedback function and run an evolutionary loop with logging enabled.

```python
from trulens.apps.gepa import TruGEPA, run_evolution

def my_relevance(prompt: str) -> float:
    return len(prompt) / 200  # replace with a real TruLens provider method

# Without logging:
fitness = TruGEPA(my_relevance, optimize_key="prompt")

# With logging — supply both app_name and app_version (omit both to disable;
# supplying only one raises a ValueError immediately):
from trulens.core import TruSession
session = TruSession()

fitness = TruGEPA(
    my_relevance,
    optimize_key="prompt",
    app_name="my_optimizer",
    app_version="v1",
)

# Works with any feedback signature — e.g. context_relevance(question, context):
# fitness = TruGEPA(
#     provider.context_relevance,
#     optimize_key="question",
#     feedback_args={\"context\": REFERENCE_CONTEXT},
# )

best_prompt, best_score, history = run_evolution(
    base_prompt="Summarize the document.",
    fitness_fn=fitness,
    mutate_fn=lambda p: p + " Be concise.",
    n_generations=5,
    population_size=4,
)
print(f"Best prompt ({best_score:.3f}): {best_prompt}")
```

--------------------------------

### Install OTEL and Zipkin Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/experimental/export_dummy_example.ipynb

Installs necessary OpenTelemetry and Zipkin Python packages. Also includes commands to start and stop a Zipkin Docker container for demonstration.

```python
# Install OTEL deps:
# ! pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

# Install zipkin python package:
# ! pip install opentelemetry-exporter-zipkin-proto-http

# Start the zipkin docker container:
# ! docker run --rm -d -p 9411:9411 --name zipkin openzipkin/zipkin

# Stop the zipkin docker container:
# ! docker stop $(docker ps -a -q --filter ancestor=openzipkin/zipkin)
```

--------------------------------

### Initialize Qdrant Client

Source: https://github.com/truera/trulens/blob/main/examples/expositional/vector_stores/qdrant/hybrid_search_rag_langchain_and_qdrant.ipynb

Initializes the Qdrant client using the provided URL and API key. Ensure QDRANT_URL and QDRANT_API_KEY environment variables are set.

```python
client = QdrantClient(
    url=QDRANT_URL,
    api_key=QDRANT_API_KEY
)
```

--------------------------------

### Initialize LLM, Index, and RAG Query Engine

Source: https://github.com/truera/trulens/blob/main/examples/expositional/use_cases/iterate_on_rag/1_rag_prototype.ipynb

Sets up the language model, creates a vector store index from loaded documents, and configures a basic RAG query engine with a system prompt. Requires OpenAI API key and a local embedding model.

```python
from llama_index import Prompt
from llama_index.core import Document
from llama_index.core import VectorStoreIndex
from llama_index.legacy import ServiceContext
from llama_index.llms.openai import OpenAI

# initialize llm
llm = OpenAI(model="gpt-3.5-turbo", temperature=0.5)

# knowledge store
document = Document(text="\n\n".join([doc.text for doc in documents]))

# service context for index
service_context = ServiceContext.from_defaults(
    llm=llm, embed_model="local:BAAI/bge-small-en-v1.5"
)

# create index
index = VectorStoreIndex.from_documents(
    [document], service_context=service_context
)


system_prompt = Prompt(
    "We have provided context information below that you may use. \n"
    "---------------------
"
    "{context_str}"
    "\n---------------------
"
    "Please answer the question: {query_str}\n"
)

# basic rag query engine
rag_basic = index.as_query_engine(text_qa_template=system_prompt)
```

--------------------------------

### Generate Test Set with Examples

Source: https://github.com/truera/trulens/blob/main/examples/experimental/generate_test_set.ipynb

Generate a test set using provided examples to guide the types of questions the application should be tested on. This helps tailor the test set to specific use cases.

```python
examples = [
    "What is sensory memory?",
    "How much information can be stored in short term memory?",
]

fewshot_test_set = test.generate_test_set(
    test_breadth=3, test_depth=2, examples=examples
)
fewshot_test_set
```

--------------------------------

### Get Snowflake Connector Version

Source: https://github.com/truera/trulens/blob/main/tests/e2e/data/simple.ipynb

Import the Snowflake connector and print its version. Ensure the connector is installed.

```python
import trulens.connectors.snowflake

print(trulens.connectors.snowflake.__version__)
```

--------------------------------

### Initialize Bedrock Runtime Client

Source: https://github.com/truera/trulens/blob/main/examples/expositional/models/bedrock/bedrock.ipynb

Set up the AWS Bedrock runtime client using boto3. Ensure you are logged in with the AWS CLI and have configured your credentials.

```python
import boto3

client = boto3.client(service_name="bedrock-runtime", region_name="us-east-1")
```

--------------------------------

### Set up ChromaDB Vector Store

Source: https://github.com/truera/trulens/blob/main/examples/experimental/pupr_e2e_cuj.ipynb

Initializes a ChromaDB client, creates or gets a collection named 'hotpotqa_contexts', and upserts the documents (contexts) with their corresponding embeddings.

```python
import chromadb

chroma_client = chromadb.Client()
vector_store = chroma_client.get_or_create_collection(name="hotpotqa_contexts")

vector_store.upsert(
    [str(i) for i in range(len(all_contexts))],
    documents=all_contexts,
    embeddings=ctx_embeddings,
)
```

--------------------------------

### Evaluate Answer Relevance with Fewshot Guidance

Source: https://github.com/truera/trulens/blob/main/examples/experimental/feedback_with_fewshot_examples.ipynb

Evaluates answer relevance using a feedback function that has been guided by few-shot examples.

```python
f_answer_relevance_with_examples(
    "What are the key considerations when starting a small business?",
    "Find a mentor who can guide you through the early stages and help you navigate common challenges.",
)
```

--------------------------------

### Setting up Benchmark Experiments

Source: https://github.com/truera/trulens/blob/main/examples/quickstart/groundtruth_dataset_persistence.ipynb

Configure benchmark experiments using `TruBenchmarkExperiment` for different feedback functions and aggregator functions. Requires `BenchmarkParams`.

```python
from trulens.benchmark.benchmark_frameworks.tru_benchmark_experiment import (
    BenchmarkParams,
)
from trulens.benchmark.benchmark_frameworks.tru_benchmark_experiment import (
    TruBenchmarkExperiment,
)
from trulens.benchmark.benchmark_frameworks.tru_benchmark_experiment import (
    create_benchmark_experiment_app,
)

benchmark_experiment = TruBenchmarkExperiment(
    feedback_fn=context_relevance_4o,
    agg_funcs=[recall_agg_func],
    benchmark_params=BenchmarkParams(temperature=0.5),
)

benchmark_experiment_mini = TruBenchmarkExperiment(
    feedback_fn=context_relevance_4o_mini,
    agg_funcs=[recall_agg_func],
    benchmark_params=BenchmarkParams(temperature=0.5),
)
```

--------------------------------

### Add Base Path to sys.path

Source: https://github.com/truera/trulens/blob/main/tests/api.ipynb

Ensures the base path is added to sys.path for tests that are not installed. This is a common setup for testing environments.

```python
from pathlib import Path
import sys

# Need this because tests don't get installed.
basepath = Path().cwd().parent.resolve()
if str(basepath) not in sys.path:
    sys.path.append(str(basepath))
    print(f"Added {basepath} to sys.path")
```

--------------------------------

### Get Cortex Provider Version

Source: https://github.com/truera/trulens/blob/main/tests/e2e/data/simple.ipynb

Import the TruLens Cortex provider and print its version. This verifies the installation and accessibility of the Cortex integration.

```python
import trulens.providers.cortex

print(trulens.providers.cortex.__version__)
```

--------------------------------

### Set up Environment Variables

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/langchain/langchain_ensemble_retriever.ipynb

Sets up the OpenAI API key and enables TruLens OpenTelemetry tracing.

```python
from getpass import getpass
import os

if not os.getenv("OPENAI_API_KEY"):
    os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")
os.environ["TRULENS_OTEL_TRACING"] = "1"

```

--------------------------------

### Install Dependencies

Source: https://github.com/truera/trulens/blob/main/examples/quickstart/deep_agents_quickstart.ipynb

Installs necessary libraries for deep agents, search, and TruLens evaluation. Use this command to set up your environment.

```bash
!pip install deepagents tavily-python trulens trulens-apps-langgraph trulens-providers-openai langchain-openai -q
```

--------------------------------

### Get TruLens Core Version

Source: https://github.com/truera/trulens/blob/main/tests/e2e/data/simple.ipynb

Import the TruLens core module and print its version. This is useful for verifying the installation of the core library.

```python
import trulens.core

print(trulens.core.__version__)
```

--------------------------------

### Start Production Monitoring Server

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/openai_agent_sdk/openai_agent_sdk_snowflake_tools/README.md

Launches the FastAPI server for production monitoring and development. It also includes instructions to start the frontend development server.

```bash
SNOWFLAKE_CONNECTION_NAME=YOUR_CONNECTION uvicorn server:app --reload --port 8000
cd frontend && npm run dev
```

--------------------------------

### Initialize LLMs for Planning and Reasoning

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/langchain/langgraph-multi-agent-snowflake-tools.ipynb

Sets up two ChatOpenAI language models: one for planning with JSON output and another for reasoning, also supporting JSON output and allowing for higher temperature for creative responses.

```python
from typing import Literal, Dict, Any
import json
from langchain_openai import ChatOpenAI

# ── LLMs ────────────────────────────────────────────────────────────────
planner_llm = ChatOpenAI(
    model_name="gpt-4o",
    response_format={"type": "json_object"},
)

# same for the reasoning/orchestrator LLM if it parses JSON too
reasoning_llm = ChatOpenAI(
    model_name="o3-mini",
    response_format={"type": "json_object"},
    temperature=1
)
```

--------------------------------

### Define Fewshot Examples for Relevance

Source: https://github.com/truera/trulens/blob/main/examples/experimental/feedback_with_fewshot_examples.ipynb

Defines a list of tuples, where each tuple contains a query-response pair and its corresponding human-labeled score, to guide the feedback function.

```python
fewshot_relevance_examples_list = [
    (
        {
            "query": "What are the key considerations when starting a small business?",
            "response": "You should focus on building relationships with mentors and industry leaders. Networking can provide insights, open doors to opportunities, and help you avoid common pitfalls.",
        },
        3,
    ),
]
```

--------------------------------

### Initialize OpenAI Client and Agent

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/openai_agent_sdk/openai_agents.ipynb

Sets up an asynchronous OpenAI client and defines an agent with specific instructions, tools, and a model configuration.

```python
openai_client = AsyncOpenAI()

agent = Agent(
    name="Research Assistant",
    instructions=(
        "You are a helpful research assistant. "
        "Use the weather tool for weather queries. "
        "Be concise in your answers."
    ),
    tools=[get_weather],
    model=OpenAIChatCompletionsModel(
        model="gpt-4o-mini",
        openai_client=openai_client,
    ),
)
```

--------------------------------

### Set up Custom App with TruApp

Source: https://github.com/truera/trulens/blob/main/examples/experimental/db_populate.ipynb

Creates an instance of a custom application (DummyApp) and wraps it with TruApp, specifying the main method to be instrumented and associating feedback metrics.

```python
from trulens.apps.app import TruApp

from examples.dev.dummy_app.app import DummyApp

# Create custom app:
app_custom = DummyApp()

# Create trulens wrapper:
trucustom = TruApp(
    app=app_custom,
    app_name="custom_app",
    # Make sure to specify using the bound method, bound to self=app.
    main_method=app_custom.respond_to_query,
    feedbacks=feedbacks,
)

```

--------------------------------

### Create Run Configuration

Source: https://github.com/truera/trulens/blob/main/examples/experimental/otel_exporter.ipynb

Defines a configuration for a TruLens run, including a unique run name, dataset details, and input/label specifications. This setup is necessary before starting a run.

```python
# Create run.

import uuid

from trulens.core.run import RunConfig

run_name = str(uuid.uuid4())
run_config = RunConfig(
    run_name=run_name,
    dataset_name="My test dataframe name",
    source_type="DATAFRAME",
    dataset_spec={"input": "custom_input"},
    label="label",
    description="desciption",
)

```

--------------------------------

### Define Simple Multi-Agent Workflow

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/langchain/otel_tru_graph_example.ipynb

Defines a LangGraph workflow with multiple nodes for generating topics, jokes, and selecting the best joke. This example showcases a basic multi-agent setup.

```python
class OverallState(TypedDict):
    topic: str
    subjects: list[str]
    jokes: Annotated[list[str], operator.add]
    best_selected_joke: str

def generate_topics(state: OverallState):
    return {"subjects": ["lions", "elephants", "penguins"]}

def generate_joke(state: OverallState):
    joke_map = {
        "lions": "Why don't lions like fast food? Because they can't catch it!",
        "elephants": "Why don't elephants use computers? They're afraid of the mouse!",
        "penguins": "Why don't penguins like talking to strangers at parties? Because they find it hard to break the ice."
    }
    return {"jokes": [joke_map[state["subject"]]]}

def continue_to_jokes(state: OverallState):
    return [Send("generate_joke", {"subject": s}) for s in state["subjects"]]

def best_joke(state: OverallState):
    return {"best_selected_joke": "penguins"}

builder = StateGraph(OverallState)
builder.add_node("generate_topics", generate_topics)
builder.add_node("generate_joke", generate_joke)
builder.add_node("best_joke", best_joke)
builder.add_edge(START, "generate_topics")
builder.add_conditional_edges("generate_topics", continue_to_jokes, ["generate_joke"])
builder.add_edge("generate_joke", "best_joke")
builder.add_edge("best_joke", END)
builder.add_edge("generate_topics", END)
graph = builder.compile()
```

--------------------------------

### Start PostgreSQL using Docker

Source: https://github.com/truera/trulens/blob/main/docs/component_guides/logging/where_to_log/log_in_postgres.md

Quickly set up a PostgreSQL instance for development or testing using Docker. This command maps the default PostgreSQL port and sets up a database named 'trulens'.

```bash
docker run -d \
    --name trulens-postgres \
    -e POSTGRES_DB=trulens \
    -e POSTGRES_USER=trulensuser \
    -e POSTGRES_PASSWORD=password \
    -p 5432:5432 \
    postgres:15-alpine
```

--------------------------------

### Run Basic Unit Tests

Source: https://github.com/truera/trulens/blob/main/DEVELOPMENT.md

Installs only the required dependencies and runs basic unit tests. This is useful for a quick check of core functionality.

```bash
# Installs only required dependencies and runs basic unit tests
make test-unit-basic
```

--------------------------------

### Deferred Feedback Evaluation Setup

Source: https://github.com/truera/trulens/blob/main/docs/component_guides/logging/logging.ipynb

Configure TruChain to use deferred feedback evaluation by setting `feedback_mode='deferred'` and starting the session's evaluator. This allows feedback to be processed asynchronously.

```python
truchain: TruChain = TruChain(
    chain,
    app_name="ChatApplication",
    app_version="chain_1",
    feedbacks=[f_lang_match],
    feedback_mode="deferred",
)

with truchain:
    chain("This will be logged by deferred evaluator.")

session.start_evaluator()
# session.stop_evaluator()
```

--------------------------------

### Display First Training Example

Source: https://github.com/truera/trulens/blob/main/examples/expositional/models/bedrock/bedrock_finetuning_experiments.ipynb

Retrieves and displays the first data sample from the prepared training set to inspect its structure.

```python
train_and_test_dataset["train"][0]
```

--------------------------------

### Log and Instrument LangChain LLM Apps with TruLens

Source: https://github.com/truera/trulens/blob/main/docs/blog/posts/release_blog_1dot.md

Install trulens-apps-langchain to instrument LangChain applications. This example demonstrates wrapping a LangChain RAG chain with TruChain for recording application runs.

```bash
pip install trulens-apps-langchain
```

```python
from langchain import hub
from langchain.chat_models import ChatOpenAI
from langchain.schema import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

retriever = vectorstore.as_retriever()

prompt = hub.pull("rlm/rag-prompt")
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

from trulens.apps.langchain import TruChain

# Wrap application
tru_recorder = TruChain(
    chain, 
    app_id='Chain1_ChatApplication'
)

# Record application runs
with tru_recorder as recording:
    chain("What is langchain?")

```

--------------------------------

### Setup Environment and Keys

Source: https://github.com/truera/trulens/blob/main/examples/experimental/virtual_example.ipynb

Configures the Python environment and loads necessary TruLens modules. This setup is tailored for running from a GitHub repository.

```python
# Setup env and keys. This is currently set up for running from github repo.

%load_ext autoreload
%autoreload 2
from pathlib import Path
import sys

base = Path().cwd()
while not (base / "trulens").exists():
    base = base.parent

print(base)

# If running from github repo, can use this:
sys.path.append(str(base))
```

--------------------------------

### Initialize TruLens Chat Engine Recorder

Source: https://github.com/truera/trulens/blob/main/examples/experimental/export_llamaindex_example.ipynb

Initializes a `TruLlama` recorder for a LlamaIndex chat engine, specifying the application name and version. This setup is necessary before starting a traced chat session.

```python
chat_engine = index.as_chat_engine(similarity_top_k=3)
tru_chat_engine_recorder = TruLlama(
    chat_engine, app_name="LlamaIndex_App", app_version="chat"
)
```

--------------------------------

### Python Import Style Example (Do This)

Source: https://github.com/truera/trulens/blob/main/docs/contributing/standards.md

Prefer importing specific modules or submodules and aliasing them if necessary. This improves clarity and can delay module loading.

```python
from trulens.schema import record as record_schema # do this instead
```

--------------------------------

### Log and Instrument LlamaIndex LLM Apps with TruLens

Source: https://github.com/truera/trulens/blob/main/docs/blog/posts/release_blog_1dot.md

Install trulens-core and trulens-apps-llamaindex to instrument LlamaIndex applications. This example shows how to wrap a LlamaIndex query engine with TruLlama for recording query executions.

```bash
pip install trulens-core trulens-apps-llamaindex
```

```python
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

from trulens.apps.llamaindex import TruLlama
from trulens.core import Feedback

tru_recorder = TruLlama(query_engine,
    app_id='LlamaIndex_App1')

with tru_recorder as recording:
    query_engine.query("What is LlamaIndex?")

```

--------------------------------

### Run Feedback Functions with Huggingface Classification Models

Source: https://github.com/truera/trulens/blob/main/docs/blog/posts/release_blog_1dot.md

Install trulens-core and trulens-providers-huggingface to use feedback functions with Huggingface models. This example defines a remote groundedness feedback function using a Huggingface provider.

```bash
pip install trulens-core trulens-providers-huggingface
```

```python
from trulens.core import Feedback
from trulens.core import Select
from trulens.providers.huggingface import Huggingface

# Define a remote HuggingFace groundedness feedback function
provider = Huggingface()
f_remote_groundedness = (
    Feedback(
        provider.groundedness_measure_with_nli,
        name="[Remote] Groundedness",
    )
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
)

```

--------------------------------

### Creating and Running Benchmark App

Source: https://github.com/truera/trulens/blob/main/examples/quickstart/groundtruth_dataset_persistence.ipynb

Create a benchmark experiment app using `create_benchmark_experiment_app` and run it as a context manager to get feedback results. Specify app name, version, and the benchmark experiment.

```python
tru_benchmark = create_benchmark_experiment_app(
    app_name="Context Relevance",
    app_version="gpt-4o",
    benchmark_experiment=benchmark_experiment,
)

with tru_benchmark as recording:
    feedback_res = tru_benchmark.app(gt_df)
```

--------------------------------

### Setup LangGraph and TruGraph

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/langchain/otel_tru_graph_example.ipynb

Initializes TruSession and checks for the availability of LangGraph and TruGraph. Sets the environment variable for OTel tracing.

```python
import operator
import os
from langgraph.graph import StateGraph, END, START
from langgraph.types import Send
from typing_extensions import TypedDict, Annotated
from trulens.core.session import TruSession


os.environ["TRULENS_OTEL_TRACING"] = "1"

session = TruSession()
session.reset_database()
# Check if LangGraph is available
try:
    from langgraph.graph import StateGraph, MessagesState, END, START
    from langchain_core.messages import HumanMessage, AIMessage
    from trulens.apps.langgraph import TruGraph
    print("✅ LangGraph and TruGraph are available!")
    LANGGRAPH_AVAILABLE = True
except ImportError as e:
    raise ImportError(f"❌ LangGraph not available: {e}")
```

--------------------------------

### Run Feedback Functions with OpenAI LLMs

Source: https://github.com/truera/trulens/blob/main/docs/blog/posts/release_blog_1dot.md

Install trulens-core and trulens-providers-openai to use feedback functions with OpenAI models. This example demonstrates setting up an OpenAI provider and defining a context relevance feedback function.

```bash
pip install trulens-core  trulens-providers-openai
```

```python
from trulens.providers.openai import OpenAI
from trulens.core import Feedback
import numpy as np

provider = OpenAI()

# Use feedback
f_context_relevance = (
    Feedback(provider.context_relevance_with_context_reasons)
    .on_input()
    .on(context)  # Refers to context defined from `select_context`
    .aggregate(np.mean)
)

```

--------------------------------

### Set up Langchain App with Bedrock LLM

Source: https://github.com/truera/trulens/blob/main/examples/expositional/models/bedrock/bedrock.ipynb

Construct a Langchain LLMChain using the Bedrock LLM and a chat prompt template. This sets up a basic conversational agent.

```python
template = "You are a helpful assistant."
system_message_prompt = SystemMessagePromptTemplate.from_template(template)
example_human = HumanMessagePromptTemplate.from_template("Hi")
example_ai = AIMessagePromptTemplate.from_template("Argh me mateys")
human_template = "{text}"
human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)

chat_prompt = ChatPromptTemplate.from_messages(
    [system_message_prompt, example_human, example_ai, human_message_prompt]
)
chain = LLMChain(llm=bedrock_llm, prompt=chat_prompt, verbose=True)

print(chain.run("What's the capital of the USA?"))
```

--------------------------------

### Log and Instrument Python LLM Apps with TruLens

Source: https://github.com/truera/trulens/blob/main/docs/blog/posts/release_blog_1dot.md

Install the trulens-core library to instrument custom Python LLM applications. This example shows how to use the @instrument decorator for methods within a custom application class.

```bash
pip install trulens-core
```

```python
from trulens.apps.app import instrument

class CustomApp:

    def __init__(self):
        self.retriever = CustomRetriever()
        self.llm = CustomLLM()
        self.template = CustomTemplate(
            "The answer to {question} is {answer}"
        )

    @instrument()
    def retrieve_chunks(self, data):
        return self.retriever.retrieve_chunks(data)

    @instrument()
    def respond_to_query(self, input):
        chunks = self.retrieve_chunks(input)
        answer = self.llm.generate(",".join(chunks))
        output = self.template.fill(question=input, answer=answer)

        return output

ca = CustomApp()

```

--------------------------------

### Basic Feedback Function Usage

Source: https://github.com/truera/trulens/blob/main/AGENTS.md

Demonstrates the basic setup for a Feedback function using shortcuts for input and output. Requires importing Feedback and a provider like OpenAI.

```python
from trulens.core import Feedback
from trulens.providers.openai import OpenAI

provider = OpenAI()
f_relevance = Feedback(provider.relevance_with_cot_reasons).on_input().on_output()
```

--------------------------------

### Example Streamlit Test Structure

Source: https://github.com/truera/trulens/blob/main/tests/unit/streamlit/README.md

Demonstrates a typical structure for writing tests for Streamlit components. It includes setup for mocking dependencies and running the Streamlit app within a test environment. Use this as a template for new tests.

```python
def test_component_with_scenario(self, mock_data):
    """Test description explaining what this verifies."""
    with MockManager.mock_dashboard_utils(mock_data):
        def test_app():
            # Your component test code
            pass

        app = AppTestHelper.create_and_run_app(test_app)
        AppTestHelper.assert_no_errors(app)
        # Add specific assertions
```

--------------------------------

### Run Feedback Functions with Ollama LLMs

Source: https://github.com/truera/trulens/blob/main/docs/blog/posts/release_blog_1dot.md

Install trulens-core and trulens-providers-litellm to use feedback functions with local LLMs like Ollama. This example shows configuring LiteLLM provider for Ollama and defining a context relevance feedback function.

```bash
pip install trulens-core trulens-providers-litellm
```

```python
from trulens.providers.litellm import LiteLLM
from trulens.core import Feedback
import numpy as np

provider = LiteLLM(
    model_engine="ollama/llama3.1:8b", api_base="http://localhost:11434"
)

# Use feedback
f_context_relevance = (
    Feedback(provider.context_relevance_with_context_reasons)
    .on_input()
    .on(context)  # Refers to context defined from `select_context`
    .aggregate(np.mean)
)

```

--------------------------------

### Display First Three Training Split Examples

Source: https://github.com/truera/trulens/blob/main/examples/experimental/cortex_finetuning_groundedness_ff.ipynb

Displays the first three examples from the training split to provide a preview of the data format.

```python
train_split[:3]
```

--------------------------------

### Install Langchain and Qdrant Libraries

Source: https://github.com/truera/trulens/blob/main/examples/expositional/vector_stores/qdrant/hybrid_search_rag_langchain_and_qdrant.ipynb

Installs the necessary Langchain community, OpenAI, and Qdrant libraries for building the RAG pipeline.

```python
!pip install langchain-community langchain-openai langchain-qdrant
```

--------------------------------

### Define Text-to-SQL Quality Metric

Source: https://github.com/truera/trulens/blob/main/examples/experimental/client_side_custom_metrics.ipynb

Implement a custom metric to evaluate the quality of text-to-SQL queries. This example uses a simple heuristic based on SQL keywords and query length. It requires no external setup beyond standard Python.

```python
def text2sql_quality(query: str, sql: str) -> float:
    """
    Custom metric to evaluate text-to-SQL quality.

    Args:
        query: The natural language query
        sql: The generated SQL query

    Returns:
        Quality score between 0 and 1
    """
    # Simple heuristic - check if SQL contains expected keywords
    if "SELECT" in sql.upper() and len(query) > 10:
        return 0.9
    elif "SELECT" in sql.upper():
        return 0.7
    else:
        return 0.3
```

--------------------------------

### Groq Llama 3 Context Filtered RAG

Source: https://github.com/truera/trulens/blob/main/examples/expositional/use_cases/context_filters.ipynb

Applies a context filter using Groq's Llama 3 - 8B model to a RAG system. This example demonstrates using a different provider and model for context relevance filtering. Requires LiteLLM setup for Groq.

```python
from trulens.providers.litellm import LiteLLM
from trulens.rag.metrics import Metric
from trulens.rag.vectorstores import RAG
from trulens.rag.utils.filters import context_filter

groq_llama3_8b_provider = LiteLLM("groq/llama3-8b-8192")

f_context_relevance_groqllama3_8b = Metric(implementation=groq_llama3_8b_provider.context_relevance)

class FilteredRAG(RAG):
    @context_filter(
        feedback=f_context_relevance_groqllama3_8b,
        threshold=0.75,
        keyword_for_prompt="query",
    )
    def retrieve(self, query: str) -> list:
        """
        Retrieve relevant text from vector store.
        """
        results = vector_store.query(query_texts=query, n_results=5)
        if "documents" in results and results["documents"]:
            return [doc for sublist in results["documents"] for doc in sublist]
        else:
            return []


filtered_rag = FilteredRAG()

filtered_rag.query("How often are environmental KPIs assessed?")
```

--------------------------------

### Initialize TruApp and Run Agent

Source: https://github.com/truera/trulens/blob/main/examples/expositional/frameworks/openai_agent_sdk/openai_agents.ipynb

Instantiates the 'AgentApp' and 'TruApp', linking the agent, its main method, and feedback metrics. Then, it runs the agent with a sample question and prints the truncated answer.

```python
agent_app = AgentApp()

tru_app = TruApp(
    agent_app,
    app_name="Research Assistant",
    app_version="v1",
    main_method=agent_app.ask,
    feedbacks=[f_answer_relevance, f_coherence],
)

questions = [
    "What is the weather in Tokyo?",
]

for q in questions:
    print(f"\nQ: {q}")
    with tru_app as recording:
        answer = await agent_app.ask(q)
    print(f"A: {answer[:500]}")
```

--------------------------------

### Installing Grit CLI

Source: https://github.com/truera/trulens/blob/main/docs/component_guides/other/trulens_eval_migration.md

Install the Grit CLI globally using npm or with an installation script.

```bash
npm install --location=global @getgrit/cli
```

```bash
curl -fsSL https://docs.grit.io/install | bash
```