W&B Weave

https://docs.wandb.ai/weave
Admin
W&B Weave is an observability and evaluation platform that helps you track, evaluate, and improve...
Tokens:518,021
Snippets:4,218
Trust Score:10
Update:4 months ago
Show doc for...
Tokens:
Raw
### Clone and Navigate to mcp_demo Example

Source: https://docs.wandb.ai/weave/guides/integrations/mcp

Clones the documentation repository and changes the directory to the mcp_demo example for further setup.

```bash
git clone https://github.com/wandb/docs
cd docs/weave/examples/mcp_demo
```

--------------------------------

### Install Dependencies and Setup API Keys

Source: https://docs.wandb.ai/weave/cookbooks/Intro_to_Weave_Hello_Trace

Installs necessary libraries and prompts for W&B and OpenAI API keys. Initializes the W&B project.

```python
# Install dependencies and imports
!pip install wandb weave openai -q

import json
import os
from getpass import getpass

from openai import OpenAI

import weave

# 🔑 Setup your API keys
# Running this cell will prompt you for your API key with `getpass` and will not echo to the terminal.
#####
print("--- ")
print("Create a W&B API key at: https://wandb.ai/settings#apikeys")
os.environ["WANDB_API_KEY"] = getpass("Enter your W&B API key: ")
print("--- ")
print("You can generate your OpenAI API key here: https://platform.openai.com/api-keys")
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")
print("--- ")
#####

# 🏠 Enter your W&B project name
weave_client = weave.init("MY_PROJECT_NAME")  # 🐝 Your W&B project name
```

--------------------------------

### Install and Import Packages, Set Up Environment

Source: https://docs.wandb.ai/weave/cookbooks/custom_model_cost

Installs necessary packages and sets up the W&B environment, including API key and project name. This is the initial setup for logging traces.

```python
%pip install wandb weave datetime --quiet
python
import os

import wandb
from google.colab import userdata

import weave

os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
name_of_wandb_project = "custom-cost-model"

wandb.login()
python
weave_client = weave.init(name_of_wandb_project)
```

--------------------------------

### Python SDK Class: Evaluation - Setup and Run

Source: https://docs.wandb.ai/weave/reference/python-sdk

Example demonstrating how to set up and run an evaluation using the Evaluation class. This includes defining examples, scoring functions, and the model to be evaluated.

```python
# Collect your examples
examples = [
     {"question": "What is the capital of France?", "expected": "Paris"},
     {"question": "Who wrote 'To Kill a Mockingbird'?", "expected": "Harper Lee"},
     {"question": "What is the square root of 64?", "expected": "8"},
]

# Define any custom scoring function
@weave.op
def match_score1(expected: str, model_output: dict) -> dict:
     # Here is where you'd define the logic to score the model output
     return {'match': expected == model_output['generated_text']}

@weave.op
def function_to_evaluate(question: str):
     # here's where you would add your LLM call and return the output
     return  {'generated_text': 'Paris'}

# Score your examples using scoring functions
evaluation = Evaluation(
     dataset=examples, scorers=[match_score1]
)

# Start tracking the evaluation
weave.init('intro-example')
# Run the evaluation
asyncio.run(evaluation.evaluate(function_to_evaluate))
```

--------------------------------

### Run OpenInference Example

Source: https://docs.wandb.ai/weave/guides/tracking/otel

Executes the TypeScript example using ts-node. Ensure you have Node.js and npm installed.

```bash
npx ts-node openinference_example.ts
```

--------------------------------

### Running the OpenLLMetry Example

Source: https://docs.wandb.ai/weave/guides/tracking/otel

This bash command shows how to execute the TypeScript example using ts-node. Ensure you have Node.js and npm installed.

```bash
npx ts-node openllmetry_example.ts
```

--------------------------------

### Realtime Voice Assistant Setup and Execution

Source: https://docs.wandb.ai/weave/guides/integrations/openai-realtime-audio

Initializes audio streams, configures a realtime agent and runner, and starts the asynchronous session for a voice assistant. Requires setting up audio devices and the OpenAI API key.

```python
channels=input_channels,
             rate=RATE,
             input=True,
             output=False,
             frames_per_buffer=CHUNK,
             input_device_index=input_device_index,
             start=False,
         )
         speaker = p.open(
             format=FORMAT,
             channels=output_channels,
             rate=RATE,
             input=False,
             output=True,
             frames_per_buffer=CHUNK,
             output_device_index=output_device_index,
             start=False,
         )
         mic.start_stream()
         speaker.start_stream()

         # Buffer audio through a queue so in-flight playback can be flushed on interrupt.
         audio_output_queue = queue.Queue()
         threading.Thread(
             target=play_audio, args=(speaker, audio_output_queue), daemon=True
         ).start()

         s_agent = RealtimeAgent(
             name="Speech Assistant",
             instructions="You are a tool using AI. Use tools to accomplish a task whenever possible"
         )

         s_runner = RealtimeRunner(s_agent, config={
             "model_settings": {
                 "model_name": "gpt-realtime",
                 "modalities": ["audio"],
                 "output_modalities": ["audio"],
                 "input_audio_format": "pcm16",
                 "output_audio_format": "pcm16",
                 "speed": 1.2,
                 "turn_detection": {
                     "prefix_padding_ms": 100,
                     "silence_duration_ms": 100,
                     "type": "server_vad",
                     "interrupt_response": True,
                     "create_response": True,
                 },
             }
         })
         print("--- Session Active (Speak into mic) ---")
         print("🎙  Mic ON (press t to toggle)")

         threading.Thread(target=start_keylistener, daemon=True).start()

         async with await s_runner.run() as session:
             # Stream mic input to the Realtime API, sending silence when muted.
             async def send_mic_audio():
                 silence = b'\x00' * CHUNK * 2  # 2 bytes per sample (16-bit PCM).
                 try:
                     while True:
                         raw_data = mic.read(CHUNK, exception_on_overflow=False)

                         if mic_enabled:
                             audio_data = np.frombuffer(raw_data, dtype=np.int16).astype(np.float64)
                             rms = np.sqrt(np.mean(audio_data**2))
                             meter = int(min(rms / 50, 50))
                             print(f"Mic Level: {'█' * meter}{' ' * (50-meter)} | 🎙 ON ", end="\r")
                             await session.send_audio(raw_data)
                         else:
                             print(f"Mic Level: {' ' * 50} | 🎙 OFF", end="\r")
                             await session.send_audio(silence)

                         await asyncio.sleep(0)  # Yield to the event loop between reads.
                 except Exception:
                     pass

             # Receive events from the session and route audio to the speaker.
             async def handle_events():
                 async for event in session:
                     if event.type == "audio":
                         audio_output_queue.put(event.audio.data)
                     elif event.type == "audio_interrupted":
                         # Flush queued AI audio so it doesn't talk over the user.
                         while not audio_output_queue.empty():
                             try:
                                 audio_output_queue.get_nowait()
                             except queue.Empty:
                                 break

             mic_task = asyncio.create_task(send_mic_audio())
             try:
                 await handle_events()
             finally:
                 mic_task.cancel()

         # Cleanup
         audio_output_queue.put(None)  # Signal playback thread to exit.
         mic.close()
         speaker.close()
         p.terminate()

     if __name__ == "__main__":
         args = parse_args()
         init_weave(args.weave_project)
         fd = sys.stdin.fileno()
         old_settings = termios.tcgetattr(fd)
         try:
             asyncio.run(main(input_device_index=args.input_device, output_device_index=args.output_device))
         finally:
             termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)
     
```

--------------------------------

### Install CrewAI and Weave

Source: https://docs.wandb.ai/weave/guides/integrations/crewai

Install the necessary libraries for CrewAI and Weave integration. This is a prerequisite for running the example.

```bash
pip install crewai weave
```

--------------------------------

### Example Keeper Hostnames with Default Names

Source: https://docs.wandb.ai/weave/guides/platform/weave-self-managed

Illustrates the expected Keeper node hostnames when using default installation and cluster names.

```text
chk-wandb-keeper-0-0.clickhouse.svc.cluster.local
chk-wandb-keeper-0-1.clickhouse.svc.cluster.local
chk-wandb-keeper-0-2.clickhouse.svc.cluster.local
```

--------------------------------

### Setup Audio Recording Stream

Source: https://docs.wandb.ai/weave/cookbooks/audio_with_weave

Configures and starts a `pyaudio` input stream. The `audio_callback` function is invoked for each chunk of audio data, sending it to the provided `RTAudioModel`.

```python
def record_audio(realtime_model: RTAudioModel) -> pyaudio.Stream:
    """Setup a Pyaudio input stream and use the RTAudioModel as a callback for streaming data."""

    def audio_callback(in_data, frame_count, time_info, status):
        realtime_model.send_audio(in_data)
        return (None, pyaudio.paContinue)

    p = pyaudio.PyAudio()
    stream = p.open(
        format=pyaudio.paInt16,
        channels=INPUT_DEVICE_CHANNELS,
        rate=SAMPLE_RATE,
        input=True,
        input_device_index=INPUT_DEVICE_INDEX,
        frames_per_buffer=CHUNK,
        stream_callback=audio_callback,
    )
    stream.start_stream()

    print("Recording started. Please begin speaking to your personal assistant...")
    return stream
```

--------------------------------

### Example Keeper Hostnames with Customized Installation Name

Source: https://docs.wandb.ai/weave/guides/platform/weave-self-managed

Shows Keeper node hostnames when the installation name is customized (e.g., 'myweave').

```text
chk-myweave-keeper-0-0.clickhouse.svc.cluster.local
chk-myweave-keeper-0-1.clickhouse.svc.cluster.local
chk-myweave-keeper-0-2.clickhouse.svc.cluster.local
```

--------------------------------

### Install Mintlify using npm

Source: https://docs.wandb.ai/weave/development

Install the Mintlify CLI globally using npm. Ensure Node.js version 19 or higher is installed.

```bash
npm i -g mintlify
```

--------------------------------

### Install GSM8K Training Environment

Source: https://docs.wandb.ai/weave/guides/integrations/verifiers

Install the pre-configured GSM8K training environment using the vf-install command.

```bash
vf-install gsm8k --from-repo
```

--------------------------------

### Install Weave and W&B

Source: https://docs.wandb.ai/weave/guides/integrations/verifiers

Installs the Weave and W&B libraries using uv.

```bash
uv pip install weave wandb
```

--------------------------------

### Install Verifiers and Dependencies

Source: https://docs.wandb.ai/weave/guides/integrations/verifiers

Clone the Verifiers repository from GitHub and install the library along with necessary dependencies using uv.

```bash
git clone https://github.com/willccbb/verifiers
cd verifiers
uv sync --all-extras && uv pip install flash-attn --no-build-isolation
```

--------------------------------

### Install Libraries with uv

Source: https://docs.wandb.ai/weave/guides/integrations/openai-realtime-audio

Install necessary libraries using the 'uv' package manager. Ensure you have 'weave', 'websockets', 'pyaudio', and 'numpy' installed.

```bash
uv add weave websockets pyaudio numpy
```

--------------------------------

### Install Libraries with pip

Source: https://docs.wandb.ai/weave/guides/integrations/openai-realtime-audio

Install necessary libraries using 'pip'. Ensure you have 'weave', 'websockets', 'pyaudio', and 'numpy' installed.

```bash
pip install weave websockets pyaudio numpy
```

--------------------------------

### Install OTEL Dependencies

Source: https://docs.wandb.ai/weave/guides/integrations/pydantic_ai

Install the necessary OpenTelemetry SDK and OTLP HTTP exporter for tracing.

```bash
pip install opentelemetry-sdk OTELemetry-exporter-otlp-proto-http
```

--------------------------------

### Install Weave with npm

Source: https://docs.wandb.ai/weave/guides/integrations/js

Install the Weave SDK and any other required libraries using npm.

```bash
npm install weave
```

--------------------------------

### Install AutoGen and Weave

Source: https://docs.wandb.ai/weave/guides/integrations/autogen

Install the necessary packages for AutoGen and Weave. Ensure you also install SDKs for your chosen LLM providers.

```bash
pip install autogen_agentchat "autogen_ext[openai,anthropic]" weave
```

--------------------------------

### Install Smolagents, OpenAI, and Weave

Source: https://docs.wandb.ai/weave/guides/integrations/smolagents

Install or upgrade the necessary libraries for Smolagents and Weave integration. Use '-qqq' to suppress output.

```bash
pip install -U smolagents openai weave -qqq
```

--------------------------------

### Install Mintlify using Yarn

Source: https://docs.wandb.ai/weave/development

Install the Mintlify CLI globally using Yarn. Ensure Node.js version 19 or higher is installed.

```bash
yarn global add mintlify
```

--------------------------------

### Run OpenTelemetry Example

Source: https://docs.wandb.ai/weave/guides/tracking/otel

Execute the OpenTelemetry example TypeScript file using Node.js.

```bash
npx ts-node opentelemetry_example.ts
```

--------------------------------

### Install and Import Libraries

Source: https://docs.wandb.ai/weave/cookbooks/codegen

Installs necessary libraries and imports them for the code generation pipeline. Includes a workaround for an OpenAI library bug.

```python
!pip install -qU autopep8 autoflake weave isort openai set-env-colab-kaggle-dotenv datasets
python
%%capture
# Temporary workaround to fix bug in openai:
# TypeError: Client.__init__() got an unexpected keyword argument 'proxies'
# See https://community.openai.com/t/error-with-openai-1-56-0-client-init-got-an-unexpected-keyword-argument-proxies/1040332/15
!pip install "httpx<0.28"
python
import ast
import os
import re
import subprocess
import tempfile
import traceback

import autopep8
import isort
from autoflake import fix_code
from datasets import load_dataset
from openai import OpenAI
from pydantic import BaseModel
from set_env import set_env

import weave
from weave import Dataset, Evaluation

set_env("WANDB_API_KEY")
set_env("OPENAI_API_KEY")
python
WEAVE_PROJECT = "codegen-cookbook-example"
weave.init(WEAVE_PROJECT)
python
client = OpenAI()
python
human_eval = load_dataset("openai_humaneval")
selected_examples = human_eval["test"][:3]
```

--------------------------------

### Install Hugging Face Hub and Weave

Source: https://docs.wandb.ai/weave/guides/integrations/huggingface

Install or upgrade the huggingface_hub and weave libraries to their latest versions. This command minimizes installation output.

```python
pip install -U huggingface_hub weave -qqq
```

--------------------------------

### Install Verifiers Core Library

Source: https://docs.wandb.ai/weave/guides/integrations/verifiers

Installs the core Verifiers library for local development and API-based models using uv.

```bash
uv add verifiers
```

--------------------------------

### Install Weave Claude Plugin

Source: https://docs.wandb.ai/weave/guides/integrations/claude_code

Install the CLI globally and then run the installer once to register the plugin with Claude Code. This process creates settings files and prompts for W&B credentials.

```bash
npm install -g weave-claude-plugin
weave-claude-plugin install
```

--------------------------------

### Install wandb and weave

Source: https://docs.wandb.ai/weave/guides/tracking/trace-to-run

Install the necessary Python libraries for W&B and Weave integration.

```bash
pip install wandb weave
```

--------------------------------

### Install Verifiers from GitHub

Source: https://docs.wandb.ai/weave/guides/integrations/verifiers

Installs the latest version of the Verifiers library directly from GitHub, including unreleased features.

```bash
uv add verifiers @ git+https://github.com/willccbb/verifiers.git
```

--------------------------------

### Install OpenAI and Weave Libraries

Source: https://docs.wandb.ai/weave/guides/integrations/inference

Install the necessary Python libraries for using the Inference API and Weave for tracing LLM applications.

```bash
pip install openai weave
```

--------------------------------

### Install necessary packages

Source: https://docs.wandb.ai/weave/cookbooks/online_monitoring

Installs Streamlit, Pandas, Plotly, and Weave. These are the core libraries required for the production dashboard.

```bash
!pip install streamlit pandas plotly weave
```

--------------------------------

### Run Mintlify Development Server

Source: https://docs.wandb.ai/weave/development

Navigate to the docs directory containing docs.json and start the local development server.

```bash
mintlify dev
```

--------------------------------

### Install and Import Packages, Set Up Environment

Source: https://docs.wandb.ai/weave/cookbooks/import_from_csv

Installs necessary packages (wandb, weave, pandas, datetime) and sets up the W&B API key. It also writes sample CSV data to a file and initializes a Weave client.

```python
%pip install wandb weave pandas datetime --quiet
python
import os

import pandas as pd
import wandb
from google.colab import userdata

import weave

## Write samples file to disk
with open("/content/import_cookbook_data.csv", "w") as f:
    f.write(
        "conversation_id,turn_index,start_time,user_input,ground_truth,answer_text\n"
    )
    f.write(
        '1234,1,2024-09-04 13:05:39,This is the beginning, ["This was the beginning"], That was the beginning\n'
    )
    f.write(
        "1235,1,2024-09-04 13:02:11,This is another trace,, That was another trace\n"
    )
    f.write(
        "1235,2,2024-09-04 13:04:19,This is the next turn,, That was the next turn\n"
    )
    f.write(
        "1236,1,2024-09-04 13:02:10,This is a 3 turn conversation,, Woah thats a lot of turns\n"
    )
    f.write(
        '1236,2,2024-09-04 13:02:30,This is the second turn, ["That was definitely the second turn"], You are correct\n'
    )
    f.write("1236,3,2024-09-04 13:02:53,This is the end,, Well good riddance!\n")

os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
name_of_file = "/content/import_cookbook_data.csv"
name_of_wandb_project = "import-weave-traces-cookbook"

wandb.login()
python
weave_client = weave.init(name_of_wandb_project)
```

--------------------------------

### Install Dependencies with pip

Source: https://docs.wandb.ai/weave/guides/integrations/openai-realtime-audio

Installs necessary libraries including weave, openai-agents, websockets, pyaudio, and numpy using pip.

```bash
pip install weave openai-agents websockets pyaudio numpy
```

--------------------------------

### Install Vercel AI SDK and OpenTelemetry Dependencies

Source: https://docs.wandb.ai/weave/guides/integrations/vercel_ai_sdk

Install the necessary Vercel AI SDK and OpenTelemetry libraries using npm.

```bash
npm install ai @ai-sdk/openai @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto @opentelemetry/resources zod
```

--------------------------------

### Install Weave Skill in Claude Code

Source: https://docs.wandb.ai/weave/guides/integrations/claude_code

Use this slash command within a Claude Code session to interactively install and configure the Weave plugin. It checks for the CLI, runs the installer, and verifies the setup.

```text
/weave:weave-install

```

--------------------------------

### Setting up and running an evaluation

Source: https://docs.wandb.ai/weave/reference/typescript-sdk/classes/evaluation

This example demonstrates how to set up an evaluation by defining a dataset, a custom scoring function, and a model. It then initializes an Evaluation object and runs the evaluation process.

```typescript
const dataset = new weave.Dataset({
  id: 'my-dataset',
  rows: [
    { question: 'What is the capital of France?', expected: 'Paris' },
    { question: 'Who wrote "To Kill a Mockingbird"?', expected: 'Harper Lee' },
    { question: 'What is the square root of 64?', expected: '8' },
  ],
});

const scoringFunction = weave.op(function isEqual({ modelOutput, datasetRow }) {
  return modelOutput == datasetRow.expected;
});

const model = weave.op(async function alwaysParisModel({ question }) {
  return 'Paris';
});

const evaluation = new weave.Evaluation({
  id: 'my-evaluation',
  dataset: dataset,
  scorers: [scoringFunction],
});

const results = await evaluation.evaluate({ model });
```

--------------------------------

### Install Claude Agent SDK and Weave

Source: https://docs.wandb.ai/weave/guides/integrations/claude_agent

Install the necessary Python packages for the Claude Agent SDK and W&B Weave.

```bash
pip install weave claude-agent-sdk
```

--------------------------------

### Log a Complete Example with Inputs, Output, and Scores

Source: https://docs.wandb.ai/weave/ref-link-python

Use the `log_example` method as a convenience to log a complete example when all data (inputs, output, and scores) is available upfront. This combines the functionality of `log_prediction` and `log_score`.

```python
ev = EvaluationLogger()
ev.log_example(
    inputs={'q': 'What is 2+2?'},
    output='4',
    scores={'correctness': 1.0, 'fluency': 0.9}
)
```

--------------------------------

### Run the MCP Demo Client and Server

Source: https://docs.wandb.ai/weave/guides/integrations/mcp

Launches the MCP demo client and server. The client will start an interactive CLI for testing features.

```bash
python example_client.py example_server.py
```

--------------------------------

### Python RAG Implementation

Source: https://docs.wandb.ai/weave/tutorial-rag

This Python code demonstrates a complete Retrieval-Augmented Generation (RAG) application setup using Weave. It includes example articles, LLM and retrieval function definitions, model creation, and evaluation setup.

```Python
from openai import OpenAI
import weave
from weave import Model
import numpy as np
import json
import asyncio

# Examples to use for evaluations
articles = [
    "Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial “tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too,” one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.",
    "Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.",
    "Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if it's stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities",
    "Rivian and Lucid shares plunge after weak EV earnings reports Shares of electric vehicle makers Rivian and Lucid fell Thursday after the companies reported stagnant production in their fourth-quarter earnings after the bell Wednesday. Rivian shares sank about 25 percent, and Lucids stock dropped around 17 percent. Rivian forecast it will make 57,000 vehicles in 2024, slightly less than the 57,232 vehicles it produced in 2023. Lucid said it expects to make 9,000 vehicles in 2024, more than the 8,428 vehicles it made in 2023.",
    "Mauritius blocks Norwegian cruise ship over fears of a potential cholera outbreak Local authorities on Sunday denied permission for the Norwegian Dawn ship, which has 2,184 passengers and 1,026 crew on board, to access the Mauritius capital of Port Louis, citing “potential health risks.” The Mauritius Ports Authority said Sunday that samples were taken from at least 15 passengers on board the cruise ship. A spokesperson for the U.S.-headquartered Norwegian Cruise Line Holdings said Sunday that 'a small number of guests experienced mild symptoms of a stomach-related illness' during Norwegian Dawns South Africa voyage.",
    "Intuitive Machines lands on the moon in historic first for a U.S. company Intuitive Machines Nova-C cargo lander, named Odysseus after the mythological Greek hero, is the first U.S. spacecraft to soft land on the lunar surface since 1972. Intuitive Machines is the first company to pull off a moon landing — government agencies have carried out all previously successful missions. The company's stock surged in extended trading Thursday, after falling 11 percent in regular trading."
]

# Define a simple LLM call
@weave.op()
def llm_call(prompt):
    client = OpenAI()
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "user", "content": prompt}
        ],
    )
    return response.choices[0].message.content

# Define a simple retrieval function
@weave.op()
def retrieve(query):
    # In a real app, this would query a vector database or other knowledge source.
    # For this example, we'll just return a relevant article based on keywords.
    if "weight loss" in query.lower():
        return articles[0]
    elif "berkshire" in query.lower():
        return articles[1]
    elif "google" in query.lower() or "epic" in query.lower():
        return articles[2]
    elif "ev" in query.lower() or "electric vehicle" in query.lower():
        return articles[3]
    elif "cruise ship" in query.lower() or "cholera" in query.lower():
        return articles[4]
    elif "moon landing" in query.lower() or "intuitive machines" in query.lower():
        return articles[5]
    else:
        return "No relevant information found."

# Define a RAG pipeline
@weave.op()
def rag_pipeline(question):
    retrieved_info = retrieve(question)
    prompt = f"Based on the following information: {retrieved_info}\n\nAnswer the question: {question}"
    return llm_call(prompt)

# Define a Model subclass for RAG
class RagModel(Model):
    def predict(self, question):
        return rag_pipeline(question)

# Create an instance of the RAG model
my_rag_model = RagModel()

# Example usage
question = "Tell me about the latest news on weight loss drugs."
answer = my_rag_model.predict(question)
print(f"Question: {question}")
print(f"Answer: {answer}")

# Example of collecting examples for evaluation
example_question = "What did Berkshire Hathaway report for Q4 earnings?"
example_answer = my_rag_model.predict(example_question)

# Define a scoring function
@weave.op()
def score_answer(example, prediction):
    # This is a placeholder for a real scoring function.
    # In a real scenario, you might use another LLM to score the answer,
    # or compare it against a ground truth answer.
    if example.answer.lower() in prediction.lower():
        return 1.0
    else:
        return 0.0

# Create an Evaluation object
# Note: In a real scenario, you would have a list of examples, each with a question and expected answer.
# For simplicity, we'll just use one example here.
evaluation = weave.Evaluation(
    "my-rag-evaluation",
    model=my_rag_model,
    examples=[
        {
            "input": {"question": example_question},
            "output": {"answer": example_answer} # In a real eval, this would be the ground truth
        }
    ],
    score_fn=score_answer
)

# Run the evaluation (this will trigger the RAG pipeline for the example)
# In a real scenario, you would run this after collecting many examples.
evaluation.run()

print("Evaluation complete. Check the Weave UI for results.")

# Example of setting parallelism to avoid rate limits
# import os
# os.environ["WEAVE_PARALLELISM"] = "3"
# print(f"WEAVE_PARALLELISM set to: {os.environ.get('WEAVE_PARALLELISM')}")

```

--------------------------------

### Install Dependencies and Configure Environment

Source: https://docs.wandb.ai/weave/cookbooks/multi-agent-structured-output

Installs necessary libraries (openai, weave, wandb) and sets environment variables for W&B API key and OpenAI API key. Initializes W&B project and fetches a Weave client.

```python
!pip install -qU openai weave wandb
python
%%capture
# Temporary workaround to fix bug in openai:
# TypeError: Client.__init__() got an unexpected keyword argument 'proxies'
# See https://community.openai.com/t/error-with-openai-1-56-0-client-init-got-an-unexpected-keyword-argument-proxies/1040332/15
!pip install "httpx<0.28"
```

```python
import base64
import json
import os
from io import BytesIO, StringIO

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import wandb
from google.colab import userdata
from openai import OpenAI

import weave
python
os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
os.environ["OPENAI_API_KEY"] = userdata.get("OPENAI_API_KEY")

wandb.login()
name_of_wandb_project = "multi-agent-structured-output"
weave.init(name_of_wandb_project)

client = OpenAI()
MODEL = "gpt-4o-2024-08-06"
```

--------------------------------

### Weave Trace Output Example

Source: https://docs.wandb.ai/weave/quickstart

This is an example of the terminal output when running a Weave-tracked function. It includes installation instructions, login status, a link to view traces on wandb.ai, and the trace URL for the specific call, followed by the JSON output.

```shell
weave:  $ pip install weave --upgrade
weave: Logged in as Weights & Biases user: example-username.
weave: View Weave data at https://wandb.ai/your-team/traces-quickstart/weave
weave: 🍩 https://wandb.ai/your-team/traces-quickstart/r/call/019ae171-7f32-7c96-8b42-931a32f900b7
{
  "dinosaurs": [
    {
      "name": "Tyrannosaurus rex",
      "common_name": "T. rex",
      "diet": "carnivore"
    },
    {
      "name": "Triceratops",
      "common_name": "Trike",
      "diet": "herbivore"
    },
    {
      "name": "Brachiosaurus",
      "common_name": "Brachi",
      "diet": "herbivore"
    }
  ]
}

```

--------------------------------

### Setup and Imports for Weave and HuggingFace Datasets

Source: https://docs.wandb.ai/weave/cookbooks/hf_dataset_evals

Initializes Weave and imports necessary libraries like datasets and wandb. Logs into Weights & Biases and applies nest_asyncio for notebook environments. Ensure WANDB_KEY, WEAVE_TEAM, and WEAVE_PROJECT are set.

```python
!pip install datasets wandb weave
python
# Initialize variables
HUGGINGFACE_DATASET = "wandb/ragbench-test-sample"
WANDB_KEY = ""
WEAVE_TEAM = ""
WEAVE_PROJECT = ""

# Init weave and required libraries
import asyncio

import nest_asyncio
import wandb
from datasets import load_dataset

import weave
from weave import Evaluation

# Login to wandb and initialize weave
wandb.login(key=WANDB_KEY)
client = weave.init(f"{WEAVE_TEAM}/{WEAVE_PROJECT}")

# Apply nest_asyncio to allow nested event loops (needed for some notebook environments)
nest_asyncio.apply()
```

--------------------------------

### Evaluate RAG Model with Context Scorers

Source: https://docs.wandb.ai/weave/guides/evaluation/builtin_scorers

Example of running a Weave Evaluation with both ContextEntityRecallScorer and ContextRelevancyScorer. This demonstrates a typical RAG evaluation setup.

```python
import asyncio
from textwrap import dedent
import weave
from weave.scorers import ContextEntityRecallScorer, ContextRelevancyScorer

class RAGModel(weave.Model):
    @weave.op()
    async def predict(self, question: str) -> str:
        """Retrieve relevant context"""
        return "Paris is the capital of France."

# Define prompts
relevancy_prompt: str = dedent("""
    Given the following question and context, rate the relevancy of the context to the question on a scale from 0 to 1.

    Question: {question}
    Context: {context}
    Relevancy Score (0-1):
    """
)
# Initialize scorers
entity_recall_scorer = ContextEntityRecallScorer()
relevancy_scorer = ContextRelevancyScorer(relevancy_prompt=relevancy_prompt)
# Create dataset
dataset = [
    {
        "question": "What is the capital of France?",
        "context": "Paris is the capital city of France."
    },
    {
        "question": "Who wrote Romeo and Juliet?",
        "context": "William Shakespeare wrote many famous plays."
    }
]
# Run evaluation
evaluation = weave.Evaluation(
    dataset=dataset,
    scorers=[entity_recall_scorer, relevancy_scorer]
)
results = asyncio.run(evaluation.evaluate(RAGModel()))
print(results)
```

--------------------------------

### Using the Converse API with Bedrock

Source: https://docs.wandb.ai/weave/guides/integrations/bedrock

Example of using the Bedrock converse API to get a response. This method is an alternative to invoke_model for specific use cases.

```python
messages = [{"role": "user", "content": [{"text": "What is the capital of France?"}]}]

response = client.converse(
    modelId="anthropic.claude-3-5-sonnet-20240620-v1:0",
    system=[{"text": "You are a helpful AI assistant."}],
    messages=messages,
    inferenceConfig={"maxTokens": 100},
)
print(response["output"]["message"]["content"][0]["text"])
```

--------------------------------

### Basic OpenAI Instrumentor Setup

Source: https://docs.wandb.ai/weave/guides/tracking/otel

This snippet demonstrates the basic setup for the OpenAIInstrumentor, including calling the main function and ensuring the provider is shut down.

```javascript
console.log("Response:", response.choices[0]?.message?.content);
  }

  (async () => {
    await main();
    await new Promise(resolve => setTimeout(resolve, 2000));
    await provider.shutdown();
  })();
```

--------------------------------

### Create and Run a Simple ADK Agent with Tracing

Source: https://docs.wandb.ai/weave/guides/integrations/google_adk

Demonstrates how to create a basic LLM agent with a tool and run it using an in-memory runner after configuring OTEL tracing. This allows for automatic tracing of agent and tool calls.

```python
from google.adk.agents import LlmAgent
from google.adk.runners import InMemoryRunner
from google.adk.tools import FunctionTool
from google.genai import types
import asyncio

```

--------------------------------

### Instantiate and Kickoff CrewAI Flow

Source: https://docs.wandb.ai/weave/guides/integrations/crewai

This snippet shows the basic setup for initializing and running a CrewAI flow. It's used to start the agentic process.

```python
flow = CustomerFeedbackFlow()
result = flow.kickoff()
print(result)
```

--------------------------------

### Python MessagesPrompt Example

Source: https://docs.wandb.ai/weave/guides/core-types/prompts

Demonstrates how to create and publish a MessagesPrompt in Python for use with OpenAI's chat completions. Ensure you have the 'weave' and 'openai' libraries installed.

```python
import weave
weave.init('intro-example')

prompt = weave.MessagesPrompt([
    {
        "role": "system",
        "content": "You are a stegosaurus, but don't be too obvious about it."
    },
    {
        "role": "user",
        "content": "What's good to eat around here?"
    }
])
weave.publish(prompt, name="dino_prompt")

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
  model="gpt-4o",
  messages=prompt.format(),
)
```

--------------------------------

### Full TypeScript Evaluation Example

Source: https://docs.wandb.ai/weave/guides/core-types/evaluations

Demonstrates a complete evaluation run from start to finish using TypeScript. It includes defining a dataset, a scoring function, and a model to be evaluated.

```typescript
import * as weave from 'weave';

// Initialize Weave
await weave.init('intro-example');

// Collect your examples into a dataset
const dataset = new weave.Dataset({
  id: 'my-dataset',
  rows: [
    {question: 'What is the capital of France?', expected: 'Paris'},
    {question: 'Who wrote "To Kill a Mockingbird"?', expected: 'Harper Lee'},
    {question: 'What is the square root of 64?', expected: '8'},
  ],
});

// Define scoring functions
const matchScore = weave.op(
  ({modelOutput, datasetRow}) => {
    return {match: modelOutput === datasetRow.expected};
  },
  {name: 'matchScore'}
);

// Define the function to evaluate
const myModel = weave.op(
  async ({question}) => {
    // here's where you would add your LLM call and return the output
    return 'Paris';
  },
  {name: 'myModel'}
);

// Create and run the evaluation
const evaluation = new weave.Evaluation({
  id: 'my-evaluation',
  dataset: dataset,
  scorers: [matchScore],
});

const results = await evaluation.evaluate({model: myModel});
console.log('Evaluation results:', results);
```

--------------------------------

### Initialize OpenAI Client and Weave Project

Source: https://docs.wandb.ai/weave/cookbooks/audio_with_weave

Sets up the OpenAI client using an API key and initializes a new Weave project for logging.

```python
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
weave.init("openai-audio-chat")
```

--------------------------------

### Generate Self-Contained HTML with Serverless Inference

Source: https://docs.wandb.ai/weave/guides/core-types/media

Creates self-contained HTML pages using Serverless Inference and logs them to Weave. This example requires OpenAI client setup and a valid API key.

```python
import weave
from weave import Content
from typing import Annotated, Literal
import openai
import wandb

prompt_template = weave.StringPrompt("""
You are a front-end web developer. Generate a single self-contained `.html` file (no external build tools) that demonstrates: "{ONE_LINE_REQUEST}".
""")

client = openai.OpenAI(
    base_url='https://api.inference.wandb.ai/v1',
    api_key=wandb.api.api_key,
    project="wandb/test-html",
)

weave.init("your-team-name/your-project-name")
weave.publish(prompt_template, name="generate_prompt")

@weave.op
def generate_html(prompt: str, template: weave.StringPrompt) -> Annotated[bytes, Content[Literal['html']]]:
    response = client.chat.completions.create(
        model="Qwen/Qwen3-Coder-480B-A35B-Instruct",
        messages=[
            {"role": "system", "content": prompt_template.format(ONE_LINE_REQUEST=prompt)},
        ],
    )
    html_content = response.choices[0].message.content
    return html_content.encode('utf-8')

prompt = "Weights & Biases UI but with multi-run selection and plots, but it looks like Windows 95. Include 5 plots with comparisons of each run, bar plots, parallel coordinates and line plots for the runs. Use mock data for the runs. Make it possible to add new plots. Give the runs names like squishy-lemon-2, fantastic-horizon-4 etc. with random adjectives & nouns."

result = generate_html(prompt, prompt_template)
```

--------------------------------

### Track Groq API Calls with @weave.op

Source: https://docs.wandb.ai/weave/guides/integrations/groq

Wrap your Groq API interaction function with `@weave.op` to start capturing data. This example shows how to recommend places to visit in a city using Groq and Weave.

```python
import os
import weave
from groq import Groq


weave.init(project_name="groq-test")

client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

@weave.op()
def recommend_places_to_visit(city: str, model: str="llama3-8b-8192"):
    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": "You are a helpful assistant meant to suggest places to visit in a city",
            },
            {
                "role": "user",
                "content": city,
            }
        ],
        model="llama3-8b-8192",
    )
    return chat_completion.choices[0].message.content


recommend_places_to_visit("New York")
recommend_places_to_visit("Paris")
recommend_places_to_visit("Kolkata")
```

--------------------------------

### Full Python Evaluation Example

Source: https://docs.wandb.ai/weave/guides/core-types/evaluations

Demonstrates a complete evaluation run from start to finish using Python. It includes defining scoring functions, a custom model, and running evaluations for both the model and a standalone function.

```python
from weave import Evaluation, Model
import weave
import asyncio
weave.init('intro-example')
examples = [
    {"question": "What is the capital of France?", "expected": "Paris"},
    {"question": "Who wrote 'To Kill a Mockingbird'?", "expected": "Harper Lee"},
    {"question": "What is the square root of 64?", "expected": "8"},
]

@weave.op()
def match_score1(expected: str, output: dict) -> dict:
    return {'match': expected == output['generated_text']}

@weave.op()
def match_score2(expected: dict, output: dict) -> dict:
    return {'match': expected == output['generated_text']}

class MyModel(Model):
    prompt: str

    @weave.op()
    def predict(self, question: str):
        # here's where you would add your LLM call and return the output
        return {'generated_text': 'Hello, ' + question + self.prompt}

model = MyModel(prompt='World')
evaluation = Evaluation(dataset=examples, scorers=[match_score1, match_score2])

asyncio.run(evaluation.evaluate(model))

@weave.op()
def function_to_evaluate(question: str):
    # here's where you would add your LLM call and return the output
    return  {'generated_text': 'some response' + question}

asyncio.run(evaluation.evaluate(function_to_evaluate("What is the capitol of France?")))
```

--------------------------------

### Initialize Weave and Trace ChatNVIDIA Calls

Source: https://docs.wandb.ai/weave/guides/integrations/nvidia_nim

Start capturing traces for ChatNVIDIA library calls by initializing Weave with a project name. This snippet demonstrates the basic setup required before invoking the ChatNVIDIA client.

```python
from langchain_nvidia_ai_endpoints import ChatNVIDIA
import weave
client = ChatNVIDIA(model="mistralai/mixtral-8x7b-instruct-v0.1", temperature=0.8, max_tokens=64, top_p=1)
weave.init('emoji-bot')

messages=[
    {
      "role": "system",
      "content": "You are AGI. You will be provided with a message, and your task is to respond using emojis only."
    }]

response = client.invoke(messages)
```

--------------------------------

### Get Current Weave Op Call

Source: https://docs.wandb.ai/weave/ref-link-python

Retrieve the Call object for the currently executing Weave Op. This is only available when called from within an Op. The returned Call object's attributes are immutable after the call starts.

```python
current_call = weave.get_current_call()
```

--------------------------------

### Create and Use a Reasoning Agent

Source: https://docs.wandb.ai/weave/guides/integrations/agno

Set up an Agno agent with reasoning tools and a finance-specific toolset. This agent can analyze stock market data and provide reasoned investment recommendations, with its thought process captured in traces.

```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.reasoning import ReasoningTools
from agno.tools.yfinance import YFinanceTools

from dotenv import load_dotenv
load_dotenv()

# Load AgnoInstrumentor from the tracin.py file
from tracing import AgnoInstrumentor

# Start instrumenting Agno
AgnoInstrumentor().instrument()

# Create a reasoning agent
reasoning_agent = Agent(
    name="Reasoning Finance Agent",
    model=OpenAIChat(id="gpt-4o"),
    tools=[
        ReasoningTools(add_instructions=True),
        YFinanceTools(
            stock_price=True,
            analyst_recommendations=True,
            company_info=True,
            company_news=True
        ),
    ],
    instructions="Use tables to display data and show your reasoning process",
    show_tool_calls=True,
    markdown=True,
)

# Use the reasoning agent
reasoning_agent.print_response(
    "Should I invest in Apple stock right now? Analyze the current situation and provide a reasoned recommendation.",
    stream=True
)
```

--------------------------------

### EasyPrompt.__init__

Source: https://docs.wandb.ai/weave/reference/python-sdk

Initializes an EasyPrompt object.

```APIDOC
## method EasyPrompt.__init__

### Description
Initializes an EasyPrompt object.

### Method
method

### Signature
`__init__(content: str | dict | list | None = None, role: str | None = None, dedent: bool = False, **kwargs: Any) -> None`

### Pydantic Fields:
* `name`: `str | None`
* `description`: `str | None`
* `ref`: `trace.refs.ObjectRef | None`
* `data`: `<class 'list'>`
* `config`: `<class 'dict'>`
* `requirements`: `<class 'dict'>`
```
Tokens:
Raw