### Install Promptfoo Client Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/observation_tools/promptfoo/promptfoo.ipynb Install the promptfoo client. This is a one-time setup step required before running evaluations. ```python promptfoo_client = EvalPromptfoo() promptfoo_client.install() ``` -------------------------------- ### Install libraries and make necessary imports Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/vector_db/FAISS.ipynb Install the required dependencies for FAISS and UpTrain. Use faiss-cpu if faiss-gpu encounters installation errors. ```python pip install uptrain faiss-gpu sentence-transformers import numpy as np import faiss from uptrain import EvalLLM, Evals from sentence_transformers import SentenceTransformer ``` -------------------------------- ### Install Dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/rag/rag_evaluations_uptrain_mistral.ipynb Install the necessary packages for FAISS, Mistral, datasets, and UpTrain. ```python %pip install faiss-cpu mistralai datasets uptrain -q ``` -------------------------------- ### Install UpTrain Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/tutorials/analyzing-failure-cases.mdx Install the UpTrain library using pip. ```python %pip install uptrain ``` -------------------------------- ### Install Required Libraries Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/vector_db/qdrant.ipynb Install the necessary packages for QDrant, UpTrain, and sentence transformers. ```python !pip install -qU qdrant-client uptrain sentence-transformers ``` -------------------------------- ### Install Documentation Dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/CONTRIBUTING.md Install the necessary dependencies for working with the documentation, including the Mintlify CLI. ```bash npm i -g mintlify mintlify install ``` -------------------------------- ### Install Required Packages Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/observation-tools/zeno.mdx Installs the necessary Zeno client and UpTrain libraries. Run this command in your environment. ```python %pip install zeno-client uptrain ``` -------------------------------- ### Install dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/vector_db/chroma.ipynb Install the necessary libraries for ChromaDB, NumPy, datasets, and UpTrain. ```python # !pip install -Uq chromadb numpy datasets uptrain ``` -------------------------------- ### Install Uptrain and LiteLLM Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/spade/evaluating_guidelines_generated_by_spade.ipynb Run these commands in your terminal or notebook to install the Uptrain and LiteLLM libraries. Ensure you have pip installed. ```python !pip install uptrain !pip install litellm ``` -------------------------------- ### Start Documentation Development Server Source: https://github.com/uptrain-ai/uptrain/blob/main/CONTRIBUTING.md Navigate to the 'docs' folder and start the local development server for previewing documentation changes. The documentation will be available at http://localhost:3000. ```bash cd docs mintlify dev ``` -------------------------------- ### Install UpTrain and OpenAI Libraries Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/experiments/llm_openai_vs_gemma7b_experiment.ipynb Install the necessary libraries for UpTrain and OpenAI. This is a prerequisite for running the experiment. ```python # %pip install openai uptrain ``` -------------------------------- ### Install UpTrain and LlamaIndex Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/framework/llamaindex-methods/evalllamaindex.mdx Install the necessary libraries for UpTrain and LlamaIndex integration. ```python pip install uptrain llama_index ``` -------------------------------- ### Install Dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/observation-tools/langfuse.mdx Install the necessary libraries for Langfuse, datasets, UpTrain, litellm, and openai. ```python %pip install langfuse datasets uptrain litellm openai --upgrade ``` -------------------------------- ### Install UpTrain and LLM dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/experiments/llm_openai_vs_claude_experiment.ipynb Install the necessary libraries for OpenAI, Anthropic, and UpTrain. ```python # %pip install openai uptrain anthropic ``` -------------------------------- ### Install UpTrain Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/getting-started/quickstart.mdx Install the UpTrain library using pip. This is the first step to begin using UpTrain for your evaluation needs. ```bash pip install uptrain ``` -------------------------------- ### Install Project Dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/experiments/data_driven_experimentation_demo.ipynb Installs necessary libraries like uptrain, openai, langchain, and faiss. Use this command in your project's environment. ```python %pip install uptrain openai langchain faiss-cpu -q ``` -------------------------------- ### Clone and Run UpTrain Dashboard Source: https://github.com/uptrain-ai/uptrain/blob/main/README.md Clone the UpTrain repository and run the bash script to start the locally hosted dashboard. Ensure Docker is installed. ```bash git clone https://github.com/uptrain-ai/uptrain cd uptrain bash run_uptrain.sh ``` -------------------------------- ### Install UpTrain Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/llms/mistral.mdx Install the UpTrain library using pip. This is the first step to using UpTrain for LLM evaluations. ```python pip install uptrain ``` ```python from uptrain import EvalLLM, Evals, Settings import json ``` -------------------------------- ### Install Dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/observation-tools/helicone.mdx Install the required Python packages for OpenAI and UpTrain. ```python %pip install openai uptrain -qU ``` -------------------------------- ### Install UpTrain and LlamaIndex dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/framework/llamaindex-methods/callback_handler/overview.mdx Install the necessary packages to enable UpTrain and LlamaIndex integration. ```bash pip install -q html2text llama-index pandas tqdm uptrain cohere ``` -------------------------------- ### Install uptrain and requests libraries Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/embedding_models/jina.ipynb Installs the necessary Python libraries for making API requests and using UpTrain. ```python %pip install requests uptrain ``` -------------------------------- ### Install UpTrain and import dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/llm_providers/ollama.ipynb Install the UpTrain package and import the necessary classes for evaluation. ```python %pip install uptrain -qU from uptrain import EvalLLM, Evals, Settings import json ``` -------------------------------- ### Install dependencies Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/experiments/embedding_models_comparison.ipynb Install the required packages for the embedding model comparison experiment. ```python %pip install -qU httpx sentence-transformers faiss ``` -------------------------------- ### Install UpTrain and LlamaIndex Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/llamaindex.ipynb Install the necessary libraries for UpTrain and LlamaIndex. Note that you might need to upgrade pip. ```python %pip install -q uptrain llama-index ``` -------------------------------- ### Execute Prompt Generation Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/spade/evaluating_guidelines_generated_by_spade.ipynb Runs the defined prompt template against example data using the configured LLM. ```python from spade_utils import * print("TASK DATA:\n", EXAMPLES[0], "\n\RESPONSE:\n", generate_response(PROMPT_TEMPLATE, LLM, EXAMPLES[0])) ``` -------------------------------- ### Install and Use Black for Code Formatting Source: https://github.com/uptrain-ai/uptrain/blob/main/CONTRIBUTING.md Install Black, an automated Python code formatter, and use it to format your source files or directories. This ensures compliance with standard Python style guides. ```bash # Install Black pip install black black[jupyter] # Format your source changes black {source_file_or_directory} # or python -m black {source_file_or_directory} ``` -------------------------------- ### Create a Zeno Project Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/observation-tools/zeno.mdx Initializes a Zeno project with specific metrics and view configurations using the ZenoClient. ```python from zeno_client import ZenoClient, ZenoMetric client = ZenoClient(ZENO_API_KEY) project = client.create_project( name="UpTrain Zeno Demo", description="Evaluation of LLMs using UpTrain", public=True, view={ "data": {"type": "markdown"}, "label": {"type": "text"}, "output": {"type": "markdown"}, }, metrics=[ ZenoMetric(name="context relevance", type="mean", columns=["score_context_relevance"]), ZenoMetric(name="response completeness", type="mean", columns=["score_response_completeness"]), ZenoMetric(name="factual accuracy", type="mean", columns=["score_factual_accuracy"]), ZenoMetric(name="response relevance", type="mean", columns=["score_response_relevance"]), ], ) ``` -------------------------------- ### Set Up API Keys Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/observation-tools/zeno.mdx Configure your Zeno and OpenAI API keys. Ensure these are kept secure. ```python ZENO_API_KEY = "sk-***************" OPENAI_API_KEY = "zen_***************" ``` -------------------------------- ### Initialize and Evaluate with UpTrain OSS Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/custom/custom_prompt_evals.ipynb Set up the EvalLLM with your OpenAI API key and response format. Then, use the evaluate method with your data and custom prompt checks. ```python from uptrain import CustomPromptEval, EvalLLM, Settings import json OPENAI_API_KEY = "sk-*****************" # Insert your OpenAI key here eval_llm = EvalLLM(settings=Settings(openai_api_key=OPENAI_API_KEY, response_format={"type":"json_object"})) results = eval_llm.evaluate( data = data, checks = [CustomPromptEval( prompt = prompt, choices = choices, choice_scores = choice_scores, prompt_var_to_column_mapping = prompt_var_to_column_mapping )] ) ``` -------------------------------- ### Install Pandas Package Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/code_eval/code_hallucination.ipynb Command to install the Pandas package in Python using pip. ```bash pip install pandas ``` -------------------------------- ### Run Next.js Development Server Source: https://github.com/uptrain-ai/uptrain/blob/main/uptrain/dashboard/frontend/dashboard/README.md Use these commands to start the development server for your Next.js project. Open http://localhost:3000 in your browser to view the application. ```bash npm run dev # or yarn dev # or pnpm dev ``` -------------------------------- ### Instantiate Chroma client Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/vector_db/chroma.ipynb Initialize an ephemeral ChromaDB client for local, in-memory vector storage. ```python # Import Chroma and instantiate a client. The default Chroma client is ephemeral, meaning it will not save to disk. import chromadb client = chromadb.Client() ``` -------------------------------- ### Initialize SubQuestionQueryEngine Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/framework/llamaindex-methods/callback_handler/subques_query.mdx Configures a vector store index and wraps it in a SubQuestionQueryEngine to handle multi-part queries. ```python vector_query_engine = VectorStoreIndex.from_documents( documents=documents, use_async=True, service_context=service_context ).as_query_engine() query_engine_tools = [ QueryEngineTool( query_engine=vector_query_engine, metadata=ToolMetadata( name="documents", description="Paul Graham essay on What I Worked On", ), ), ] query_engine = SubQuestionQueryEngine.from_defaults( query_engine_tools=query_engine_tools, service_context=service_context, use_async=True, ) response = query_engine.query( "How was Paul Grahams life different before, during, and after YC?" ) ``` -------------------------------- ### Initialize EvalLLM and Run Evaluation Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/context_awareness/relevance.ipynb Set up the EvalLLM with your OpenAI API key and run an evaluation check for context relevance. Ensure you have the 'uptrain' library installed and your API key is correctly configured. ```python from uptrain import EvalLLM, Evals import json OPENAI_API_KEY = "sk-********************" # Insert your OpenAI key here eval_llm = EvalLLM(openai_api_key=OPENAI_API_KEY) res = eval_llm.evaluate( data = data, checks = [Evals.CONTEXT_RELEVANCE] ) ``` -------------------------------- ### Example Output Log Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/experiments/embedding_models_comparison.ipynb This is an example of the log output during the evaluation process, showing the progress of sending evaluation requests to the Uptrain server. ```log Output: 2024-04-01 10:41:21.664 | INFO  | uptrain.framework.remote:log_and_evaluate:669 - Sending evaluation request for rows 0 to <50 to the Uptrain server 2024-04-01 10:41:43.813 | INFO  | uptrain.framework.remote:log_and_evaluate:669 - Sending evaluation request for rows 50 to <100 to the Uptrain server 2024-04-01 10:42:12.790 | INFO  | uptrain.framework.remote:log_and_evaluate:669 - Sending evaluation request for rows 100 to <150 to the Uptrain server 2024-04-01 10:42:50.607 | INFO  | uptrain.framework.remote:log_and_evaluate:669 - Sending evaluation request for rows 150 to <200 to the Uptrain server 2024-04-01 10:43:16.171 | INFO  | uptrain.framework.remote:log_and_evaluate:669 - Sending evaluation request for rows 200 to <250 to the Uptrain server 2024-04-01 10:43:38.115 | INFO  | uptrain.framework.remote:log_and_evaluate:669 - Sending evaluation request for rows 250 to <300 to the Uptrain server ``` -------------------------------- ### Initialize OpenAI Client with Helicone Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/observation-tools/helicone.mdx Set up the OpenAI client to route requests through the Helicone gateway. ```python import os from openai import OpenAI import uuid import requests update_headers = { 'Authorization': f'Bearer {HELICONE_API_KEY}', 'Content-Type': 'application/json', } client = OpenAI( api_key=OPENAI_API_KEY, # Replace with your OpenAI API key base_url="http://oai.hconeai.com/v1", # Set the API endpoint default_headers= { # Optionally set default headers or set per request (see below) "Helicone-Auth": f"Bearer {HELICONE_API_KEY}", } ) ``` -------------------------------- ### Initialize UpTrain Client and Evaluate Data Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/context_awareness/context_utlization.ipynb Configures the API client with an access token and executes a response completeness evaluation. ```python from uptrain import Evals, APIClient, Settings UPTRAIN_API_KEY = "up-********************" # Insert your UpTrain API key here uptrain_client = APIClient( Settings( uptrain_access_token=UPTRAIN_API_KEY, response_format={"type": "json_object"} ) ) res = uptrain_client.log_and_evaluate( "Sample-response-completeness-wrt-context-evals", data = data, checks = [Evals.RESPONSE_COMPLETENESS_WRT_CONTEXT] ) print(json.dumps(res, indent=3)) ``` -------------------------------- ### Install RAG Libraries Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/vector_db/milvus.ipynb Installs the required Python packages for Milvus, UpTrain, and related models. Ensure you have Python 3.11 or compatible version. ```python %pip install pymilvus uptrain milvus-model ``` -------------------------------- ### Initialize Uptrain API Client Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/tutorials/uptrain-api-client.mdx Configure the client with your access token and server URL. ```python from uptrain import APIClient, Settings settings = Settings( uptrain_access_token=YOUR_API_KEY, uptrain_server_url="https://demo.uptrain.ai" ) client = APIClient(settings) ``` -------------------------------- ### Example Evaluation Run Result Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/openai_evals.ipynb This is an example of the JSON output returned by the `get_run` method, showing the status, dataset, and checkset information for an evaluation. ```json { 'run_id': 'b1ce5260eca94d2a93231f2c53bc66a6', 'created_at': '2024-02-09T08:43:08.618160', 'updated_at': '2024-02-09T08:43:08.629075', 'status': 'running', 'result': '', 'dataset': 'qna-dataset', 'checkset': 'qna-checkset', 'dataset_version': 2, 'checkset_version': 2 } ``` -------------------------------- ### Initialize UpTrain API Client Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/tutorials/experiments-evaluation-tutorial.mdx Create an API client instance using an API key retrieved from environment variables. ```python UPTRAIN_API_KEY = "up-***************" import os UPTRAIN_API_KEY = os.environ.get("UPTRAIN_API_KEY") client = APIClient(uptrain_api_key=UPTRAIN_API_KEY) ``` -------------------------------- ### Example Run Result Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/custom/model_grade_score.ipynb An example of the JSON response when retrieving run results, showing the run ID, status, and associated dataset and checkset. ```json { "run_id": "6c6baddb2b46443ab762e4a913820159", "created_at": "2024-01-10T12:01:08.238637", "updated_at": "2024-01-10T12:01:50.346376", "status": "completed", "result": "", "dataset": "qna-dataset", "checkset": "qna-checkset", "dataset_version": 5, "checkset_version": 5 } ``` -------------------------------- ### Setup VectorDB with OpenAI Embeddings Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/experiments/data_driven_experimentation_demo.ipynb Initializes a Faiss vector store with OpenAI embeddings for efficient data retrieval. Requires all_splits from the previous step. ```python from langchain.vectorstores.faiss import FAISS from langchain.embeddings import OpenAIEmbeddings vectorstore = FAISS.from_documents(documents=all_splits, embedding= OpenAIEmbeddings()) ``` -------------------------------- ### Example Evaluation Output for Conversation Satisfaction Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/conversation/conversation_satisfaction.ipynb This is an example of the structured output from a ConversationSatisfaction evaluation. It includes the conversation turns, the satisfaction score, and a detailed explanation for the score. ```json [ { "conversation": [ { "role": "patient", "content": "Help" }, { "role": "nurse", "content": "what do you need" }, { "role": "patient", "content": "Having chest pain" }, { "role": "nurse", "content": "please call 102" }, { "role": "patient", "content": "Thank you nurse" } ], "score_conversation_satisfaction": 1.0, "explanation_conversation_satisfaction": "1. The patient starts the conversation with \"Help\", indicating a sense of urgency or distress.\n2. The nurse responds by asking what the patient needs, showing a willingness to help.\n3. The patient expresses that they are having chest pain, a serious medical concern.\n4. The nurse advises the patient to call 102, indicating a sense of urgency and concern for the patient's well-being.\n5. The patient responds with \"Thank you nurse\", indicating gratitude and satisfaction with the assistance provided.\n\nBased on the conversation, it can be inferred that the patient looks highly satisfied in the conversation (A) because the nurse responded promptly and appropriately to the patient's medical concern, and the patient expressed gratitude at the end of the conversation.\n\n1.0\n1.0" }, { "conversation": [ { "role": "patient", "content": "Help" }, { "role": "nurse", "content": "what do you need" }, { "role": "patient", "content": "Having chest pain" }, { "role": "nurse", "content": "Sorry, I am not sure what that means" }, { "role": "patient", "content": "You don't understand. Do something! I am having severe pain in my chest" } ], "score_conversation_satisfaction": 0.0, "explanation_conversation_satisfaction": "1. The patient starts the conversation with a simple \"Help\" which indicates urgency and distress.\n2. The nurse responds by asking what the patient needs, showing willingness to help.\n3. The patient then explains that they are having chest pain, which is a serious medical concern.\n4. The nurse responds by saying they are not sure what that means, which could be perceived as dismissive or unhelpful.\n5. The patient then expresses frustration and urgency by saying \"Do something! I am having severe pain in my chest.\"\n\nBased on the conversation, it is clear that the patient is not at all satisfied in the conversation. The nurse's response did not address the patient's urgent medical concern, and the patient's frustration is evident in their final statement.\n\n0.0\n0.0" } ] ``` -------------------------------- ### Initialize UpTrain API Client and Evaluate Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/conversation/guideline_adherence.ipynb Configures the API client with an access token and executes an evaluation check for conversation guideline adherence. ```python from uptrain import APIClient, Settings, ConversationGuidelineAdherence UPTRAIN_API_KEY = "up-*******************************" # Insert your UpTrain API key here uptrain_client = APIClient( Settings( uptrain_access_token=UPTRAIN_API_KEY, ) ) res = uptrain_client.log_and_evaluate( "Emergency-Contact-Information", data=data, checks=[ConversationGuidelineAdherence(guideline="Provide emergency contact information if the patient is in distress", guideline_name="Emergency Contact Information")], ) print(json.dumps(res, indent=3)) ``` -------------------------------- ### Example Evaluation Output Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/framework/llamaindex-methods/callback_handler/rag_query.mdx This is an example of the evaluation output captured by the UpTrain callback handler for each query, including Context Relevance, Factual Accuracy, and Response Completeness scores. ```bash Question: What did Paul Graham do growing up? Context Relevance Score: 0.0 Factual Accuracy Score: 1.0 Response Completeness Score: 0.0 Question: When and how did Paul Graham's mother die? Context Relevance Score: 0.0 Factual Accuracy Score: 1.0 Response Completeness Score: 0.0 Question: What, in Paul Graham's opinion, is the most distinctive thing about YC? Context Relevance Score: 1.0 Factual Accuracy Score: 1.0 Response Completeness Score: 1.0 Question: When and how did Paul Graham meet Jessica Livingston? Context Relevance Score: 1.0 Factual Accuracy Score: 1.0 Response Completeness Score: 0.5 Question: What is Bel, and when and where was it written? Context Relevance Score: 1.0 Factual Accuracy Score: 1.0 Response Completeness Score: 0.0 ``` -------------------------------- ### Sample Datapoint Input Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/tutorials/analyzing-failure-cases.mdx A simplified example of a datapoint used for evaluation. ```json { "question": "Can FedL send a delivery to Bangalore?", "context": "FedL was established in 2020. Using FedL you can send deliveries to over 1000+ cities in India including major cities like Bangalore, Mumbai and Delhi. Recently we crossed a milestone by completing 1 million deliveries. Using FedL you can deliver any goods under 10kg(not more than that) to anyone whether it be your friends or family. P.S.: We can't deliver electronic devices. You can also use FedL to deliver a car.", "response": "Yes, FedL can send a delivery to Bangalore.", "cited_context": "FedL was established in 2020.", } ``` -------------------------------- ### Initialize UpTrain API Client and Run Evaluation Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/context_awareness/conciseness.ipynb Configures the API client with an access token and executes a context conciseness evaluation on provided data. ```python from uptrain import Evals, APIClient, Settings UPTRAIN_API_KEY = "up-********************" # Insert your UpTrain API key here uptrain_client = APIClient( Settings( uptrain_access_token=UPTRAIN_API_KEY, response_format={"type": "json_object"} ) ) res = uptrain_client.log_and_evaluate( "Sample-context-conciseness-test", data = data, checks = [Evals.CONTEXT_CONCISENESS] ) print(json.dumps(res, indent=3)) ``` -------------------------------- ### Create Dataset Folder and Download Data Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/integrations/framework/llamaindex-methods/evalllamaindex.mdx Prepare a directory for your dataset and download sample data (NYC Wikipedia text) if it doesn't exist. ```python url = "https://uptrain-assets.s3.ap-south-1.amazonaws.com/data/nyc_text.txt" if not os.path.exists('nyc_wikipedia'): os.makedirs('nyc_wikipedia') dataset_path = os.path.join('./nyc_wikipedia', "nyc_text.txt") if not os.path.exists(dataset_path): import httpx r = httpx.get(url) with open(dataset_path, "wb") as f: f.write(r.content) ``` -------------------------------- ### Sample Evaluation Result Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/llms/ollama.mdx Example JSON output structure from an evaluation run. ```json [ { 'context': 'France, context for its exquisite pastries and fashion, has ' "a capital city called Paris. It's a place where people speak " 'French and enjoy baguettes. I once heard that the Eiffel ' "Tower was built by aliens, but don't quote me on that.", 'explanation_context_relevance': '{\n' ' "Reasoning": "The extracted ' 'context can answer the given query ' 'because it provides information about ' "France's capital city, Paris. This " 'information is relevant to answering ' 'the question about the capital of ' 'France.",\n' ' "Choice": "A"\n' '}', 'explanation_response_conciseness': '{\n' ' "Reasoning": "The given ' 'response provides information about ' 'the capital of France, but it lacks ' 'the detail that could be expected ' 'in this context.",\n' ' "Choice": "B"\n' '}', 'explanation_response_relevance': 'Response Precision: 0.5{\n' ' "Reasoning": "The given response ' 'provides information about the ' 'capital of France, but it lacks the ' 'detail that could be expected in this ' 'context.",\n' ' "Choice": "B"\n' '}\n' 'Response Recall: 1.0{\n' ' "Reasoning": "The given response ' 'adequately answers the question about ' 'the capital of France because it ' 'provides the correct answer, i.e., ' } ``` -------------------------------- ### Evaluation Output Logs Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/query_quality/sub_query_completeness.ipynb Example of the console output generated during the evaluation process. ```text Output: /Users/shreyanshjain/anaconda3/lib/python3.11/site-packages/lazy_loader/__init__.py:185: RuntimeWarning: subpackages can technically be lazily loaded, but it causes the package to be eagerly loaded even if it is already lazily loaded.So, you probably shouldn't use subpackages with this lazy feature. warnings.warn(msg, RuntimeWarning) 2024-02-28 16:51:36.406 | INFO  | uptrain.framework.evalllm:evaluate_on_server:216 - Sending evaluation request for rows 0 to <50 to the Uptrain ``` -------------------------------- ### Import Required Libraries and Initialize Client Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/rag/rag_evaluations_uptrain_mistral.ipynb Import necessary libraries and initialize the Mistral client using an environment variable. ```python from mistralai.client import MistralClient from mistralai.models.chat_completion import ChatMessage from datasets import load_dataset import requests import numpy as np import faiss import os import json mistral_api_key= os.environ["MISTRAL_API_KEY"] client = MistralClient(api_key=mistral_api_key) ``` -------------------------------- ### Runtime Warning Output Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/openai_evals.ipynb Example of a runtime warning message encountered during package loading. ```text /Users/dhruvchawla/Work/uptrain-v1/.venv/lib/python3.11/site-packages/lazy_loader/__init__.py:185: RuntimeWarning: subpackages can technically be lazily loaded, but it causes the package to be eagerly loaded even if it is already lazily loaded.So, you probably shouldn't use subpackages with this lazy feature. warnings.warn(msg, RuntimeWarning) ``` -------------------------------- ### Initialize UpTrain Settings Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/open_source_evaluator_tutorial.ipynb Create the settings object and EvalLLM instance using the Mistral model. ```python from uptrain import Settings settings = Settings(model = 'mistral/mistral-tiny', mistral_api_key=MISTRAL_API_KEY) eval_llm = EvalLLM(settings) ``` -------------------------------- ### Response Matching Output Format Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/predefined-evaluations/ground-truth-comparison/response-matching.mdx Example JSON structure returned by the evaluation process. ```json [ { "response_match_precision": 1.0, "response_match_recall": 0.5, "score_response_match": 0.57, "response_match_method": "llm" } ] ``` -------------------------------- ### Initialize UpTrain and API authentication Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/experiments/llm_openai_vs_claude_experiment.ipynb Import required modules and configure API keys for OpenAI and Anthropic. ```python import polars as pl import os from uptrain import EvalLLM, Evals, ResponseMatching import openai from anthropic import Anthropic OPENAI_API_KEY = "sk-*****************" anthropic = Anthropic(api_key="sk-ant-****************") ``` -------------------------------- ### Visualize Results with Streamlit Source: https://github.com/uptrain-ai/uptrain/blob/main/docs/tutorials/openai-evals.mdx Starts a local Streamlit dashboard to view evaluation logs. ```python from uptrain.dashboard import StreamlitRunner runner = StreamlitRunner(settings.logs_folder) runner.start() ``` -------------------------------- ### Initialize UpTrain Settings Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/open_source_evaluator_tutorial.ipynb Create a Settings object with the specified model and API key, then initialize the EvalLLM instance. ```python from uptrain import Settings settings = Settings(model = 'claude-2.1', anthropic_api_key=ANTHROPIC_API_KEY) eval_llm = EvalLLM(settings) ``` -------------------------------- ### UpTrain Execution Logs Source: https://github.com/uptrain-ai/uptrain/blob/main/examples/integrations/observation_tools/langfuse.ipynb Example output logs generated during the execution of UpTrain evaluation pipelines. ```text /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/lazy_loader/__init__.py:185: RuntimeWarning: subpackages can technically be lazily loaded, but it causes the package to be eagerly loaded even if it is already lazily loaded.So, you probably shouldn't use subpackages with this lazy feature. warnings.warn(msg, RuntimeWarning) 2024-02-21 04:27:32.472 | WARNING  | uptrain.operators.language.llm:fetch_responses:234 - Detected a running event loop, scheduling requests in a separate thread. 100%|██████████| 4/4 [00:01<00:00, 2.47it/s] 2024-02-21 04:27:34.153 | WARNING  | uptrain.operators.language.llm:fetch_responses:234 - Detected a running event loop, scheduling requests in a separate thread. 100%|██████████| 4/4 [00:01<00:00, 2.33it/s] 2024-02-21 04:27:35.873 | WARNING  | uptrain.operators.language.llm:fetch_responses:234 - Detected a running event loop, scheduling requests in a separate thread. 0%| | 0/4 [00:00