### Create Leva Prompt Configuration (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This Ruby code shows how to define a Leva prompt, including system and user prompts, versioning, and metadata. This prompt can then be associated with an experiment to guide the LLM's response.

```ruby
prompt = Leva::Prompt.create!(
  name: "Sentiment Analysis",
  version: 1,
  system_prompt: "You are an expert at analyzing text and returning the sentiment.",
  user_prompt: "Please analyze the following text and return the sentiment as Positive, Negative, or Neutral.\n\n{{TEXT}}",
  metadata: { model: "gpt-4", temperature: 0.5 }
)
```

--------------------------------

### Install Leva Engine Gem

Source: https://context7.com/kieranklaassen/leva/llms.txt

Instructions to add the Leva gem to your Rails project's Gemfile, install it, and run migrations. This sets up the necessary database tables for Leva's functionality.

```ruby
# Gemfile
gem 'leva'
```

```bash
bundle install
rails leva:install:migrations
rails db:migrate
```

```ruby
# config/routes.rb
Rails.application.routes.draw do
  mount Leva::Engine => "/leva"
  # your other routes...
end
```

--------------------------------

### Install Leva Migrations and Run Migrations (Bash)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

These bash commands are used to install Leva's database migrations and then apply them to your database. This is necessary for Leva to manage its data within your Rails application.

```bash
rails leva:install:migrations
rails db:migrate
```

--------------------------------

### Add Leva Gem to Gemfile (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This snippet demonstrates how to add the Leva gem to your application's Gemfile for installation. It's a standard Ruby gem installation process.

```ruby
gem 'leva'
```

--------------------------------

### Ruby: Run a Single Record Evaluation

Source: https://context7.com/kieranklaassen/leva/llms.txt

Provides an example of evaluating a single record from a dataset, which is useful for testing purposes. It details how to run the evaluation and access the prediction, parsed predictions, and evaluation scores for that specific record.

```ruby
# Evaluate just one record (useful for testing)
dataset_record = dataset.dataset_records.first
runner = SentimentRun.new
evaluators = [SentimentAccuracyEval.new]

Leva.run_single_evaluation(
  experiment: experiment,
  run: runner,
  evals: evaluators,
  dataset_record: dataset_record
)

# Access results for that record
runner_result = dataset_record.runner_results.last
runner_result.prediction
# => "<sentiment>Positive</sentiment>"

runner_result.parsed_predictions
# => ["Positive"]

evaluation_result = runner_result.evaluation_results.first
evaluation_result.score
# => 1.0
```

--------------------------------

### Ruby: Calculate Semantic Similarity

Source: https://context7.com/kieranklaassen/leva/llms.txt

Evaluates semantic similarity between a prediction and ground truth using embeddings. It requires placeholder functions for getting embeddings and calculating cosine similarity. The output is a score between 0.0 and 1.0.

```ruby
class SemanticSimilarityEval < Leva::BaseEval
  def evaluate(runner_result, recordable)
    prediction = runner_result.parsed_predictions.first
    ground_truth = recordable.ground_truth

    # Example: Use embeddings to calculate semantic similarity
    # Replace with actual embedding service
    similarity = calculate_cosine_similarity(
      get_embedding(prediction),
      get_embedding(ground_truth)
    )

    # Return score between 0.0 and 1.0
    similarity
  end

  private

  def get_embedding(text)
    # Placeholder: implement actual embedding API call
    # e.g., OpenAI embeddings, Sentence-BERT, etc.
    [0.1, 0.2, 0.3] # dummy embedding
  end

  def calculate_cosine_similarity(vec1, vec2)
    # Implement cosine similarity calculation
    dot_product = vec1.zip(vec2).map { |a, b| a * b }.sum
    magnitude1 = Math.sqrt(vec1.map { |x| x ** 2 }.sum)
    magnitude2 = Math.sqrt(vec2.map { |x| x ** 2 }.sum)
    dot_product / (magnitude1 * magnitude2)
  end
end
```

--------------------------------

### Run Leva Experiment (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This Ruby code demonstrates how to set up and run an experiment in Leva. It involves creating an experiment instance with a dataset, instantiating a run and evaluation classes, and then initiating the evaluation process.

```ruby
experiment = Leva::Experiment.create!(name: "Sentiment Analysis", dataset: dataset)

run = SentimentRun.new
evals = [SentimentAccuracyEval.new, SentimentF1Eval.new]

Leva.run_evaluation(experiment: experiment, run: run, evals: evals)
```

--------------------------------

### Ruby: Create and Execute an Experiment

Source: https://context7.com/kieranklaassen/leva/llms.txt

Shows how to create and execute a Leva experiment. This involves defining the experiment's name, description, dataset, prompt, runner class, and evaluator classes. The code demonstrates both synchronous and asynchronous execution, and how to check experiment status.

```ruby
# Create experiment with runner and evaluators
experiment = Leva::Experiment.create!(
  name: "Sentiment Analysis Baseline v1",
  description: "Testing GPT-4 on customer review sentiment",
  dataset: dataset,
  prompt: prompt,
  runner_class: "SentimentRun",
  evaluator_classes: ["SentimentAccuracyEval", "SentimentF1Eval"]
)

# Instantiate runner and evaluators
runner = SentimentRun.new
evaluators = [
  SentimentAccuracyEval.new,
  SentimentF1Eval.new
]

# Run evaluation (processes all dataset records)
Leva.run_evaluation(
  experiment: experiment,
  run: runner,
  evals: evaluators
)

# Check experiment status
experiment.reload
experiment.status
# => "completed"

# Experiments are automatically queued in background when created via UI
# or by calling after_create callback
experiment = Leva::Experiment.create!(
  name: "Async Experiment",
  dataset: dataset,
  prompt: prompt,
  runner_class: "SentimentRun",
  evaluator_classes: ["SentimentAccuracyEval"]
)
# => Automatically enqueues ExperimentJob.perform_later(experiment.id)
```

--------------------------------

### Create and Populate a Leva Dataset (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This Ruby code demonstrates how to create a new Leva dataset and add records to it using a `Recordable` model. It illustrates the process of initializing a dataset and populating it with data for evaluation.

```ruby
dataset = Leva::Dataset.create(name: "Sentiment Analysis Dataset")
dataset.add_record TextContent.create(text: "I love this product!", expected_label: "Positive")
dataset.add_record TextContent.create(text: "Terrible experience", expected_label: "Negative")
dataset.add_record TextContent.create(text: "It's ok", expected_label: "Neutral")
```

--------------------------------

### Ruby: Create and Manage Prompts with Liquid Templating

Source: https://context7.com/kieranklaassen/leva/llms.txt

Demonstrates how to create and manage prompts using Liquid templating in Ruby. It includes creating prompts with version control, updating them, and accessing metadata. Prompts can incorporate dynamic data from context and runner results.

```ruby
# Create a prompt with version control
prompt = Leva::Prompt.create!(
  name: "Sentiment Analysis v1",
  system_prompt: "You are an expert at analyzing customer feedback and determining sentiment. Be concise and accurate.",
  user_prompt: <<~PROMPT,
    Analyze the sentiment of the following customer review and respond with ONLY one word: Positive, Negative, or Neutral.

    Review: {{ text }}

    Format your response as: <sentiment>YOUR_ANSWER</sentiment>
  PROMPT,
  metadata: {
    model: "gpt-4",
    temperature: 0.3,
    max_tokens: 10
  }
)

# Prompts auto-increment version on each save
prompt.version
# => 1

prompt.update!(user_prompt: "Updated prompt with better instructions...")
prompt.version
# => 2

# Access prompt configuration
prompt.metadata['model']
# => "gpt-4"
```

```ruby
# Create prompt using both record and runner context
prompt = Leva::Prompt.create!(
  name: "Context-Aware Sentiment Analysis",
  system_prompt: "You are a sentiment analysis expert. Consider the context and metadata.",
  user_prompt: <<~PROMPT,
    Analyze the sentiment of this review. Note that we found {{ similar_texts_count }} similar reviews.

    Review Text: {{ text }}
    Expected Label: {{ expected_label }}
    Analysis Time: {{ analysis_timestamp }}

    Provide your sentiment classification: <sentiment>YOUR_ANSWER</sentiment>
  PROMPT,
  metadata: {
    model: "gpt-4-turbo",
    temperature: 0.5,
    top_p: 0.9
  }
)
```

--------------------------------

### Run Leva Experiment with Prompt (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This Ruby code demonstrates how to create and run a Leva experiment that utilizes a predefined prompt. The experiment is configured with a dataset and the prompt object, and then the evaluation is executed with the specified run and eval classes.

```ruby
experiment = Leva::Experiment.create!(
  name: "Sentiment Analysis with LLM",
  dataset: dataset,
  prompt: prompt
)

run = SentimentRun.new
evals = [SentimentAccuracyEval.new, SentimentF1Eval.new]

Leva.run_evaluation(experiment: experiment, run: run, evals: evals)
```

--------------------------------

### Leva Web UI Routes in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

Defines the mounting point for the Leva engine and lists available web UI routes for interacting with datasets, records, experiments, and prompts. It also provides the base URL for accessing the workbench interface.

```ruby
# Mount point (from routes.rb)
mount Leva::Engine => "/leva"

# Available routes:
# GET  /leva                                    # Workbench dashboard
# GET  /leva/datasets                           # List all datasets
# GET  /leva/datasets/:id                       # Dataset detail
# GET  /leva/datasets/:id/records               # List dataset records
# GET  /leva/datasets/:id/records/:record_id    # Record detail
# GET  /leva/experiments                        # List all experiments
# GET  /leva/experiments/:id                    # Experiment detail with results
# POST /leva/experiments/:id/rerun              # Clear and re-run experiment
# GET  /leva/prompts                            # List all prompts
# POST /leva/workbench/run                      # Execute runner interactively
# POST /leva/workbench/run_all_evals            # Run all evaluators on result
# POST /leva/workbench/run_evaluator            # Run specific evaluator

# Navigate to http://localhost:3000/leva for the UI
```

--------------------------------

### Generate Leva Runner Scaffold

Source: https://context7.com/kieranklaassen/leva/llms.txt

This command generates a basic scaffold for a new runner in the Leva evaluation framework. Runners define the logic for executing models against datasets.

```bash
# Generate a runner scaffold
rails generate leva:runner sentiment
```

--------------------------------

### Mount Leva Engine in Rails Routes (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This Ruby code shows how to mount the Leva engine in your Rails application's `config/routes.rb` file. This makes the Leva UI accessible at the specified path, typically '/leva'.

```ruby
Rails.application.routes.draw do
  mount Leva::Engine => "/leva"
  # your other routes...
end
```

--------------------------------

### Compare Multiple Experiments in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

Compares different prompts or models by fetching experiments ordered by creation date. It iterates through each experiment, displaying its name, prompt version, runner class, status, and average scores grouped by evaluator class.

```ruby
# Compare different prompts or models
experiments = Leva::Experiment.where(dataset: dataset).order(created_at: :desc)

experiments.each do |exp|
  puts "\n#{exp.name} (#{exp.prompt.version})"
  puts "Runner: #{exp.runner_class}"
  puts "Status: #{exp.status}"

  exp.evaluation_results.group_by(&:evaluator_class).each do |eval_class, results|
    avg = results.sum(&:score) / results.size.to_f
    puts "  #{eval_class.split('::').last}: #{avg.round(3)}"
  end
end

# Example output:
# Sentiment Analysis Baseline v1 (1)
# Runner: SentimentRun
# Status: completed
#   SentimentAccuracyEval: 0.857
#   SentimentF1Eval: 0.823
#
# Sentiment Analysis Improved v2 (2)
# Runner: OpenaiSentimentRun
# Status: completed
#   SentimentAccuracyEval: 0.942
#   SentimentF1Eval: 0.916
```

--------------------------------

### Define Recordable Model for Leva Datasets (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This Ruby code defines a model compatible with Leva. By including `Leva::Recordable`, the model gains methods like `ground_truth`, `index_attributes`, `show_attributes`, and `to_llm_context` required for dataset management and display within Leva.

```ruby
class TextContent < ApplicationRecord
  include Leva::Recordable

  # @return [String] The ground truth label for the record
  def ground_truth
    expected_label
  end

  # @return [Hash] A hash of attributes to be displayed in the dataset records index
  def index_attributes
    {
      text: text,
      expected_label: expected_label,
      created_at: created_at.strftime('%Y-%m-%d %H:%M:%S')
    }
  end

  # @return [Hash] A hash of attributes to be displayed in the dataset record show view
  def show_attributes
    {
      text: text,
      expected_label: expected_label,
      created_at: created_at.strftime('%Y-%m-%d %H:%M:%S')
    }
  end

  # @return [Hash] A hash of attributes to be displayed in the dataset record show view
  def to_llm_context
    {
      text: text,
      expected_label: expected_label,
      created_at: created_at.strftime('%Y-%m-%d %H:%M:%S')
    }
  end
end
```

--------------------------------

### Background Job for Experiment Execution in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

Defines the `ExperimentJob` for asynchronously processing experiment runs. This job finds the experiment, checks its status to prevent duplicates, updates the status to 'running', and then queues individual `RunEvalJob` instances for each dataset record with a staggered delay.

```ruby
# When experiment is created, ExperimentJob is automatically queued
class Leva::ExperimentJob < ApplicationJob
  queue_as :default

  def perform(experiment_id)
    experiment = Leva::Experiment.find(experiment_id)

    # Prevent duplicate runs
    return if experiment.running? || experiment.completed?

    experiment.update!(status: :running)

    # Queue individual record jobs with stagger to prevent thundering herd
    experiment.dataset.dataset_records.each_with_index do |record, index|
      Leva::RunEvalJob.set(wait: (index * 3).seconds).perform_later(
        experiment.id,
        record.id
      )
    end
  end
end
```

--------------------------------

### Process Each Record Independently with Leva::RunEvalJob

Source: https://context7.com/kieranklaassen/leva/llms.txt

This Ruby code defines a background job that processes individual dataset records for a given experiment. It instantiates a runner and evaluators, executes a single evaluation, and updates the experiment status if all records are processed. Dependencies include Leva::Experiment, Leva::DatasetRecord, and Leva's evaluation execution logic.

```ruby
class Leva::RunEvalJob < ApplicationJob
  queue_as :default

  def perform(experiment_id, dataset_record_id)
    experiment = Leva::Experiment.find(experiment_id)
    dataset_record = Leva::DatasetRecord.find(dataset_record_id)

    # Instantiate runner and evaluators
    runner = experiment.runner_class.constantize.new
    evaluators = experiment.evaluator_classes.map { |e| e.constantize.new }

    # Execute
    Leva.run_single_evaluation(
      experiment: experiment,
      run: runner,
      evals: evaluators,
      dataset_record: dataset_record
    )

    # Mark complete if all records processed
    if experiment.runner_results.count == experiment.dataset.dataset_records.count
      experiment.update!(status: :completed)
    end
  end
end
```

--------------------------------

### Generate Leva Runner (Bash)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This bash command uses the Rails generator to create a new runner class for Leva. This runner will contain the logic for executing your language model inference.

```bash
rails generate leva:runner sentiment
```

--------------------------------

### Ruby: Rerun an Experiment

Source: https://context7.com/kieranklaassen/leva/llms.txt

Explains how to rerun an existing Leva experiment. This involves clearing previous results and re-executing the evaluation. The code demonstrates how to achieve this via the API by destroying previous results and updating the status, or by using a controller action.

```ruby
# Clear previous results and re-execute
experiment = Leva::Experiment.find(1)

# Via API (clears results and requeues)
experiment.runner_results.destroy_all
experiment.update!(status: :pending)

runner = experiment.runner_class.constantize.new
evaluators = experiment.evaluator_classes.map { |e| e.constantize.new }

Leva.run_evaluation(
  experiment: experiment,
  run: runner,
  evals: evaluators
)

# Via controller action (POST /experiments/:id/rerun)
# Automatically handles cleanup and requeueing
```

--------------------------------

### Add Records to a Leva Dataset

Source: https://context7.com/kieranklaassen/leva/llms.txt

This code demonstrates how to create a new Leva dataset and add existing ActiveRecord records to it. It also shows how to iterate through dataset records and access the underlying model and its ground truth.

```ruby
# Create a new dataset
dataset = Leva::Dataset.create!(
  name: "Sentiment Analysis Dataset",
  description: "Customer reviews with labeled sentiment"
)

# Add existing ActiveRecord records to the dataset
dataset.add_record(TextContent.create!(
  text: "I love this product! Amazing quality and fast shipping.",
  expected_label: "Positive"
))

dataset.add_record(TextContent.create!(
  text: "Terrible experience. Product broke after one week.",
  expected_label: "Negative"
))

dataset.add_record(TextContent.create!(
  text: "It's ok. Nothing special but does the job.",
  expected_label: "Neutral"
))

# List all records in a dataset
dataset.dataset_records.each do |record|
  puts record.recordable.text
  puts record.ground_truth
end

# Access the underlying model via polymorphic association
dataset.dataset_records.first.recordable
# => #<TextContent id: 1, text: "I love this product!..., expected_label: "Positive">

```

--------------------------------

### Simulate Sentiment Analysis Runner in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

Implements a basic sentiment analysis runner by simulating LLM API calls. It parses text content, determines sentiment (Positive, Negative, Neutral) based on keywords, and returns the result in a structured XML format. It also includes optional context gathering for LLM prompts.

```ruby
class SentimentRun < Leva::BaseRun
  # Required: Implement model execution logic
  # @param record [ActiveRecord::Base] The recordable object (e.g., TextContent)
  # @return [String] The raw model output/prediction
  def execute(record)
    # Example: Call an LLM API with the prompt
    prompt_template = Liquid::Template.parse(@prompt.user_prompt)
    rendered_prompt = prompt_template.render(merged_llm_context)

    # Simulate API call (replace with actual LLM integration)
    text = record.text.downcase
    sentiment = case
    when text.match?(/(love|great|excellent|awesome|fantastic)/)
      "Positive"
    when text.match?(/(hate|terrible|awful|horrible|bad)/)
      "Negative"
    else
      "Neutral"
    end

    # Return output in structured format for parsing
    "<sentiment>#{sentiment}</sentiment>"
  end

  # Optional: Provide runner-specific context for LLM prompts
  # Use for expensive computations that shouldn't be in the model's to_llm_context
  # @param record [ActiveRecord::Base] The recordable object
  # @return [Hash] Additional context merged with record's to_llm_context
  def to_llm_context(record)
    {
      # Example: Expensive database query
      similar_texts_count: record.class.where(
        "text LIKE ?", "%#{record.text.split.first}%"
      ).count,
      analysis_timestamp: Time.current.iso8601
    }
  end
end
```

--------------------------------

### Implement Leva Evaluation Logic (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This Ruby code shows how to implement an evaluation method within a Leva eval class. The `evaluate` method compares the model's `prediction` against the `record`'s ground truth and returns a score along with the ground truth value.

```ruby
class SentimentAccuracyEval < Leva::BaseEval
  def evaluate(prediction, record)
    score = prediction == record.expected_label ? 1.0 : 0.0
    [score, record.expected_label]
  end
end

class SentimentF1Eval < Leva::BaseEval
  def evaluate(prediction, record)
    # Calculate F1 score
    # ...
    [f1_score, record.f1_score]
  end
end
```

--------------------------------

### Generate Leva Eval (Bash)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This bash command generates a new evaluation class for Leva. This class will contain the logic for scoring the predictions made by your language model.

```bash
rails generate leva:eval sentiment_accuracy
```

--------------------------------

### OpenAI Sentiment Analysis Runner with Real API in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

Integrates with the OpenAI API to perform sentiment analysis. This runner renders system and user prompts using Liquid templating and sends them to the specified OpenAI model. It includes error handling for API calls and logs any exceptions.

```ruby
# app/runners/openai_sentiment_run.rb
require 'openai'

class OpenaiSentimentRun < Leva::BaseRun
  def execute(record)
    client = OpenAI::Client.new(access_token: ENV['OPENAI_API_KEY'])

    # Access merged context (record context + runner context)
    context = merged_llm_context

    # Render prompts with Liquid templating
    system_prompt = Liquid::Template.parse(@prompt.system_prompt).render(context)
    user_prompt = Liquid::Template.parse(@prompt.user_prompt).render(context)

    # Call OpenAI API
    response = client.chat(
      parameters: {
        model: @prompt.metadata['model'] || 'gpt-4',
        temperature: @prompt.metadata['temperature'] || 0.7,
        messages: [
          { role: 'system', content: system_prompt },
          { role: 'user', content: user_prompt }
        ]
      }
    )

    response.dig('choices', 0, 'message', 'content')
  rescue => e
    Rails.logger.error "OpenAI API error: #{e.message}"
    "Error: #{e.message}"
  end
end
```

--------------------------------

### Analyze Leva Experiment Results (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This Ruby code shows how to analyze the results of a Leva experiment. It iterates through the evaluation results, groups them by evaluator class, and calculates the average score for each evaluator.

```ruby
experiment.evaluation_results.group_by(&:evaluator_class).each do |evaluator_class, results|
  average_score = results.average(&:score)
  puts "#{evaluator_class.capitalize} Average Score: #{average_score}"
end
```

--------------------------------

### Multi-Model Runner Comparison in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

This Ruby code defines a Leva runner that executes multiple language models for a given record and returns their results. It iterates through a list of model identifiers, calls a private `call_model` method for each, and aggregates the results into a JSON string. This allows for direct comparison of different models' outputs.

```ruby
# app/runners/multi_model_run.rb
class MultiModelRun < Leva::BaseRun
  def execute(record)
    models = ['gpt-4', 'gpt-3.5-turbo', 'claude-3-opus']
    results = {}

    models.each do |model|
      results[model] = call_model(record, model)
    end

    # Return all results for comparison
    results.to_json
  end

  private

  def call_model(record, model)
    # Implementation for each model
    # ...
  end
end
```

--------------------------------

### Implement Leva Runner Logic (Ruby)

Source: https://github.com/kieranklaassen/leva/blob/main/README.md

This Ruby code defines the `execute` method within a Leva runner class. This method is responsible for taking a record as input and returning the model's prediction, which could involve API calls or local model execution.

```ruby
class SentimentRun < Leva::BaseRun
  def execute(record)
    # Your model execution logic here
    # This could involve calling an API, running a local model, etc.
    # Return the model's output
  end
end
```

--------------------------------

### Analyze Experiment Evaluation Results in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

Retrieves all evaluation results for a given experiment, groups them by evaluator, and calculates the average scores. It also shows how to access detailed results per record, including predictions and ground truth.

```ruby
# Get all evaluation results for an experiment
experiment = Leva::Experiment.find(1)

# Group by evaluator and calculate average scores
experiment.evaluation_results.group_by(&:evaluator_class).each do |evaluator_class, results|
  scores = results.map(&:score)
  average = scores.sum / scores.size.to_f

  puts "#{evaluator_class}: #{average.round(3)}"
  # => SentimentAccuracyEval: 0.857
  # => SentimentF1Eval: 0.823
end

# Access detailed results per record
experiment.runner_results.includes(:evaluation_results).each do |runner_result|
  puts "Record: #{runner_result.dataset_record.recordable.text.truncate(40)}"
  puts "Prediction: #{runner_result.parsed_predictions.first}"
  puts "Ground Truth: #{runner_result.ground_truth}"

  runner_result.evaluation_results.each do |eval_result|
    puts "  #{eval_result.evaluator_class}: #{eval_result.score}"
  end
end
```

--------------------------------

### Export Experiment Results to CSV in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

Exports detailed experiment results, including record ID, text, prediction, ground truth, accuracy, and F1 score, to a CSV file. It uses the `csv` library to generate the CSV data and `File.write` to save it.

```ruby
require 'csv'

experiment = Leva::Experiment.find(1)

csv_data = CSV.generate do |csv|
  csv << ["Record ID", "Text", "Prediction", "Ground Truth", "Accuracy", "F1 Score"]

  experiment.runner_results.includes(:evaluation_results, dataset_record: :recordable).each do |rr|
    record = rr.dataset_record.recordable
    accuracy = rr.evaluation_results.find_by(evaluator_class: "SentimentAccuracyEval")&.score
    f1 = rr.evaluation_results.find_by(evaluator_class: "SentimentF1Eval")&.score

    csv << [
      record.id,
      record.text.truncate(100),
      rr.parsed_predictions.first,
      rr.ground_truth,
      accuracy,
      f1
    ]
  end
end

File.write("experiment_#{experiment.id}_results.csv", csv_data)
```

--------------------------------

### Time-Based Performance Evaluation (Latency) in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

This Ruby code defines a Leva evaluator that calculates latency in milliseconds and scores it. It uses the `created_at` and `updated_at` timestamps from the `runner_result` to determine execution duration. The scoring mechanism provides a perfect score for latencies under 1 second, decaying linearly to 0 at 10 seconds.

```ruby
# app/evals/latency_eval.rb
class LatencyEval < Leva::BaseEval
  def evaluate(runner_result, recordable)
    # Check if runner stored execution time in metadata
    start_time = runner_result.created_at
    end_time = runner_result.updated_at

    latency_ms = (end_time - start_time) * 1000

    # Score based on latency (lower is better)
    # Perfect score (1.0) for < 1s, linear decay to 0 at 10s
    [1.0 - (latency_ms - 1000) / 9000.0, 0.0].max
  end
end
```

--------------------------------

### Make Models Evaluable with Leva::Recordable Concern

Source: https://context7.com/kieranklaassen/leva/llms.txt

This snippet demonstrates how to include the Leva::Recordable concern in an ActiveRecord model to make it compatible with the Leva evaluation framework. It defines essential methods for ground truth, display attributes, and LLM context.

```ruby
# app/models/text_content.rb
class TextContent < ApplicationRecord
  include Leva::Recordable

  # Required: Return the expected/ground truth value for evaluation
  def ground_truth
    expected_label
  end

  # Required: Attributes displayed in dataset records table listing
  def index_attributes
    {
      text: text.truncate(50),
      expected_label: expected_label
    }
  end

  # Required: Attributes displayed in dataset record detail view
  def show_attributes
    {
      text: text,
      expected_label: expected_label,
      created_at: created_at.strftime("%Y-%m-%d %H:%M:%S"),
      updated_at: updated_at.strftime("%Y-%m-%d %H:%M:%S")
    }
  end

  # Required: Context hash passed to LLM prompts via Liquid templating
  # Available in prompts as {{ text }}, {{ expected_label }}, etc.
  def to_llm_context
    {
      text: text,
      expected_label: expected_label
    }
  end

  # Optional: Regex pattern to extract structured predictions from model output
  # Example: Extracts "Positive" from "<sentiment>Positive</sentiment>"
  def extract_regex_pattern
    /<sentiment>(.*?)</sentiment>/
  end
end
```

--------------------------------

### Generate Sentiment Accuracy Evaluator in Bash

Source: https://context7.com/kieranklaassen/leva/llms.txt

A bash command to generate a scaffold for a new sentiment accuracy evaluator within the Leva framework. This command sets up the basic file structure and class definition for implementing custom evaluation logic.

```bash
# Generate an evaluator scaffold
rails generate leva:eval sentiment_accuracy
```

--------------------------------

### Implement Advanced Sentiment Evaluator with F1 Score in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

Implements an advanced sentiment evaluator that calculates the F1 score for each prediction and stores them for batch calculation. This evaluator handles multi-class sentiment and includes a simplified F1 calculation, storing intermediate results in the experiment's metadata.

```ruby
# app/evals/sentiment_f1_eval.rb
class SentimentF1Eval < Leva::BaseEval
  def evaluate(runner_result, recordable)
    prediction = runner_result.parsed_predictions.first
    ground_truth = recordable.ground_truth

    # Store predictions for batch F1 calculation
    @experiment.metadata ||= {}
    @experiment.metadata['predictions'] ||= []
    @experiment.metadata['predictions'] << {
      predicted: prediction,
      actual: ground_truth
    }
    @experiment.save!

    # Calculate F1 for this specific prediction
    calculate_f1(prediction, ground_truth)
  end

  private

  def calculate_f1(predicted, actual)
    return 1.0 if predicted == actual

    # For multi-class, calculate per-class F1
    # This is simplified; implement full confusion matrix for production
    true_positive = (predicted == actual && actual == "Positive") ? 1 : 0
    false_positive = (predicted != actual && predicted == "Positive") ? 1 : 0
    false_negative = (predicted != actual && actual == "Positive") ? 1 : 0

    precision = true_positive.to_f / (true_positive + false_positive + 0.001)
    recall = true_positive.to_f / (true_positive + false_negative + 0.001)

    2 * (precision * recall) / (precision + recall + 0.001)
  end
end
```

--------------------------------

### Implement Basic Sentiment Accuracy Evaluator in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

Defines a basic sentiment accuracy evaluator that compares a runner's prediction against the ground truth from the recordable object. It returns a score of 1.0 if they match and 0.0 otherwise, providing a simple binary accuracy measure.

```ruby
# app/evals/sentiment_accuracy_eval.rb
class SentimentAccuracyEval < Leva::BaseEval
  # Required: Compare prediction against ground truth
  # @param runner_result [Leva::RunnerResult] The result from runner execution
  # @param recordable [ActiveRecord::Base] The original record being evaluated
  # @return [Float] Score typically between 0.0 and 1.0
  def evaluate(runner_result, recordable)
    # Extract parsed prediction using regex pattern
    prediction = runner_result.parsed_predictions.first
    ground_truth = recordable.ground_truth

    # Binary accuracy: 1.0 for match, 0.0 for mismatch
    prediction == ground_truth ? 1.0 : 0.0
  end
end
```

--------------------------------

### Conditional Evaluation with Confidence Weighting in Ruby

Source: https://context7.com/kieranklaassen/leva/llms.txt

This Ruby code defines a custom Leva evaluator that weights accuracy based on prediction confidence extracted from the runner's output. It parses a 'confidence: X.XX' pattern from the prediction string. The output is a score between 0.0 and 1.0, representing weighted accuracy.

```ruby
# app/evals/confidence_weighted_accuracy_eval.rb
class ConfidenceWeightedAccuracyEval < Leva::BaseEval
  def evaluate(runner_result, recordable)
    prediction = runner_result.parsed_predictions.first
    ground_truth = recordable.ground_truth

    # Extract confidence if present
    confidence_match = runner_result.prediction.match(/confidence:\s*(\d+\.?\d*)/)
    confidence = confidence_match ? confidence_match[1].to_f : 1.0

    # Weight accuracy by confidence
    is_correct = prediction == ground_truth ? 1.0 : 0.0
    is_correct * confidence
  end
end
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.