### Define Use Case Summary Prompts in Scala

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Configures the prompt parts for generating use cases, including scenario guidelines and example use cases.

```scala
def getUseCaseSummaryPromptParts(person: Personas.Person): UseCaseSummaryPromptParts = {
  UseCaseSummaryPromptParts(
    evidenceTypeDescription = "situations where users share facts about themselves",
    scenarioGuidelines = """Create scenarios where:
      - The user naturally shares personal information
      - The facts are specific and memorable
      - Later recalling these facts would be useful""",
    exampleUseCases = Some(List(
      EvidenceUseCase(1, "Work Preferences", "User mentions their favorite project management tool"),
      EvidenceUseCase(2, "Personal Details", "User shares their dietary restrictions")
    ))
  )
}
```

--------------------------------

### Unlicense for Sample Code

Source: https://github.com/salesforceairesearch/convomem/blob/main/how_to_license.md

This text should be placed in a LICENSE.txt file for sample, demo, and example code. It signifies that the software is free and unencumbered, released into the public domain.

```text
This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means.

In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
```

--------------------------------

### Configure and Run Benchmark Evaluations with Evaluator

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Defines custom evaluation logic by extending the Evaluator class and provides command-line execution examples.

```scala
import com.salesforce.crmmembench.evaluation._
import com.salesforce.crmmembench.evaluation.memory._
import com.salesforce.crmmembench.questions.evidence.generators.UserFactsEvidenceGenerator
import com.salesforce.crmmembench.LLM_endpoints.{Gemini, OpenAI}

// Define a custom evaluator
object MyCustomEvaluator extends Evaluator {
  // Configure test case generation
  override def testCasesGenerator: TestCasesGenerator =
    new BatchedTestCasesGenerator(new UserFactsEvidenceGenerator(evidenceCount = 3))

  // Configure memory system
  override def memoryFactory: MemoryAnswererFactory = LongContextMemoryFactory

  // Configure the answering model
  override def model = Gemini.flash

  // Configure the judge model for verification
  override def judgeModel = Gemini.flash

  // Optional helper model for block-based processing
  override def helperModel = Some(Gemini.flashLite)

  // Configure parallelism
  override def testCaseThreads: Int = 40
}

// Run the evaluation
// MyCustomEvaluator.runEvaluation()

// Pre-built evaluators for common configurations
// object Evaluate1UserFactsEvidenceLargeContext extends Evaluator { ... }
// object Evaluate2UserFactsEvidenceMem0 extends Mem0Evaluator { ... }
// object Evaluate3UserFactsEvidenceBlockBased extends BlockBasedEvaluator { ... }

// Run via Gradle command line:
// ./gradlew run --args="evaluate --type user_facts --memory long_context"

// Or run specific evaluator class directly:
// ./gradlew findAndRun -PclassName=Evaluate2UserFactsEvidenceLargeContext
```

--------------------------------

### Initialize and Use LLMModel Instances

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Instantiate pre-configured LLM models from different providers like Gemini and OpenAI. Use the `generateContent` method for text generation and handle success or failure responses. Access model details and token usage information.

```scala
import com.salesforce.crmmembench.LLM_endpoints.{LLMModel, LLMResponse, Gemini, OpenAI, TokenUsage}
import scala.util.{Success, Failure}

// Use pre-configured model instances
val geminiFlash: LLMModel = Gemini.flash        // gemini-2.5-flash with rate limiting
val geminiPro: LLMModel = Gemini.pro            // gemini-2.5-pro with rate limiting
val geminiFlashLite: LLMModel = Gemini.flashLite // gemini-2.5-flash-lite
val gpt4o: LLMModel = OpenAI.gpt4o              // gpt-4o
val gpt4oMini: LLMModel = OpenAI.gpt4oMini      // gpt-4o-mini

// JSON-mode variants for structured output
val geminiFlashJson: LLMModel = Gemini.flashJson
val geminiProJson: LLMModel = Gemini.proJson

// Generate content with a model
geminiFlash.generateContent("What is the capital of France?") match {
  case Success(response: LLMResponse) =>
    println(s"Answer: ${response.content}")
    println(s"Model: ${response.modelName}")
    println(s"Cost: $$${response.cost}")

    // Token usage breakdown
    response.tokenUsage.foreach { usage: TokenUsage =>
      println(s"Input tokens: ${usage.inputTokens}")
      println(s"Output tokens: ${usage.outputTokens}")
      println(s"Total tokens: ${usage.totalTokens}")
      println(s"Cached tokens: ${usage.cachedInputTokens.getOrElse(0)}")
      println(s"Cache hit rate: ${usage.cacheHitRate.getOrElse(0.0)}%")
    }

  case Failure(exception) =>
    println(s"Error: ${exception.getMessage}")
}

// Model introspection
println(s"Model name: ${geminiFlash.getModelName}")   // "gemini-2.5-flash"
println(s"Provider: ${geminiFlash.getProvider}")       // "gemini"
```

--------------------------------

### Execute Benchmarks via Gradle

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Commands to set environment variables and trigger specific evaluation tasks using Gradle.

```bash
# Set required API keys
echo "GOOGLE_AI_API_KEY=your_gemini_key" > .env
echo "OPENAI_API_KEY=your_openai_key" >> .env

# Run user facts evaluation with long context memory
./gradlew findAndRun -PclassName=Evaluate2UserFactsEvidenceLargeContext

# Run with different models
./gradlew findAndRun -PclassName=Evaluate3UserFactsEvidenceLargeContextFlash    # Gemini Flash Lite
./gradlew findAndRun -PclassName=Evaluate5UserFactsEvidenceLargeContextPro      # Gemini Pro
./gradlew findAndRun -PclassName=Evaluate5UserFactsEvidenceLargeContexto4       # GPT-4o

# Run with mem0 memory system
./gradlew findAndRun -PclassName=Evaluate2UserFactsEvidenceMem0
```

--------------------------------

### Initialize and Use Mem0MemoryAnswerer

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Shows how to set up Mem0MemoryAnswerer for external RAG with vector search, including user ID prefix for isolation. Ensure Mem0Client is correctly configured.

```scala
import com.salesforce.crmmembench.memory.mem0.{Mem0MemoryAnswerer, Mem0Client}
import com.salesforce.crmmembench.questions.evidence.{Conversation, Message}
import com.salesforce.crmmembench.LLM_endpoints.Gemini

// Create mem0 memory answerer with custom configuration
val mem0Memory = new Mem0MemoryAnswerer(
  model = Gemini.flash,
  mem0Client = new Mem0Client(),
  userIdPrefix = "my-benchmark"  // Unique prefix for thread isolation
)

// Initialize the mem0 connection
mem0Memory.initialize()

// Create conversations with required IDs
val conversations = List(
  Conversation(
    messages = List(
      Message("User", "I'm allergic to peanuts and shellfish."),
      Message("Assistant", "I'll remember your food allergies.")
    ),
    id = Some("conv-health-001"),
    containsEvidence = Some(true)
  ),
  Conversation(
    messages = List(
      Message("User", "My budget for the project is $10,000."),
      Message("Assistant", "Noted - your project budget is $10,000.")
    ),
    id = Some("conv-budget-001"),
    containsEvidence = Some(true)
  )
)

// Bulk add conversations (includes indexing wait time)
mem0Memory.addConversations(conversations)

// Answer question using mem0's semantic search
val result = mem0Memory.answerQuestion(
  question = "What foods should I avoid due to allergies?",
  testCaseId = "test-mem0-001"
)

// Result includes memory system responses for debugging
result.answer.foreach(println)
result.memorySystemResponses.foreach {
  response =>
    println(s"Mem0 search response: ${response.take(500)}...")
}

// Get unique user ID for this instance
println(s"Mem0 user ID: ${mem0Memory.getUserId}")

// Cleanup all memories for this user
mem0Memory.cleanup()
```

--------------------------------

### Run ConvoMem Evaluation

Source: https://github.com/salesforceairesearch/convomem/blob/main/README.md

Configures the environment with API keys and executes the evaluation process using Gradle.

```bash
# Set API keys
echo "GOOGLE_AI_API_KEY=your_gemini_key" > .env

# Run evaluation
./gradlew run --args="evaluate --type user_facts --memory long_context"
```

--------------------------------

### Execute Short Runner via Gradle

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Command to execute the short runner for evidence generation.

```bash
./gradlew run -PmainClass=...GenerateMyEvidence1Short
```

--------------------------------

### Initialize Evidence Generators

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Instantiate various evidence generators for different categories, specifying the desired count of evidence items and changes. Load pre-generated evidence items from resources.

```scala
import com.salesforce.crmmembench.questions.evidence.generators._
import com.salesforce.crmmembench.questions.evidence._
import com.salesforce.crmmembench.questions.evidence.generation.EvidencePersistence

// Create evidence generators for different categories
val userFactsGen = new UserFactsEvidenceGenerator(evidenceCount = 3)
val changingEvidenceGen = new ChangingEvidenceGenerator(evidenceCount = 2, changeCount = 3)
val assistantFactsGen = new AssistantFactsEvidenceGenerator(evidenceCount = 2)
val preferenceGen = new PreferenceEvidenceGenerator(evidenceCount = 1)
val abstentionGen = new AbstentionEvidenceGenerator(evidenceCount = 2)
val implicitConnectionGen = new ImplicitConnectionEvidenceGenerator(evidenceCount = 2)

// Load pre-generated evidence items from resources
val evidenceItems: List[EvidenceItem] = userFactsGen.loadEvidenceItems()

println(s"Loaded ${evidenceItems.size} evidence items")

// Each evidence item contains:
evidenceItems.headOption.foreach {
  item =>
    println(s"Question: ${item.question}")
    println(s"Answer: ${item.answer}")
    println(s"Category: ${item.category}")
    println(s"Scenario: ${item.scenario_description.getOrElse("N/A")}")
    println(s"Evidence messages: ${item.message_evidences.size}")
    println(s"Conversations: ${item.conversations.size}")

    // Evidence messages that must appear in conversations
    item.message_evidences.foreach {
      msg =>
        println(s"  ${msg.speaker}: ${msg.text}")
    }
}

// Get generator metadata
println(s"Evidence type: ${userFactsGen.getEvidenceTypeName}")  // "User facts"
println(s"Evidence count: ${userFactsGen.getEvidenceCount()}")   // 3

// Generate new evidence (requires LLM API access)
// userFactsGen.generateEvidence()  // Generates and saves to resources
```

--------------------------------

### Initialize and Use LongContextMemoryAnswerer

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Demonstrates creating a LongContextMemoryAnswerer, adding conversations, and answering questions. Ensure the model endpoint is correctly configured.

```scala
import com.salesforce.crmmembench.evaluation.memory._
import com.salesforce.crmmembench.questions.evidence.{Conversation, Message}
import com.salesforce.crmmembench.LLM_endpoints.Gemini

// Create a long-context memory answerer (puts full history in prompt)
val longContextMemory = new LongContextMemoryAnswerer(model = Gemini.flash)

// Create sample conversations
val conversation1 = Conversation(
  messages = List(
    Message("User", "My favorite color is blue."),
    Message("Assistant", "Got it! I'll remember that your favorite color is blue.")
  ),
  id = Some("conv-001"),
  containsEvidence = Some(true)
)

val conversation2 = Conversation(
  messages = List(
    Message("User", "I have a meeting scheduled for 3pm tomorrow."),
    Message("Assistant", "I've noted your meeting at 3pm tomorrow.")
  ),
  id = Some("conv-002"),
  containsEvidence = Some(true)
)

// Initialize the memory system
longContextMemory.initialize()

// Add conversations to memory
longContextMemory.addConversation(conversation1)
longContextMemory.addConversation(conversation2)

// Or add multiple at once
longContextMemory.addConversations(List(conversation1, conversation2))

// Answer questions based on stored conversations
val result: AnswerResult = longContextMemory.answerQuestion(
  question = "What is my favorite color?",
  testCaseId = "test-001"
)

// Process the answer result
result.answer match {
  case Some(answer) =>
    println(s"Answer: $answer")
    println(s"Input tokens: ${result.inputTokens.getOrElse(0)}")
    println(s"Output tokens: ${result.outputTokens.getOrElse(0)}")
    println(s"Cost: $$${result.cost.getOrElse(0.0)}")
    println(s"Retrieved conversations: ${result.retrievedConversationIds}")
  case None =>
    println("Failed to get an answer")
}

// Get memory system type identifier
println(s"Memory type: ${longContextMemory.getMemoryType}")  // "long_context"

// Clear memory when done
longContextMemory.clearMemory()

// Cleanup resources
longContextMemory.cleanup()
```

--------------------------------

### Run with Custom JVM Settings

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Executes an evaluation task with increased heap memory allocation.

```bash
JAVA_OPTS="-Xmx8g" ./gradlew findAndRun -PclassName=Evaluate6UserFactsEvidenceLargeContext
```

--------------------------------

### Generate and Load Benchmark Dataset

Source: https://github.com/salesforceairesearch/convomem/blob/main/README.md

Use the Scala evaluation framework to generate test cases by loading evidence items and irrelevant conversations.

```scala
import com.salesforce.crmmembench.questions.evidence.generators.UserFactsEvidenceGenerator
import com.salesforce.crmmembench.evaluation.{ConversationLoader, TestCasesGenerator}

// Create evidence generator with 3 facts to track
val evidenceGenerator = new UserFactsEvidenceGenerator(3)

// Load evidence items from resources
val evidenceItems = evidenceGenerator.loadEvidenceItems()

// Load irrelevant conversations for mixing
val irrelevantConversations = ConversationLoader.loadIrrelevantConversations()

// Create test cases with standard mixing strategy
val testCasesGen = TestCasesGenerator.createStandard(evidenceGenerator)
val testCases = testCasesGen.generateTestCases()
```

--------------------------------

### Create Standard Test Cases Generator

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Create a standard test case generator by providing an evidence generator and a list of context sizes. This strategy generates test cases with varying amounts of irrelevant context.

```scala
import com.salesforce.crmmembench.evaluation._
import com.salesforce.crmmembench.questions.evidence.generators.UserFactsEvidenceGenerator
import com.salesforce.crmmembench.Config

// Create an evidence generator
val evidenceGenerator = new UserFactsEvidenceGenerator(evidenceCount = 2)

// Create test case generators with different strategies
val standardGen = TestCasesGenerator.createStandard(
  evidenceGenerator = evidenceGenerator,
  contextSizes = List(2, 4, 6, 10, 20, 50, 100)  // Number of total conversations
)

val batchedGen = TestCasesGenerator.createBatched(
  evidenceGenerator = evidenceGenerator,
  contextSizes = Config.Evaluation.CONTEXT_SIZES  // Default: 2 to 300
)

val stitchingGen = TestCasesGenerator.createStitching(
  evidenceGenerator = evidenceGenerator,
  threshold = 50,           // Use standard for < 50, batched for >= 50
  maxEvidencePerBatch = 100
)

// Generate test cases
val testCases: List[TestCase] = standardGen.generateTestCases()

println(s"Generated ${testCases.size} test cases")

// Each test case contains:
testCases.headOption.foreach {
  tc =>
    println(s"Test case ID: ${tc.id}")
    println(s"Conversation count: ${tc.conversationCount}")
    println(s"Context size: ${tc.contextSize}")
    println(s"Question: ${tc.evidenceItem.question}")
    println(s"Expected answer: ${tc.evidenceItem.answer}")
    println(s"Evidence conversations: ${tc.evidenceItem.conversations.size}")
}

// Get generator metadata
println(s"Generator type: ${standardGen.generatorType}")       // "User facts Generator"
println(s"Generator class: ${standardGen.generatorClassType}") // "standard"
println(s"Answering evaluation: ${standardGen.getAnsweringEvaluation}")
```

--------------------------------

### Define Evidence Core Generation Prompts in Scala

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Configures the prompt parts for generating evidence cores, defining guidelines for questions, answers, and evidence messages.

```scala
def getEvidenceCorePromptParts(person: Personas.Person, useCase: EvidenceUseCase): EvidenceCorePromptParts = {
  EvidenceCorePromptParts(
    evidenceTypeDescription = "user-stated facts that should be remembered",
    questionGuidelines = "Ask directly about the fact the user shared",
    answerGuidelines = "The answer should be exactly what the user stated",
    evidenceGuidelines = "User should state the fact clearly and naturally",
    exampleCore = Some(GeneratedEvidenceCore(
      question = "What's my favorite project management tool?",
      answer = "Asana",
      message_evidences = List(
        Message("User", "I've been using Asana for years - it's definitely my favorite project management tool.")
      )
    ))
  )
}
```

--------------------------------

### Utility and Maintenance Tasks

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Administrative tasks for verifying connectivity, resetting services, and inventory management.

```bash
./gradlew testAllModels
```

```bash
./gradlew resetMem0
```

```bash
./gradlew evidenceInventory
```

--------------------------------

### Generate Evidence Data

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Tasks for generating test data and filler conversations for the benchmark.

```bash
./gradlew runUserFactsShort
```

```bash
./gradlew generateConversations
```

--------------------------------

### Apache License 2.0 Boilerplate

Source: https://github.com/salesforceairesearch/convomem/blob/main/how_to_license.md

This boilerplate should be added as a comment to all Salesforce-authored source code and configuration files. It includes copyright information, license details, and disclaimers.

```text
/*
 * Copyright (c) 2025, Salesforce, Inc.
 * SPDX-License-Identifier: Apache-2
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
```

--------------------------------

### Generate Evidence Core

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Generates and prints the evidence core for a given person and use case to verify question and answer clarity.

```scala
val useCase = useCases.head
val core = generator.generateEvidenceCore(person, useCase)

println(s"Question: ${core.question}")
println(s"Answer: ${core.answer}")
core.message_evidences.foreach { msg =>
  println(s"[${msg.speaker}]: ${msg.text}")
}
```

--------------------------------

### Implement Production Runners

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Defines production-grade runner objects for full-scale evidence generation.

```scala
package com.salesforce.crmmembench.questions.evidence.runners

// For generators with evidenceCount parameter, create runners for 1-6
object GenerateMyEvidence1 {
  def main(args: Array[String]): Unit = {
    new MyEvidenceGenerator(1).generateEvidence()
  }
}

object GenerateMyEvidence2 {
  def main(args: Array[String]): Unit = {
    new MyEvidenceGenerator(2).generateEvidence()
  }
}

// ... up to 6
```

--------------------------------

### Run Evaluation Tasks

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Executes specific evaluation classes using the Gradle findAndRun task.

```bash
./gradlew findAndRun -PclassName=Evaluate3UserFactsEvidenceBlockBased
```

```bash
./gradlew findAndRun -PclassName=Evaluate2ChangingEvidenceLargeContext
```

```bash
./gradlew findAndRun -PclassName=Evaluate2AbstentionEvidenceLargeContext
```

```bash
./gradlew findAndRun -PclassName=Evaluate1PreferenceEvidenceLargeContext
```

```bash
./gradlew findAndRun -PclassName=Evaluate2ImplicitConnectionEvidenceLargeContext
```

--------------------------------

### Define Conversation Generation Prompts in Scala

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Configures the prompt parts for generating conversations based on specific use cases and evidence cores.

```scala
def getConversationPromptParts(person: Personas.Person, useCase: EvidenceUseCase, evidenceCore: GeneratedEvidenceCore): ConversationPromptParts = {
  ConversationPromptParts(
    evidenceType = "user facts memory test",
    scenarioDescription = "Testing if the AI remembers facts the user shares",
    useCaseScenario = Some(useCase.scenario_description),
    evidenceMessages = evidenceCore.message_evidences,
    question = evidenceCore.question,
    answer = evidenceCore.answer,
    evidenceCount = 1
  )
}
```

--------------------------------

### Initialize MultithreadedEvaluator

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Configures parallel test execution with batching and specific LLM endpoints. Requires predefined test cases and an answering evaluation strategy.

```scala
import com.salesforce.crmmembench.evaluation._
import com.salesforce.crmmembench.evaluation.memory._
import com.salesforce.crmmembench.questions.evidence.generation.DefaultAnsweringEvaluation
import com.salesforce.crmmembench.LLM_endpoints.Gemini

// Create a multithreaded evaluator
val evaluator = new MultithreadedEvaluator(
  caseType = "user_facts",
  memoryFactory = LongContextMemoryFactory,
  model = Some(Gemini.flash),
  helperModel = None,
  testCaseThreads = 40,
  answeringEvaluation = DefaultAnsweringEvaluation,
  testCaseGeneratorType = "batched",
  judgeModel = Gemini.flash,
  evidenceCount = 3
)

// Generate test cases first
val evidenceGenerator = new com.salesforce.crmmembench.questions.evidence.generators.UserFactsEvidenceGenerator(3)
val testCasesGenerator = new BatchedTestCasesGenerator(evidenceGenerator)
val testCases: List[TestCase] = testCasesGenerator.generateTestCases()

// Run the evaluation
// evaluator.runEvaluation(testCases)

// Results are automatically:
// - Logged to console with progress updates
// - Exported to CSV in logs/[evidence_type]/[timestamp]/
// - Include accuracy, cost, and latency metrics per context size

// Output structure:
// logs/
//   user_facts/
//     2024-01-15_14-30-00/
//       results.json       - Detailed per-question results
//       summary.txt        - Accuracy by context size
//       costs.csv          - Token usage and costs
```

--------------------------------

### Implement Short Runners

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Defines short-run objects for evidence generation to validate end-to-end functionality with a limited dataset.

```scala
package com.salesforce.crmmembench.questions.evidence.runners.short

object GenerateMyEvidence1Short {
  def main(args: Array[String]): Unit = {
    val generator = new MyEvidenceGenerator(1) {
      override protected val runShort: Boolean = true
    }
    generator.generateEvidence()
  }
}

object GenerateMyEvidence2Short {
  def main(args: Array[String]): Unit = {
    val generator = new MyEvidenceGenerator(2) {
      override protected val runShort: Boolean = true
    }
    generator.generateEvidence()
  }
}
```

--------------------------------

### Define Evidence and Conversation Schemas

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Defines the core data structures for messages, conversations, and evidence items. Includes configuration classes for different test types.

```scala
import com.salesforce.crmmembench.questions.evidence._

// Message - a single utterance in a conversation
val userMessage = Message(
  speaker = "User",      // "User" or "Assistant"
  text = "My budget for the renovation is $50,000."
)

// Conversation - a complete dialogue with metadata
val conversation = Conversation(
  messages = List(
    Message("User", "I'm planning a kitchen renovation."),
    Message("Assistant", "That sounds exciting! What's your budget?"),
    Message("User", "My budget is $50,000."),
    Message("Assistant", "Great, that gives us good options to work with.")
  ),
  id = Some("conv-12345"),
  containsEvidence = Some(true),
  model_name = Some("gemini-2.5-flash")
)

// GeneratedEvidenceCore - intermediate result from LLM generation
val evidenceCore = GeneratedEvidenceCore(
  question = "What is my renovation budget?",
  answer = "$50,000",
  message_evidences = List(
    Message("User", "My budget is $50,000.")
  ),
  model_name = Some("gemini-2.5-pro")
)

// EvidenceItem - complete test item with all metadata
val evidenceItem = EvidenceItem(
  question = "What is my renovation budget?",
  answer = "$50,000",
  message_evidences = List(
    Message("User", "My budget is $50,000.")
  ),
  conversations = List(conversation),
  category = "Home Improvement",
  scenario_description = Some("User discusses kitchen renovation plans with AI assistant"),
  personId = Some("person-001"),
  use_case_model_name = Some("gemini-2.5-flash"),
  core_model_name = Some("gemini-2.5-pro")
)

// EvidencePayload - collection for serialization
val payload = EvidencePayload(
  evidence_items = List(evidenceItem),
  checkpoint = Some("abc123def")  // Git commit hash
)

// Evidence configurations for different test types
val userFactsConfig = UserFactsEvidenceConfig(count = 3)
val changingConfig = ChangingEvidenceConfig(evidenceCount = 2, changeCount = 3)
val abstentionConfig = AbstentionEvidenceConfig(evidenceCount = 2)
val preferenceConfig = PreferenceEvidenceConfig(evidenceCount = 1)
val implicitConfig = ImplicitConnectionEvidenceConfig(evidenceCount = 2)

// Access config properties
println(s"Resource path: ${userFactsConfig.resourcePath}")
// "src/main/resources/questions/evidence/user_evidence/3_evidence"
```

--------------------------------

### Test Use Case Generation for Diversity

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Generate and inspect at least 20 use cases to ensure sufficient diversity in the test scenarios.

```scala
// In QuickGeneratorTest or similar utility
val generator = new MyEvidenceGenerator(1)
val person = getTestPerson()
// Generate 20 use cases to properly assess diversity
val useCases = generator.generateUseCases(person).take(20)

// Inspect ALL generated use cases for diversity
useCases.zipWithIndex.foreach { case (uc, i) =>
  println(s"\n${i+1}. Category: ${uc.category}")
  println(s"   Scenario: ${uc.scenario_description}")
}
```

--------------------------------

### Generate Conversations from Core

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Generates conversations based on the evidence core and inspects the message count.

```scala
val conversations = generator.generateConversationsFromCore(person, useCase, core)
conversations.foreach { conv =>
  println(s"Conversation has ${conv.messages.length} messages")
  // Inspect first few messages
}
```

--------------------------------

### Define a New Evidence Generator Class

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Extend the EvidenceGenerator class to create a custom generator, ensuring the evidence type name is defined for logging purposes.

```scala
package com.salesforce.crmmembench.questions.evidence.generators

class MyEvidenceGenerator(evidenceCount: Int) 
  extends EvidenceGenerator(MyEvidenceConfig(evidenceCount)) {
  
  // Define the evidence type name for logging
  def getEvidenceTypeName: String = "my_evidence"
  
  // Define prompt parts for each phase...
}
```

--------------------------------

### Configure Verification Checks in Scala

Source: https://github.com/salesforceairesearch/convomem/blob/main/src/main/scala/com/salesforce/crmmembench/questions/evidence/generators/README.md

Overrides the verification check method to define custom combinations of evidence-based verification strategies.

```scala
override protected def getVerificationChecks(): List[VerificationCheck] = {
  // Default: verify with evidence + verify without evidence
  super.getVerificationChecks()
  
  // No verification (e.g., temporal evidence, conversations)
  List.empty
  
  // Add partial evidence check for multi-fact scenarios
  super.getVerificationChecks() ++ List(new VerifyWithPartialEvidence())
  
  // Custom combination
  List(
    new VerifyWithEvidence(requiredPasses = 5),
    new VerifyWithoutEvidence(),
    new VerifyWithPartialEvidence()
  )
}
```

--------------------------------

### Configure ConvoMem Evaluation Parameters

Source: https://github.com/salesforceairesearch/convomem/blob/main/README.md

Modify the evaluation settings such as context sizes, thread count, and caching behavior in the configuration file.

```scala
object Config {
  val DEBUG = false  // Verbose logging
  val CONTEXT_SIZES = List(2, 4, 6, 10, 20, 30, 50, 70, 100, 150, 200, 300)
  val EVALUATION_THREADS = 40
  val USE_CACHED_TEST_CASES = true  // Use pre-generated test cases
}
```

--------------------------------

### Cite the ConvoMem Benchmark

Source: https://github.com/salesforceairesearch/convomem/blob/main/README.md

Use this BibTeX entry to cite the ConvoMem research paper in academic publications.

```bibtex
@article{convomem2025,
  title={ConvoMem: A Comprehensive Benchmark for Conversational Memory},
  author={[Authors]},
  year={2025}
}
```

--------------------------------

### Load Irrelevant Conversations with ConversationLoader

Source: https://context7.com/salesforceairesearch/convomem/llms.txt

Retrieves filler conversations for testing purposes, optionally limiting the total count.

```scala
import com.salesforce.crmmembench.evaluation.ConversationLoader
import com.salesforce.crmmembench.questions.evidence.Conversation

// Load all irrelevant conversations grouped by person
val conversationsByPerson: Map[String, List[Conversation]] =
  ConversationLoader.loadIrrelevantConversations()

println(s"Loaded conversations from ${conversationsByPerson.size} personas")

// Total conversation count
val totalConversations = conversationsByPerson.values.map(_.size).sum
println(s"Total irrelevant conversations: $totalConversations")

// Each conversation has proper metadata
conversationsByPerson.values.flatten.headOption.foreach { conv =>
  println(s"Conversation ID: ${conv.id.getOrElse("N/A")}")
  println(s"Contains evidence: ${conv.containsEvidence.getOrElse(false)}")  // Always false
  println(s"Message count: ${conv.messages.size}")
  println(s"Model: ${conv.model_name.getOrElse("N/A")}")

  // Sample messages
  conv.messages.take(2).foreach { msg =>
    println(s"  ${msg.speaker}: ${msg.text.take(100)}...")
  }
}

// Load a limited number of conversations
val limitedConversations = ConversationLoader.loadIrrelevantConversations(count = 1000)
val limitedTotal = limitedConversations.values.map(_.size).sum
println(s"Limited load: $limitedTotal conversations")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.