### Setup WorkArena Environment

Source: https://github.com/servicenow/workarena/wiki/WorkArena-‐-Human-Evaluation-Study

Installs Mamba, creates and activates a Python 3.12 environment, and installs necessary packages including browsergym-core and workarena-core. Ensure you replace 'PASSWORD_HERE' with the actual password provided.

```bash
export SNOW_INSTANCE_URL="https://dev253956.service-now.com/" && \
export SNOW_INSTANCE_UNAME="admin" && \
export SNOW_INSTANCE_PWD='PASSWORD_HERE' && \
mkdir workarena-human-eval && cd workarena-human-eval && \
if [[ $(uname -m) == 'arm64' ]]; then \
  curl -Ls https://micro.mamba.pm/api/micromamba/osx-arm64/latest | tar -xvj bin/micromamba; \
else \
  curl -Ls https://micro.mamba.pm/api/micromamba/osx-64/latest | tar -xvj bin/micromamba; \
fi && \
export MAMBA_ROOT_PREFIX=$(pwd)/micromamba && \
eval "$(./bin/micromamba shell hook -s posix)" && \
micromamba env create -n "human-eval" python=3.12 --channel conda-forge -y && \
micromamba activate human-eval && \
pip install --upgrade browsergym-core==0.3.4 && \
pip install --upgrade https://alexdrouin.com/workarena.zip && \
playwright install && \
export PATH=$PATH:micromamba/envs/human-eval/bin/ && \
echo "Installation complete."
```

--------------------------------

### Verify WorkArena Installation

Source: https://github.com/servicenow/workarena/wiki/WorkArena-‐-Human-Evaluation-Study

Runs the WorkArena evaluation tool to confirm that the environment setup was successful. Kill the process with CTRL-C if it starts.

```bash
workarena-human-eval --email YOUR_EMAIL --log test.log --curriculum random
```

--------------------------------

### Install OpenAI Package

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Installs the necessary OpenAI Python package. Run this command before importing the library.

```python
!pip install openai
```

--------------------------------

### Install BrowserGym WorkArena Package

Source: https://github.com/servicenow/workarena/blob/main/README.md

Install the WorkArena package within the BrowserGym environment using pip. This is a prerequisite for using WorkArena.

```bash
pip install browsergym-workarena
```

--------------------------------

### Start WorkArena Human Evaluation

Source: https://github.com/servicenow/workarena/wiki/WorkArena-‐-Human-Evaluation-Study

Initiates the human evaluation task using the specified email, log file, and curriculum file. Ensure 'YOUR_CURRICULUM_FILE' points to the correct file provided by the session lead.

```bash
workarena-human-eval --email YOUR_EMAIL --log ~/LASTNAME.log --curriculum YOUR_CURRICULUM_FILE
```

--------------------------------

### Install Playwright for Browser Automation

Source: https://github.com/servicenow/workarena/blob/main/README.md

Install Playwright, a tool for browser automation, which is required for WorkArena's interactive components. This command ensures all necessary browser binaries are downloaded.

```bash
playwright install
```

--------------------------------

### Get Most Frequent Items in a List

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Calculates and returns the most frequent item(s) from a given list. It uses the `Counter` class for efficient frequency counting.

```python
def get_most_frequent_items(my_list):
    # Count the frequency of each item in the list
    frequency = Counter(my_list)

    # Find the maximum frequency
    max_frequency = max(frequency.values())

    # Get all items with the maximum frequency
    most_frequent = [
        item for item, count in frequency.items() if count == max_frequency
    ]

    return most_frequent
```

--------------------------------

### Initialize OpenAI Client and Chat Function

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Initializes the OpenAI client with an API key from environment variables and defines a helper function to interact with the chat completions API.

```python
import json
import openai
import os


client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])


def chat(messages, model="gpt-4-1106-preview"):
    return (
        client.chat.completions.create(model=model, messages=messages)
        .choices[0]
        .message
    )
```

--------------------------------

### Run WorkArena-L1 Demo with BrowserGym

Source: https://github.com/servicenow/workarena/blob/main/README.md

Executes WorkArena-L1 tasks using BrowserGym. This script shuffles atomic tasks and uses cheat functions to automatically solve each one. For agent evaluation, replace cheat calls with env.step().

```python
import random

from browsergym.core.env import BrowserEnv
from browsergym.workarena import ATOMIC_TASKS
from time import sleep


random.shuffle(ATOMIC_TASKS)
for task in ATOMIC_TASKS:
    print("Task:", task)

    # Instantiate a new environment
    env = BrowserEnv(task_entrypoint=task,
                    headless=False)
    env.reset()

    # Cheat functions use Playwright to automatically solve the task
    env.chat.add_message(role="assistant", msg="On it. Please wait...")
    cheat_messages = []
    env.task.cheat(env.page, cheat_messages)

    # Send cheat messages to chat
    for cheat_msg in cheat_messages:
        env.chat.add_message(role=cheat_msg["role"], msg=cheat_msg["message"])

    # Post solution to chat
    env.chat.add_message(role="assistant", msg="I'm done!")

    # Validate the solution
    reward, stop, message, info = env.task.validate(env.page, cheat_messages)
    if reward == 1:
        env.chat.add_message(role="user", msg="Yes, that works. Thanks!")
    else:
        env.chat.add_message(role="user", msg=f"No, that doesn't work. {info.get('message', '')}")

    sleep(3)
    env.close()
```

--------------------------------

### Run WorkArena-L2 Demo with BrowserGym

Source: https://github.com/servicenow/workarena/blob/main/README.md

Executes WorkArena-L2 tasks using BrowserGym. This script samples L2 tasks and uses cheat functions to solve each subtask. To sample L3 tasks, modify the filter on line 6. For agent evaluation, replace cheat calls with env.step().

```python
import random

from browsergym.core.env import BrowserEnv
from browsergym.workarena import get_all_tasks_agents
 
AGENT_L2_SAMPLED_SET = get_all_tasks_agents(filter="l2")
 
AGENT_L2_SAMPLED_TASKS, AGENT_L2_SEEDS = [sampled_set[0] for sampled_set in AGENT_L2_SAMPLED_SET], [
    sampled_set[1] for sampled_set in AGENT_L2_SAMPLED_SET
]
from time import sleep

for (task, seed) in zip(AGENT_L2_SAMPLED_TASKS, AGENT_L2_SEEDS):
    print("Task:", task)

    # Instantiate a new environment
    env = BrowserEnv(task_entrypoint=task,
                    headless=False)
    env.reset()

    # Cheat functions use Playwright to automatically solve the task
    env.chat.add_message(role="assistant", msg="On it. Please wait...")
    
    for i in range(len(env.task)):
        sleep(1)
        env.task.cheat(page=env.page, chat_messages=env.chat.messages, subtask_idx=i)
        sleep(1)
        reward, done, message, info = env.task.validate(page=env.page, chat_messages=env.chat.messages)
   
    if reward == 1:
        env.chat.add_message(role="user", msg="Yes, that works. Thanks!")
    else:
        env.chat.add_message(role="user", msg=f"No, that doesn't work. {info.get('message', '')}")

    sleep(3)
    env.close()
```

--------------------------------

### Cite WorkArena++ Project

Source: https://github.com/servicenow/workarena/blob/main/README.md

Use this BibTeX entry to cite the WorkArena++ paper, which focuses on compositional planning and reasoning. Include all provided fields for accurate citation.

```bibtex
@misc{boisvert2024workarenacompositionalplanningreasoningbased,
      title={WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks},
      author={Léo Boisvert and Megh Thakkar and Maxime Gasse and Massimo Caccia and Thibault Le Sellier De Chezelles and Quentin Cappart and Nicolas Chapados and Alexandre Lacoste and Alexandre Drouin},
      year={2024},
      eprint={2407.05291},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2407.05291}
}
```

--------------------------------

### Print Alternative Answers

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Prints the generated alternative answers for each item in the knowledge base. Assumes the 'kb' variable has been updated with 'alternative_answers'.

```python
for kb_i in kb:
    print(kb_i["alternative_answers"])
```

--------------------------------

### Print Generated Questions

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Iterates through the processed knowledge base entries and prints the generated questions for each article.

```python
for i in range(len(kb)):
    print(f"Article {i}:", kb[i]["questions"])
```

--------------------------------

### Load and Clear Knowledge Base Questions

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Loads a JSON knowledge base and clears all existing questions and alternative answers from each article. This is useful for regenerating questions.

```python
import concurrent.futures
import re

from collections import Counter
from tqdm.notebook import tqdm


# Reload the generated KB
kb = json.load(open("knowledge_base.json", "r"))
print("Loaded", len(kb), "articles")


# Clear all questions
for kb_i in kb:
    if "questions" in kb_i:
        kb_i["questions"] = []
        kb_i["alternative_answers"] = []
```

--------------------------------

### Cite WorkArena Project

Source: https://github.com/servicenow/workarena/blob/main/README.md

Use this BibTeX entry to cite the original WorkArena paper. Ensure all fields are included for proper academic referencing.

```bibtex
@misc{workarena2024,
      title={WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?},
      author={Alexandre Drouin and Maxime Gasse and Massimo Caccia and Issam H. Laradji and Manuel Del Verme and Tom Marty and Léo Boisvert and Megh Thakkar and Quentin Cappart and David Vazquez and Nicolas Chapados and Alexandre Lacoste},
      year={2024},
      eprint={2403.07718},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```

--------------------------------

### Save Knowledge Base to JSON

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Saves the entire knowledge base, including the newly generated alternative answers, to a JSON file named 'knowledge_base.json'.

```python
json.dump(kb, open("knowledge_base.json", "w"))
```

--------------------------------

### Parallel Question Generation

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Uses a ThreadPoolExecutor to concurrently generate questions for multiple knowledge base entries. It submits tasks to generate questions for each article and collects the results.

```python
# Assuming kb is a list of dictionaries
with concurrent.futures.ThreadPoolExecutor() as executor:
    # Prepare the futures
    futures = [
        executor.submit(
            lambda x: {
                **x,
                "questions": generate_question(x["article"], x["item"], x["value"]),
            },
            kb_i,
        )
        for kb_i in kb
    ]

    # Use tqdm to create a progress bar
    kb = [
        x.result()
        for x in tqdm(concurrent.futures.as_completed(futures), total=len(futures))
    ]
```

--------------------------------

### Generate Knowledge Base Article

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Generates a knowledge-base article based on a specific fact, embedding it within fabricated related information. Ensures the fact is included verbatim and the output is in HTML format. Includes retry logic for API calls.

```python
def generate_article(item, value, all_facts, n_retries=5):
    prompt = f"""
<task>
You are in charge of writing knowledge-base articles to document important company and workplace information.
Your articles will be used as reference by the employees of the company.
</task>

Here is a list of facts in the knowledge base:
<all_facts>
{all_facts}
</all_facts>

You need to write an article about this specific fact:
<article_fact>
The {item} is {value}.
</article_fact>

<instructions>
* Generate a knowledge-base article that contains the <article_fact> as-is. Do not modify or add to its text.
* Hide the fact inside a bunch of other related information, but do not source the related information from <all_facts>. Make stuff up.
* Make sure that nothing you write contradicts <all_facts>
* You must use HTML format. Generate only the <body> tag.
* Don't include information about the knowledge base itself or headers like \"welcome to our knowledge base\".
</instructions>

    """
    messages = [
        {
            "role": "system",
            "content": "You are a ServiceNow system administrator in charge of writing knowledge base articles to help employees in your organization.",
        },
        {"role": "user", "content": prompt},
    ]

    print("... attempting to produce article")
    for retry in range(n_retries):
        try:
            print(f"... try {retry}")
            messages.append({"role": "assistant", "content": chat(messages).content})
            article = (
                messages[-1]["content"]
                .replace("<body>", "")
                .replace("</body>", "")
                .replace("```HTML", "")
                .replace("```", "")
                .replace("<strong>", "")
                .replace("</strong>", "")
                .strip()
            )

            # Validate that the fact is included without modification
            assert (
                f"the {item} is {value}".lower() in article.lower()
            ), f'Error: Could not find the string "The {item} is {value}" in this article.'
            print("... valid article found.")
            break
        except AssertionError as e:
            messages.append(
                {"role": "user", "content": str(e) + "\nDon't apologize and try again."}
            )

    if messages[-1]["role"] == "user":
        raise RuntimeError(
            f"Failed to produce a valid article after {n_retries} retries."
        )

    return article
```

--------------------------------

### Validate and Refine Questions

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Iterates through questions, checks if they are answerable from an article, and collects feedback for ambiguous questions. It then formats feedback for a language model to improve clarity.

```python
            # Validate questions
            bad_questions = []
            bad_answers = []
            for q in questions:
                print("... testing:", q, end=" ")
                success, answers = is_question_answerable(article, q, value)
                if not success:
                    bad_questions.append(q)
                    bad_answers.append(answers)
                    print("FAIL")
                    print("... preemptively stopping to give feedback")
                    break
                else:
                    good_questions.add(q)
                    print("PASS")

            # Give feedback and retry
            if len(bad_questions) > 0:
                feedback = ""
                for q, a in zip(bad_questions, bad_answers):
                    feedback += "<ambiguous_question>\n"
                    feedback += "    <statement>\n"
                    feedback += "        " + q + "\n"
                    feedback += "    </statement>\n"
                    feedback += "    <incorrect_answers>\n"
                    feedback += "\n".join("        " + x for x in a) + "\n"
                    feedback += "    </incorrect_answers>\n"
                    feedback += "</ambiguous_question>\n"
                print(feedback)

                # print(f"... {len(bad_questions)} questions are ambiguous, fixing them.")
                bad_questions = "\n".join(bad_questions)
                messages.append(
                    {
                        "role": "user",
                        "content": f"""
                The following questions are too ambiguous. I gave them to many company employees,
along with the article and they were not able to answer with exactly \"{value}\".
Please improve their clarity, especially the formatting instructions.

                {feedback}

                Try again. Do not apologize.""",
                    }
                )

            print(f"... gathered {len(good_questions)} good questions")

        except AssertionError as e:
            print(f"... Error:", e)
            messages.append({"role": "user", "content": f"Error: {e}"})

    print("... we have enough good questions. stopping.")
    return list(good_questions)[:10]
```

--------------------------------

### Update Knowledge Base with Alternative Answers

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Iterates through a knowledge base (kb) and adds a new field 'alternative_answers' to each entry by calling the `alternative_answers` function. Requires the 'kb' variable to be defined and populated.

```python
for kb_i in tqdm(kb, total=len(kb)):
    kb_i["alternative_answers"] = alternative_answers(
        kb_i["article"], kb_i["item"], kb_i["value"], kb_i["questions"]
    )
```

--------------------------------

### Clean and Save Knowledge Base to JSON

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Cleans the generated articles by removing specific HTML tags and markdown formatting, then saves the processed facts to a JSON file named 'knowledge_base.json'. This step is crucial for preparing the data for further use or display.

```python
for f in facts:
    f["article"] = (
        f["article"]
        .replace("```HTML", "")
        .replace("```", "")
        .replace("<strong>", "")
        .replace("</strong>", "")
        .strip()
    )

json.dump(facts, open("knowledge_base.json", "w"))
```

--------------------------------

### Define Facts for Article Generation

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

A list of dictionaries, where each dictionary contains an 'item' and its corresponding 'value', used as input for generating knowledge base articles.

```python
facts = [
    {"item": "password to conference room A-561", "value": "roo918k"},
    {"item": "address of office #456", "value": "42, Pizza street, New York, USA"},
    {"item": "number of employees in department X", "value": "75"},
    {"item": "CEO's name", "value": "Alex Johnson"},
    {"item": "year of company establishment", "value": "1998"},
    {"item": "Wi-Fi network name in office #456", "value": "Office456_WiFi"},
    {"item": "Wi-Fi password for office #456", "value": "456SecureNet!"},
    {"item": "average annual revenue", "value": "$50 million"},
    {"item": "number of branches worldwide", "value": "18"},
    {"item": "name of the head of HR department", "value": "Samantha Green"},
    {"item": "brand of coffee machine in kitchen #3", "value": "Delonghi Magnifica"},
    {"item": "company's stock ticker symbol", "value": "COMPX"},
    {"item": "company's main product", "value": "Advanced Analytics Software"},
    {"item": "color of the carpet in conference room A-561", "value": "Navy Blue"},
    {"item": "annual budget for marketing department", "value": "$2 million"},
    {"item": "capacity of conference room B-762", "value": "30 people"},
    {"item": "number of floors in the main office building", "value": "10"},
    {"item": "type of plants in the lobby of office #456", "value": "Ficus Lyrata"},
    {
        "item": "catering service provider for office events",
        "value": "Gourmet Caterers",
    },
    {"item": "software used for payroll management", "value": "QuickBooks Payroll"},
    {"item": "number of parking spaces in office #456 parking lot", "value": "150"},
    {"item": "company's first product", "value": "Data Analysis Toolkit"},
    {"item": "brand of computers used in IT department", "value": "Lenovo ThinkPad"},
    {"item": "annual CSR budget", "value": "$500,000"},
]
```

--------------------------------

### Generate Question Variants using LLM

Source: https://github.com/servicenow/workarena/blob/main/scripts/generate_knowledge_base.ipynb

Generates multiple rephrased questions for a given fact and article using a language model. It includes strict validation checks to ensure questions are unambiguous, do not reveal the answer, and follow specific formatting instructions.

```python
def generate_question(article, item, value, n_questions=10, n_retries=5):
    prompt = f"""
    Here is an article taken from a company knowledge base.

    <article>
    {article}
    </article>

    This article contains the following fact:
    <fact>
    The {item} is {value}.
    </fact>

    Here is a question about this fact to which the answer is \"{value}\":
    <question>
    What is the {item}?
    </question>

    Produce {n_questions} rephrasings of this question.

    <instructions>
    * Make sure that your questions are precise and unambiguous.
    * It must be clear that the questions are asking about \"{item}\" and
      their answer must still be exactly \"{value}\".
    * Make sure that you provide clear and specific instructions on the expected
      format for the answer (e.g., Day, Month, Year).
    * You cannot, in any circumstances, reveal information from the answer in the question or instructions.
    * Make sure they are questions that end with a question mark.
    * Make sure that your questions do not mention the article (e.g., \"in the article\").
    * Answer with one per line and do not number them.
    </instructions>

    <example>
    Suppose the question is \"On which day was the company founded?\" and the answer is \"January 26, 2024",
    then a good rephrasing would be \"When was the company founded? Answer with Month Day, Year\".
    </example>

    <example>
    Suppose the question is \"What kind of fridge do we have in the cafeteria?\" and the answer is \"LG X456",
    then a good rephrasing would be \"Which type of fridge is in the cafeteria? Answer with Brand followed by Model name\".
    </example>

    <example>
    Suppose the question is \"Where is the company headquarter located?\" and the answer is \"123, Banana Street, Montreal, Canada",
    then a good rephrasing would be \"What's the address of the company headquarter? Answer with Number, Street, City, Country\".
    </example>

    <example>
    Suppose the question is \"What were the total sales of the company in 2023?\" and the answer is \"150B$\",
    then a good rephrasing would be \"What do the company sales for 2023 sum up to? Answer with NumberB$, where B is billions\".
    </example>
    """
    good_questions = set([])
    messages = [{"role": "user", "content": prompt}]
    while len(good_questions) < n_questions:
        try:
            # Generate questions
            questions = chat(messages).content

            # Parse output
            # ... check the article is not mentioned
            assert (
                "article" not in questions.lower()
            ), "You are not allowed to mention the article in your rephrasing of the questions."
            # ... check the number of actual questions included
            assert (
                questions.count("?") == n_questions
            ), f"I couldn't find {n_questions} question marks in the output. Make sure all questions end with ?"
            # ... heuristic to detect numbered questions
            assert not all(
                f"{x}." in questions for x in range(4)
            ), "The questions appear to be numbered, but they should not."
            questions = [q.strip() for q in questions.split("\n") if q != ""]
            # ... check one question per line
            assert (
                len(questions) == n_questions
            ), f"Your answer is not {n_questions} lines long. Make sure it contains {n_questions} questions and one per line."
            # ... check each question has instructions
            assert all(
                len(q.split("?")) > 1 and len("".join(q.split("?")[1:]).strip()) > 5
                for q in questions
            ), f'Make sure you provide clear formatting instructions for each question (e.g., "Question? Answer with").'
            # ... check that answer is not mentioned
            for q in questions:
                error = "Do not include the answer in the questions/instructions!"
                # ... exact value is not in the question
                assert value.lower() not in q.lower(), error

```

--------------------------------

### Validate Questions and Provide Feedback

Source: https://github.com/servicenow/workarena/blob/main/scripts/generate_knowledge_base.ipynb

Iterates through questions, validates their answerability against an article and expected value, and collects feedback for ambiguous questions. Stops processing if a question is unanswerable.

```python
            # Validate questions
            bad_questions = []
            bad_answers = []
            for q in questions:
                print("... testing:", q, end=" ")
                success, answers = is_question_answerable(article, q, value)
                if not success:
                    bad_questions.append(q)
                    bad_answers.append(answers)
                    print("FAIL")
                    print("... preemptively stopping to give feedback")
                    break
                else:
                    good_questions.add(q)
                    print("PASS")

            # Give feedback and retry
            if len(bad_questions) > 0:
                feedback = ""
                for q, a in zip(bad_questions, bad_answers):
                    feedback += "<ambiguous_question>\n"
                    feedback += "    <statement>\n"
                    feedback += "        " + q + "\n"
                    feedback += "    </statement>\n"
                    feedback += "    <incorrect_answers>\n"
                    feedback += "\n".join("        " + x for x in a) + "\n"
                    feedback += "</ambiguous_question>\n"
                print(feedback)

                # print(f"... {len(bad_questions)} questions are ambiguous, fixing them.")
                bad_questions = "\n".join(bad_questions)
                messages.append(
                    {
                        "role": "user",
                        "content": f"""
                The following questions are too ambiguous. I gave them to many company employees,
along with the article and they were not able to answer with exactly \"{value}\".
                Please improve their clarity, especially the formatting instructions.

                {feedback}

                Try again. Do not apologize.""",
                    }
                )

            print(f"... gathered {len(good_questions)} good questions")

        except AssertionError as e:
            print(f"... Error:", e)
            messages.append({"role": "user", "content": f"Error: {e}"})

    print("... we have enough good questions. stopping.")
    return list(good_questions)[:10]
```

--------------------------------

### Generate Articles for Facts

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Iterates through a list of facts, generates an article for each fact using the `generate_article` function, and prints the item, value, and generated article. Ensure the `generate_article` function is defined and accessible.

```python
print("Number of facts:", len(facts))

for i, f in enumerate(facts):
    f["article"] = generate_article(f["item"], f["value"], all_facts=facts)
    print("... Article", i + 1, f["item"], f["value"])
    print(f["article"])
    print("\n" * 2)
```

--------------------------------

### Generate Alternative Answers Function

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Defines a function to generate alternative phrasings for an answer using a chat model. It includes validation to ensure the output meets specific criteria, such as not being numbered and having the correct number of unique answers.

```python
def alternative_answers(article, item, value, questions, n_answers=10):
    questions = "\n".join(questions)

    prompt = f"""
    Here is a set of questions:
    <questions>
    {questions}
    </questions>

    The exact answer to all of these questions is:
    <answer>
    {value}
    </answer>

    Give me {n_answers} other ways to spell out the answer.
    Don't add context words around it or anything, just reformulate it.

    <example>
    Initial value: 5.5/10
    Reformulated: 5.5 out of 10
    Reformulated: 55%
    Reformulated: fifty-five percent
    </example>

    <example>
    Initial value: Tesla Model X
    Reformulated: Model X by Tesla
    Reformulated: Tesla's Model X
    </example>

    <example>
    Initial value: 150$
    Reformulated: one hundred fifty dollars
    Reformulated: 150.00$
    Reformulated: $150
    </example>

    Answer with one per line. Don't number them.

    "
    messages = [{"role": "user", "content": prompt}]

    while True:
        try:
            answers = chat(messages).content

            # Validation
            # ... heuristic to detect numbered questions
            assert not all(
                f"{x}." in answers for x in range(4)
            ), "The answers appear to be numbered, but they should not."
            answers = [a.strip() for a in answers.split("\n") if a != ""]
            # ... check one question per line
            assert (
                len(answers) == n_answers
            ), f"Your response is not {answers} lines long. Make sure it contains {answers} answers and one per line."
            assert (
                len(set(answers)) == n_answers
            ), f"You provided duplicate values. There were only {len(set(answers))} unique values."
            break
        except AssertionError as e:
            print(f"... Error:", e)
            messages.append({"role": "user", "content": f"Error: {e}"})

    return answers
```

--------------------------------

### Generate Rephrased Questions for a Fact

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Generates multiple rephrased questions for a given fact within an article. It uses a language model and includes strict validation checks to ensure questions are unambiguous, do not reveal the answer, and follow specific formatting instructions.

```python
def generate_question(article, item, value, n_questions=10, n_retries=5):
    prompt = f"""
    Here is an article taken from a company knowledge base.

    <article>
    {article}
    </article>

    This article contains the following fact:
    <fact>
    The {item} is {value}.
    </fact>

    Here is a question about this fact to which the answer is \"{value}\":
    <question>
    What is the {item}?
    </question>

    Produce {n_questions} rephrasings of this question.

    <instructions>
    * Make sure that your questions are precise and unambiguous.
    * It must be clear that the questions are asking about \"{item}\" and
      their answer must still be exactly \"{value}\".
    * Make sure that you provide clear and specific instructions on the expected
      format for the answer (e.g., Day, Month, Year).
    * You cannot, in any circumstances, reveal information from the answer in the question or instructions.
    * Make sure they are questions that end with a question mark.
    * Make sure that your questions do not mention the article (e.g., \"in the article\").
    * Answer with one per line and do not number them.
    </instructions>

    <example>
    Suppose the question is \"On which day was the company founded?\" and the answer is \"January 26, 2024\",
    then a good rephrasing would be \"When was the company founded? Answer with Month Day, Year\".
    </example>

    <example>
    Suppose the question is \"What kind of fridge do we have in the cafeteria?\" and the answer is \"LG X456\",
    then a good rephrasing would be \"Which type of fridge is in the cafeteria? Answer with Brand followed by Model name\".
    </example>

    <example>
    Suppose the question is \"Where is the company headquarter located?\" and the answer is \"123, Banana Street, Montreal, Canada\",
    then a good rephrasing would be \"What's the address of the company headquarter? Answer with Number, Street, City, Country\".
    </example>

    <example>
    Suppose the question is \"What were the total sales of the company in 2023?\" and the answer is \"150B$\",
    then a good rephrasing would be \"What do the company sales for 2023 sum up to? Answer with NumberB$, where B is billions\".
    </example>
    """
    good_questions = set([])
    messages = [{"role": "user", "content": prompt}]
    while len(good_questions) < n_questions:
        try:
            # Generate questions
            questions = chat(messages).content

            # Parse output
            # ... check the article is not mentioned
            assert (
                "article" not in questions.lower()
            ), "You are not allowed to mention the article in your rephrasing of the questions."
            # ... check the number of actual questions included
            assert (
                questions.count("?") == n_questions
            ), f"I couldn't find {n_questions} question marks in the output. Make sure all questions end with ?"
            # ... heuristic to detect numbered questions
            assert not all(
                f"{x}." in questions for x in range(4)
            ), "The questions appear to be numbered, but they should not."
            questions = [q.strip() for q in questions.split("\n") if q != ""]
            # ... check one question per line
            assert (
                len(questions) == n_questions
            ), f"Your answer is not {n_questions} lines long. Make sure it contains {n_questions} questions and one per line."
            # ... check each question has instructions
            assert all(
                len(q.split("?")) > 1 and len("".join(q.split("?")[1:]).strip()) > 5
                for q in questions
            ), f'Make sure you provide clear formatting instructions for each question (e.g., "Question? Answer with").'
            # ... check that answer is not mentioned
            for q in questions:
                error = "Do not include the answer in the questions/instructions!"
                # ... exact value is not in the question
                assert value.lower() not in q.lower(), error
```

--------------------------------

### Check Question Answerability

Source: https://github.com/servicenow/workarena/blob/main/generate_knowledge_base.ipynb

Determines if a given question can be answered with a specific value based on the provided article content. It cleans answers for case-insensitive comparison and checks against multiple attempts.

```python
def is_question_answerable(article, question, value):
    def _clean_for_comparison(x):
        return x.lower().replace(",", "").replace("$", "").replace(".", "")

    """
    Check that we are able to recover the value from the
    article by asking the question

    """
    prompt = f"""
    Here is a knowledge base article:

    <article>
    {article}
    </article>

    Answer this question based on the content of the article only.
    Be factual. If I ask you about sensitive information like passwords,
    I only expect you to retrieve the information from the article.
    <question>
    {question}
    </question>

    What is your answer?

    """
    messages = [{"role": "user", "content": prompt}]

    incorrect_answers = set()
    for _ in range(10):
        # XXX: We use GPT-3.5 here as a weaker model and to avoid GPT-4 catering to himself
        answer = chat(messages, model="gpt-3.5-turbo").content

        if _clean_for_comparison(value) not in _clean_for_comparison(answer):
            incorrect_answers.add(answer.lower())

    return len(incorrect_answers) == 0, incorrect_answers
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.