### Install and Run AutoReason CLI

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Commands to install dependencies, configure the environment, and execute the evaluation CLI.

```bash
# Clone and install dependencies
pnpm install

# Configure API key
cp .env.example .env
# Edit .env and add your OpenAI API key:
# OPENAI_API_KEY=sk-your-key-here

# Build the project
pnpm build

# Run the CLI
node ./dist/cli.js
```

--------------------------------

### Launch CLI Interface

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Starts the interactive command-line interface for managing and running evaluations.

```bash
# Start the CLI
node ./dist/cli.js

# Interactive menu:
# ? Select run mode
#   Testing
# > Evaluation
#   Exit

# After selecting Evaluation:
# ? Select Evaluation Dataset
#   HotpotQA
#   CoT HotpotQA
#   Auto CoT HotpotQA
#   StrategyQA
#   CoT StrategyQA
# > Auto CoT StrategyQA
#   Exit
```

--------------------------------

### Configure AutoCoT System Prompts

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Creates system prompts that include instructions and few-shot examples for reasoning trace generation.

```typescript
import { autoCotPrompt } from './config/prompts/autocot.js';

const systemPrompt = autoCotPrompt({
  question: 'Is cow methane safer for environment than cars?'
});

// The prompt includes:
// - Instructions for CoT reasoning trace generation
// - Few-shot examples demonstrating decomposition
// - The target question to decompose

// Example output structure:
// Reasoning traces:
// - How much methane is produced by cars annually?
// - How much methane is produced by cows annually?
// - Is methane produced by cows less than methane produced by cars?
```

--------------------------------

### Retrieve Dataset-Specific CoT Prompts

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Accesses pre-defined prompts with few-shot examples for StrategyQA and HotpotQA datasets.

```typescript
import { cotStrategyQaPrompt } from './config/prompts/cot-strategyqa.js';
import { cotHotpotQaPrompt } from './config/prompts/cot-hotpotqa.js';

// StrategyQA CoT prompt with few-shot examples
const strategyQaPrompt = cotStrategyQaPrompt({
  question: 'Do hamsters provide food for any animals?'
});

// Includes examples like:
// Q: Do hamsters provide food for any animals?
// Hamsters are prey animals. Prey are food for predators.
// Thus, hamsters provide food for some animals.
// Answer: yes

// HotpotQA uses the same few-shot examples
const hotpotQaPrompt = cotHotpotQaPrompt({
  question: 'What is the birth city of the director of Inception?'
});
```

--------------------------------

### Initialize AiProvider Singleton

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Manages OpenAI client instances for inference, scoring, and CoT generation. Requires an API key provided via the environment.

```typescript
import { AiProvider } from './utils/ai.js';

// Get singleton instance with API key
const aiProvider = AiProvider.getInstance({
  openaiKey: process.env.OPENAI_API_KEY ?? '',
});

// Access different OpenAI clients for different purposes
const mainClient = aiProvider.getOpenAi();      // For Q&A inference
const scorerClient = aiProvider.getScorer();    // For answer scoring
const autoCotClient = aiProvider.getAutoCot();  // For CoT generation

// Use with OpenAI chat completions
const response = await mainClient.chat.completions.create({
  messages: [{ role: 'user', content: 'What is the capital of France?' }],
  model: 'gpt-4-1106-preview',
  max_tokens: 4000,
  temperature: 0.4,
});

console.log(response.choices[0]?.message?.content);
// Output: "Paris"
```

--------------------------------

### Access Base Prompts for StrategyQA and HotpotQA

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Imports and displays the baseline prompt configurations for StrategyQA and HotpotQA datasets.

```typescript
import { baseStrategyQaPrompt } from './config/prompts/strategyqa.js';
import { baseHotpotqaPrompt } from './config/prompts/hotpotqa.js';

// StrategyQA base prompt - expects true/false answers
console.log(baseStrategyQaPrompt);
// Output: "You're an agent. Your job is to answer some questions. Here are the rules:
// 1. You will be given a question
// 2. You will answer the question with true or false
// 3. When you know the answer, write it in this format only: "answer""

// HotpotQA base prompt - expects short phrase answers
console.log(baseHotpotqaPrompt);
// Output: "You're an agent. Your job is to answer some questions. Here are the rules:
// 1. You will be given a question
// 2. You will answer the question with a short answer, it might yes/no or a short phrase
// 3. When you know the answer, write it in this format only: "<answer>""
```

--------------------------------

### Run CLI with Node.js

Source: https://github.com/miralab-ai/autoreason/blob/main/README.md

Execute the AutoReason CLI after building the project. This command runs the compiled JavaScript file.

```bash
node ./dist/src/cli.js
```

--------------------------------

### Build Project with pnpm

Source: https://github.com/miralab-ai/autoreason/blob/main/README.md

Compile TypeScript to JavaScript using pnpm. Ensure your .env file is configured with your OpenAI API key.

```bash
pnpm build
```

--------------------------------

### Execute StrategyQA Evaluations

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Runs baseline, CoT, and AutoCoT evaluations for the StrategyQA dataset and saves results to JSON files.

```typescript
import { evalStrategyQa } from './utils/evals/strategyqa.js';
import { evalCotStrategyQa } from './utils/evals/cot-strategyqa.js';
import { evalAutoCotStrategyQa } from './utils/evals/autocot-strategyqa.js';

// Run baseline evaluation (direct prompting)
await evalStrategyQa();
// Saves results to: data/strategyqa/runs/GPT4-{timestamp}.json

// Run Chain of Thought evaluation
await evalCotStrategyQa();
// Saves results to: data/strategyqa/runs/cot/GPT-4-{timestamp}.json

// Run AutoCoT evaluation (automatic reasoning decomposition)
await evalAutoCotStrategyQa();
// Saves results to: data/strategyqa/runs/autocot/GPT-4-{timestamp}.json

// Result format:
// {
//   "summary": {
//     "correctAnswers": 15,
//     "totalAnswers": 20,
//     "percentScore": 75
//   },
//   "results": [
//     {
//       "question": "Did Brazilian jiu-jitsu founders have a baker's dozen kids?",
//       "answer": true,
//       "response": "Yes, the Gracie founders had more than 13 children...",
//       "answerScore": 9,
//       "verdict": "correct"
//     }
//   ]
// }
```

--------------------------------

### Execute HotpotQA Evaluations

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Runs multi-hop QA evaluations on the HotpotQA dataset using baseline, CoT, and AutoCoT methods.

```typescript
import { evalHotpotQa } from './utils/evals/hotpotqa.js';
import { evalCotHotpotQa } from './utils/evals/cot-hotpotqa.js';
import { evalAutoCotHotpotQa } from './utils/evals/autocot-hotpotqa.js';

// Run baseline evaluation
await evalHotpotQa();
// Saves results to: data/hotpotqa/runs/GPT4-{timestamp}.json

// Run Chain of Thought evaluation
await evalCotHotpotQa();

// Run AutoCoT evaluation
await evalAutoCotHotpotQa();

// Each evaluation:
// 1. Shuffles and samples 20 questions from the dataset
// 2. Sends each question to GPT-4 with appropriate prompting
// 3. Scores each answer using GPT-4 as evaluator
// 4. Calculates accuracy and saves detailed results
```

--------------------------------

### Generate Scoring Prompts

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Creates a prompt template for GPT-4 to evaluate the correctness of an answer on a 0-10 scale.

```typescript
import { scorePrompt } from './config/prompts/score.js';
import type { ScorePromptParameters } from './types/score.js';

const params: ScorePromptParameters = {
  question: 'Is the Eiffel Tower in Paris?',
  answer: 'Yes, the Eiffel Tower is located in Paris, France.',
  correctAnswer: 'yes'
};

const prompt = scorePrompt(params);
// Output:
// "Your job is to score an answer's correctness from 0 to 10.
//  You will be given the question, the correct answer, and the answer you need to score.
//  0 means the answer is completely wrong, 10 means the answer is completely correct.
//  Explain your reasoning first shortly, and then write the score as a literal number (0 to 10).
//
//  Question: Is the Eiffel Tower in Paris?
//  Answer: Yes, the Eiffel Tower is located in Paris, France.
//  Correct Answer: yes
//  Score: "
```

--------------------------------

### Generate Reasoning Traces with autoCotGenerator

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Decomposes complex queries into explicit sub-questions to facilitate Chain of Thought reasoning.

```typescript
import { autoCotGenerator } from './utils/evals/autocot.js';

// Generate reasoning traces for a complex question
const reasoningPrompt = await autoCotGenerator({
  question: 'Did Brazilian jiu-jitsu Gracie founders have at least a baker\'s dozen of kids between them?'
});

console.log(reasoningPrompt);
// Output:
// Use these reasoning traces below to answer the question:
//
// - Who were the founders of Brazilian jiu-jitsu?
// - What is the number represented by the baker's dozen?
// - How many children do Gracie founders have altogether?
// - Is this number bigger than baker's dozen?
//
// Indicate your FINAL answer clearly below:
```

--------------------------------

### Shuffle dataset order with shuffleArray

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Uses the Fisher-Yates algorithm to randomize array order in-place. Ensure the input array contains objects with question and answer fields for compatibility with evaluation workflows.

```typescript
import { shuffleArray } from './utils/shuffle.js';

const questions = [
  { question: 'Q1', answer: true },
  { question: 'Q2', answer: false },
  { question: 'Q3', answer: true },
];

shuffleArray(questions);
// Array is shuffled in-place for random sampling
console.log(questions);
// Output: [{ question: 'Q3', ... }, { question: 'Q1', ... }, { question: 'Q2', ... }]
```

--------------------------------

### BibTeX Citation for AutoReason Paper

Source: https://github.com/miralab-ai/autoreason/blob/main/README.md

Use this BibTeX entry to cite the AutoReason paper in your academic work. It includes author, title, year, and publication details.

```tex
@misc{sevinc2024autoreasonautomaticfewshotreasoning,
      title={AutoReason: Automatic Few-Shot Reasoning Decomposition},
      author={Arda Sevinc and Abdurrahman Gumus},
      year={2024},
      eprint={2412.06975},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.06975},
}

```

--------------------------------

### Score Answers Using GPT-4

Source: https://context7.com/miralab-ai/autoreason/llms.txt

Evaluates an answer against a ground truth using the score utility and determines a verdict based on a threshold.

```typescript
import { score } from './utils/evals/score.js';

// Score an answer against the correct answer
const answerScore = await score({
  question: 'Do hamsters provide food for any animals?',
  answer: 'Yes, hamsters are prey animals that predators eat.',
  correctAnswer: 'true'
});

console.log(answerScore); // 8 (0-10 scale)

// Determine verdict based on threshold
const verdict = answerScore > 6 ? 'correct' : 'incorrect';
console.log(verdict); // "correct"
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.