DataSleuth

https://plustorg.github.io/datasleuth/
Admin
DataSleuth is a modular AI-powered research engine that builds LLM-powered pipelines to transform...
Tokens:32,639
Snippets:186
Trust Score:8.3
Update:9 months ago
Show doc for...
Tokens:
Raw
### Clone Repository and Setup Development Environment (Bash)

Source: https://plustorg.github.io/datasleuth/media/CONTRIBUTING

This snippet shows how to clone the @plust/datasleuth repository, navigate into the project directory, install dependencies, set up the development environment, and create a new feature branch.

```bash
git clone https://github.com/YOUR_USERNAME/datasleuth.git
cd datasleuth
npm install
npm run setup:dev
git checkout -b feature/your-feature-name
```

--------------------------------

### Install @plust/datasleuth with npm

Source: https://plustorg.github.io/datasleuth/index

Installs the @plust/datasleuth package using npm. This is the initial step to integrate the library into your project for building LLM-powered research pipelines.

```bash
npm install @plust/datasleuth
```

--------------------------------

### Isolate Research Step for Testing

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Illustrates how to run individual steps of the Datasleuth research pipeline in isolation, such as testing only the search step. This technique helps in pinpointing issues within complex pipelines.

```typescript
// Test just the search step
const result = await research({
  query: "Your query",
  outputSchema: schema,
  steps: [searchWeb({ provider: googleSearch })]
});
```

--------------------------------

### Handle ConfigurationError in TypeScript

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Demonstrates how to catch and handle ConfigurationError in TypeScript. It shows how to access error messages, details, and suggestions provided by the error object to help resolve configuration issues.

```typescript
try {
  const results = await research({ /*...*/
});
} catch (error) {
  if (error instanceof ConfigurationError) {
    console.error(`Configuration error: ${error.message}`);
    console.error(`Details: ${JSON.stringify(error.details)}`);
    console.error(`Fix suggestions: ${error.suggestions.join('\n')}`);
  }
}
```

--------------------------------

### Testing 'myFunction' with Jest (TypeScript)

Source: https://plustorg.github.io/datasleuth/media/CONTRIBUTING

Provides an example of how to write unit tests for a function using Jest. It covers testing successful execution, handling empty input, and managing error cases.

```typescript
describe('myFunction', () => {
  it('should process valid input correctly', () => {
    // Test implementation
  });
  
  it('should handle empty input gracefully', () => {
    // Test implementation
  });
  
  it('should throw appropriate error for invalid input', () => {
    // Test implementation
  });
});
```

--------------------------------

### Basic Research Pipeline with @plust/datasleuth

Source: https://plustorg.github.io/datasleuth/media/index

Execute a basic research pipeline using the `research` function from @plust/datasleuth. This example demonstrates setting up a simple query, defining an output schema with Zod, and using an OpenAI model for processing.

```javascript
import { research } from '@plust/datasleuth';
import { z } from 'zod';
import { openai } from '@ai-sdk/openai';

// Define the structure of your research results
const outputSchema = z.object({
  summary: z.string(),
  keyFindings: z.array(z.string()),
  sources: z.array(z.string().url()),
});

// Execute research
const results = await research({
  query: 'Latest advancements in quantum computing',
  outputSchema,
  defaultLLM: openai('gpt-4o'),
});

console.log(results);

```

--------------------------------

### Advanced Research with Custom Pipeline in @plust/datasleuth

Source: https://plustorg.github.io/datasleuth/index

Illustrates advanced research capabilities using @plust/datasleuth with a custom pipeline. This example configures a search provider (Google), defines a complex output schema, and specifies a series of steps including planning, web searching, content extraction, and iterative evaluation.

```javascript
import {
  research,
  plan,
  searchWeb,
  extractContent,
  evaluate,
  repeatUntil,
} from '@plust/datasleuth';
import { z } from 'zod';
import { google } from '@plust/search-sdk';
import { openai } from '@ai-sdk/openai';

// Configure a search provider
const googleSearch = google.configure({
  apiKey: process.env.GOOGLE_API_KEY,
  cx: process.env.GOOGLE_CX,
});

// Define complex output schema
const outputSchema = z.object({
  summary: z.string(),
  threats: z.array(z.string()),
  opportunities: z.array(z.string()),
  timeline: z.array(
    z.object({
      year: z.number(),
      event: z.string(),
    })
  ),
  sources: z.array(
    z.object({
      url: z.string().url(),
      reliability: z.number().min(0).max(1),
    })
  ),
});

// Execute research with custom pipeline steps
const results = await research({
  query: 'Impact of climate change on agriculture',
  outputSchema,
  steps: [
    plan({ llm: openai('gpt-4o') }),
    searchWeb({ provider: googleSearch, maxResults: 10 }),
    extractContent({ selector: 'article, .content, main' }),
    repeatUntil(evaluate({ criteriaFn: (data) => data.sources.length > 15 }), [
      searchWeb({ provider: googleSearch }),
      extractContent(),
    ]),
  ],
  config: {
    errorHandling: 'continue',
    timeout: 60000, // 1 minute
  },
});
```

--------------------------------

### Immutable State Transformation Example (TypeScript)

Source: https://plustorg.github.io/datasleuth/media/CONTRIBUTING

Demonstrates the pattern of immutable state transformation, where new state objects are created instead of modifying existing ones. This is crucial for predictable state management.

```typescript
function executeStep(state: ResearchState, options: StepOptions): Promise<ResearchState> {
  return {
    ...state,
    data: {
      ...state.data,
      newData: processedResult
    }
  };
}
```

--------------------------------

### Configure Datasleuth to Continue on Error

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Shows how to configure the Datasleuth research pipeline to continue execution even when errors occur in individual steps. This is achieved by setting the `errorHandling` configuration to 'continue' and `continueOnError` to true.

```typescript
research({
  query: "Your query",
  outputSchema: schema,
  config: {
    errorHandling: 'continue',
    continueOnError: true
  }
});
```

--------------------------------

### Advanced Research Pipeline with Custom Steps in @plust/datasleuth

Source: https://plustorg.github.io/datasleuth/media/index

Perform advanced research with a custom pipeline configuration in @plust/datasleuth. This example showcases using multiple pipeline steps like planning, web searching, content extraction, and conditional repetition, along with custom output schema and error handling.

```javascript
import {
  research,
  plan,
  searchWeb,
  extractContent,
  evaluate,
  repeatUntil,
} from '@plust/datasleuth';
import { z } from 'zod';
import { google } from '@plust/search-sdk';
import { openai } from '@ai-sdk/openai';

// Configure a search provider
const googleSearch = google.configure({
  apiKey: process.env.GOOGLE_API_KEY,
  cx: process.env.GOOGLE_CX,
});

// Define complex output schema
const outputSchema = z.object({
  summary: z.string(),
  threats: z.array(z.string()),
  opportunities: z.array(z.string()),
  timeline: z.array(
    z.object({
      year: z.number(),
      event: z.string(),
    })
  ),
  sources: z.array(
    z.object({
      url: z.string().url(),
      reliability: z.number().min(0).max(1),
    })
  ),
});

// Execute research with custom pipeline steps
const results = await research({
  query: 'Impact of climate change on agriculture',
  outputSchema,
  steps: [
    plan({ llm: openai('gpt-4o') }),
    searchWeb({ provider: googleSearch, maxResults: 10 }),
    extractContent({ selector: 'article, .content, main' }),
    repeatUntil(evaluate({ criteriaFn: (data) => data.sources.length > 15 }), [
      searchWeb({ provider: googleSearch }),
      extractContent(),
    ]),
  ],
  config: {
    errorHandling: 'continue',
    timeout: 60000, // 1 minute
  },
});

```

--------------------------------

### Configure Debug Logging Level in Datasleuth

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Shows how to enable verbose logging for the Datasleuth research process by setting the `logLevel` configuration option to 'debug'. This helps in debugging by providing detailed information about the pipeline's execution.

```typescript
research({
  query: "Your query",
  outputSchema: schema,
  config: {
    logLevel: 'debug' // 'error', 'warn', 'info', 'debug', 'trace'
  }
});
```

--------------------------------

### Provide Default LLM for Research in TypeScript

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Shows how to provide a default Language Model (LLM) when initializing the research function in TypeScript. This is useful when no LLM is specified for a particular step.

```typescript
import { openai } from '@ai-sdk/openai';

research({
  query: "Your query",
  defaultLLM: openai('gpt-4o'),
  outputSchema: schema
});
```

--------------------------------

### Specify LLM for Specific Research Steps in TypeScript

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Illustrates how to specify an LLM for individual steps within a research process in TypeScript. This allows for granular control over which LLM is used for different parts of the workflow.

```typescript
steps: [
  plan({ llm: openai('gpt-4o') }),
  // other steps...
]
```

--------------------------------

### Transform String Numbers to Numbers with Zod Preprocess

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Shows how to use Zod's `preprocess` function to transform string representations of numbers into actual numbers, addressing type mismatch validation errors. This is particularly helpful when dealing with data that may contain numeric values as strings.

```typescript
// Transform string numbers to actual numbers
const schema = z.object({
  value: z.preprocess(
    (val) => typeof val === 'string' ? Number(val) : val,
    z.number()
  )
});
```

--------------------------------

### Execute Research with Custom Schema and Providers (JavaScript)

Source: https://plustorg.github.io/datasleuth/functions/research

This example demonstrates how to use the 'research' function from '@plust/datasleuth' to perform a search. It includes defining a custom output schema using Zod, configuring a search provider (Google), and specifying a default language model (OpenAI). The function orchestrates the query, search, and result validation based on the provided schema.

```javascript
import { research } from '@plust/datasleuth';
import { z } from 'zod';
import { openai } from '@ai-sdk/openai';
import { google } from '@plust/search-sdk';

// Define your output schema
const outputSchema = z.object({
  summary: z.string(),
  keyFindings: z.array(z.string()),
  sources: z.array(z.string().url())
});

// Configure your search provider
const searchProvider = google.configure({
  apiKey: process.env.GOOGLE_API_KEY,
  cx: process.env.GOOGLE_CX
});

// Execute research
const results = await research({
  query: "Latest advancements in quantum computing",
  outputSchema,
  defaultLLM: openai('gpt-4o'),
  defaultSearchProvider: searchProvider
});
```

--------------------------------

### Handle Optional and Default Fields with Zod Schema

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Demonstrates how to make fields optional or provide default values in a Zod schema to resolve missing field validation errors. This is useful when certain fields might not always be present in the data.

```typescript
const schema = z.object({
  summary: z.string(),
  findings: z.array(z.string()).optional(), // Make optional
  // or provide default value:
  sources: z.array(z.string().url()).default([])
});
```

--------------------------------

### Custom Array Validation with Zod Refine

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Demonstrates how to add custom validation logic to an array field using Zod's `refine` method. This example ensures that an array of items is not empty, addressing complex validation issues with nested structures.

```typescript
const schema = z.object({
  items: z.array(z.string()).refine(
    (items) => items.length > 0,
    { message: "Items array cannot be empty" }
  )
});
```

--------------------------------

### Orchestrate Function

Source: https://plustorg.github.io/datasleuth/functions/orchestrate

This function creates an orchestration step that uses agents to make decisions. It can handle an entire research process from start to finish by adaptively choosing the right tools.

```APIDOC
## Function orchestrate

### Description
Creates an orchestration step that uses agents to make decisions. This step uses an LLM to dynamically select and execute research tools based on the current state. It can handle an entire research process from start to finish by adaptively choosing the right tools.

### Method
N/A (This is a function within a library, not a direct HTTP API endpoint)

### Parameters
#### Options (OrchestrateOptions)
Configuration for the orchestration.
- **model** (LanguageModelV1) - Required - LLM model to use for orchestration (from the AI library).
- **searchProvider** (any) - Optional - Default search provider to use for search-dependent steps.
- **tools** (Record<string, ResearchStep>) - Optional - Map of tool names to step functions that can be used by the agent.
- **customPrompt** (string) - Optional - Custom prompt for the orchestration agent.
- **maxIterations** (number) - Optional - Maximum number of iterations.
- **exitCriteria** ((state: ResearchState) => boolean | Promise<boolean>) - Optional - Optional function that determines when to exit orchestration.
- **includeInResults** (boolean) - Optional - Whether to include the orchestration results in the final output.
- **continueOnError** (boolean) - Optional - Whether to continue if a tool execution fails.
- **toolSelectorFn** ((state: ResearchState, availableTools: string[]) => Promise<{ toolName: string; reasoning: string }>) - Optional - Custom tool selection function (if provided, uses this instead of LLM).
- **retry** ({ maxRetries?: number; baseDelay?: number }) - Optional - Retry configuration.
  - **maxRetries** (number) - Optional - Maximum number of retries.
  - **baseDelay** (number) - Optional - Base delay between retries in ms.

### Returns
ResearchStep - An orchestration step for the research pipeline.

### Example
```javascript
import { research, orchestrate } from '@plust/datasleuth';
import { openai } from '@ai-sdk/openai';
import { google } from '@plust/search-sdk';

const results = await research({
  query: 'Impact of climate change on agriculture',
  outputSchema: schema,
  steps: [
    orchestrate({
      model: openai('gpt-4o'),
      searchProvider: google.configure({ apiKey: process.env.GOOGLE_API_KEY }),
      maxIterations: 15,
      continueOnError: true,
      exitCriteria: (state) => state.data.summary !== undefined
    })
  ]
});
```
```

--------------------------------

### Datasleuth `plan` Pipeline Step

Source: https://plustorg.github.io/datasleuth/index

The `plan` step generates a research plan using Large Language Models (LLMs). It allows customization of the LLM, system prompt, temperature, and whether the plan should be included in the final results.

```typescript
plan({
  llm?: LanguageModel;        // LLM model to use (falls back to defaultLLM)
  customPrompt?: string;      // Custom system prompt
  temperature?: number;       // LLM temperature (0.0-1.0)
  includeInResults?: boolean; // Whether to include plan in results
}): ResearchStep
```

--------------------------------

### Basic Research with @plust/datasleuth

Source: https://plustorg.github.io/datasleuth/index

Demonstrates a basic research query using the default pipeline of @plust/datasleuth. It requires defining an output schema using Zod and specifying a default LLM provider like OpenAI.

```javascript
import { research } from '@plust/datasleuth';
import { z } from 'zod';
import { openai } from '@ai-sdk/openai';

// Define the structure of your research results
const outputSchema = z.object({
  summary: z.string(),
  keyFindings: z.array(z.string()),
  sources: z.array(z.string().url()),
});

// Execute research
const results = await research({
  query: 'Latest advancements in quantum computing',
  outputSchema,
  defaultLLM: openai('gpt-4o'),
});

console.log(results);
```

--------------------------------

### LLM Integration with Vercel AI SDK

Source: https://plustorg.github.io/datasleuth/media/index

Demonstrates how to integrate @plust/datasleuth with the Vercel AI SDK to leverage different LLM providers for various research steps like planning, analysis, and fact-checking. It shows defining output schemas and configuring LLM providers such as OpenAI and Anthropic.

```APIDOC
## LLM Integration with Vercel AI SDK

### Description
This section details how to use the `@plust/datasleuth` library with the Vercel AI SDK to integrate various Large Language Models (LLMs) into your research workflow. It covers defining schemas for the expected output and specifying different LLM providers for distinct stages of the research process.

### Method
`research` function from `@plust/datasleuth`

### Endpoint
N/A (Client-side library)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **query** (string) - Required - The primary research question.
- **outputSchema** (zod schema) - Required - The schema defining the structure of the research output.
- **steps** (Array<object>) - Required - An array of research steps, which can include LLM configurations:
  - **plan** (object) - Configures the planning step:
    - **llm** (object) - The LLM provider configuration (e.g., `openai('gpt-4o')`).
    - **temperature** (number) - Optional - Controls the randomness of the output.
  - **analyze** (object) - Configures the analysis step:
    - **llm** (object) - The LLM provider configuration.
    - **focus** (string) - Optional - Specifies the area of focus for the analysis.
    - **depth** (string) - Optional - Defines the depth of the analysis (e.g., 'comprehensive').
  - **factCheck** (object) - Configures the fact-checking step:
    - **llm** (object) - The LLM provider configuration.
    - **threshold** (number) - Optional - The confidence threshold for fact-checking.
    - **includeEvidence** (boolean) - Optional - Whether to include evidence in the fact-check results.
  - **summarize** (object) - Configures the summarization step:
    - **llm** (object) - The LLM provider configuration.
    - **format** (string) - Optional - The desired format of the summary (e.g., 'structured').
    - **maxLength** (number) - Optional - The maximum length of the summary.

### Request Example
```json
{
  "query": "Advancements in gene editing technologies",
  "outputSchema": "z.object({...})",
  "steps": [
    {
      "plan": {
        "llm": "openai('gpt-4o')",
        "temperature": 0.4
      }
    },
    {
      "analyze": {
        "llm": "anthropic('claude-3-opus-20240229')",
        "focus": "ethical-considerations",
        "depth": "comprehensive"
      }
    },
    {
      "factCheck": {
        "llm": "openai('gpt-4o')",
        "threshold": 0.8,
        "includeEvidence": true
      }
    },
    {
      "summarize": {
        "llm": "anthropic('claude-3-sonnet-20240229')",
        "format": "structured",
        "maxLength": 2000
      }
    }
  ]
}
```

### Response
#### Success Response (200)
- **results** (object) - The structured output based on the provided `outputSchema`.

#### Response Example
```json
{
  "summary": "...",
  "analysis": {
    "insights": [
      "..."
    ]
  },
  "factChecks": [
    {
      "statement": "...",
      "isValid": true
    }
  ]
}
```
```

--------------------------------

### Pipeline Step: searchWeb

Source: https://plustorg.github.io/datasleuth/index

Searches the web using configured search providers. It allows specifying the provider, maximum results, language, region, and safe search settings.

```APIDOC
## searchWeb(options)

### Description
Executes a web search using a provided search provider and allows filtering results by count, language, region, and safety.

### Method
Pipeline Step Configuration

### Endpoint
N/A (Used within `research` function's `steps` array)

### Parameters
#### Request Body (within `steps` array)
- **provider** (SearchProvider) - Required - The configured search provider (e.g., Google, SerpApi).
- **maxResults** (number) - Optional - The maximum number of search results to return.
- **language** (string) - Optional - The language code for the search query (e.g., 'en' for English).
- **region** (string) - Optional - The region code for the search query (e.g., 'US' for United States).
- **safeSearch** ('off' | 'moderate' | 'strict') - Optional - Controls the level of safe search filtering.
- **useQueriesFromPlan** (boolean) - Optional - If true, uses queries generated from a preceding 'plan' step.

### Request Example
```json
{
  "name": "searchWeb",
  "provider": "google.configure({ apiKey: 'YOUR_GOOGLE_API_KEY' })",
  "maxResults": 10,
  "language": "en",
  "region": "US",
  "safeSearch": "moderate"
}
```

### Response
#### Success Response (200)
N/A (This is a configuration object for a pipeline step.)

#### Response Example
N/A
```

--------------------------------

### Summarize Research Step

Source: https://plustorg.github.io/datasleuth/index

Synthesizes information into concise summaries using specified options.

```APIDOC
## POST /research/summarize

### Description
Synthesizes information into concise summaries.

### Method
POST

### Endpoint
/research/summarize

### Parameters
#### Request Body
- **llm** (LanguageModel) - Optional - LLM model to use
- **maxLength** (number) - Optional - Maximum summary length
- **format** ('paragraph' | 'bullet' | 'structured') - Optional - The format of the summary
- **includeInResults** (boolean) - Optional - Whether to include summary in results

### Request Example
```json
{
  "llm": "default",
  "maxLength": 500,
  "format": "paragraph",
  "includeInResults": true
}
```

### Response
#### Success Response (200)
- **summary** (string) - The generated summary

#### Response Example
```json
{
  "summary": "This is a concise summary of the research."
}
```
```

--------------------------------

### DataSleuth Research Steps API

Source: https://plustorg.github.io/datasleuth/media/index

Documentation for various research steps available in the DataSleuth library.

```APIDOC
## `summarize(options?)`

### Description
Synthesizes information into concise summaries.

### Method
`summarize`

### Parameters
#### Options
- **llm** (LanguageModel) - Optional - LLM model to use
- **maxLength** (number) - Optional - Maximum summary length
- **format** ('paragraph' | 'bullet' | 'structured') - Optional - Format of the summary
- **includeInResults** (boolean) - Optional - Whether to include summary in results

### Returns
`ResearchStep`

## `evaluate(options)`

### Description
Evaluates current state against specified criteria.

### Method
`evaluate`

### Parameters
#### Options
- **criteriaFn** ((state) => boolean | Promise<boolean>) - Required - Evaluation function
- **criteriaName** (string) - Optional - Name for this evaluation
- **confidenceThreshold** (number) - Optional - Confidence threshold (0.0-1.0)

### Returns
`ResearchStep`

## `repeatUntil(conditionStep, stepsToRepeat, options?)`

### Description
Repeats steps until a condition is met.

### Method
`repeatUntil`

### Parameters
- **conditionStep** (ResearchStep) - Required - Step that evaluates condition
- **stepsToRepeat** (ResearchStep[]) - Required - Steps to repeat
#### Options
- **maxIterations** (number) - Optional - Maximum iterations
- **throwOnMaxIterations** (boolean) - Optional - Throw error on max iterations

### Returns
`ResearchStep`

## `parallel(options)`

### Description
Executes multiple research tracks concurrently.

### Method
`parallel`

### Parameters
#### Options
- **tracks** (TrackOptions[]) - Required - Array of research tracks
- **mergeFunction** (MergeFunction) - Optional - Function to merge results
- **continueOnTrackError** (boolean) - Optional - Continue if a track fails

### Returns
`ResearchStep`

## `track(options)`

### Description
Creates an isolated research track.

### Method
`track`

### Parameters
#### Options
- **name** (string) - Required - Track name
- **steps** (ResearchStep[]) - Required - Steps to execute in this track
- **initialData** (any) - Optional - Initial data for this track

### Returns
`ResearchStep`

## `orchestrate(options)`

### Description
Uses AI agents to make dynamic decisions about research steps.

### Method
`orchestrate`

### Parameters
#### Options
- **llm** (LanguageModel) - Required - LLM model for orchestration
- **tools** (Record<string, ResearchStep>) - Required - Available tools for agent
- **customPrompt** (string) - Optional - Custom orchestration prompt
- **maxIterations** (number) - Optional - Maximum iterations
- **exitCriteria** ((state) => boolean | Promise<boolean>) - Optional - Exit condition

### Returns
`ResearchStep`

## `transform(options?)`

### Description
Ensures research output matches the expected schema structure.

### Method
`transform`

### Parameters
#### Options
- **llm** (LanguageModel) - Optional - LLM model to use (falls back to defaultLLM)
- **allowMissingWithDefaults** (boolean) - Optional - Auto-fix missing fields with defaults
- **useLLM** (boolean) - Optional - Use LLM for intelligent transformation
- **temperature** (number) - Optional - LLM temperature (0.0-1.0)
- **systemPrompt** (string) - Optional - Custom system prompt
- **transformFn** ((state) => any) - Optional - Custom transformation function

### Returns
`ResearchStep`
```

--------------------------------

### Inspect Pipeline State with Custom Evaluation Step

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Demonstrates adding a custom evaluation step within the Datasleuth pipeline to inspect and log the intermediate state of the research process. This is useful for understanding and debugging complex pipeline flows.

```typescript
steps: [
  // other steps...
  evaluate({
    criteriaFn: (state) => {
      console.log('Current state:', JSON.stringify(state.data, null, 2));
      return true; // Always continue
    }
  })
]
```

--------------------------------

### Execute Research with Agent Orchestration in JavaScript

Source: https://plustorg.github.io/datasleuth/media/index

This snippet demonstrates how to configure search providers (Google, SerpApi) and execute a research task using AI agents for dynamic step selection. It utilizes Zod for schema definition and OpenAI's GPT-4o model for orchestration. The research process involves defining a query, an output schema, and custom orchestration steps with specific tools and exit criteria.

```javascript
import {
  research,
  orchestrate,
  searchWeb,
  extractContent,
  analyze,
  transform,
} from '@plust/datasleuth';
import { z } from 'zod';
import { google, serpapi } from '@plust/search-sdk';
import { openai } from '@ai-sdk/openai';

// Configure search providers
const webSearch = google.configure({ apiKey: process.env.GOOGLE_API_KEY });
const academicSearch = serpapi.configure({
  apiKey: process.env.SERPAPI_KEY,
  engine: 'google_scholar',
});

// Execute research with orchestration
const results = await research({
  query: 'Emerging technologies in renewable energy storage',
  outputSchema: z.object({
    marketOverview: z.string(),
    technologies: z.array(
      z.object({
        name: z.string(),
        maturityLevel: z.enum(['research', 'emerging', 'growth', 'mature']),
        costEfficiency: z.number().min(1).max(10),
        scalabilityPotential: z.number().min(1).max(10),
        keyPlayers: z.array(z.string()),
      })
    ),
    forecast: z.object({
      shortTerm: z.string(),
      mediumTerm: z.string(),
      longTerm: z.string(),
    }),
    sources: z.array(
      z.object({
        url: z.string().url(),
        type: z.enum(['academic', 'news', 'company', 'government']),
        relevance: z.number().min(0).max(1),
      })
    ),
  }),
  steps: [
    orchestrate({
      llm: openai('gpt-4o'),
      tools: {
        searchWeb: searchWeb({ provider: webSearch }),
        searchAcademic: searchWeb({ provider: academicSearch }),
        extractContent: extractContent(),
        analyze: analyze(),
        // Add your custom tools here
      },
      customPrompt: `  
        You are conducting market research on emerging renewable energy storage technologies.  
        Your goal is to build a comprehensive market overview with technical assessment.  
      `,
      maxIterations: 15,
      exitCriteria: (state) =>
        state.metadata.confidenceScore > 0.85 &&
        state.data.dataPoints?.length > 20,
    }),
  ],
});

```

--------------------------------

### Process Content in Chunks with TypeScript

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Provides a TypeScript code snippet for processing large amounts of content in smaller, manageable chunks. This helps mitigate memory usage issues during large content processing by iterating through the data in fixed-size segments.

```typescript
// Process content in 10KB chunks
const chunkSize = 10 * 1024;
for (let i = 0; i < content.length; i += chunkSize) {
  const chunk = content.substring(i, i + chunkSize);
  // Process chunk...
}
```

--------------------------------

### Datasleuth `searchWeb` Pipeline Step

Source: https://plustorg.github.io/datasleuth/index

The `searchWeb` step facilitates web searches using configured providers. It supports specifying the number of results, language, region, safe search settings, and optionally using queries derived from a research plan.

```typescript
searchWeb({
  provider: SearchProvider;   // Configured search provider
  maxResults?: number;        // Maximum results to return
  language?: string;          // Language code (e.g., 'en')
  region?: string;            // Region code (e.g., 'US')
  safeSearch?: 'off' | 'moderate' | 'strict';
  useQueriesFromPlan?: boolean; // Use queries from research plan
}): ResearchStep
```

--------------------------------

### Handle Multiple Data Types with Zod Union

Source: https://plustorg.github.io/datasleuth/media/troubleshooting

Illustrates the use of Zod's `union` type to define fields that can accept multiple data types, solving type mismatch issues. This is useful for fields like dates that might be represented as strings or Date objects.

```typescript
const schema = z.object({
  date: z.union([z.string(), z.date()])
});
```

--------------------------------

### Pipeline Step: plan

Source: https://plustorg.github.io/datasleuth/index

Creates a research plan using LLMs. This step can be included in the research pipeline to pre-define research tasks.

```APIDOC
## plan(options?)

### Description
Generates a research plan using a specified Language Model, allowing for custom prompts and control over the output.

### Method
Pipeline Step Configuration

### Endpoint
N/A (Used within `research` function's `steps` array)

### Parameters
#### Request Body (within `steps` array)
- **llm** (LanguageModel) - Optional - The LLM model to use for planning. Falls back to the `defaultLLM` if not provided.
- **customPrompt** (string) - Optional - A custom system prompt to guide the plan generation.
- **temperature** (number) - Optional - The temperature setting for the LLM (0.0 to 1.0).
- **includeInResults** (boolean) - Optional - Whether to include the generated plan in the final research results.

### Request Example
```json
{
  "name": "plan",
  "llm": "openai('gpt-4o')",
  "customPrompt": "Create a detailed research plan for investigating the impact of climate change on agriculture.",
  "includeInResults": true
}
```

### Response
#### Success Response (200)
N/A (This is a configuration object for a pipeline step.)

#### Response Example
N/A
```

--------------------------------

### Orchestrate AI Research Steps with Datasleuth

Source: https://plustorg.github.io/datasleuth/index

This snippet demonstrates how to use Datasleuth's `research` function to orchestrate AI agents for dynamic research. It configures search providers (Google and SerpAPI), defines a detailed output schema using Zod, and specifies research steps including web searches, content extraction, and analysis. The orchestration step utilizes OpenAI's GPT-4o model with custom prompts and exit criteria.

```typescript
import {
  research,
  orchestrate,
  searchWeb,
  extractContent,
  analyze,
  transform,
} from '@plust/datasleuth';
import { z } from 'zod';
import { google, serpapi } from '@plust/search-sdk';
import { openai } from '@ai-sdk/openai';

// Configure search providers
const webSearch = google.configure({ apiKey: process.env.GOOGLE_API_KEY });
const academicSearch = serpapi.configure({
  apiKey: process.env.SERPAPI_KEY,
  engine: 'google_scholar',
});

// Execute research with orchestration
const results = await research({
  query: 'Emerging technologies in renewable energy storage',
  outputSchema: z.object({
    marketOverview: z.string(),
    technologies: z.array(
      z.object({
        name: z.string(),
        maturityLevel: z.enum(['research', 'emerging', 'growth', 'mature']),
        costEfficiency: z.number().min(1).max(10),
        scalabilityPotential: z.number().min(1).max(10),
        keyPlayers: z.array(z.string()),
      })
    ),
    forecast: z.object({
      shortTerm: z.string(),
      mediumTerm: z.string(),
      longTerm: z.string(),
    }),
    sources: z.array(
      z.object({
        url: z.string().url(),
        type: z.enum(['academic', 'news', 'company', 'government']),
        relevance: z.number().min(0).max(1),
      })
    ),
  }),
  steps: [
    orchestrate({
      llm: openai('gpt-4o'),
      tools: {
        searchWeb: searchWeb({ provider: webSearch }),
        searchAcademic: searchWeb({ provider: academicSearch }),
        extractContent: extractContent(),
        analyze: analyze(),
        // Add your custom tools here
      },
      customPrompt: `  
        You are conducting market research on emerging renewable energy storage technologies.
        Your goal is to build a comprehensive market overview with technical assessment.
      `,
      maxIterations: 15,
      exitCriteria: (state) =>
        state.metadata.confidenceScore > 0.85 &&
        state.data.dataPoints?.length > 20,
    }),
  ],
});
```

--------------------------------

### LLM Integration with Vercel AI SDK using Datasleuth

Source: https://plustorg.github.io/datasleuth/index

Demonstrates how to integrate Datasleuth with the Vercel AI SDK to leverage various LLM providers like OpenAI and Anthropic for different research steps. It defines an output schema using Zod and specifies LLM configurations for planning, analysis, fact-checking, and summarization.

```typescript
import {
  research,
  plan,
  analyze,
  factCheck,
  summarize,
} from '@plust/datasleuth';
import { z } from 'zod';
import { openai } from '@ai-sdk/openai';
import { anthropic } from '@ai-sdk/anthropic';

// Define your output schema
const outputSchema = z.object({
  summary: z.string(),
  analysis: z.object({
    insights: z.array(z.string()),
  }),
  factChecks: z.array(
    z.object({
      statement: z.string(),
      isValid: z.boolean(),
    })
  ),
});

// Use different LLM providers for different steps
const results = await research({
  query: 'Advancements in gene editing technologies',
  outputSchema,
  steps: [
    // Use OpenAI for research planning
    plan({
      llm: openai('gpt-4o'),
      temperature: 0.4,
    }),

    // Use Anthropic for specialized analysis
    analyze({
      llm: anthropic('claude-3-opus-20240229'),
      focus: 'ethical-considerations',
      depth: 'comprehensive',
    }),

    // Use OpenAI for fact checking
    factCheck({
      llm: openai('gpt-4o'),
      threshold: 0.8,
      includeEvidence: true,
    }),

    // Use Anthropic for final summarization
    summarize({
      llm: anthropic('claude-3-sonnet-20240229'),
      format: 'structured',
      maxLength: 2000,
    }),
  ],
});
```

--------------------------------

### Create Initial State for Research Pipeline (JavaScript)

Source: https://plustorg.github.io/datasleuth/functions/createInitialState

The createInitialState function initializes the state object for a research pipeline. It requires a research query string and a Zod schema defining the expected output structure. This function returns a ResearchState object populated with the provided query and schema.

```javascript
import { z } from 'zod';
import { createInitialState } from '@plust/datasleuth';

const outputSchema = z.object({
  summary: z.string(),
  findings: z.array(z.string())
});

const initialState = createInitialState(
  "What are the latest advancements in renewable energy?",
  outputSchema
);
```

--------------------------------

### Options Pattern for Function Configuration (TypeScript)

Source: https://plustorg.github.io/datasleuth/media/CONTRIBUTING

Shows the options pattern for providing optional configuration to functions. This allows for flexible function behavior with sensible defaults.

```typescript
interface MyOptions {
  param1?: string;
  param2?: number;
}

function myFunction(options: MyOptions = {}) {
  const { param1 = 'default', param2 = 42 } = options;
  // Implementation
}
```

--------------------------------

### Pipeline Step: orchestrate

Source: https://plustorg.github.io/datasleuth/index

An orchestration step that uses LLMs to dynamically decide which research steps to execute. It configures tools and provides custom prompts for the agent's behavior.

```APIDOC
## orchestrate(options)

### Description
Configures an orchestration step for the research pipeline, allowing an LLM to dynamically select and execute available tools based on a custom prompt and criteria.

### Method
Orchestration Step Configuration

### Endpoint
N/A (Used within `research` function's `steps` array)

### Parameters
#### Request Body (within `steps` array)
- **llm** (LanguageModel) - Required - The Language Model to use for orchestration.
- **tools** (object) - Required - An object mapping tool names to their configurations (e.g., `searchWeb`, `extractContent`).
- **customPrompt** (string) - Optional - A custom prompt to guide the LLM's decision-making process.
- **maxIterations** (number) - Optional - The maximum number of iterations the orchestration should run.
- **exitCriteria** (function) - Optional - A function that determines when the orchestration should stop based on the current state.

### Request Example
```json
{
  "name": "orchestrate",
  "llm": "openai('gpt-4o')",
  "tools": {
    "searchWeb": "searchWeb({ provider: google.configure({ apiKey: 'YOUR_GOOGLE_API_KEY' }) })",
    "searchAcademic": "searchWeb({ provider: serpapi.configure({ apiKey: 'YOUR_SERPAPI_KEY', engine: 'google_scholar' }) })",
    "extractContent": "extractContent()",
    "analyze": "analyze()"
  },
  "customPrompt": "You are an AI assistant performing market research...",
  "maxIterations": 15,
  "exitCriteria": "(state) => state.metadata.confidenceScore > 0.85 && state.data.dataPoints?.length > 20"
}
```

### Response
#### Success Response (200)
N/A (This is a configuration object for a pipeline step.)

#### Response Example
N/A
```

--------------------------------

### Interface PlanOptions for Research Planning

Source: https://plustorg.github.io/datasleuth/interfaces/PlanOptions

Defines configuration options for the research planning step in Datasleuth. It allows customization of the language model, system prompt, temperature, result inclusion, and retry settings for LLM calls.

```typescript
interface PlanOptions {
  customPrompt?: string;
  llm?: LanguageModelV1;
  temperature?: number;
  includeInResults?: boolean;
  retry?: { maxRetries?: number; baseDelay?: number };
}
```

--------------------------------

### Extract Content Options

Source: https://plustorg.github.io/datasleuth/interfaces/ExtractContentOptions

Configuration options for the content extraction step in Datasleuth.

```APIDOC
## Interface ExtractContentOptions

Options for the content extraction step.

### Properties

*   **selectors** (string) - Optional - CSS selectors to extract content from.
*   **selector** (string) - Optional - Alias for `selectors` (for backwards compatibility).
*   **maxUrls** (number) - Optional - Maximum number of URLs to process.
*   **maxContentLength** (number) - Optional - Maximum content length per URL (characters).
*   **includeInResults** (boolean) - Optional - Whether to include the extracted content in the final results.
*   **timeout** (number) - Optional - Timeout for each URL fetch in milliseconds.
*   **retry** (object) - Optional - Fetch retry configuration.
    *   **maxRetries** (number) - Optional - Maximum number of retries.
    *   **baseDelay** (number) - Optional - Base delay between retries in ms.
*   **minContentLength** (number) - Optional - Minimum content length to consider a successful extraction.
*   **continueOnError** (boolean) - Optional - Whether to continue if some URLs fail to extract.
*   **requireSuccessful** (boolean) - Optional - Whether to require at least one successful extraction.
*   **[key: string]** (any) - Optional - Allows for additional arbitrary properties.
```

--------------------------------

### Pipeline Step: extractContent

Source: https://plustorg.github.io/datasleuth/index

Extracts content from web pages. This step can target specific content using CSS selectors and limit the amount of content extracted.

```APIDOC
## extractContent(options?)

### Description
Extracts specific content from provided URLs, with options to filter by CSS selectors, URL count, and content length.

### Method
Pipeline Step Configuration

### Endpoint
N/A (Used within `research` function's `steps` array)

### Parameters
#### Request Body (within `steps` array)
- **selectors** (string) - Optional - CSS selectors to target specific content elements on the page.
- **maxUrls** (number) - Optional - The maximum number of unique URLs to process.
- **maxContentLength** (number) - Optional - The maximum length of content to extract per URL.
- **includeInResults** (boolean) - Optional - Whether to include the extracted content in the final research results.

### Request Example
```json
{
  "name": "extractContent",
  "selectors": "main p, article li",
  "maxUrls": 5,
  "maxContentLength": 2048,
  "includeInResults": true
}
```

### Response
#### Success Response (200)
N/A (This is a configuration object for a pipeline step.)

#### Response Example
N/A
```

--------------------------------

### Factory Function Pattern for Creating Steps (TypeScript)

Source: https://plustorg.github.io/datasleuth/media/CONTRIBUTING

Illustrates the factory function pattern used in @plust/datasleuth for creating reusable steps. This pattern abstracts the creation logic and allows for configuration through options.

```typescript
export function myStep(options: MyStepOptions = {}): ReturnType<typeof createStep> {
  return createStep('MyStep', executeMyStep, options);
}
```

--------------------------------

### Repeat Until Research Step

Source: https://plustorg.github.io/datasleuth/index

Repeats a series of steps until a specific condition is met.

```APIDOC
## POST /research/repeatUntil

### Description
Repeats a series of steps until a specific condition is met.

### Method
POST

### Endpoint
/research/repeatUntil

### Parameters
#### Request Body
- **conditionStep** (ResearchStep) - Required - The step that evaluates the condition
- **stepsToRepeat** (ResearchStep[]) - Required - The steps to repeat
- **maxIterations** (number) - Optional - Maximum number of iterations
- **throwOnMaxIterations** (boolean) - Optional - Whether to throw an error if max iterations is reached

### Request Example
```json
{
  "conditionStep": {"name": "CheckStatus"},
  "stepsToRepeat": [{"name": "ProcessData"}],
  "maxIterations": 10,
  "throwOnMaxIterations": false
}
```

### Response
#### Success Response (200)
- **finalState** (any) - The state after repeating steps

#### Response Example
```json
{
  "finalState": {"data": "processed"}
}
```
```

--------------------------------

### PipelineConfig Properties Explained

Source: https://plustorg.github.io/datasleuth/interfaces/PipelineConfig

Detailed explanation of the properties within the PipelineConfig interface. This includes the mandatory 'steps' property and various optional properties for controlling pipeline execution flow and behavior.

```typescript
// steps: ResearchStep[]
// errorHandling?: "stop" | "continue" | "rollback"
// maxRetries?: number
// retryDelay?: number
// backoffFactor?: number
// continueOnError?: boolean
// timeout?: number
// logLevel?: LogLevel
```

--------------------------------

### Datasleuth Pipeline Step: plan() API in TypeScript

Source: https://plustorg.github.io/datasleuth/media/index

The `plan` function is a pipeline step used to generate a research plan using Large Language Models (LLMs). It allows for specifying the LLM, a custom prompt, temperature, and whether to include the plan in the final results. It returns a `ResearchStep` object.

```typescript
plan({
  llm?: LanguageModel;        // LLM model to use (falls back to defaultLLM)
  customPrompt?: string;      // Custom system prompt
  temperature?: number;       // LLM temperature (0.0-1.0)
  includeInResults?: boolean; // Whether to include plan in results
}): ResearchStep

```
Tokens:
Raw