Olostep

https://docs.olostep.com
Admin
Olostep is a web data API for AI and research agents that enables web search, real-time data...
Tokens:111,404
Snippets:849
Trust Score:7.4
Update:6 months ago
Show doc for...
Tokens:
Raw
### Initiate Batch Processing

Source: https://docs.olostep.com/examples/price-monitoring

Example of preparing a batch array and starting the processing task.

```python
# Prepare batch array
batch_array = [
{
    "custom_id": "product_123",
    "url": "https://www.amazon.it/dp/B0CHF6Z393/?coliid=INQXTGFQF4FM4&colid=1R0NGA5NR5LSZ&psc=1&ref_=list_c_wl_lv_vv_lig_dp_it"
},
{
    "custom_id": "product_124",
    "url": "https://www.amazon.it/dp/B0CHMJL774/?coliid=I6CFYA5EHVHE2&colid=1R0NGA5NR5LSZ&psc=1&ref_=list_c_wl_lv_vv_lig_dp_it"
}
]

# Start batch processing
batch_id = start_batch(batch_array)
print(f"Started batch: {batch_id}")
```

--------------------------------

### Install Dependencies

Source: https://docs.olostep.com/integrations/mastra

Install both the Olostep tools and the core Mastra package.

```bash
npm install @olostep/mastra-tools @mastra/core
```

--------------------------------

### Install Olostep SDK

Source: https://docs.olostep.com/sdks/python

Installation commands for various Python package managers.

```bash
pip install olostep
```

```bash
pip3 install olostep
```

```bash
poetry add olostep
```

```bash
uv pip install olostep
```

--------------------------------

### Install Olostep SDK

Source: https://docs.olostep.com/sdks/node-js

Install the package via npm.

```bash
npm install olostep
```

--------------------------------

### Start a Batch with Node.js

Source: https://docs.olostep.com/features/batches

Initiate a batch job using Node.js by sending a POST request to the Olostep API with an array of items and a specified parser. This example demonstrates the basic payload structure.

```javascript
const API_URL = 'https://api.olostep.com/v1'
const payload = {
  items: [
    { custom_id: 'item-1', url: 'https://www.google.com/search?q=stripe&gl=us&hl=en' },
    { custom_id: 'item-2', url: 'https://www.google.com/search?q=paddle&gl=us&hl=en' },
  ],
  parser: { id: '@olostep/google-search' }
}
const res = await fetch(`${API_URL}/batches`, {
  method: 'POST',

```

--------------------------------

### Install x402 SDK

Source: https://docs.olostep.com/x402

Install the necessary SDKs for Node.js or Python to handle x402 payment flows automatically.

```bash
# Node.js
npm install x402-fetch viem

# Python  
pip install x402 eth-account
```

--------------------------------

### Install Olostep CLI

Source: https://docs.olostep.com/sdks/cli

Install the CLI globally using npm.

```bash
npm install -g olostep-cli
```

--------------------------------

### Install Apify CLI

Source: https://docs.olostep.com/integrations/apify

Install the Apify command-line interface globally and verify the installation.

```bash
npm install -g apify-cli
apify --version
```

--------------------------------

### Manual Installation

Source: https://docs.olostep.com/integrations/mcp-server

Install the Olostep MCP package globally via npm.

```bash
npm install -g olostep-mcp
```

--------------------------------

### cURL Installation Note

Source: https://docs.olostep.com/features/maps

Note on cURL installation for command-line usage. The macOS built-in cURL is sufficient.

```bash
# macOS: builtin curl is fine
```

--------------------------------

### Install Olostep Mastra Tools

Source: https://docs.olostep.com/integrations/mastra

Commands to install the required package using various package managers.

```bash
npm install @olostep/mastra-tools
```

```bash
yarn add @olostep/mastra-tools
```

```bash
pnpm add @olostep/mastra-tools
```

--------------------------------

### Example Input Configuration

Source: https://docs.olostep.com/integrations/apify

JSON structure for configuring the Olostep actor operation.

```json
{
  "operation": "scrape",
  "apiKey": "YOUR_OLostep_API_KEY",
  "url_to_scrape": "https://example.com",
  "formats": "markdown"
}
```

--------------------------------

### Start a Batch with Python

Source: https://docs.olostep.com/features/batches

Use this Python script to compose a list of items, start a batch job with a specified parser, and monitor its completion. Ensure you replace '<YOUR_API_KEY>' with your actual API key.

```python
import requests
import hashlib
import time

API_KEY = "<YOUR_API_KEY>"  # Replace with your actual API key
API_URL = "https://api.olostep.com/v1"


# Step 1: Utilities
def create_hash_id(url):
    return hashlib.sha256(url.encode()).hexdigest()[:16]


def compose_items_array():
    urls = [
        "https://www.google.com/search?q=ecommerce+platform&gl=us&hl=en",
        "https://www.google.com/search?q=payment+gateway&gl=us&hl=en",
        "https://www.google.com/search?q=stripe&gl=us&hl=en",
        "https://www.google.com/search?q=paddle&gl=us&hl=en",
        "https://www.google.com/search?q=merchant+of+record&gl=us&hl=en",
        "https://www.google.com/search?q=saas+payments&gl=us&hl=en",
        "https://www.google.com/search?q=digital+river&gl=us&hl=en",
        "https://www.google.com/search?q=subscription+billing&gl=us&hl=en",
        "https://www.google.com/search?q=online+payments&gl=us&hl=en",
        "https://www.google.com/search?q=braintree&gl=us&hl=en"
    ]

    # Add the parser configuration to each item
    items = []
    for url in urls:
        items.append({
            "custom_id": create_hash_id(url),
            "url": url,
        })

    return items


def start_batch(items):
    payload = {
        "items": items,
        "parser": {"id": "@olostep/google-search"}
    }
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }
    response = requests.post(f"{API_URL}/batches", headers=headers, json=payload)
    response.raise_for_status()
    return response.json()["id"]


# Step 3: Wait for completion
def check_batch_status(batch_id):
    headers = {"Authorization": f"Bearer {API_KEY}"}
    response = requests.get(
        f"{API_URL}/batches/{batch_id}", 
        headers=headers
    )
    response.raise_for_status()
    return response.json()["status"]


def wait_until_complete(batch_id):
    print("Waiting for batch to complete...")
    while True:
        status = check_batch_status(batch_id)
        print("Status:", status)
        if status == "completed":
            print("Batch completed!")
            return
        time.sleep(10)


# Step 4: Get items
def get_completed_items(batch_id):
    headers = {"Authorization": f"Bearer {API_KEY}"}
    response = requests.get(f"{API_URL}/batches/{batch_id}/items", headers=headers)
    response.raise_for_status()
    return response.json()["items"]


# Step 5: Retrieve content with format specified
def retrieve_content(retrieve_id):
    url = f"{API_URL}/retrieve"
    headers = {"Authorization": f"Bearer {API_KEY}"}
    params = {
        "retrieve_id": retrieve_id,
        "formats": ["markdown", "json"]
    }
    response = requests.get(url, headers=headers, params=params)
    response.raise_for_status()
    return response.json()


# Step 6: Run end-to-end flow
if __name__ == "__main__":
    print("Composing batch...")
    items = compose_items_array()

    print("Starting batch...")
    batch_id = start_batch(items)
    print("Batch ID:", batch_id)
    print(f"You can check the status of the batch at: {API_URL}/batches/{batch_id}?token={API_KEY}")

    wait_until_complete(batch_id)

    print("Fetching completed items...")
    completed_items = get_completed_items(batch_id)

    print("Retrieving content...")
    for item in completed_items:
        retrieve_id = item["retrieve_id"]
        print(f"\nRetrieving content for item {item['retrieve_id']}...")
        content = retrieve_content(retrieve_id)
        print(f"\n---\nURL: {item['url']}\nCustom ID: {item['custom_id']}\n")
        #print("Markdown:\n", content.get("markdown_content", "[No markdown found]"))
        print("JSON:\n", content.get("json_content", "[No JSON found]"))

        # If you want to see the parsed content specifically
        if "parsed_content" in content:
            print("\nParsed Content:")
            print(content["parsed_content"])

```

--------------------------------

### Install langchain-olostep with Poetry

Source: https://docs.olostep.com/integrations/langchain

Install the Olostep LangChain integration using Poetry. Poetry is a dependency management tool for Python.

```bash
poetry add langchain-olostep
```

--------------------------------

### Complete upload example

Source: https://docs.olostep.com/features/files

This example demonstrates the complete process of uploading a JSON file with the purpose set to 'context'. It involves three steps: creating an upload URL, uploading the file data, and completing the upload.

```APIDOC
## File Upload API

This API allows for uploading files in multiple steps.

### Step 1: Create Upload URL

#### POST /files

##### Description
Initiates a file upload by creating a pre-signed URL for uploading the file content. It requires the filename and purpose.

##### Method
POST

##### Endpoint
/files

##### Request Body
- **filename** (string) - Required - The name of the file to be uploaded.
- **purpose** (string) - Required - The purpose of the file (e.g., 'context').

##### Request Example
```json
{
  "filename": "user-data.json",
  "purpose": "context"
}
```

##### Response (200)
- **id** (string) - The unique identifier for the file.
- **upload_url** (string) - The URL to which the file content should be uploaded.

##### Response Example
```json
{
  "id": "file_abc123xyz789",
  "upload_url": "https://example.com/upload/path/to/file"
}
```

### Step 2: Upload File Content

#### PUT [upload_url]

##### Description
Uploads the actual file content to the pre-signed URL obtained in Step 1.

##### Method
PUT

##### Endpoint
[upload_url] (obtained from POST /files response)

##### Request Body
- The file content (e.g., JSON data) to be uploaded.

##### Request Example
```json
{
  "users": [
    {"id": 1, "name": "Alice", "role": "admin"},
    {"id": 2, "name": "Bob", "role": "user"}
  ]
}
```

##### Response (200)
- (No specific response body detailed, typically indicates success of the PUT request)

### Step 3: Complete Upload

#### POST /files/{file_id}/complete

##### Description
Marks the file upload as complete after the content has been successfully uploaded.

##### Method
POST

##### Endpoint
/files/{file_id}/complete

##### Parameters
#### Path Parameters
- **file_id** (string) - Required - The ID of the file being completed.

##### Response (200)
- **id** (string) - The ID of the completed file.
- **bytes** (integer) - The size of the uploaded file in bytes.

##### Response Example
```json
{
  "id": "file_abc123xyz789",
  "bytes": 1024
}
```
```

--------------------------------

### Install Python Requests Library

Source: https://docs.olostep.com/features/maps

Install the requests library for making HTTP requests in Python. This is a prerequisite for using the Olostep API with Python.

```python
# pip install requests

import requests
```

--------------------------------

### Start a Crawl

Source: https://docs.olostep.com/features/crawls

Initiate a website crawl by providing the starting URL and other optional parameters to control the crawl's scope and behavior.

```APIDOC
## POST /v1/crawls

### Description
Crawl a website and get the content from all subpages (or limit the depth of the crawl). Use special patterns to crawl specific pages (e.g. `/blog/**`). Pass a `webhook_url` to get notified when the crawl is completed. Search query to only find specific pages and sort by relevance.

### Method
POST

### Endpoint
https://api.olostep.com/v1/crawls

### Parameters
#### Request Body
- **start_url** (string) - Required - The starting URL for the crawl.
- **max_pages** (integer) - Required - The maximum number of pages to crawl.
- **include_urls** (array of strings) - Optional - URL patterns to include in the crawl.
- **exclude_urls** (array of strings) - Optional - URL patterns to exclude from the crawl.
- **max_depth** (integer) - Optional - The maximum depth of the crawl.
- **include_external** (boolean) - Optional - Whether to include external links.
- **include_subdomain** (boolean) - Optional - Whether to include subdomains.
- **search_query** (string) - Optional - A search query to filter pages.
- **top_n** (integer) - Optional - Sort results by relevance and return top N pages.
- **webhook_url** (string) - Optional - A URL to send notifications to upon completion.
- **timeout** (integer) - Optional - The timeout for the crawl in seconds.

### Request Example
```json
{
  "start_url": "https://sugarbooandco.com",
  "max_pages": 100,
  "include_urls": ["/**"],
  "exclude_urls": ["/collections/**"]
}
```

### Response
#### Success Response (200)
- **id** (string) - The unique identifier for the crawl.
- **status** (string) - The current status of the crawl (e.g., 'pending', 'running', 'completed').
- **pages_count** (integer) - The number of pages crawled so far.

#### Response Example
```json
{
  "id": "crawl_abc123",
  "status": "pending",
  "pages_count": 0
}
```
```

--------------------------------

### Start a Crawl Job

Source: https://docs.olostep.com/features/crawls

Initiate a crawl by providing a starting URL and configuration parameters such as page limits and URL patterns.

```python
import time, json

API_URL = 'https://api.olostep.com'
API_KEY = '<YOUR_API_KEY>'
HEADERS = { 
    'Content-Type': 'application/json',
    'Authorization': f'Bearer {API_KEY}' 
}

data = {
  "start_url": "https://sugarbooandco.com",
  "max_pages": 100,
  "include_urls": ["/**"],
  "exclude_urls": ["/collections/**"],
  "include_external": False
}

res = requests.post(f"{API_URL}/v1/crawls", headers=HEADERS, json=data)
crawl = res.json()
print(json.dumps(crawl, indent=2))
```

```js
const API_URL = 'https://api.olostep.com'
const res = await fetch(`${API_URL}/v1/crawls`, {
  method: 'POST',
  headers: { 'Authorization': 'Bearer <YOUR_API_KEY>', 'Content-Type': 'application/json' },
  body: JSON.stringify({
    start_url: 'https://sugarbooandco.com',
    max_pages: 100,
    include_urls: ['/**'],
    exclude_urls: ['/collections/**']
  })
})
console.log(await res.json())
```

```bash
curl -s -X POST "https://api.olostep.com/v1/crawls" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "start_url": "https://sugarbooandco.com",
    "max_pages": 100,
    "include_urls": ["/**"],
    "exclude_urls": ["/collections/**"]
  }'
```

--------------------------------

### Complete Research Agent Example

Source: https://docs.olostep.com/integrations/mastra

This example demonstrates building a research agent using Mastra and Olostep. It covers initializing integrations, creating an agent, discovering web pages with `createMap`, scraping content with `batchScrape`, and summarizing findings.

```typescript
import { Mastra } from '@mastra/core';
import { Agent } from '@mastra/core';
import { createOlostepIntegration } from '@olostep/mastra-tools';

// Create and register Olostep integration
const olostep = createOlostepIntegration();
olostep.registerApis();

// Initialize Mastra
export const mastra = new Mastra({
  config: {
    integrations: [olostep],
    // ... other config
  },
});

// Create research agent
const researchAgent = new Agent({
  name: 'research-assistant',
  instructions: `
    You are a research assistant that can search, extract, and structure web data.
    When users ask you to research a topic:
    1. Use Olostep's createMap to discover relevant pages
    2. Use batchScrape to extract content from multiple sources
    3. Analyze and summarize the findings
    4. Present structured research reports
  `,
  model: 'openai/gpt-4',
});

// Use the agent
async function researchTopic(topic: string) {
  // Step 1: Discover relevant pages
  const mapResult = await mastra.callApi({
    integrationName: 'olostep',
    api: 'createMap',
    payload: {
      data: {
        apiKey: process.env.OLOSTEP_API_KEY!,
        url: `https://example.com/search?q=${topic}`,
        top_n: 20,
      }
    }
  });

  // Step 2: Scrape discovered pages
  const batchResult = await mastra.callApi({
    integrationName: 'olostep',
    api: 'batchScrape',
    payload: {
      data: {
        apiKey: process.env.OLOSTEP_API_KEY!,
        batch_array: mapResult.urls.slice(0, 10).map(url => ({ url })),
        formats: ['markdown'],
      }
    }
  });

  // Step 3: Analyze with agent
  const summary = await researchAgent.generate({
    messages: [{
      role: 'user',
      content: `Based on this research data, provide a comprehensive summary of ${topic}`
    }]
  });

  return summary;
}
```

--------------------------------

### Complete file upload example

Source: https://docs.olostep.com/features/files

This example demonstrates the complete process of uploading a JSON file with the purpose set to 'context'. It involves creating an upload URL, uploading the data, and then completing the upload.

```python
import requests
import json

API_KEY = "<YOUR_API_KEY>"
API_URL = "https://api.olostep.com/v1"

# Step 1: Create upload URL
create_response = requests.post(
    f"{API_URL}/files",
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
    json={"filename": "user-data.json", "purpose": "context"}
)
upload_data = create_response.json()
file_id = upload_data["id"]
upload_url = upload_data["upload_url"]

# Step 2: Prepare and upload JSON data
json_data = {
    "users": [
        {"id": 1, "name": "Alice", "role": "admin"},
        {"id": 2, "name": "Bob", "role": "user"}
    ]
}

upload_response = requests.put(
    upload_url,
    data=json.dumps(json_data),
    headers={"Content-Type": "application/json"}
)
upload_response.raise_for_status()

# Step 3: Complete the upload
complete_response = requests.post(
    f"{API_URL}/files/{file_id}/complete",
    headers={"Authorization": f"Bearer {API_KEY}"}
)
file_info = complete_response.json()

print(f"File uploaded successfully: {file_info['id']}")
print(f"File size: {file_info['bytes']} bytes")
```

```javascript
const API_URL = 'https://api.olostep.com/v1'

// Step 1: Create upload URL
const createRes = await fetch(`${API_URL}/files`, {
  method: 'POST',
  headers: { 'Authorization': 'Bearer <YOUR_API_KEY>', 'Content-Type': 'application/json' },
  body: JSON.stringify({ filename: 'user-data.json', purpose: 'context' })
})
const uploadData = await createRes.json()
const fileId = uploadData.id
const uploadUrl = uploadData.upload_url

// Step 2: Prepare and upload JSON data
const jsonData = {
  users: [
    { id: 1, name: 'Alice', role: 'admin' },
    { id: 2, name: 'Bob', role: 'user' }
  ]
}

await fetch(uploadUrl, {
  method: 'PUT',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(jsonData)
})

// Step 3: Complete the upload
const completeRes = await fetch(`${API_URL}/files/${fileId}/complete`, {
  method: 'POST',
  headers: { 'Authorization': 'Bearer <YOUR_API_KEY>' }
})
const fileInfo = await completeRes.json()

console.log(`File uploaded successfully: ${fileInfo.id}`)
console.log(`File size: ${fileInfo.bytes} bytes`)
```

--------------------------------

### Start Web Crawl

Source: https://docs.olostep.com/x402

Start a new crawl using the /v1/crawls endpoint. This endpoint uses dynamic pricing based on request parameters.

```bash
curl -X POST 'https://api.olostep.com/x402/v1/crawls' \
  -H 'Content-Type: application/json' \
  -H 'X-Payment: {{paymentHeader}}' \
  -d '{
    "start_url": "example",
    "max_pages": 123,
    "include_urls": "",
    "exclude_urls": "",
    "max_depth": 123,
    "include_external": true,
    "include_subdomain": true,
    "search_query": "example",
    "top_n": 123,
    "webhook_url": "example",
    "timeout": 123
  }'
```

--------------------------------

### Install langchain-olostep with pip

Source: https://docs.olostep.com/integrations/langchain

Install the Olostep LangChain integration using pip. This is the standard method for adding Python packages to your project.

```bash
pip install langchain-olostep
```

--------------------------------

### Install Node Fetch for Node.js

Source: https://docs.olostep.com/features/maps

Install the node-fetch library for making HTTP requests in Node.js. Supports both ESM and CommonJS module systems.

```javascript
// npm install node-fetch

// ESM
import fetch from 'node-fetch'

// CommonJS
const fetch = require('node-fetch')
```

--------------------------------

### Sources List Example

Source: https://docs.olostep.com/features/answers

The list of source URLs provided in the response result.

```json
[
    "https://www.harrypotter.com/writing-by-jk-rowling",
    "https://stories.jkrowling.com/book-news/",
    "https://deadline.com/2024/09/jk-rowling-writing-futuristic-novel-1236093909/",
    "https://www.reddit.com/r/FantasticBeasts/comments/1cl1shn/jk_rowling_may_2024_ive_got_six_more_books_in_my/",
    "https://www.jkrowling.com/news/"
]
```

--------------------------------

### Run CLI without global installation

Source: https://docs.olostep.com/sdks/cli

Execute the CLI directly using npx.

```bash
npx -y olostep-cli@latest --help
```

--------------------------------

### Create Crawl

Source: https://docs.olostep.com/integrations/apify

Follow links and scrape multiple pages from a start URL.

```APIDOC
## POST /crawl

### Description
Follow links and scrape multiple pages from a start URL.

### Request Body
- **operation** (constant) - Required - Must be "crawl"
- **apiKey** (string) - Required - Your Olostep API key
- **start_url** (string) - Required - Starting URL for the crawl
- **max_pages** (integer) - Optional - Max pages to crawl (default: 10)
- **follow_links** (boolean) - Optional - Follow on-page links (default: true)
- **formats** (dropdown) - Optional - One of: Markdown, HTML, JSON, Text (default: markdown)
- **country** (string) - Optional - Optional country code
- **parser** (string) - Optional - Optional parser ID

### Response
- **crawl_id** (object) - Status and details of the crawl
- **status** (object) - Current status
- **start_url** (string) - Starting URL
- **max_pages** (integer) - Max pages configured
- **follow_links** (boolean) - Follow links setting
- **created** (object) - Creation timestamp
- **formats** (object) - Output formats
```

--------------------------------

### List files response structure

Source: https://docs.olostep.com/features/files

Example JSON response for a list files request.

```json
{
  "object": "list",
  "data": [
    {
      "id": "file_abc123xyz789",
      "object": "file",
      "created": 1760329882,
      "filename": "my-data.json",
      "bytes": 1024,
      "purpose": "context",
      "status": "completed"
    }
  ]
}
```

--------------------------------

### GET /v1/schedules/{schedule_id}

Source: https://docs.olostep.com/api-reference/schedules/get

Retrieves a single schedule by its ID. The schedule ID must start with 'schedule_'.

```APIDOC
## GET /v1/schedules/{schedule_id}

### Description
Retrieves a single schedule by its ID.

### Method
GET

### Endpoint
https://api.olostep.com/v1/schedules/{schedule_id}

### Parameters
#### Path Parameters
- **schedule_id** (string) - Required - Unique identifier for the schedule. Must start with 'schedule_'.

### Response
#### Success Response (200)
- **schedule** (object) - The schedule object containing details like team_id, schedule_id, type, endpoint, payload, cron_expression, execute_at, expression_timezone, text, schedule_name, schedule_group, created_at, updated_at, state, and method.

#### Response Example
{
  "schedule": {
    "team_id": "team_123",
    "schedule_id": "schedule_abc",
    "type": "recurring",
    "endpoint": "https://api.example.com/webhook",
    "payload": {},
    "cron_expression": "* * * * *",
    "expression_timezone": "UTC",
    "text": "every minute",
    "schedule_name": "my-schedule",
    "schedule_group": "default",
    "created_at": "2023-01-01T00:00:00Z",
    "updated_at": "2023-01-01T00:00:00Z",
    "state": "ENABLED",
    "method": "POST"
  }
}
```

--------------------------------

### Content Retrieval

Source: https://docs.olostep.com/sdks/python

Retrieve content using a retrieve ID. This example shows how to get content in multiple formats, including HTML, Markdown, text, and JSON.

```python
from olostep import Olostep

client = Olostep(api_key="your-api-key")

# Get content by retrieve ID
result = client.retrieve.get(retrieve_id="ret_123")

# Get multiple formats
result = client.retrieve.get(retrieve_id="ret_123", formats=["html", "markdown", "text", "json"])
```

--------------------------------

### Initialize Sync Client

Source: https://docs.olostep.com/sdks/python

Basic initialization of the synchronous Olostep client.

```python
from olostep import Olostep

# Provide the API key either via passing in the 'api_key' parameter or
# by setting the OLOSTEP_API_KEY environment variable

# The sync client handles resource management automatically
# No explicit close needed - resources are cleaned up after each operation
client = Olostep(api_key="YOUR_REAL_KEY")
scrape_result = client.scrapes.create(url_to_scrape="https://example.com")
```

--------------------------------

### Metadata Schema Example

Source: https://docs.olostep.com/api-reference/batches/create

Example of the metadata object structure for storing additional key-value pairs.

```yaml
order_id: '12345'
customer_name: John Doe
priority: high
processed: 'true'
```

--------------------------------

### Enable Logging

Source: https://docs.olostep.com/sdks/python

Enable logging to debug issues with the Olostep SDK. This example shows how to configure basic logging and set the logger level.

```python
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("olostep")
logger.setLevel(logging.INFO)  # Use DEBUG for verbose output
```

--------------------------------

### Initialize Async Client

Source: https://docs.olostep.com/sdks/python

Initialize the asynchronous client using a context manager for resource cleanup.

```python
from olostep import AsyncOlostep

# Provide the API key either via passing in the 'api_key' parameter or
# by setting the OLOSTEP_API_KEY environment variable

# RESOURCE MANAGEMENT
# ===================
# The SDK supports two usage patterns for resource management:

# 1. Context Manager (Recommended for one-off usage):
#    Automatically handles resource cleanup
async with AsyncOlostep(api_key="YOUR_REAL_KEY") as client:
    scrape_result = await client.scrapes.create(url_to_scrape="https://example.com")
```

--------------------------------

### Initialize Olostep Client

Source: https://docs.olostep.com/sdks/node-js

Configure the client with API credentials, custom base URLs, timeout settings, and retry logic.

```ts
import Olostep from 'olostep';

const client = new Olostep({
  apiKey: 'your_api_key',
  apiBaseUrl: 'https://api.olostep.com/v1',  // optional
  timeoutMs: 150000,  // 150 seconds (optional)
  retry: {
    maxRetries: 3,
    initialDelayMs: 1000
  },
  userAgent: 'MyApp/1.0'  // optional
});
```

```ts
import Olostep from 'olostep';

const client = new Olostep({
  api_key: 'your_api_key',
  api_base_url: 'https://api.olostep.com/v1',  // optional
  timeout_ms: 150000,  // 150 seconds (optional)
  retry: {
    max_retries: 3,
    initial_delay_ms: 1000
  },
  user_agent: 'MyApp/1.0'  // optional
});
```

--------------------------------

### Site Mapping with AsyncOlostep

Source: https://docs.olostep.com/sdks/python

Extract all discoverable URLs from a website using `maps.create`. Iterate through the returned URLs.

```python
import asyncio
from olostep import AsyncOlostep

async def main():
    async with AsyncOlostep(api_key="your-api-key") as client:
        # Extract all links from a website
        maps = await client.maps.create(url="https://example.com")

        # Get all discovered URLs
        urls = []
        async for url in maps.urls():
            urls.append(url)
            if len(urls) >= 10:  # Limit for demo
                break

        print(f"Found {len(urls)} URLs")

asyncio.run(main())
```

--------------------------------

### Example Markdown Output Structure

Source: https://docs.olostep.com/examples/crawl-matching-pages

This is an example of the markdown output generated by the script. It includes the URL, title, and content for each crawled page, separated by '---'.

```markdown
URL: https://stripe.com/blog/using-ml-to-detect-and-respond-to-performance-degradations

## Using ML to detect and respond to performance degradations

By Jane Smith, Senior Engineer at Stripe

At Stripe, we process millions of API requests every day...

---

URL: https://stripe.com/blog/building-robust-payment-systems

## Building a robust payment system

By John Doe, Engineering Manager

Reliability is at the core of Stripe's infrastructure...

---
```

--------------------------------

### GET /websites/olostep/batch_items

Source: https://docs.olostep.com/api-reference/batches/items

Retrieves the list of items processed for a batch. The `retrieve_id` can be used with the Retrieve Endpoint to get the content. Metadata, if provided during batch creation, is included.

```APIDOC
## GET /websites/olostep/batch_items

### Description
Retrieves the list of items processed for a batch. You can then use the `retrieve_id` to get the content with the Retrieve Endpoint.

### Method
GET

### Endpoint
/websites/olostep/batch_items

### Parameters
#### Query Parameters
- **batch_id** (string) - Required - The ID of the batch to retrieve items from.

### Response
#### Success Response (200)
- **items** (array) - A list of processed items, each containing a `retrieve_id` and optional `metadata`.
  - **retrieve_id** (string) - The ID to use with the Retrieve Endpoint.
  - **metadata** (object) - Optional metadata attached to the item during batch creation.

#### Response Example
```json
{
  "items": [
    {
      "retrieve_id": "item_abc123",
      "metadata": {
        "key1": "value1"
      }
    },
    {
      "retrieve_id": "item_def456"
    }
  ]
}
```
```

--------------------------------

### Get Answer OpenAPI Specification

Source: https://docs.olostep.com/api-reference/answers/get

Defines the GET /v1/answers/{answer_id} endpoint, including the required answer_id path parameter and the expected JSON response structure.

```yaml
openapi: 3.0.3
info:
  title: Answers API
  version: 1.0.0
servers:
  - url: https://api.olostep.com
security: []
paths:
  /v1/answers/{answer_id}:
    get:
      summary: Get Answer
      description: This endpoint retrieves a previously completed answer by its ID.
      parameters:
        - name: answer_id
          in: path
          required: true
          description: Unique identifier for the answer to be retrieved.
          schema:
            type: string
      responses:
        '200':
          description: Successful response with the answer object.
          content:
            application/json:
              schema:
                type: object
                properties:
                  id:
                    type: string
                  object:
                    type: string
                  created:
                    type: integer
                  metadata:
                    type: object
                  task:
                    type: string
                  result:
                    type: object
                    properties:
                      json_content:
                        type: string
                      json_hosted_url:
                        type: string
                      sources:
                        type: array
                        items:
                          type: string
        '402':
          description: Invalid API Key
        '404':
          description: Answer Not Found
        '500':
          description: Internal Server Error
      security:
        - Authorization: []
components:
  securitySchemes:
    Authorization:
      type: http
      scheme: bearer
      description: >-
        Bearer authentication header of the form Bearer <token>, where <token>
        is your auth token.
```

--------------------------------

### Example Olostep API Response

Source: https://docs.olostep.com/examples

This is an example of a successful JSON response from the Olostep API after requesting markdown content. It includes details about the scrape job and the extracted content, if available.

```json
{
    "id": "scrape_63x2e5sf5r",
    "object": "scrape",
    "created": 1740341743,
    "metadata": {},
    "retrieve_id": "63x2e5sf5r",
    "url_to_scrape": "https://www.nea.com/team",
    "result": {
        "html_content": null,
        "markdown_content": "NEA ….",
        "text_content": null,
        "json_content": null,
        "llm_extract": null,
        "screenshot_hosted_url": null,
        "html_hosted_url": null,
        "markdown_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/markDown_63x2e5sf5r.txt",
        "json_hosted_url": null,
        "text_hosted_url": null,
        "links_on_page": [],
        "page_metadata": {
            "status_code": 200,
            "title": ""
        }
    }
}

```

--------------------------------

### OpenAPI Specification for Get Map

Source: https://docs.olostep.com/api-reference/maps/get

This OpenAPI 3.0.3 specification defines the GET /v1/maps/{map_id} endpoint for retrieving map data. It includes details on parameters, responses, and security schemes.

```yaml
openapi: 3.0.3
info:
  title: Maps API
  version: 1.0.0
servers:
  - url: https://api.olostep.com
security: []
paths:
  /v1/maps/{map_id}:
    get:
      summary: Get Map
      description: Retrieve a previously completed map by its ID.
      parameters:
        - name: map_id
          in: path
          required: true
          description: Unique identifier for the map to be retrieved.
          schema:
            type: string
      responses:
        '200':
          description: Successful response with the map object.
          content:
            application/json:
              schema:
                type: object
                properties:
                  id:
                    type: string
                    description: Unique identifier for this map
                  urls_count:
                    type: integer
                    description: Number of URLs in the current response
                  urls:
                    type: array
                    items:
                      type: string
                    description: Array of URLs found on the page
                  cursor:
                    type: string
                    nullable: true
                    description: >-
                      Pagination cursor to retrieve the next set of URLs. If
                      null, all URLs have been retrieved.
        '401':
          description: Invalid API Key
        '404':
          description: Map Not Found
        '500':
          description: Internal Server Error
      security:
        - Authorization: []
components:
  securitySchemes:
    Authorization:
      type: http
      scheme: bearer
      description: >-
        Bearer authentication header of the form Bearer <token>, where <token>
        is your auth token.

```

--------------------------------

### Verify Olostep Installation with Kilo

Source: https://docs.olostep.com/integrations/kilo

Run the Kilo command in your terminal to verify that the Olostep tools are available after configuration.

```bash
kilo
```

--------------------------------

### Claude Desktop Configuration

Source: https://docs.olostep.com/integrations/mcp-server

Configuration for integrating the Olostep MCP server into Claude Desktop.

```bash
{
  "mcpServers": {
    "mcp-server-olostep": {
      "command": "npx",
      "args": ["-y", "olostep-mcp"],
      "env": {
        "OLOSTEP_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}
```

```bash
npx -y @smithery/cli install @olostep/olostep-mcp-server --client claude
```

--------------------------------

### Example Request to Check Supported Countries

Source: https://docs.olostep.com/examples/geo

This `curl` command demonstrates how to request the supported countries for the Perplexity parser. Ensure the `parser` parameter is correctly set.

```bash
curl "https://api.olostep.com/v1/countries?service=batches&parser=@olostep/perplexity-results"
```

--------------------------------

### Start a Website Crawl

Source: https://docs.olostep.com/api-reference/crawls/create

Initiates a web crawl for a given website. You can specify various parameters to control the crawl process, including the parser to use, start URL, maximum pages, and exclusion/inclusion rules.

```APIDOC
## POST /websites/olostep

### Description
Starts a web crawl for a given website. This endpoint allows for detailed configuration of the crawl process, including the parser to be used for extracting structured data from each page.

### Method
POST

### Endpoint
/websites/olostep

### Parameters
#### Request Body
- **start_url** (string) - Required - The starting URL for the crawl.
- **max_pages** (number) - Required - The maximum number of pages to crawl.
- **parser_name** (string) - Optional - The name of the parser to run on each page (e.g., "@olostep/extract-emails"). Automatically adds `json` to `formats` when set.
- **max_depth** (number) - Optional - The maximum depth of the crawl.
- **exclude_urls** (array) - Optional - A list of URLs to exclude from the crawl.
- **include_urls** (array) - Optional - A list of URLs to include in the crawl.
- **include_external** (boolean) - Optional - Whether to include external links.
- **search_query** (string) - Optional - A search query to apply during crawling.
- **top_n** (number) - Optional - The top N results to consider.
- **webhook** (string) - Optional - A webhook URL to send notifications to.
- **follow_robots_txt** (boolean) - Optional - Whether to follow the robots.txt file.

### Request Example
```json
{
  "start_url": "https://example.com",
  "max_pages": 100,
  "parser_name": "@olostep/extract-emails"
}
```

### Response
#### Success Response (200)
- **id** (string) - Crawl ID.
- **object** (string) - The kind of object. "crawl" for this endpoint.
- **status** (string) - Crawl status (`in_progress` or `completed`).
- **created** (number) - Created time in epoch.
- **start_date** (string) - Created time in date format.
- **start_url** (string) - The starting URL of the crawl.
- **max_pages** (number) - The maximum number of pages crawled.
- **max_depth** (number) - The maximum depth of the crawl.
- **exclude_urls** (array) - List of excluded URLs.
- **include_urls** (array) - List of included URLs.
- **include_external** (boolean) - Whether external links were included.
- **search_query** (string) - The search query used.
- **top_n** (number) - The top N results considered.
- **current_depth** (number) - The current depth of the crawl process.
- **pages_count** (number) - Count of pages crawled.
- **webhook** (string) - The webhook URL.
- **follow_robots_txt** (boolean) - Whether robots.txt was followed.

#### Response Example
```json
{
  "id": "crawl-12345",
  "object": "crawl",
  "status": "in_progress",
  "created": 1678886400,
  "start_date": "2023-03-15T10:00:00Z",
  "start_url": "https://example.com",
  "max_pages": 100,
  "max_depth": 5,
  "exclude_urls": [],
  "include_urls": [],
  "include_external": false,
  "search_query": null,
  "top_n": null,
  "current_depth": 1,
  "pages_count": 10,
  "webhook": null,
  "follow_robots_txt": true
}
```

### Error Handling
- **400 Bad Request**: Returned due to incorrect or missing parameters.
- **500 Internal Server Error**: Returned in case of server-side errors.
```

--------------------------------

### Configure Kilo with Olostep MCP Server (Local/Stdio)

Source: https://docs.olostep.com/integrations/kilo

Add the Olostep MCP Server configuration to your Kilo project's .kilocode/mcp.json file for local or stdio hosting. Replace YOUR_OLOSTEP_API_KEY with your actual API key.

```json
{
  "mcpServers": {
    "olostep": {
      "command": "npx",
      "args": ["-y", "olostep-mcp"],
      "env": {
        "OLOSTEP_API_KEY": "YOUR_OLOSTEP_API_KEY"
      }
    }
  }
}
```

--------------------------------

### Get Schedule API

Source: https://docs.olostep.com/features/schedules

Retrieve a single schedule by its ID.

```APIDOC
## Get a Schedule

### Description
Retrieve a single schedule by its ID.

### Method
GET

### Endpoint
/schedules/{schedule_id}

### Parameters
#### Path Parameters
- **schedule_id** (string) - Required - The ID of the schedule to retrieve.

### Request Example (Python)
```python
import requests
import json

API_KEY = "<YOUR_API_KEY>"
API_URL = "https://api.olostep.com/v1"
schedule_id = "schedule_abc123xyz"

headers = {
    "Authorization": f"Bearer {API_KEY}"
}

response = requests.get(f"{API_URL}/schedules/{schedule_id}", headers=headers)
print(json.dumps(response.json(), indent=2))
```

### Request Example (Node.js)
```javascript
const API_URL = 'https://api.olostep.com/v1'
const scheduleId = 'schedule_abc123xyz'

const res = await fetch(`${API_URL}/schedules/${scheduleId}`, {
  headers: { 'Authorization': 'Bearer <YOUR_API_KEY>' }
})
console.log(await res.json())
```

### Request Example (cURL)
```bash
curl -s -X GET "https://api.olostep.com/v1/schedules/schedule_abc123xyz" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY"
```
```

--------------------------------

### Start a Batch Request with Python

Source: https://docs.olostep.com/examples/price-monitoring

Sends a batch of URLs to the Olostep API for processing. Requires an API key and supports optional custom parsers to return structured JSON data.

```python
import requests

def start_batch(batch_array):
    payload = {
        "batch_array": batch_array,                 # Array of items to process
        "batch_country": "IT",                      # Country code for the batch
        "parser": "@olostep/amazon-it-product"      # Optional: Specify a custom parser so you only get the JSON data you need
    }

    headers = {
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json"
    }

    response = requests.post(
        "https://api.olostep.com/v1/batches",
        headers=headers,
        json=payload
    )

    return response.json()["id"]
```

--------------------------------

### GET /v1/files/{file_id}

Source: https://docs.olostep.com/api-reference/files/get

Retrieve metadata for a file by its ID.

```APIDOC
## GET /v1/files/{file_id}

### Description
Retrieve metadata for a file by its ID.

### Method
GET

### Endpoint
/v1/files/{file_id}

### Parameters
#### Path Parameters
- **file_id** (string) - Required - Unique identifier for the file to retrieve.

### Request Example
(No request body for GET requests)

### Response
#### Success Response (200)
- **id** (string) - File ID
- **object** (string) - The kind of object. "file" for this endpoint.
- **created** (integer) - Created epoch timestamp
- **filename** (string) - The filename of the uploaded file
- **bytes** (integer) - File size in bytes
- **purpose** (string) - The purpose of the file
- **status** (string) - File status (e.g., 'pending', 'completed')

#### Response Example
```json
{
  "id": "file-abc123xyz",
  "object": "file",
  "created": 1678886400,
  "filename": "example.txt",
  "bytes": 1024,
  "purpose": "fine-tune",
  "status": "completed"
}
```

#### Error Responses
- **401** - Invalid API key
- **404** - File not found
- **500** - Internal server error
```
Tokens:
Raw