ScrapeGraphAI

https://docs.scrapegraphai.com/
Admin
ScrapeGraphAI is a suite of AI-powered, LLM-driven web scraping tools designed to extract structured...
Tokens:170,440
Snippets:1,559
Trust Score:8.3
Update:2 months ago
Show doc for...
Tokens:
Raw
### SmartScraper Quick Start

Source: https://docs.scrapegraphai.com/services/smartscraper

Examples of how to use the SmartScraper service in Python, JavaScript, cURL, and CLI.

```APIDOC
## SmartScraper Quick Start

### Description
Examples of how to use the SmartScraper service in Python, JavaScript, cURL, and CLI.

### Python
```python
from scrapegraph_py import Client

client = Client(api_key="your-api-key")

response = client.smartscraper(
    website_url="https://scrapegraphai.com/",
    user_prompt="Extract info about the company"
)
```

### JavaScript
```javascript
import { smartScraper } from 'scrapegraph-js';

const apiKey = 'your-api-key';

const response = await smartScraper(apiKey, {
  website_url: 'https://scrapegraphai.com',
  user_prompt: 'What does the company do?',
});

if (response.status === 'error') {
  console.error(response.error);
} else {
  console.log(response.data);
}
```

### cURL
```bash
curl -X 'POST' \
  'https://api.scrapegraphai.com/v1/smartscraper' \
  -H 'accept: application/json' \
  -H 'SGAI-APIKEY: your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{ 
    "website_url": "https://example-shop.com/products/456",
    "user_prompt": "Extract product details including name, price, availability, specifications, and customer ratings."
  }'
```

### CLI
```bash
just-scrape smart-scraper https://scrapegraphai.com/ -p "Extract info about the company"
```
```

--------------------------------

### Start Crawl cURL Example

Source: https://docs.scrapegraphai.com/api-reference/endpoint/crawl/start

Example request using cURL to start a crawl with specific configuration parameters.

```bash
curl -X POST https://v2-api.scrapegraphai.com/api/crawl \
  -H "SGAI-APIKEY: $SGAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scrapegraphai.com/",
    "formats": [{ "type": "markdown" }],
    "maxPages": 5,
    "maxDepth": 2,
    "includePatterns": ["/blog/*"],
    "excludePatterns": ["/admin/*"]
  }'
```

--------------------------------

### Crawl Quick Start

Source: https://docs.scrapegraphai.com/services/crawl

Examples for initiating and polling a crawl job across Python, JavaScript, and cURL.

```python
import time
from scrapegraph_py import ScrapeGraphAI, MarkdownFormatConfig

# reads SGAI_API_KEY from env, or pass explicitly: ScrapeGraphAI(api_key="...")
sgai = ScrapeGraphAI()

start = sgai.crawl.start(
    "https://scrapegraphai.com/",
    formats=[MarkdownFormatConfig()],
    max_pages=5,
    max_depth=2,
)

if start.status != "success":
    print("Failed:", start.error)
else:
    crawl_id = start.data.id
    print("Crawl started:", crawl_id)

    while True:
        time.sleep(2)
        status = sgai.crawl.get(crawl_id)
        if status.status != "success":
            break
        print(f"{status.data.finished}/{status.data.total} - {status.data.status}")
        if status.data.status in ("completed", "failed"):
            for page in status.data.pages:
                print(f"  {page.url} - {page.status}")
            break
```

```javascript
import { ScrapeGraphAI } from "scrapegraph-js";

const sgai = ScrapeGraphAI();

const start = await sgai.crawl.start({
  url: "https://scrapegraphai.com/",
  formats: [{ type: "markdown" }],
  maxPages: 5,
  maxDepth: 2,
});

if (start.status !== "success" || !start.data) {
  console.error("Failed:", start.error);
} else {
  const crawlId = start.data.id;
  console.log("Crawl started:", crawlId);

  while (true) {
    await new Promise((r) => setTimeout(r, 2000));
    const status = await sgai.crawl.get(crawlId);
    if (status.status !== "success" || !status.data) break;
    console.log(`${status.data.finished}/${status.data.total} - ${status.data.status}`);
    if (status.data.status === "completed" || status.data.status === "failed") {
      for (const p of status.data.pages) console.log(`  ${p.url} - ${p.status}`);
      break;
    }
  }
}
```

```bash
# Start a crawl
curl -X POST https://v2-api.scrapegraphai.com/api/crawl \
  -H "SGAI-APIKEY: $SGAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scrapegraphai.com/",
    "formats": [{ "type": "markdown" }],
    "maxPages": 5,
    "maxDepth": 2
  }'

# Check status (replace :id with the crawl id returned above)
curl -X GET https://v2-api.scrapegraphai.com/api/crawl/:id \
  -H "SGAI-APIKEY: $SGAI_API_KEY"

# Fetch pages with resolved scrape results
curl -X GET "https://v2-api.scrapegraphai.com/api/crawl/:id/pages?limit=50&cursor=0" \
  -H "SGAI-APIKEY: $SGAI_API_KEY"
```

--------------------------------

### Quick Start Scrape Requests

Source: https://docs.scrapegraphai.com/services/scrape

Basic implementation examples for fetching markdown content from a URL using Python, JavaScript, and cURL.

```python
from scrapegraph_py import ScrapeGraphAI, MarkdownFormatConfig

# reads SGAI_API_KEY from env, or pass explicitly: ScrapeGraphAI(api_key="...")
sgai = ScrapeGraphAI()

res = sgai.scrape(
    "https://example.com",
    formats=[MarkdownFormatConfig()],
)

if res.status == "success":
    md = res.data.results.get("markdown", {}).get("data", [])
    print(md[0] if md else None)
else:
    print("Failed:", res.error)
```

```javascript
import { ScrapeGraphAI } from "scrapegraph-js";

// reads SGAI_API_KEY from env, or pass explicitly: ScrapeGraphAI({ apiKey: "..." })
const sgai = ScrapeGraphAI();

const res = await sgai.scrape({
  url: "https://example.com",
  formats: [{ type: "markdown" }],
});

if (res.status === "success") {
  console.log(res.data?.results.markdown?.data?.[0]);
} else {
  console.error(res.error);
}
```

```bash
curl -X POST https://v2-api.scrapegraphai.com/api/scrape \
  -H "SGAI-APIKEY: $SGAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": [{ "type": "markdown" }]
  }'
```

--------------------------------

### Quick Start

Source: https://docs.scrapegraphai.com/sdks/python

A basic example demonstrating how to initialize the ScrapeGraphAI client and perform a scrape operation. The API key is read from the environment variable SGAI_API_KEY by default.

```python
from scrapegraph_py import ScrapeGraphAI

sgai = ScrapeGraphAI()

result = sgai.scrape("https://example.com")

if result.status == "success":
    print(result.data.results["markdown"]["data"])
else:
    print(result.error)
```

--------------------------------

### Install just-scrape Skill

Source: https://docs.scrapegraphai.com/integrations/hermes

Install the skill using the Hermes registry or the skills.sh CLI.

```bash
hermes skills install skills-sh/scrapegraphai/just-scrape/just-scrape
```

```bash
npx skills add https://github.com/scrapegraphai/just-scrape --skill just-scrape
```

--------------------------------

### Install Python SDK

Source: https://docs.scrapegraphai.com/contribute/opensource

Install the Python SDK for convenient API integration. This is a recommended way to use the service.

```bash
pip install scrapegraph-py
```

--------------------------------

### Install JavaScript SDK

Source: https://docs.scrapegraphai.com/contribute/opensource

Install the JavaScript SDK for convenient API integration. This is a recommended way to use the service.

```bash
npm install scrapegraph-js
```

--------------------------------

### SmartCrawler Quick Start Examples

Source: https://docs.scrapegraphai.com/services/smartcrawler

Initiate a crawl with AI-powered extraction. Specify URL, prompt, depth, and rules for path inclusion/exclusion. Ensure your API key is set.

```python
from scrapegraph_py import Client

client = Client(api_key="your-api-key")

response = client.crawl(
    url="https://scrapegraphai.com/",
    prompt="Extract info about the company",
    depth=2,
    breadth=None,
    max_pages=10,
    rules={
        "include_paths": ["/about", "/services/*"],
        "exclude_paths": ["/admin/*", "/api/**"],
        "same_domain": True
    }
)
```

```javascript
import { crawl } from 'scrapegraph-js';

const apiKey = 'your-api-key';

const response = await crawl(apiKey, {
  url: 'https://scrapegraphai.com/',
  prompt: 'Extract info about the company',
  depth: 2,
  max_pages: 10,
  rules: {
    include_paths: ['/about', '/services/*'],
    exclude_paths: ['/admin/*', '/api/**'],
    same_domain: true
  }
});

if (response.status === 'success') {
  console.log(response.data);
}
```

```bash
curl -X 'POST' \
  'https://api.scrapegraphai.com/v1/crawl' \
  -H 'accept: application/json' \
  -H 'SGAI-APIKEY: your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{ \
    "url": "https://scrapegraphai.com/", \
    "prompt": "Extract info about the company", \
    "depth": 2, \
    "breadth": null, \
    "max_pages": 10, \
    "rules": { \
      "include_paths": ["/about", "/services/*"], \
      "exclude_paths": ["/admin/*", "/api/**"], \
      "same_domain": true \
    } \
  }'
```

```bash
just-scrape crawl https://scrapegraphai.com/ -p "Extract info about the company" \
  --depth 2 --max-pages 10 \
  --rules '{"include_paths":["/about","/services/*"],"exclude_paths":["/admin/*","/api/**"],"same_domain":true}'
```

--------------------------------

### Install x402 Dependencies

Source: https://docs.scrapegraphai.com/integrations/orthogonal

Install the necessary libraries for using the x402 gateway with Node.js or Python.

```bash
# Node.js
npm install x402-fetch viem

# Python
pip install x402 eth-account
```

--------------------------------

### Local ScrapeGraph MCP Installation and Setup

Source: https://docs.scrapegraphai.com/services/mcp-server/claude

Install and configure ScrapeGraph MCP for local use with Claude Desktop. Set the SGAI_API_KEY environment variable before running the server.

```bash
pip install scrapegraph-mcp
export SGAI_API_KEY=your-api-key
```

```json
{
  "mcpServers": {
    "scrapegraph-mcp-local": {
      "command": "python3",
      "args": ["-m", "scrapegraph_mcp.server"],
      "env": { "SGAI_API_KEY": "YOUR_API_KEY" }
    }
  }
}
```

--------------------------------

### Python FetchConfig Example

Source: https://docs.scrapegraphai.com/services/additional-parameters/proxy

Example of initializing a `FetchConfig` object in Python with various parameters set.

```Python
from scrapegraph_py import FetchConfig

config = FetchConfig(
    mode="js",             # Fetch mode
    stealth=True,          # Stealth proxy
    timeout=15000,         # 15s timeout
    wait=2000,             # Wait 2s after page load
    scrolls=3,             # Scroll 3 times
    country="us",          # Route through US proxies
    headers={"Accept-Language": "en-US"},
    cookies={"session": "abc123"},
    mock=False,
)
```

--------------------------------

### Install Hermes Agent

Source: https://docs.scrapegraphai.com/integrations/hermes

One-liner installation script for macOS, Linux, or WSL2.

```bash
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
```

--------------------------------

### Install just-scrape CLI

Source: https://docs.scrapegraphai.com/knowledge-base/cli/getting-started

Install the just-scrape CLI globally using npm, pnpm, yarn, or bun. Alternatively, use npx or bunx to run commands without a global installation.

```bash
npm install -g just-scrape
```

```bash
pnpm add -g just-scrape
```

```bash
yarn global add just-scrape
```

```bash
bun add -g just-scrape
```

```bash
npx just-scrape --help
```

```bash
bunx just-scrape --help
```

--------------------------------

### Install LiteLLM with proxy support

Source: https://docs.scrapegraphai.com/integrations/litellm

Install the LiteLLM package including the necessary proxy dependencies.

```bash
pip install 'litellm[proxy]'
```

--------------------------------

### JavaScript FetchConfig Example

Source: https://docs.scrapegraphai.com/services/additional-parameters/proxy

Example of creating a `FetchConfig` object in JavaScript with custom proxy and fetch settings.

```JavaScript
const fetchConfig = {
  mode: 'js',
  stealth: true,
  timeout: 15000,
  wait: 2000,
  scrolls: 3,
  country: 'us',
  headers: { 'Accept-Language': 'en-US' },
  cookies: { session: 'abc123' },
  mock: false,
};
```

--------------------------------

### Initialize Crawl Tools

Source: https://docs.scrapegraphai.com/integrations/vercel_ai

Registers a suite of crawling tools including start, get, stop, and resume functionality.

```ts
import { crawlTools } from "@scrapegraph-ai/ai-sdk";

const tools = {
  ...crawlTools(),
};
```

--------------------------------

### Get Monitor Details

Source: https://docs.scrapegraphai.com/api-reference/endpoint/monitor/manage

Fetch the complete configuration and details for a specific monitor using its cronId. This provides a full view of the monitor's setup.

```http
GET https://v2-api.scrapegraphai.com/api/monitor/:cronId
```

```bash
curl -X GET https://v2-api.scrapegraphai.com/api/monitor/d9a09a07-... \
  -H "SGAI-APIKEY: $SGAI_API_KEY"
```

--------------------------------

### Verify CLI setup

Source: https://docs.scrapegraphai.com/services/cli/introduction

Check the current API credit balance.

```bash
just-scrape credits    # check your credit balance
```

--------------------------------

### Install and Log in with Orthogonal CLI

Source: https://docs.scrapegraphai.com/integrations/orthogonal

Install the Orthogonal CLI globally for command-line access. Log in using your browser or by explicitly providing your API key.

```bash
npm install -g @orth/cli

# Option 1: browser-based login (recommended)
orth login

# Option 2: pass the key explicitly
export ORTHOGONAL_API_KEY="orth_live_xxxxxxxxxxxx"
```

--------------------------------

### Install OpenClaw

Source: https://docs.scrapegraphai.com/integrations/openclaw

Installs the OpenClaw daemon on macOS or Linux systems.

```bash
curl -fsSL https://openclaw.ai/install.sh | bash
```

--------------------------------

### Install dependencies

Source: https://docs.scrapegraphai.com/developer-guides/llm-sdks-and-frameworks/anthropic

Install the required npm packages for ScrapeGraphAI, Anthropic, and Zod.

```bash
npm install scrapegraph-js @anthropic-ai/sdk zod
```

--------------------------------

### Install and Add Local MCP Server

Source: https://docs.scrapegraphai.com/services/mcp-server/codex

Install the local package and register the stdio-based MCP server in Codex.

```bash
pip install scrapegraph-mcp
codex mcp add scrapegraph-mcp-local \
  --env SGAI_API_KEY=YOUR_API_KEY \
  -- python3 -m scrapegraph_mcp.server
```

--------------------------------

### Install just-scrape CLI

Source: https://docs.scrapegraphai.com/services/cli

Install the `just-scrape` CLI globally using your preferred package manager.

```bash
npm install -g just-scrape
```

```bash
pnpm add -g just-scrape
```

```bash
yarn global add just-scrape
```

```bash
bun add -g just-scrape
```

--------------------------------

### Install ScrapeGraphAI Core Package

Source: https://docs.scrapegraphai.com/contribute/opensource

Install the core library that powers the API service. This is recommended for most use cases.

```bash
pip install scrapegraphai
```

--------------------------------

### Install ScrapeGraphAI and OpenAI dependencies

Source: https://docs.scrapegraphai.com/developer-guides/llm-sdks-and-frameworks/openai

Install the necessary packages via npm.

```bash
npm install scrapegraph-js openai
```

--------------------------------

### Example Monitor Creation Response

Source: https://docs.scrapegraphai.com/api-reference/endpoint/monitor/create

This is an example of a successful response when creating a monitor. It includes a unique `cronId` for managing the monitor, a `scheduleId`, the configured `interval`, the `status` of the monitor, and a normalized `config` object reflecting the request details.

```json
{
  "cronId": "f0b171db-b288-4380-9cad-1bcac916348e",
  "scheduleId": "scd_4u8PPLytsJ2niWBVEAPkLHkfjVge",
  "interval": "*/30 * * * *",
  "status": "active",
  "config": {
    "url": "https://example.com",
    "name": "Doc test monitor",
    "formats": [{ "mode": "normal", "type": "markdown" }],
    "interval": "*/30 * * * *"
  },
  "createdAt": "2026-04-23T11:11:37.487Z",
  "updatedAt": "2026-04-23T11:11:37.487Z"
}
```

--------------------------------

### Example cURL Request to Create Monitor

Source: https://docs.scrapegraphai.com/api-reference/endpoint/monitor/create

This example demonstrates how to create a monitor using cURL. It includes the necessary headers for API key authentication and content type, along with a JSON payload specifying the URL, monitor name, fetch interval, and desired format.

```bash
curl -X POST https://v2-api.scrapegraphai.com/api/monitor \
  -H "SGAI-APIKEY: $SGAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "name": "Homepage watch",
    "interval": "*/30 * * * *",
    "formats": [{ "type": "markdown" }]
  }'
```

--------------------------------

### Install just-scrape Skill

Source: https://docs.scrapegraphai.com/knowledge-base/cli/ai-agent-skill

Install the just-scrape skill for AI agents using bunx. This command adds the skill to your agent's available tools.

```bash
bunx skills add https://github.com/ScrapeGraphAI/just-scrape
```

--------------------------------

### Run just-scrape CLI without installation

Source: https://docs.scrapegraphai.com/services/cli

Execute `just-scrape` commands directly using `npx` or `bunx` without a global installation.

```bash
npx just-scrape --help
```

```bash
bunx just-scrape --help
```

--------------------------------

### Install LangGraph and ScrapeGraph Dependencies

Source: https://docs.scrapegraphai.com/integrations/langgraph

Install the necessary libraries for LangGraph, LangChain, OpenAI, and ScrapeGraph. This command ensures all required packages are available for integration.

```bash
pip install langchain langchain-openai langgraph scrapegraph-py
```

--------------------------------

### Python SDK - Asynchronous Pagination Example

Source: https://docs.scrapegraphai.com/cookbook/examples/pagination

This asynchronous example fetches paginated URLs in parallel for improved performance. It utilizes `AsyncScrapeGraphAI` and `asyncio.gather` to manage concurrent requests.

```python
#!/usr/bin/env python3
"""
Extract Pagination Example (Async)

Fetch every page in parallel with AsyncScrapeGraphAI.
"""

import asyncio
import json
import time
from typing import List, Optional

from dotenv import load_dotenv
from pydantic import BaseModel

from scrapegraph_py import AsyncScrapeGraphAI

load_dotenv()


class ProductInfo(BaseModel):
    name: str
    price: Optional[str] = None
    rating: Optional[str] = None
    image_url: Optional[str] = None
    description: Optional[str] = None


class ProductList(BaseModel):
    products: List[ProductInfo]


async def main():
    base_url = "https://www.amazon.in/s?k=tv"
    prompt = "Extract all product info including name, price, rating, image_url, and description"
    schema = ProductList.model_json_schema()
    urls = [f"{base_url}&page={i}" for i in range(1, 4)]

    start = time.time()
    async with AsyncScrapeGraphAI() as sgai:
        tasks = [sgai.extract(prompt, url=u, schema=schema) for u in urls]
        results = await asyncio.gather(*tasks)

    all_products: list[dict] = []
    for url, res in zip(urls, results):
        if res.status != "success":
            print(f"Page failed: {url} - {res.error}")
            continue
        products = (res.data.json_data or {}).get("products", [])
        print(f"{url} -> {len(products)} products ({res.elapsed_ms}ms)")
        all_products.extend(products)

    print(f"\nDone in {time.time() - start:.1f}s. Total: {len(all_products)} products")
    print(json.dumps(all_products[:3], indent=2, ensure_ascii=False))


if __name__ == "__main__":
    asyncio.run(main())

```

--------------------------------

### Start LiteLLM Proxy

Source: https://docs.scrapegraphai.com/integrations/litellm

Set the required API keys and launch the LiteLLM proxy using the configuration file.

```bash
export OPENAI_API_KEY="your-openai-key"
export SGAI_API_KEY="your-scrapegraph-key"

litellm --config config.yaml
```

--------------------------------

### Python Quick Start for Markdownify

Source: https://docs.scrapegraphai.com/services/markdownify

Initialize the client and call the markdownify method with the website URL.

```python
from scrapegraph_py import Client

client = Client(api_key="your-api-key")

response = client.markdownify(
    website_url="https://example.com/article"
)
```

--------------------------------

### Discovering and Inspecting Endpoints (API)

Source: https://docs.scrapegraphai.com/integrations/orthogonal

Examples of how to use the Orthogonal API to search for endpoints, get their details, and generate integration code.

```APIDOC
## Discovering and inspecting endpoints
```bash
# Natural-language search
curl -X POST 'https://api.orthogonal.com/v1/search' \
  -H "Authorization: Bearer $ORTHOGONAL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "prompt": "extract structured data from a webpage", "limit": 5 }'

# Full parameter schema for a specific endpoint
curl -X POST 'https://api.orthogonal.com/v1/details' \
  -H "Authorization: Bearer $ORTHOGONAL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "api": "scrapegraph", "path": "/v1/extract" }'

# Code snippets in any supported format
curl -X POST 'https://api.orthogonal.com/v1/integrate' \
  -H "Authorization: Bearer $ORTHOGONAL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "api": "scrapegraph", "path": "/v1/extract", "format": "all" }'
```

The `format` field on `/v1/integrate` accepts `orth-sdk`, `run-api`, `curl`, `x402-fetch`, `x402-python`, or `all`.
```

--------------------------------

### SearchScraper - AI Extraction Mode

Source: https://docs.scrapegraphai.com/services/searchscraper

This example demonstrates how to use the searchscraper function to perform an AI-powered search and extract structured data. Set `extraction_mode=True` (or omit it, as it's the default) to get AI-processed results.

```APIDOC
## searchscraper

### Description
Performs an intelligent search across multiple web sources and extracts information using advanced LLM models.

### Method
POST

### Endpoint
/v1/searchscraper

### Parameters
#### Request Body
- **user_prompt** (string) - Required - The query to search for.
- **num_results** (integer) - Optional - The number of search results to retrieve. Defaults to 5.
- **extraction_mode** (boolean) - Optional - Set to `True` for AI-extracted data (default), `False` for raw markdown content.

### Request Example
```json
{
  "user_prompt": "Latest research papers on machine learning and neural networks",
  "num_results": 5,
  "extraction_mode": true
}
```

### Response
#### Success Response (200)
- **search_results** (array) - A list of search results, each containing extracted information.
- **reference_urls** (array) - A list of URLs from which the information was retrieved.

#### Response Example
```json
{
  "search_results": [
    {
      "title": "Example Research Paper Title",
      "snippet": "An AI-generated snippet summarizing the paper...",
      "url": "http://example.com/paper1"
    }
  ],
  "reference_urls": [
    "http://example.com/source1"
  ]
}
```
```

--------------------------------

### SearchScraper - AI Extraction Mode

Source: https://docs.scrapegraphai.com/services/searchscraper

This snippet demonstrates how to use the SearchScraper service in AI Extraction Mode to get structured information from web search results. It shows examples in Python, JavaScript, cURL, and CLI.

```APIDOC
## SearchScraper - AI Extraction Mode

### Description
Uses AI to extract and structure specific information from web search results. This is the default mode and costs 10 credits per page.

### Method
POST

### Endpoint
`https://api.scrapegraphai.com/v1/searchscraper`

### Parameters
#### Request Body
- **user_prompt** (string) - Required - The user's query to search for.
- **num_results** (integer) - Optional - The number of websites to search (default is 3).
- **extraction_mode** (boolean) - Optional - Set to `true` to enable AI extraction (default).

### Request Example (cURL)
```bash
curl -X 'POST' \
  'https://api.scrapegraphai.com/v1/searchscraper' \
  -H 'accept: application/json' \
  -H 'SGAI-APIKEY: your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{ \
    "user_prompt": "What are the key features and pricing of ChatGPT Plus?", \
    "num_results": 5, \
    "extraction_mode": true \
  }'
```

### Response
#### Success Response (200)
- **status** (string) - Indicates the status of the operation ('success' or 'error').
- **data** (object) - Contains the extracted information if successful.
- **error** (string) - Contains the error message if the status is 'error'.

### Response Example (JSON)
```json
{
  "status": "success",
  "data": {
    "extracted_data": [
      {
        "url": "https://example.com/page1",
        "content": "Extracted information from page 1..."
      },
      {
        "url": "https://example.com/page2",
        "content": "Extracted information from page 2..."
      }
    ]
  }
}
```
```

--------------------------------

### Configure OpenClaw

Source: https://docs.scrapegraphai.com/integrations/openclaw

Runs the onboarding wizard and checks the status of the local gateway.

```bash
openclaw onboard --install-daemon
```

```bash
openclaw gateway status
```

--------------------------------

### Initialize ScrapeGraphAI Client

Source: https://docs.scrapegraphai.com/knowledge-base/account/api-keys

Examples for initializing the client in Python and JavaScript, showing both explicit key passing and environment variable usage.

```python
from scrapegraph_py import ScrapeGraphAI

# Reads SGAI_API_KEY from env, or pass explicitly:
sgai = ScrapeGraphAI(api_key="your-api-key")

res = sgai.extract("Extract the title", url="https://example.com")
```

```javascript
import { ScrapeGraphAI } from "scrapegraph-js";

// Reads SGAI_API_KEY from env, or pass explicitly:
const sgai = ScrapeGraphAI({ apiKey: "your-api-key" });

const res = await sgai.extract({
  url: "https://example.com",
  prompt: "Extract the title",
});
```

--------------------------------

### Install LangChain and ScrapeGraph dependencies

Source: https://docs.scrapegraphai.com/integrations/langchain

Install the necessary packages to use ScrapeGraph with LangChain.

```bash
pip install langchain langchain-openai scrapegraph-py
```

--------------------------------

### Async Monitor Creation in Python

Source: https://docs.scrapegraphai.com/services/monitor

Create a monitor using asynchronous operations in Python. This example demonstrates setting up an asynchronous monitor with a Markdown format configuration.

```python
import asyncio
from scrapegraph_py import AsyncScrapeGraphAI, MarkdownFormatConfig

async def main():
    async with AsyncScrapeGraphAI() as sgai:
        res = await sgai.monitor.create(
            "https://example.com",
            "0 * * * *",
            name="async watch",
            formats=[MarkdownFormatConfig()],
        )
        if res.status == "success":
            print(res.data.cron_id)

asyncio.run(main())
```

--------------------------------

### Install ScrapeGraphAI and Google Generative AI SDK

Source: https://docs.scrapegraphai.com/developer-guides/llm-sdks-and-frameworks/gemini

Install the necessary npm packages for ScrapeGraphAI and the Google Generative AI SDK. Ensure you have Node.js version 20 or higher, or install `dotenv` for older versions.

```bash
npm install scrapegraph-js @google/genai
```

```bash
SGAI_APIKEY=your_scrapegraph_key
GEMINI_API_KEY=your_gemini_key
```

--------------------------------

### Create and Manage Monitors with Python SDK

Source: https://docs.scrapegraphai.com/sdks/python

Demonstrates how to create, list, get, update, pause, resume, and delete monitors using the ScrapeGraphAI Python SDK. Requires ScrapeGraphAI and MarkdownFormatConfig to be imported.

```python
from scrapegraph_py import ScrapeGraphAI, MarkdownFormatConfig

sgai = ScrapeGraphAI()

mon = sgai.monitor.create(
    "https://example.com",
    "0 * * * *",                 # cron expression (positional)
    name="Price Monitor",
    formats=[MarkdownFormatConfig()],
    webhook_url="https://example.com/webhook",
)

cron_id = mon.data.cron_id

sgai.monitor.list()
sgai.monitor.get(cron_id)
sgai.monitor.update(cron_id, interval="0 */6 * * *")
sgai.monitor.pause(cron_id)
sgai.monitor.resume(cron_id)
sgai.monitor.delete(cron_id)
```

--------------------------------

### Quick Start with ScrapeGraphAI

Source: https://docs.scrapegraphai.com/sdks/javascript

Basic implementation for initializing the client and performing a scrape operation.

```javascript
import { ScrapeGraphAI } from "scrapegraph-js";

// reads SGAI_API_KEY from env, or pass explicitly: ScrapeGraphAI({ apiKey: "..." })
const sgai = ScrapeGraphAI();

const res = await sgai.scrape({
  url: "https://example.com",
  formats: [{ type: "markdown" }],
});

if (res.status === "success") {
  console.log(res.data?.results.markdown?.data?.[0]);
} else {
  console.error(res.error);
}
```

--------------------------------

### Install ScrapeGraphAI SDK

Source: https://docs.scrapegraphai.com/sdks/javascript

Commands to install the latest version of the ScrapeGraphAI SDK using various package managers.

```bash
# npm
npm i scrapegraph-js@latest     # pins a version >= 2.1.0

# pnpm
pnpm add scrapegraph-js@latest

# yarn
yarn add scrapegraph-js@latest

# bun
bun add scrapegraph-js@latest
```

--------------------------------

### Create a Page-Change Monitor

Source: https://docs.scrapegraphai.com/services/cli

Sets up a monitor to track changes on a specific URL at a defined interval. Options include setting a name, output format, webhook URL, fetch mode, and stealth mode.

```bash
# Create a monitor
just-scrape monitor create --url <url> --interval <interval>
```

```bash
just-scrape monitor create --url <url> --interval 1h --name "My Monitor"
```

```bash
just-scrape monitor create --url <url> --interval 30m -f markdown,links --webhook-url <url>
```

```bash
just-scrape monitor create --url <url> --interval 1d -m js --stealth
```

--------------------------------

### Quickstart Agno Agent with ScrapeGraphTools

Source: https://docs.scrapegraphai.com/integrations/agno

Initialize an Agno agent with the OpenAIChat model and enable all ScrapeGraphTools. This allows the agent to use any ScrapeGraph endpoint as needed.

```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.scrapegraph import ScrapeGraphTools

agent = Agent(
    model=OpenAIChat(id="gpt-4o"),
    tools=[ScrapeGraphTools(all=True)],
    markdown=True,
)

agent.print_response(
    "Use smartscraper on https://example.com to extract the page title and main heading. Return them as JSON.",
    stream=True,
)
```

--------------------------------

### Basic Scrape Tool Usage

Source: https://docs.scrapegraphai.com/integrations/vercel_ai

Demonstrates initializing a model with a single scrape tool and setting a step limit.

```ts
import { openai } from "@ai-sdk/openai";
import { generateText, stepCountIs } from "ai";
import { scrapeTool } from "@scrapegraph-ai/ai-sdk";

const { text } = await generateText({
  model: openai("gpt-5-nano"),
  prompt:
    "Scrape Hacker News and write a short, concise summary of what people are talking about today.",
  tools: {
    scrape: scrapeTool(),
  },
  stopWhen: stepCountIs(3),
});

console.log(text);
```

--------------------------------

### Verify API Key and Setup

Source: https://docs.scrapegraphai.com/knowledge-base/cli/getting-started

Check your ScrapeGraphAI credit balance to confirm that your API key is valid and correctly configured.

```bash
just-scrape credits
```

--------------------------------

### List History Request Example

Source: https://docs.scrapegraphai.com/api-reference/endpoint/history

Example cURL request to list history entries filtered by service and limit.

```bash
curl -X GET "https://v2-api.scrapegraphai.com/api/history?service=scrape&limit=5" \
  -H "SGAI-APIKEY: $SGAI_API_KEY"
```

--------------------------------

### Quick Start: Markdown Mode (CLI)

Source: https://docs.scrapegraphai.com/services/searchscraper

Use this command-line interface command to retrieve raw markdown content. Specify the user prompt, number of results, and the --no-extraction flag.

```bash
just-scrape search-scraper "Latest developments in artificial intelligence" --num-results 3 --no-extraction
```

--------------------------------

### Install LlamaIndex and ScrapeGraphAI

Source: https://docs.scrapegraphai.com/integrations/llamaindex

Install the necessary packages for LlamaIndex and ScrapeGraphAI. Ensure you are using ScrapeGraphAI version 2.0.1 or higher.

```bash
pip install -U llama-index
pip install "scrapegraph-py>=2.0.1"
```

--------------------------------

### Initialize and Use JavaScript SDK

Source: https://docs.scrapegraphai.com/install

Basic extraction usage for the JavaScript SDK.

```javascript
import scrapegraphai from "scrapegraph-js";

const sgai = scrapegraphai({ apiKey: "your-api-key-here" });

const { data } = await sgai.extract(
  "https://scrapegraphai.com",
  { prompt: "What does the company do?" }
);

console.log(data);
```

--------------------------------

### Install LangChain Integration

Source: https://docs.scrapegraphai.com/contribute/opensource

Install the official LangChain integration for LLM workflows. This package allows seamless integration with LangChain.

```bash
pip install langchain-scrapegraph
```

--------------------------------

### Smithery CLI Installation for ScrapeGraph MCP

Source: https://docs.scrapegraphai.com/services/mcp-server/claude

Install ScrapeGraph MCP as a client using the Smithery CLI. This is an alternative provisioning method.

```bash
npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude
```

--------------------------------

### Configure Fetch Settings with Python SDK

Source: https://docs.scrapegraphai.com/sdks/python

Illustrates how to configure fetch settings using the `FetchConfig` object, including mode, stealth, timeouts, wait times, scrolls, country targeting, custom headers, cookies, and mocking.

```python
from scrapegraph_py import FetchConfig

config = FetchConfig(
    mode="js",            # "auto" (default), "fast", "js"
    stealth=True,         # Residential proxies / anti-bot headers (+5 credits)
    timeout=30000,        # 1,000–60,000 ms
    wait=2000,            # 0–30,000 ms
    scrolls=3,            # 0–100
    country="us",         # ISO 3166-1 alpha-2
    headers={"X-Custom": "header"},
    cookies={"session": "abc"},
    mock=False,           # Or a MockConfig object for testing
)
```

--------------------------------

### Example Smart Scraper Response

Source: https://docs.scrapegraphai.com/services/smartscraper

This is an example of a successful response from the Smart Scraper API, showing the structured data extracted from a webpage.

```json
{
  "request_id": "sg-req-abc123",
  "status": "completed",
  "website_url": "https://scrapegraphai.com/",
  "user_prompt": "Extract info about the company",
  "result": {
    "company_name": "ScrapeGraphAI",
    "description": "ScrapeGraphAI is a powerful AI scraping API designed for efficient web data extraction to power LLM applications and AI agents...",
    "features": [
      "Effortless, cost-effective, and AI-powered data extraction",
      "Handles proxy rotation and rate limits",
      "Supports a wide variety of websites"
    ],
    "contact_email": "contact@scrapegraphai.com",
    "social_links": {
      "github": "https://github.com/ScrapeGraphAI/Scrapegraph-ai",
      "linkedin": "https://www.linkedin.com/company/101881123",
      "twitter": "https://x.com/scrapegraphai"
    }
  },
  "error": ""
}
```

--------------------------------

### Initialize and Use Python SDK

Source: https://docs.scrapegraphai.com/install

Basic extraction usage. The API key can also be set via the SGAI_API_KEY environment variable.

```python
from scrapegraph_py import ScrapeGraphAI

sgai = ScrapeGraphAI(api_key="your-api-key-here")

# Extract data from a website
res = sgai.extract(
    "Extract information about the company",
    url="https://scrapegraphai.com",
)
print(res.data.json_data if res.status == "success" else res.error)
```

--------------------------------

### Markdownify API Response Example

Source: https://docs.scrapegraphai.com/services/markdownify

An example of a successful response from the Markdownify API, including request details and the converted markdown content.

```json
{
  "request_id": "sg-req-md456",
  "status": "completed",
  "website_url": "https://example.com/article",
  "result": "# Understanding AI-Powered Web Scraping\n\nWeb scraping has evolved significantly with the advent of AI technologies...\n\n## Key Benefits\n\n- Improved accuracy\n- Intelligent extraction\n- Structured output\n\n![AI Scraping Process](https://example.com/images/ai-scraping.png)\n\n> AI-powered scraping represents the future of web data extraction.\n\n### Getting Started\n\n1. Choose your target website\n2. Define extraction goals\n3. Select appropriate tools\n",
  "error": ""
}
```

--------------------------------

### Install ScrapeGraphAI Python SDK

Source: https://docs.scrapegraphai.com/sdks/python

Install the ScrapeGraphAI Python SDK using pip or uv. Ensure you are using version 2.1.0 or later.

```bash
pip install "scrapegraph-py>=2.1.0"
# or
uv add "scrapegraph-py>=2.1.0"
```

--------------------------------

### Create a new monitor in JavaScript

Source: https://docs.scrapegraphai.com/services/monitor

Initiate a new web monitor using the `monitor.create` method in JavaScript. Configure the URL, name, cron interval, and output formats. Optionally, provide a `webhookUrl` to receive notifications on each tick.

```javascript
import { ScrapeGraphAI } from "scrapegraph-js";

const sgai = ScrapeGraphAI();

const res = await sgai.monitor.create({
  url: "https://example.com",
  name: "Homepage watch",
  interval: "*/30 * * * *",    // every 30 minutes
  formats: [{ type: "markdown" }],
  webhookUrl: "https://your-server.com/webhook", // optional
});

if (res.status === "success") {
  console.log("Monitor id:", res.data?.cronId);
}
```

--------------------------------

### Install ScrapeGraphAI JavaScript SDK

Source: https://docs.scrapegraphai.com/knowledge-base/ai-tools/cursor

Install the latest ScrapeGraphAI JavaScript SDK using npm. Ensure your API key is set in the environment.

```bash
npm i scrapegraph-js@latest
```

--------------------------------

### SmartScraper Initialization with User Prompt

Source: https://docs.scrapegraphai.com/services/smartscraper

Initialize SmartScraper with a user prompt for extracting information. This is a basic setup for initiating a scraping task.

```python
SmartScraper(user_prompt='Extract info about the company',
  )
```

--------------------------------

### Create and Manage Monitors

Source: https://docs.scrapegraphai.com/services/cli/commands

Commands for creating, listing, getting, updating, pausing, resuming, and deleting page-change monitors. Use `--interval` with units like 'h', 'm', 'd'. Specify output format with flags like `-f markdown,screenshot`.

```bash
just-scrape monitor create --url <url> --interval <interval>    # e.g. '1h', '30m', '1d'
```

```bash
just-scrape monitor create --url <url> --interval 1h --name "My Monitor"
```

```bash
just-scrape monitor create --url <url> --interval 30m --webhook-url <url>
```

```bash
just-scrape monitor create --url <url> --interval 1d -f markdown,screenshot
```

```bash
just-scrape monitor list                                        # list all monitors
```

```bash
just-scrape monitor get --id <id>                               # get monitor details
```

```bash
just-scrape monitor update --id <id> --interval 2h              # update interval
```

```bash
just-scrape monitor pause --id <id>                             # pause
```

```bash
just-scrape monitor resume --id <id>                            # resume
```

```bash
just-scrape monitor delete --id <id>                            # delete
```

```bash
just-scrape monitor activity --id <id>                          # paginated tick history
```

```bash
just-scrape monitor activity --id <id> --limit 50               # ticks per page (max 100)
```

```bash
just-scrape monitor activity --id <id> --cursor <cursor>        # paginate with a cursor
```
Tokens:
Raw