PyScrappy

https://pyscrappy.vercel.app
Admin
PyScrappy is a Python web scraping toolkit and MCP server that provides generic and built-in...
Tokens:17,289
Snippets:162
Trust Score:7.8
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install PyScrappy optional extras

Source: https://pyscrappy.vercel.app/docs/installation

Installs optional extras for browser support, DataFrame support, MCP server, stealth, and everything. Each extra is installed separately; the 'browser' extra also requires running 'playwright install chromium'.

```bash
# Browser support (for JavaScript-rendered pages)
pip install 'pyscrappy[browser]'
playwright install chromium
 
# DataFrame support (.to_dataframe())
pip install 'pyscrappy[dataframe]'
 
# MCP server (use PyScrappy's scrapers as AI-agent tools)
pip install 'pyscrappy[mcp]'
 
# Stealth (TLS-fingerprint impersonation to bypass anti-bot filters)
pip install 'pyscrappy[stealth]'
 
# Everything
pip install 'pyscrappy[all]'
```

--------------------------------

### Run pyscrappy-mcp and install the mcp extra

Source: https://pyscrappy.vercel.app/docs/mcp-register

Run the MCP server manually to see startup errors. If it fails, the error message usually indicates the missing `mcp` extra, which you can install with `pip install 'pyscrappy[mcp]'`.

```bash
pyscrappy-mcp
```

```bash
pip install 'pyscrappy[mcp]'
```

--------------------------------

### Add pyscrappy MCP server via uvx

Source: https://pyscrappy.vercel.app/docs/mcp-register

Alternative command to add the server using uvx without a prior install. Requires uv to be installed.

```bash
claude mcp add pyscrappy -- uvx --from "pyscrappy[mcp]" pyscrappy-mcp
```

--------------------------------

### Install PyScrappy MCP extra and pull Ollama model

Source: https://pyscrappy.vercel.app/docs/mcp-local

Install PyScrappy with the MCP extra and pull the qwen2.5 model locally. Requires Python 3.10+.

```bash
pip install 'pyscrappy[mcp]'
ollama pull qwen2.5
```

--------------------------------

### Image Search Scraper Example

Source: https://pyscrappy.vercel.app/docs/scrapers-data

Searches and downloads images. Use max_images to limit count and download_to to specify a directory. The example searches for 'golden retriever' and downloads up to 10 images to './dogs'.

```python
from pyscrappy import ImageSearchScraper
 
with ImageSearchScraper() as iss:
    result = iss.scrape(query="golden retriever", max_images=10, download_to="./dogs")
```

--------------------------------

### Verify PyScrappy installation

Source: https://pyscrappy.vercel.app/docs/installation

Imports PyScrappy and prints its version to confirm the installation.

```python
import pyscrappy
print(pyscrappy.__version__)
```

--------------------------------

### Run PyScrappy over HTTP with pyscrappy-mcp --http --port 8000

Source: https://pyscrappy.vercel.app/docs/mcp-selfhost

Installs the MCP extra and starts the server over Streamable HTTP on port 8000, binding to 127.0.0.1 by default. The endpoint is served at http://127.0.0.1:8000/mcp.

```bash
pip install 'pyscrappy[mcp]'
pyscrappy-mcp --http --port 8000
```

--------------------------------

### Start Claude Code session

Source: https://pyscrappy.vercel.app/docs/mcp-register

Start a session to see the tools. After running this, use the /mcp command inside the session.

```bash
claude
```

--------------------------------

### Install PyScrappy MCP server with pip

Source: https://pyscrappy.vercel.app/docs/mcp

Installs the pyscrappy-mcp stdio MCP server. Requires Python 3.10+.

```bash
pip install 'pyscrappy[mcp]'
```

--------------------------------

### Install PyScrappy with pip

Source: https://pyscrappy.vercel.app/docs/installation

Installs the core PyScrappy package from PyPI. Requires Python 3.9+.

```bash
pip install pyscrappy
```

--------------------------------

### Deploy PyScrappy with pip install and pyscrappy-mcp --http --host 0.0.0.0 --port 8000

Source: https://pyscrappy.vercel.app/docs/mcp-selfhost

Deploys PyScrappy on any host that runs a long-lived Python process (Render, Railway, Fly.io, VPS). Install the extra and run the HTTP command as the start command.

```bash
pip install 'pyscrappy[mcp]'
pyscrappy-mcp --http --host 0.0.0.0 --port 8000
```

--------------------------------

### Run PyScrappy MCP server with uvx

Source: https://pyscrappy.vercel.app/docs/mcp

Runs the MCP server on demand without a local install.

```bash
uvx --from "pyscrappy[mcp]" pyscrappy-mcp
```

--------------------------------

### Advertise a plugin entry point in pyproject.toml

Source: https://pyscrappy.vercel.app/docs/plugins

Declares the entry point in the pyscrappy.scrapers group for a pyscrappy-reddit package. Once installed, PyScrappy discovers the scraper lazily without needing @register_scraper.

```toml
# pyproject.toml of your pyscrappy-reddit package
[project.entry-points."pyscrappy.scrapers"]
reddit = "pyscrappy_reddit:RedditScraper"
```

--------------------------------

### Bind to all interfaces with pyscrappy-mcp --http --host 0.0.0.0 --port 8000

Source: https://pyscrappy.vercel.app/docs/mcp-selfhost

Starts the HTTP server bound to all interfaces so other machines can connect. Use this instead of the default 127.0.0.1 binding for remote access.

```bash
pyscrappy-mcp --http --host 0.0.0.0 --port 8000
```

--------------------------------

### Result envelope example

Source: https://pyscrappy.vercel.app/docs/mcp-tools

Shows the structure returned by every MCP tool: data (list of items), count, scraper, source_urls, and errors.

```json
{
  "data": [ { "...": "item fields depend on the source" } ],
  "count": 5,
  "scraper": "crypto",
  "source_urls": ["https://api.coingecko.com/..."],
  "errors": []
}
```

--------------------------------

### Key-free Public API Scrapers Example

Source: https://pyscrappy.vercel.app/docs/scrapers-data

Uses WeatherScraper, CryptoScraper, CurrencyScraper, and DictionaryScraper with no API key. These rely on public APIs such as Open-Meteo, CoinGecko, exchange rates, and Free Dictionary.

```python
from pyscrappy import WeatherScraper, CryptoScraper, CurrencyScraper, DictionaryScraper
 
WeatherScraper().scrape(location="Tokyo, Japan")
CryptoScraper().scrape(query="bitcoin, ethereum")
CurrencyScraper().scrape(base="USD", to="EUR,GBP", amount=100)
DictionaryScraper().scrape(word="serendipity")
```

--------------------------------

### Apply ScraperConfig to AmazonScraper

Source: https://pyscrappy.vercel.app/docs/configuration

Passes a ScraperConfig to AmazonScraper to apply the same configuration to any scraper. The example sets timeout to 30.0 seconds, max_retries to 5, and rate_limit to 1.0 seconds per domain.

```python
from pyscrappy import ScraperConfig, AmazonScraper
 
config = ScraperConfig(timeout=30.0, max_retries=5, rate_limit=1.0)
 
with AmazonScraper(config) as scraper:
    result = scraper.scrape(query="laptop")
```

--------------------------------

### Scrape IKEA with per-country pricing using pyscrappy

Source: https://pyscrappy.vercel.app/docs/scrapers-ecommerce

Scrapes IKEA with per-country pricing by passing the store's country and lang parameters. The example uses country="gb" and lang="en", with alternatives like "us"/"en" or "de"/"de". The scrape method is called with a query and max_results, and the result is converted to a DataFrame and printed.

```python
from pyscrappy import IKEAScraper
 
with IKEAScraper(country="gb", lang="en") as scraper:   # or "us"/"en", "de"/"de", …
    result = scraper.scrape(query="desk", max_results=24)
    df = result.to_dataframe()
    print(df.head())
```

--------------------------------

### Configure ScraperConfig with GenericScraper

Source: https://pyscrappy.vercel.app/docs/configuration

Creates a ScraperConfig with all options and uses it with GenericScraper. The config controls timeouts, retries, rate limiting, proxies, browser behavior, and caching. The example shows the default values for each option.

```python
from pyscrappy import ScraperConfig, GenericScraper
 
config = ScraperConfig(
    timeout=20.0,            # request timeout in seconds
    max_retries=3,           # retry failed requests
    rate_limit=2.0,          # seconds between requests per domain
    proxy="http://...",      # proxy URL, or a list to rotate through
    scraper_api=None,        # route via a scraping-API service (see Proxies)
    headless=True,           # browser runs headless
    render_js="auto",        # auto-detect if JS rendering is needed
    cache_ttl=0,             # response cache TTL in seconds (0 = disabled)
)
 
with GenericScraper(config) as gs:
    result = gs.scrape(url="https://example.com")
```

--------------------------------

### Render JavaScript with render_js=true

Source: https://pyscrappy.vercel.app/docs/mcp-tools

Pass render_js=true to scrape_url to load the page in a headless browser before scraping. Requires the browser extra (pip install 'pyscrappy[browser]' and playwright install chromium); otherwise returns an error.

```json
{ "url": "https://github.com/torvalds", "render_js": true }
```

--------------------------------

### ZomatoScraper: scrape city listings

Source: https://pyscrappy.vercel.app/docs/scrapers-food

Use ZomatoScraper to scrape restaurant listings for a city. The example scrapes up to 20 results for Bangalore.

```python
from pyscrappy import ZomatoScraper
 
with ZomatoScraper() as scraper:
    result = scraper.scrape(city="bangalore", max_results=20)
```

--------------------------------

### Run pyscrappy chat with a local model

Source: https://pyscrappy.vercel.app/docs/mcp-local

Start a chat session with the built-in agent, letting the local model call PyScrappy scrapers as tools. The model decides which scraper to call and answers from the structured result.

```bash
pyscrappy chat --model qwen2.5 "what's the current AAPL quote?"
```

--------------------------------

### Configure PyScrappy MCP server in Claude Desktop

Source: https://pyscrappy.vercel.app/docs/mcp-register

Add PyScrappy to Claude Desktop's configuration file. If other servers exist, add 'pyscrappy' as another key inside the existing 'mcpServers' object. If the server fails to start, replace the command with the absolute path from 'which pyscrappy-mcp' (macOS/Linux) or 'where pyscrappy-mcp' (Windows).

```json
{
  "mcpServers": {
    "pyscrappy": {
      "command": "pyscrappy-mcp"
    }
  }
}
```

--------------------------------

### OpenClaw MCP server configuration for PyScrappy

Source: https://pyscrappy.vercel.app/docs/mcp-register

Configure PyScrappy as an MCP server in OpenClaw's openclaw.json. The first block uses a pre-installed command; the second runs via uvx without prior installation. After restarting OpenClaw, verify the 22 tools appear in the MCP/tools panel.

```json
{
  "mcpServers": {
    "pyscrappy": {
      "command": "pyscrappy-mcp"
    }
  }
}
```

```json
{
  "mcpServers": {
    "pyscrappy": {
      "command": "uvx",
      "args": ["--from", "pyscrappy[mcp]", "pyscrappy-mcp"]
    }
  }
}
```

--------------------------------

### pyscrappy chat with remote Ollama host

Source: https://pyscrappy.vercel.app/docs/mcp-local

Point pyscrappy chat at a remote or containerized Ollama endpoint using --host. The example uses a local network IP and the llama3.1 model.

```bash
pyscrappy chat --host http://192.168.1.50:11434 --model llama3.1 "top 5 crypto prices"
```

--------------------------------

### Verbose pyscrappy chat with tool call output

Source: https://pyscrappy.vercel.app/docs/mcp-local

Run pyscrappy chat with verbose output to see each tool call (name and arguments) printed to stderr. The example shows the expected output for a prompt that triggers two tool calls.

```bash
pyscrappy chat -v --model qwen2.5 "define 'ephemeral' and get today's weather in Tokyo"
```

```text
  → define_word({"word": "ephemeral"})
  → get_weather({"location": "Tokyo"})
```

--------------------------------

### NewsScraper (RSS feeds) Example

Source: https://pyscrappy.vercel.app/docs/scrapers-data

Fetches news articles from an RSS feed URL. Can also pass site_url for auto-discovery or article_url for full-text extraction. Iterates over result.data to access article titles.

```python
from pyscrappy import NewsScraper
 
with NewsScraper() as ns:
    result = ns.scrape(
        feed_url="https://rss.nytimes.com/services/xml/rss/nyt/World.xml"
    )
    for article in result.data[:5]:
        print(article["title"])
```

--------------------------------

### Impersonate a browser with ScraperConfig(impersonate="chrome")

Source: https://pyscrappy.vercel.app/docs/stealth

Configure a GenericScraper to impersonate a real browser's TLS fingerprint. Requires the optional `curl_cffi` backend: `pip install 'pyscrappy[stealth]'`. Impersonation applies only to the synchronous path; setting it on an async client raises an explicit error.

```python
from pyscrappy import ScraperConfig, GenericScraper
 
# needs the optional extra:  pip install 'pyscrappy[stealth]'
config = ScraperConfig(impersonate="chrome")   # or "chrome124", "safari", "firefox"
 
with GenericScraper(config) as gs:
    result = gs.scrape("https://example.com")
```

--------------------------------

### Configure PyScrappy MCP server in OpenCode

Source: https://pyscrappy.vercel.app/docs/mcp-register

Add PyScrappy to OpenCode's configuration file (opencode.json). If you already installed 'pyscrappy[mcp]' via pip, you can use the console script directly: "command": ["pyscrappy-mcp"]. After restarting OpenCode, verify with /mcp.

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "pyscrappy": {
      "type": "local",
      "command": ["uvx", "--from", "pyscrappy[mcp]", "pyscrappy-mcp"],
      "enabled": true
    }
  }
}
```

--------------------------------

### Add pyscrappy MCP server via claude mcp add

Source: https://pyscrappy.vercel.app/docs/mcp-register

Run this command in your terminal (not inside a claude session) to add the pyscrappy server to Claude Code. This is the standard method using the installed pyscrappy-mcp executable.

```bash
claude mcp add pyscrappy pyscrappy-mcp
```

--------------------------------

### Wikipedia Scraper Example

Source: https://pyscrappy.vercel.app/docs/scrapers-data

Uses WikipediaScraper in a with-block to fetch a summary for a query. The mode parameter can be 'full', 'summary', 'paragraphs', or 'headers'. The result's data[0]['text'] holds the scraped content.

```python
from pyscrappy import WikipediaScraper
 
with WikipediaScraper() as ws:
    result = ws.scrape(query="Python (programming language)", mode="summary")
    print(result.data[0]["text"])
```

--------------------------------

### Enable caching with cache_ttl

Source: https://pyscrappy.vercel.app/docs/caching

Enables in-memory caching for successful GET responses by setting cache_ttl to a positive number of seconds. Repeated requests for the same URL and query params within the TTL are served from cache, skipping the network and rate limiter. Caching is disabled by default (cache_ttl=0).

```python
from pyscrappy import WikipediaScraper, ScraperConfig
 
config = ScraperConfig(cache_ttl=300)   # cache for 5 minutes
 
with WikipediaScraper(config) as ws:
    ws.scrape(query="Python")   # fetched over the network
    ws.scrape(query="Python")   # served from cache
```

--------------------------------

### IMDBScraper (OMDb API) Example

Source: https://pyscrappy.vercel.app/docs/scrapers-data

Scrapes IMDB data via the free OMDb API. Requires OMDB_API_KEY environment variable or api_key argument. Can search by title or IMDB ID. Outputs a DataFrame with title, year, rating, genre.

```python
from pyscrappy import IMDBScraper
 
with IMDBScraper() as scraper:          # reads OMDB_API_KEY from the environment
    result = scraper.scrape(query="inception")      # search by title
    result = scraper.scrape(query="tt1375666")      # or look up an IMDB id
    df = result.to_dataframe()
    print(df[["title", "year", "rating", "genre"]])
```

--------------------------------

### Create a Selector and run CSS/XPath/BeautifulSoup-style searches

Source: https://pyscrappy.vercel.app/docs/selectors

Create a Selector from an HTML string and run CSS, XPath, BeautifulSoup-style, and text searches. CSS supports ::text and ::attr(name) pseudo-elements; XPath text() and @attr expressions return strings.

```python
from pyscrappy import Selector
 
page = Selector(html)                              # or navigate any HTML string
page.css(".title::text").getall()                  # CSS with ::text / ::attr(name)
page.xpath("//a/@href").getall()                    # XPath (elements, text(), @attr)
page.find_all("h2", class_="title")                 # BeautifulSoup-style search
page.find_by_text("Add to cart", tag="button")      # search by text content
```

--------------------------------

### List MCP servers in Claude Code

Source: https://pyscrappy.vercel.app/docs/mcp-register

Run this from your shell to verify the server is connected. You should see the output shown in the next block.

```bash
claude mcp list
```

--------------------------------

### Custom CSS selectors with GenericScraper

Source: https://pyscrappy.vercel.app/docs/quickstart

Uses GenericScraper to extract specific fields from a page using CSS selectors. The example scrapes Hacker News titles and scores.

```python
from pyscrappy import GenericScraper
 
with GenericScraper() as gs:
    result = gs.scrape(
        url="https://news.ycombinator.com",
        selectors={"title": ".titleline a", "score": ".score"},
    )
    for item in result.data:
        print(item["title"], item.get("score", ""))
```

--------------------------------

### Run MCP Inspector for PyScrappy

Source: https://pyscrappy.vercel.app/docs/mcp-register

Launch the official MCP Inspector to test the raw PyScrappy server. Opens a browser UI where all 22 tools are listed under the Tools tab; select one, fill arguments, and click Run to see the ScrapeToolResult.

```bash
npx @modelcontextprotocol/inspector pyscrappy-mcp
```

--------------------------------

### pyscrappy chat CLI options

Source: https://pyscrappy.vercel.app/docs/mcp-local

Usage syntax for pyscrappy chat. Defaults: --model qwen2.5, --host http://localhost:11434, --max-steps 8, -v off.

```bash
pyscrappy chat [--model MODEL] [--host HOST] [--max-steps N] [-v] "your prompt"
```

--------------------------------

### StockScraper (Yahoo Finance) Example

Source: https://pyscrappy.vercel.app/docs/scrapers-data

Scrapes stock data from Yahoo Finance. Supports modes 'quote', 'history', or 'profile', and period values like '1mo', '1y'. Converts result to DataFrame with to_dataframe().

```python
from pyscrappy import StockScraper
 
with StockScraper() as ss:
    result = ss.scrape(symbol="AAPL", mode="history", period="1mo")
    df = result.to_dataframe()
    print(df.head())
```

--------------------------------

### UberEatsScraper: city listings and full menus

Source: https://pyscrappy.vercel.app/docs/scrapers-food

Use UberEatsScraper with a locale to list restaurants for a city and fetch full menus. The workflow is two steps: .scrape(city=...) returns restaurants with a store url, then .get_menu(store_url) returns menu items and prices. Set locale to the country you're querying (e.g. "gb", "us", "in"). The city must be one where Uber Eats delivers, or the result will be empty.

```python
from pyscrappy import UberEatsScraper
 
with UberEatsScraper(locale="gb") as scraper:
    result = scraper.scrape(city="London", max_results=30)
 
    store = result.data[0]
    menu = scraper.get_menu(store["url"])   # items with prices
    for item in menu.data[0]["menu"]:
        print(item["name"], item["price"])
```

--------------------------------

### Open MCP panel in Claude Code

Source: https://pyscrappy.vercel.app/docs/mcp-register

Run this inside the Claude session to open the MCP panel, then select pyscrappy to see the 22 tools.

```bash
/mcp
```

--------------------------------

### Configure OpenCode remote MCP server in opencode.json

Source: https://pyscrappy.vercel.app/docs/mcp-selfhost

Configures OpenCode to use PyScrappy as a remote MCP server. Place this in opencode.json and replace your-host with your server's address.

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "pyscrappy": {
      "type": "remote",
      "url": "http://your-host:8000/mcp",
      "enabled": true
    }
  }
}
```

--------------------------------

### Expected output of claude mcp list

Source: https://pyscrappy.vercel.app/docs/mcp-register

This is the expected output when the server is connected successfully.

```text
pyscrappy: pyscrappy-mcp  ✓ Connected
```

--------------------------------

### Adaptive selector usage with auto_save and adaptive

Source: https://pyscrappy.vercel.app/docs/adaptive

First run saves a fingerprint under an adaptive_id; later runs with adaptive=True relocate the element if the selector fails. Requires passing url= for site namespacing. Without adaptive=True, a broken selector still returns empty.

```python
from pyscrappy import Selector
 
# First run: match normally and remember this element under an id.
page = Selector(html_v1, url="https://shop.example.com")
price = page.css(".price", auto_save=True, adaptive_id="price").get()
 
# Later, after a redesign renamed ".price" — heal instead of breaking:
page = Selector(html_v2, url="https://shop.example.com")
result = page.css(".price", adaptive=True, adaptive_id="price")
print(result.get(), "→ confidence:", result.adaptive_confidence)
```

--------------------------------

### Scrape any URL with scrape()

Source: https://pyscrappy.vercel.app/docs/quickstart

Performs a fast static fetch of a URL and prints the page title and word count from the result. Does not run JavaScript; use render_js=True for client-side rendered pages.

```python
from pyscrappy import scrape
 
result = scrape("https://en.wikipedia.org/wiki/Web_scraping")
print(result.data[0]["metadata"]["title"])
print(result.data[0]["text"]["word_count"])
```

--------------------------------

### Configure proxy or rotating list with ScraperConfig

Source: https://pyscrappy.vercel.app/docs/proxies

Configure PyScrappy to use a single proxy or a rotating list of proxies for both HTTP and browser backends. Set the proxy parameter to a string for one proxy or a list of strings for rotation. Use this when a scraper reports it is being blocked by anti-bot systems.

```python
from pyscrappy import ScraperConfig, AmazonScraper
 
# Single proxy
config = ScraperConfig(proxy="http://user:pass@host:port")
 
# Rotating list (one picked per request)
config = ScraperConfig(proxy=["http://p1:8080", "http://p2:8080"])
 
with AmazonScraper(config) as scraper:
    result = scraper.scrape(query="laptop")
```

--------------------------------

### Set OMDB_API_KEY via environment for PyScrappy MCP in OpenCode

Source: https://pyscrappy.vercel.app/docs/mcp-register

Provide the OMDb API key for the lookup_movie tool in OpenCode using the environment key. Only this tool requires the key; all others work without it.

```json
{
  "mcp": {
    "pyscrappy": {
      "type": "local",
      "command": ["uvx", "--from", "pyscrappy[mcp]", "pyscrappy-mcp"],
      "environment": { "OMDB_API_KEY": "your-key" }
    }
  }
}
```

--------------------------------

### Bridge remote server for Claude Desktop/OpenClaw with mcp-proxy

Source: https://pyscrappy.vercel.app/docs/mcp-selfhost

Configures Claude Desktop or OpenClaw to use mcp-proxy as a bridge to the remote HTTP endpoint. These clients expect a local command, so mcp-proxy forwards stdio to the remote server.

```json
{
  "mcpServers": {
    "pyscrappy": {
      "command": "mcp-proxy",
      "args": ["http://your-host:8000/mcp"]
    }
  }
}
```

--------------------------------

### Read values from a SelectorList with .get(), .getall(), and .text()

Source: https://pyscrappy.vercel.app/docs/selectors

Read values from a SelectorList with .get() (first), .getall() (all), or .text(). Selections are chainable, so you can select within a previous selection.

```python
first = page.css(".product")[0]
first.css(".price::text").get()        # chainable — select within a selection
page.css(".title").text()              # ["Alpha", "Beta"]
```

--------------------------------

### Set OMDB_API_KEY via env for PyScrappy MCP

Source: https://pyscrappy.vercel.app/docs/mcp-register

Provide the OMDb API key for the lookup_movie tool in Claude Desktop, Claude Code, or OpenClaw using the env key. Only this tool requires the key; all others work without it.

```json
{
  "mcpServers": {
    "pyscrappy": {
      "command": "pyscrappy-mcp",
      "env": { "OMDB_API_KEY": "your-key" }
    }
  }
}
```

--------------------------------

### Scrape Amazon and Newegg with pyscrappy

Source: https://pyscrappy.vercel.app/docs/scrapers-ecommerce

Scrapes Amazon and Newegg using the pyscrappy library. Amazon is used as a general marketplace, while Newegg targets electronics and computer hardware. Each scraper is used as a context manager and the scrape method is called with a query and max_pages parameter.

```python
from pyscrappy import AmazonScraper, NeweggScraper, IKEAScraper
 
# Amazon: general marketplace
with AmazonScraper() as scraper:
    result = scraper.scrape(query="laptop", max_pages=2)
 
# Newegg: electronics / computer hardware
with NeweggScraper() as scraper:
    result = scraper.scrape(query="graphics card", max_pages=2)
```

--------------------------------

### Expose a scraper as a typed MCP tool with mcp_tools

Source: https://pyscrappy.vercel.app/docs/plugins

Adds an mcp_tools mapping to expose the scraper as a dedicated MCP tool named search_reddit. The tool's schema is derived from the scrape() signature, making subreddit required and sort optional with default "hot".

```python
@register_scraper("reddit")
class RedditScraper(BaseScraper):
    mcp_tools = {"search_reddit": "scrape"}   # tool name -> method
 
    def scrape(self, subreddit: str, sort: str = "hot") -> ScrapeResult:
        ...
```

--------------------------------

### Search elements with find_all() and find_by_text()

Source: https://pyscrappy.vercel.app/docs/selectors

Search for elements by tag, class, and attributes with find_all(), or by text content with find_by_text(). find_by_text() does a substring match by default; pass tag and exact=True to require an exact match on a specific tag.

```python
page.find_all("a", class_="nav")                    # by tag + class + attributes
page.find_by_text("Sold out")                        # substring match, any tag
page.find_by_text("Sold out", tag="span", exact=True)  # exact text, specific tag
```

--------------------------------

### Register and use a custom scraper with BaseScraper

Source: https://pyscrappy.vercel.app/docs/plugins

Defines a Reddit scraper by subclassing BaseScraper, registering it with @register_scraper("reddit"), and implementing scrape() to return a ScrapeResult. The second block shows immediate usage via get_scraper("reddit")() and calling scrape(subreddit="python").

```python
from pyscrappy import BaseScraper, register_scraper
from pyscrappy.core.models import ScrapeResult, ScrapeMetadata
 
@register_scraper("reddit")
class RedditScraper(BaseScraper):
    def scrape(self, subreddit: str, **kwargs) -> ScrapeResult:
        soup = self.fetch_and_parse(f"https://old.reddit.com/r/{subreddit}/")
        posts = [
            {"title": a.get_text(strip=True), "url": a["href"]}
            for a in soup.select("a.title")
        ]
        return ScrapeResult(data=posts, metadata=ScrapeMetadata(scraper="reddit"))
```

```python
from pyscrappy import get_scraper
 
with get_scraper("reddit")() as s:
    result = s.scrape(subreddit="python")
    print(result.to_markdown())
```

--------------------------------

### Configure scraping-API service with ScraperConfig

Source: https://pyscrappy.vercel.app/docs/proxies

Route requests through a scraping-API service such as ScraperAPI, ScrapeOps, or ScrapingBee. Set the scraper_api parameter with the provider, api_key, and optional render_js flag. This is the reliable way to use scrapers marked as needing a proxy, like Instagram, Twitter/X, and Spotify.

```python
from pyscrappy import ScraperConfig, AmazonScraper
 
config = ScraperConfig(scraper_api={
    "provider": "scraperapi",   # or "scrapeops", "scrapingbee"
    "api_key": "YOUR_KEY",
    "render_js": True,          # optional
})
 
with AmazonScraper(config) as scraper:
    result = scraper.scrape(query="laptop")
```
Tokens:
Raw