### Install and build the mcp-server-scraper project

Source: https://github.com/ofershap/mcp-server-scraper/blob/main/README.md

These bash commands outline the development workflow for the mcp-server-scraper project. They cover installing dependencies, running type checks, building the project, and executing tests.

```bash
npm install
npm run typecheck
npm run build
npm test
```

--------------------------------

### Run mcp-server-scraper from the command line

Source: https://github.com/ofershap/mcp-server-scraper/blob/main/README.md

This command initiates the mcp-server-scraper to scrape content from a URL. It requires Node.js and npm to be installed. The output will be the scraped content in markdown format.

```bash
npx mcp-server-scraper
```

--------------------------------

### Extract Links from Web Page

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Extracts all anchor links from a web page and resolves relative URLs to their absolute paths. It filters out anchor links (starting with '#') and javascript pseudo-links. The function returns an array of objects, each containing the resolved href and the link's text content. Dependencies include `linkedom`.

```typescript
export interface PageLink {
  href: string;
  text: string;
}

export async function extractLinks(url: string): Promise<PageLink[]> {
  const html = await fetchPage(url);
  const { document } = parseHTML(html);
  const anchors = Array.from(document.querySelectorAll("a[href]"));
  const links: PageLink[] = [];
  const baseUrl = new URL(url);

  for (const a of anchors) {
    const href = a.getAttribute("href");
    if (!href || href.startsWith("#") || href.startsWith("javascript:")) continue;

    try {
      const resolved = new URL(href, baseUrl).href;
      links.push({ href: resolved, text: (a.textContent ?? "").trim() });
    } catch {
      // skip invalid URLs
    }
  }
  return links;
}

// Usage example
const links = await extractLinks("https://example.com/docs");
// [
//   { href: "https://example.com/page1", text: "Link 1" },
//   { href: "https://example.com/page2", text: "Link 2" },
//   { href: "https://example.com/relative", text: "Relative Link" }
// ]
```

--------------------------------

### Configure mcp-server-scraper for Cursor

Source: https://github.com/ofershap/mcp-server-scraper/blob/main/README.md

This JSON configuration snippet shows how to integrate mcp-server-scraper with Cursor. It specifies the command to run ('npx') and its arguments ('-y', 'mcp-server-scraper') to enable scraping functionality within Cursor.

```json
{
  "mcpServers": {
    "scraper": {
      "command": "npx",
      "args": ["-y", "mcp-server-scraper"]
    }
  }
}
```

--------------------------------

### Scrape URL Content with Mozilla Readability

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Fetches content from a given URL and extracts readable text using the Mozilla Readability library. It returns structured data including the title, main content, excerpt, byline, site name, and content length. Dependencies include `@mozilla/readability` and `linkedom`.

```typescript
import { Readability } from "@mozilla/readability";
import { parseHTML } from "linkedom";

export interface ScrapedContent {
  title: string;
  content: string;
  excerpt: string;
  byline: string;
  siteName: string;
  length: number;
}

export async function scrapeUrl(url: string): Promise<ScrapedContent> {
  const html = await fetchPage(url);
  const { document } = parseHTML(html);
  const reader = new Readability(document as unknown as Document);
  const article = reader.parse();

  if (!article) {
    throw new Error("Could not extract readable content from the page");
  }

  const textContent = article.textContent?.trim() ?? "";
  return {
    title: article.title ?? "",
    content: textContent,
    excerpt: article.excerpt ?? "",
    byline: article.byline ?? "",
    siteName: article.siteName ?? "",
    length: article.length ?? textContent.length,
  };
}

// Usage example
const content = await scrapeUrl("https://example.com/article");
console.log(content.title);   // "Article Title"
console.log(content.excerpt); // "Brief summary of the article..."
console.log(content.content); // Full extracted text content
```

--------------------------------

### Search Page Content (TypeScript)

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Searches for a query string within the text content of a web page. It fetches and parses the HTML, extracts body text, and performs a case-insensitive search, returning an array of matching lines. Dependencies include fetchPage and parseHTML. It does not execute JavaScript.

```typescript
export async function searchPage(url: string, query: string): Promise<string[]> {
  const html = await fetchPage(url);
  const { document } = parseHTML(html);
  const text = document.body?.textContent ?? "";
  const lines = text
    .split("\n")
    .map((l: string) => l.trim())
    .filter(Boolean);
  const queryLower = query.toLowerCase();
  return lines.filter((line: string) =>
    line.toLowerCase().includes(queryLower)
  );
}
```

--------------------------------

### Configure mcp-server-scraper for VS Code

Source: https://github.com/ofershap/mcp-server-scraper/blob/main/README.md

This JSON configuration allows you to integrate mcp-server-scraper into your VS Code environment via MCP settings. It specifies the command and arguments needed to activate the scraping server.

```json
{
  "mcp": {
    "servers": {
      "scraper": {
        "command": "npx",
        "args": ["-y", "mcp-server-scraper"]
      }
    }
  }
}
```

--------------------------------

### Batch Scrape Multiple URLs with MCP Tool

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Scrapes multiple URLs in parallel, returning the title and excerpt for each. Individual URL scraping failures are reported without halting the entire batch process. This is ideal for surveying content across many web pages efficiently.

```typescript
server.tool(
  "scrape_multiple",
  "Batch scrape multiple URLs. Returns title and excerpt for each. Failures are reported per URL without failing the whole batch.",
  {
    urls: z.array(z.string().url()).describe("Array of URLs to scrape"),
  },
  async ({ urls }) => {
    const results = await scrapeMultiple(urls);
    // Returns: Array<{ url, title, excerpt, error? }>
    return { content: [{ type: "text", text: formattedResults }] };
  }
);

// Example usage via AI assistant:
// "Scrape these 5 URLs and give me a summary of each"

// Example response format:
// 1. **Getting Started Guide** — Learn how to set up the project...
//    https://example.com/docs/getting-started
//
// 2. **API Reference** — Complete API documentation...
//    https://example.com/docs/api
//
// 3. https://example.com/404 — Error: HTTP 404: Not Found
```

--------------------------------

### Search Page Content with MCP Tool

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Searches a given URL for a query string within its text content. It returns all lines that contain the query in a case-insensitive manner. This tool is useful for finding specific mentions of terms within web pages.

```typescript
server.tool(
  "search_page",
  "Search for a query string within the page text. Returns matching lines (one per line). Use for finding mentions of a term.",
  {
    url: z.string().url().describe("The URL to search"),
    query: z.string().describe("The search query"),
  },
  async ({ url, query }) => {
    const lines = await searchPage(url, query);
    // Returns: Array<string> - lines containing the query
    return { content: [{ type: "text", text: formattedResults }] };
  }
);

// Example usage via AI assistant:
// "Search this page for mentions of 'authentication'"

// Example response format:
// 1. Line one with authentication
// 2. Line three with authentication token

// If no matches found:
// No lines containing "authentication" found.
```

--------------------------------

### MCP Tool: scrape_url

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Extracts clean, readable text content from a URL using Mozilla Readability. It returns the page title, excerpt, byline, site name, and main content. This tool is best suited for articles, documentation pages, and blog posts.

```APIDOC
## POST /scrape_url

### Description
Extracts clean, readable text content from a URL using Mozilla Readability. Returns the page title, excerpt, byline, site name, and main content. Best suited for articles, documentation pages, and blog posts.

### Method
POST

### Endpoint
/scrape_url

### Parameters
#### Query Parameters
- **url** (string) - Required - The URL to scrape

### Request Body
(Not applicable for this tool, parameters are passed as query parameters)

### Request Example
(Not applicable, use query parameters)

### Response
#### Success Response (200)
- **content** (Array<{ type: string, text: string }>) - The scraped content, formatted as markdown.

#### Response Example
```json
{
  "content": [
    {
      "type": "text",
      "text": "# Article Title\n*Author Name*\n*Site Name*\n\n> Article excerpt or summary\n\n---\n\nMain content of the article...\n\n_(12345 characters)_"
    }
  ]
}
```
```

--------------------------------

### MCP Tool: extract_metadata

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Extracts page metadata including the title, description, Open Graph tags (og:title, og:description, og:image), canonical URL, and favicon. This is useful for SEO analysis or generating link previews.

```APIDOC
## POST /extract_metadata

### Description
Extracts page metadata: title, description, Open Graph tags (og:title, og:description, og:image), canonical URL, and favicon.

### Method
POST

### Endpoint
/extract_metadata

### Parameters
#### Query Parameters
- **url** (string) - Required - The URL to extract metadata from

### Request Body
(Not applicable for this tool, parameters are passed as query parameters)

### Request Example
(Not applicable, use query parameters)

### Response
#### Success Response (200)
- **content** (Array<{ type: string, text: string }>) - A text representation of the extracted metadata.

#### Response Example
```json
{
  "content": [
    {
      "type": "text",
      "text": "**Title:** Page Title\n**Description:** Page description here\n**og:title:** OG Title\n**og:description:** OG Description\n**og:image:** https://example.com/image.png\n**Canonical:** https://example.com/canonical\n**Favicon:** /favicon.ico"
    }
  ]
}
```
```

--------------------------------

### Extract Page Metadata (TypeScript)

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Extracts metadata such as title, description, Open Graph tags, canonical URL, and favicon from a given URL. It relies on fetchPage and parseHTML functions (not provided) and returns a PageMetadata object. Limitations include not executing JavaScript.

```typescript
export interface PageMetadata {
  title: string;
  description: string;
  ogTitle: string;
  ogDescription: string;
  ogImage: string;
  canonical: string;
  favicon: string;
}

export async function extractMetadata(url: string): Promise<PageMetadata> {
  const html = await fetchPage(url);
  const { document } = parseHTML(html);

  const getMeta = (name: string): string => {
    const el = document.querySelector(
      `meta[property="${name}"], meta[name="${name}"]`
    );
    return el?.getAttribute("content") ?? "";
  };

  return {
    title: document.querySelector("title")?.textContent ?? "",
    description: getMeta("description"),
    ogTitle: getMeta("og:title"),
    ogDescription: getMeta("og:description"),
    ogImage: getMeta("og:image"),
    canonical: document.querySelector("link[rel='canonical']")?.getAttribute("href") ?? "",
    favicon: document.querySelector("link[rel='icon'], link[rel='shortcut icon']")?.getAttribute("href") ?? "",
  };
}
```

--------------------------------

### Scrape Multiple URLs in Parallel (TypeScript)

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Batches scraping of multiple URLs concurrently using Promise.allSettled. It returns an array of results, indicating success (with URL, title, excerpt) or failure (with URL and an error message). Dependencies include a scrapeUrl function. This function does not execute JavaScript.

```typescript
export async function scrapeMultiple(
  urls: string[]
): Promise<{ url: string; title: string; excerpt: string; error?: string }[]> {
  const results = await Promise.allSettled(
    urls.map(async (url) => {
      const content = await scrapeUrl(url);
      return {
        url,
        title: content.title,
        excerpt: content.excerpt,
      };
    })
  );

  return results.map((r, i) => {
    if (r.status === "fulfilled") return r.value;
    return {
      url: urls[i] ?? "",
      title: "",
      excerpt: "",
      error: String((r.reason as Error).message),
    };
  });
}
```

--------------------------------

### MCP Tool: extract_metadata

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Defines the 'extract_metadata' tool for the MCP Server Scraper. This tool extracts various metadata from a given URL, including the page title, description, Open Graph tags (og:title, og:description, og:image), canonical URL, and favicon. It is useful for SEO analysis and generating link previews.

```typescript
server.tool(
  "extract_metadata",
  "Extract page metadata: title, description, Open Graph tags (og:title, og:description, og:image), canonical URL, and favicon.",
  {
    url: z.string().url().describe("The URL to extract metadata from"),
  },
  async ({ url }) => {
    const meta = await extractMetadata(url);
    // Returns: { title, description, ogTitle, ogDescription, ogImage, canonical, favicon }
    return { content: [{ type: "text", text: formattedMetadata }] };
  }
);

// Example usage via AI assistant:
// "What's the OG image and description for this URL?"

// Example response format:
// **Title:** Page Title
// **Description:** Page description here
// **og:title:** OG Title
// **og:description:** OG Description
// **og:image:** https://example.com/image.png
// **Canonical:** https://example.com/canonical
// **Favicon:** /favicon.ico
```

--------------------------------

### MCP Tool: scrape_url

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Defines the 'scrape_url' tool for the MCP Server Scraper. This tool uses Mozilla Readability to extract clean text content, title, excerpt, byline, site name, and length from a given URL. It's best suited for articles, documentation, and blog posts.

```typescript
server.tool(
  "scrape_url",
  "Extract clean, readable text content from a URL using Mozilla Readability. Returns title, excerpt, and main content. Best for articles, docs, and blog posts.",
  {
    url: z.string().url().describe("The URL to scrape"),
  },
  async ({ url }) => {
    const content = await scrapeUrl(url);
    // Returns: { title, content, excerpt, byline, siteName, length }
    return { content: [{ type: "text", text: formattedMarkdown }] };
  }
);

// Example usage via AI assistant:
// "Scrape the API docs from https://docs.example.com and summarize them"

// Example response format:
// # Article Title
// *Author Name*
// *Site Name*
//
// > Article excerpt or summary
//
// ---
//
// Main content of the article...
//
// _(12345 characters)_
```

--------------------------------

### MCP Tool: extract_links

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Extracts all links from a given URL, including their href URLs and anchor text. It automatically resolves relative URLs to absolute paths and skips anchor links (#) and javascript: links.

```APIDOC
## POST /extract_links

### Description
Extracts all links from a page with their href and anchor text. Resolves relative URLs. Skips anchors and javascript: links.

### Method
POST

### Endpoint
/extract_links

### Parameters
#### Query Parameters
- **url** (string) - Required - The URL to extract links from

### Request Body
(Not applicable for this tool, parameters are passed as query parameters)

### Request Example
(Not applicable, use query parameters)

### Response
#### Success Response (200)
- **content** (Array<{ type: string, text: string }>) - A text representation of the extracted links, formatted as a list.

#### Response Example
```json
{
  "content": [
    {
      "type": "text",
      "text": "1. [Getting Started](https://example.com/docs/getting-started)\n2. [API Reference](https://example.com/docs/api)\n3. [Examples](https://example.com/docs/examples)\n4. [GitHub Repository](https://github.com/example/project)"
    }
  ]
}
```
```

--------------------------------

### MCP Tool: extract_links

Source: https://context7.com/ofershap/mcp-server-scraper/llms.txt

Defines the 'extract_links' tool for the MCP Server Scraper. This tool extracts all links from a given URL, including their href and anchor text. It automatically resolves relative URLs to absolute paths and skips anchor links and javascript: links.

```typescript
server.tool(
  "extract_links",
  "Extract all links from a page with their href and anchor text. Resolves relative URLs. Skips anchors and javascript: links.",
  {
    url: z.string().url().describe("The URL to extract links from"),
  },
  async ({ url }) => {
    const links = await extractLinks(url);
    // Returns: Array<{ href: string, text: string }>
    return { content: [{ type: "text", text: formattedLinks }] };
  }
);

// Example usage via AI assistant:
// "Extract all links from this page"

// Example response format:
// 1. [Getting Started](https://example.com/docs/getting-started)
// 2. [API Reference](https://example.com/docs/api)
// 3. [Examples](https://example.com/docs/examples)
// 4. [GitHub Repository](https://github.com/example/project)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.