Jina Reader

https://github.com/jina-ai/reader
Admin
Jina Reader is an API-first service that converts URLs, PDFs, and documents into LLM-friendly...
Tokens:109,349
Snippets:687
Trust Score:10
License:Apache-2.0
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### First-time setup commands

Source: https://github.com/jina-ai/reader/blob/main/CONTRIBUTING.md

Clones the repository, installs dependencies, and optionally starts the local MinIO bucket cache with Docker Compose. Requires Node.js 22+ and Docker only if using the bucket-cached storage mode.

```bash
git clone git@github.com:jina-ai/reader.git
cd reader
npm install
# Optional: only if you want the local bucket cache
docker compose up -d
```

--------------------------------

### Initialize and start a Crawler

Source: https://github.com/jina-ai/reader/blob/main/fixtures/sample.html

Initializes a crawler instance and starts crawling a given URL. This is a basic example from the 'Section Four: Code' section of the guide.

```javascript
const crawler = new Crawler();
crawler.start('https://example.com');
```

--------------------------------

### Set up local development environment

Source: https://github.com/jina-ai/reader/blob/main/README.md

Sets up the local development environment: clone the repository, install dependencies, and optionally start a MinIO bucket cache with Docker Compose. After configuring environment variables, run the development server with npm run dev.

```bash
git clone git@github.com:jina-ai/reader.git
cd reader
npm install
# Optional, for bucket-cached mode:
docker compose up -d
```

```bash
npm run dev
```

--------------------------------

### Run the development server

Source: https://github.com/jina-ai/reader/blob/main/CONTRIBUTING.md

Starts the MinIO service (if needed) and runs the development server. Requires environment variables to be exported beforehand.

```bash
docker compose up -d
npm run dev
```

--------------------------------

### insertAsyncHookMiddleware() method

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/entry-points.md

Installs a middleware seeding AsyncLocalContext per request: traceId (randomUUID), traceT0 (Date), googleTraceId (first segment of the x-cloud-trace-context header).

```ts
@runOnce()
override insertAsyncHookMiddleware()
```

--------------------------------

### Usage: fromCtx with mock context

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/turndown-tweakable-options.md

Usage example showing how to create options from a mock context, equivalent to setting the X-Md-Heading-Style and X-Md-Link-Style headers. The result is an object with headingStyle and linkStyle properties.

```ts
import { TurnDownTweakableOptions } from './src/dto/turndown-tweakable-options';

// Equivalent of headers: X-Md-Heading-Style: atx, X-Md-Link-Style: referenced
const opts = TurnDownTweakableOptions.fromCtx(mockCtx);
// → { headingStyle: 'atx', linkStyle: 'referenced' }
```

--------------------------------

### Create a Crawled record

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/db-models.md

Shows how to create a Crawled record using the static from method. The example sets required fields and logs the urlPathDigest property.

```ts
import { Crawled, SERPResult } from './src/db/models';

const record = Crawled.from({
    _id: '…digest…', url: 'https://example.com', urlPathDigest: '…',
    createdAt: new Date(), expireAt: new Date(Date.now() + 3.6e6), traits: ['content'],
});
console.log(record.urlPathDigest);
```

--------------------------------

### listen(port) method

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/entry-points.md

Starts the h2c server on port and, when an alternative (plain h1) server exists, also listens on port + 1.

```ts
override listen(port: number)
```

--------------------------------

### GET /{q}

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/endpoints.md

Performs a web search using the query in the path. Requires authentication.

```APIDOC
## GET /{q}

### Description
Performs a web search using the query specified in the path. Requires authentication for actual searches.

### Method
GET

### Endpoint
https://s.jina.ai/{q}

### Parameters
#### Path Parameters
- **q** (string) - Required - The search query.

#### Query Parameters
- **type** (string) - Optional - Search type: `web`, `images`, or `news`. Default is `web`.
- **count** (integer) - Optional - Number of results (0-20). Default is 10.
- **num** (integer) - Optional - Alias for `count`.
- **provider** (string) - Optional - Search provider: `google`, `bing`, or `reader` (local index).
- **engine** (string) - Optional - Alias for `provider`.
- **gl** (string) - Optional - Two-letter country code for results.
- **hl** (string) - Optional - Language code for results.
- **location** (string) - Optional - Location filter.
- **page** (integer) - Optional - Page number for pagination.
- **fallback** (boolean) - Optional - Relaxes the query when no results match.
- **nfpr** (boolean) - Optional - Disables "Did you mean" suggestion.
- **site** (string) - Optional - Repeated; filters results to specific sites (e.g., `?site=jina.ai&site=github.com`).

### Request Example
```
curl -H 'Authorization: Bearer YOUR_TOKEN' 'https://s.jina.ai/Who%20will%20win%202024%20US%20presidential%20election%3F'
```

### Response
#### Success Response (200)
Returns an array of formatted page objects with title, url, description, content, images, links, and favicon (if `X-With-Favicons: 1` is set). The response format depends on the `Accept` header: JSON, text/plain, or SSE stream.

#### Response Example
```
[
  {
    "title": "Example Title",
    "url": "https://example.com",
    "description": "Example description",
    "content": "Full page content",
    "images": [],
    "links": [],
    "favicon": "https://example.com/favicon.ico"
  }
]
```

#### Error Responses
- **401** - Authentication required. Anonymous requests raise an `AuthenticationRequiredError`.
- **422** - `No search results available for query …` (AssertionFailureError).
```

--------------------------------

### GET https://r.jina.ai/{url}

Source: https://github.com/jina-ai/reader/blob/main/README.md

Fetches a single URL and converts it to an LLM-friendly input format.

```APIDOC
## [GET] https://r.jina.ai/{url}

### Description
Fetches a single URL and converts it to an LLM-friendly input format (markdown by default).

### Method
GET

### Endpoint
https://r.jina.ai/{url}

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL to fetch and convert.

### Request Example
https://r.jina.ai/https://en.wikipedia.org/wiki/Artificial_intelligence

### Response
#### Success Response (200)
- **body** (string) - The converted content in the requested format (e.g., markdown).

#### Response Example
```markdown
---
title: "Example Domain"
description: "This domain is for use in illustrative examples."
url: "https://example.com/"
---

## Example Domain

This domain is for use in illustrative examples in documents. ...
```
```

--------------------------------

### GET https://r.jina.ai/{url} (with X-Respond-With header)

Source: https://github.com/jina-ai/reader/blob/main/README.md

Fetches a single URL with configurable output format via request headers.

```APIDOC
## [GET] https://r.jina.ai/{url} (with X-Respond-With header)

### Description
Fetches a single URL and returns the content in a format specified by the `X-Respond-With` request header.

### Method
GET

### Endpoint
https://r.jina.ai/{url}

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL to fetch.

#### Request Headers
- **X-Respond-With** (string) - Optional - Specifies the output format. Supported values: `markdown`, `html`, `text`, `screenshot`, `pageshot`, `frontmatter`.

### Request Example
```bash
curl -H 'X-Respond-With: frontmatter' 'https://r.jina.ai/https://example.com'
```

### Response
#### Success Response (200)
- **body** (string) - The content in the requested format.

#### Response Example
```markdown
---
title: "Example Domain"
description: "This domain is for use in illustrative examples."
url: "https://example.com/"
---

## Example Domain

This domain is for use in illustrative examples in documents. ...
```
```

--------------------------------

### Usage: from with object literal

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/turndown-tweakable-options.md

Usage example showing how to create options directly from an object using the from method. Demonstrates setting headingStyle, bulletListMarker, linkStyle, and linkReferenceStyle, then accessing the headingStyle property.

```ts
import { TurnDownTweakableOptions } from './src/dto/turndown-tweakable-options';

const md = TurnDownTweakableOptions.from({
    headingStyle: 'atx',
    bulletListMarker: '-',
    linkStyle: 'referenced',
    linkReferenceStyle: 'full',
});
console.log(md.headingStyle); // 'atx'
```

--------------------------------

### Search API curl examples

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/endpoints.md

Three curl commands demonstrating the Search API. The first performs a basic search, the second adds site filtering with repeated site parameters, and the third requests JSON output with no-content and favicons, using count, provider, and type parameters.

```bash
curl -H 'Authorization: Bearer YOUR_TOKEN' 'https://s.jina.ai/Who%20will%20win%202024%20US%20presidential%20election%3F'
```

```bash
curl -H 'Authorization: Bearer YOUR_TOKEN' 'https://s.jina.ai/When%20was%20Jina%20AI%20founded%3F?site=jina.ai&site=github.com'
```

```bash
curl -H 'Authorization: Bearer YOUR_TOKEN' -H 'Accept: application/json' \
     -H 'X-Respond-With: no-content' -H 'X-With-Favicons: 1' \
     'https://s.jina.ai/llm?count=5&provider=bing&type=news'
```

--------------------------------

### Standalone server instantiation and boot (dry-run / PORT)

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/entry-points.md

Module-level instantiation and boot code, identical in all three standalone entry-point files (crawl.ts, search.ts, serp.ts). Resolves the server from the container and exports it as default; when run as the main module, it waits for serviceReady and either terminates immediately in dry-run mode (NODE_ENV containing 'dry-run') or starts the h2c server on the PORT environment variable, defaulting to 3000.

```typescript
const instance = container.resolve(CrawlStandAloneServer);
export default instance;

if (require.main === module) {
    if (process.env.NODE_ENV?.includes('dry-run')) {
        instance.serviceReady().then(() => finalizer.terminate());
    } else {
        instance.serviceReady().then((s) => s.h2c().listen(parseInt(process.env.PORT || '') || 3000));
    }
}
```

--------------------------------

### Deep research with link summary footer (x-retain-links: text)

Source: https://github.com/jina-ai/reader/blob/main/cookbooks.md

Use for deep-research or agentic models that need URLs only once. Moves links into a deduplicated footer, keeps anchor text inline, and retains image alt text. Swap 'x-retain-links: text' for 'x-retain-links: gpt-oss' to get citation tokens like 【1†...】, which auto-enables the summary footer. The 'x-preset: research' shortcut applies a similar setup with all links inline and chunks at h3; 'x-preset: spider' also collects the full link inventory.

```bash
curl https://r.jina.ai/https://example.com/article \
  -H 'x-retain-links: text' \
  -H 'x-with-links-summary: true' \
  -H 'x-retain-images: alt'
```

--------------------------------

### GET https://r.jina.ai/{url}

Source: https://github.com/jina-ai/reader/blob/main/README.md

GET request to fetch a page with optional headers for waiting on network idle or a specific element.

```APIDOC
## [GET] https://r.jina.ai/{url}

### Description
Fetches a web page and converts it to markdown. Supports various headers to control the output, including waiting for network idle or a specific element.

### Method
GET

### Endpoint
https://r.jina.ai/{url}

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL of the page to fetch.

#### Headers
- **x-timeout** (integer) - Optional - Timeout in seconds to wait for network idle or until timeout.
- **x-wait-for-selector** (string) - Optional - CSS selector to wait for before returning the content.

### Request Example
```bash
curl 'https://r.jina.ai/https://example.com/' -H 'x-timeout: 10'
```

### Response
#### Success Response (200)
- The markdown content of the fetched page.

#### Response Example
```markdown
[Fetched markdown content]
```
```

--------------------------------

### Embed the crawl server with tsyringe

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/entry-points.md

Boots the crawl server by resolving CrawlStandAloneServer from the DI container, waiting for service readiness, and starting an h2c listener on port 4000. Requires 'reflect-metadata' and tsyringe.

```ts
import 'reflect-metadata';
import { container } from 'tsyringe';
import instance, { CrawlStandAloneServer } from './src/stand-alone/crawl';

async function boot() {
    const server = container.resolve(CrawlStandAloneServer);
    await server.serviceReady();
    server.h2c().listen(4000);
    console.log('reader crawl API on http://localhost:4000');
}
```

--------------------------------

### Building BaseAuthDTO from a Koa context

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/base-auth.md

Shows how to build a BaseAuthDTO from a Koa context carrying a bearer token. The example imports from tsyringe and the base-auth module, then logs the bearerToken, ip, and isInternal fields.

```typescript
import { container } from 'tsyringe';
import { BaseAuthDTO } from './src/dto/base-auth';

// Build from a Koa context carrying a bearer token
const auth = BaseAuthDTO.from({ [RPC_CALL_ENVIRONMENT]: ctx, _token: undefined });
console.log(auth.bearerToken, auth.ip, auth.isInternal);
```

--------------------------------

### Request frontmatter output with X-Respond-With header

Source: https://github.com/jina-ai/reader/blob/main/README.md

This example shows how to request the 'frontmatter' output format from the Reader API using the 'X-Respond-With' header. The response includes a YAML front matter block with title, description, and URL, followed by the markdown content. The default response uses a custom header format instead.

```bash
curl -H 'X-Respond-With: frontmatter' 'https://r.jina.ai/https://example.com'
```

```markdown
---
title: "Example Domain"
description: "This domain is for use in illustrative examples."
url: "https://example.com/"
---

## Example Domain

This domain is for use in illustrative examples in documents. ...
```

--------------------------------

### CrawlStandAloneServer class shape (KoaServer singleton)

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/entry-points.md

Common class shape for the three standalone entry-point servers (crawl, search, serp). Each class extends KoaServer, is a singleton, and composes the same RPCRegistry shim, differing only in the RPC host injected (crawlerHost, searcherHost, or serpHost) and which routes survive the init filter. The h2c() method swaps httpServer for an http2.createServer wrapping the Koa callback; listen() also starts an alternative HTTP server on port+1.

```typescript
@singleton()
export class CrawlStandAloneServer extends KoaServer {
    logger = this.globalLogger.child({ service: this.constructor.name });

    httpAlternativeServer?: typeof this['httpServer'];
    assets = new Map<string, WalkOutEntity>();

    constructor(
        protected globalLogger: GlobalLogger,
        protected registry: RPCRegistry,
        protected crawlerHost: CrawlerHost,          // search: searcherHost; serp: serpHost
        protected threadLocal: AsyncLocalContext,
        protected threads: ThreadedServiceRegistry,
    ) {
        super(...arguments);
    }

    h2c() { /* swap httpServer for an http2.createServer wrapping the koa callback */ }
    override async init() { /* walkForAssets(); dependencyReady(); (search/serp: drop crawl routes); super.init() */ }
    async walkForAssets() { /* FsWalk.walkOut(<root>/public) into this.assets */ }
    override listen(port: number) { /* super.listen + listen(port + 1) on httpAlternativeServer */ }
    makeAssetsServingController() { /* serve files from public/ with mimeOfExt + Content-Length */ }
    registerRoutes(): void { /* koaCompress + assets + registry.makeShimController() */ }
    @runOnce() override insertAsyncHookMiddleware() { /* threadLocal.setup({traceId: randomUUID(), traceT0, googleTraceId}) */ }
    @Finalizer() override async standDown() { /* close httpAlternativeServer idle connections, then super.standDown() */ }
}
```

--------------------------------

### search (GET|POST / and /{q})

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/serp-host.md

The search method is the primary RPC method of SerpHost, exposed via HTTP GET and POST on the root path and the /{q} path. It returns raw search engine result metadata without fetching page content.

```APIDOC
## [GET|POST] / and /{q}

### Description
Performs a search using the SERP API, returning raw search-engine result metadata (title, URL, description, images, date, sitelinks) without fetching page content. Supports web, images, and news search variants, with optional provider, engine, and pagination parameters.

### Method
GET or POST

### Endpoint
`/` (when `q` is provided as a query parameter) or `/{q}` (when `q` is provided as a path parameter)

### Parameters
#### Query Parameters
- **type** (string) - Optional - Search variant: `web`, `images`, or `news`. Default: `web`.
- **q** (string) - Optional - The search query. Required when using the `/` endpoint without a path parameter.
- **provider** (string) - Optional - Search provider: `google` or `bing`. (Alias: `engine`)
- **engine** (string) - Optional - Search engine: `google` or `bing`. (Alias: `provider`)
- **num** (number) - Optional - Number of results to return. Must be between 0 and 20.
- **gl** (string) - Optional - Country code for search results (e.g., `us`). Must be a valid world country code.
- **hl** (string) - Optional - Language code for search results (e.g., `en`). Must be a valid world language code.
- **location** (string) - Optional - Location for search results.
- **page** (number) - Optional - Page number for pagination.
- **fallback** (boolean) - Optional - If `true`, retries with trimmed query term groups on failure.
- **nfpr** (boolean) - Optional - If `true`, disables personalization.

### Request Example
```
GET /?q=example&type=web&provider=google&num=5
```

### Response
#### Success Response (200)
Returns an array of search result objects, each containing metadata such as title, URL, description, images, date, and sitelinks. The response includes `X-Usage-Tokens` header and metadata via `assignMeta`.

#### Response Example
```json
[
  {
    "title": "Example Result",
    "url": "https://example.com",
    "description": "Example description"
  }
]
```

### Error Handling
- If `q` is not provided on the `/` endpoint, a `ParamValidationError` is thrown with message "Required but not provided".
- If `type` is not one of `web`, `images`, `news`, a validation error is thrown.
- If `num` is outside 0-20, a validation error is thrown.
- If `gl` is not a valid country code, a validation error is thrown.
- If `hl` is not a valid language code, a validation error is thrown.
```

--------------------------------

### Crawl API usage examples (curl)

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/endpoints.md

Examples of using the Crawl API with curl: basic anonymous crawl, header-based options, JSON output via POST, SSE streaming, and direct PDF upload. The examples show the expected request format and headers.

```bash
# Basic crawl (anonymous)
curl 'https://r.jina.ai/https://en.wikipedia.org/wiki/Artificial_intelligence'

# Crawl with options as headers
curl -H 'X-Respond-With: markdown' -H 'X-Engine: browser' \
     -H 'Authorization: Bearer YOUR_TOKEN' \
     'https://r.jina.ai/https://example.com/page'

# JSON output via POST
curl -X POST 'https://r.jina.ai/' \
  -H 'Content-Type: application/json' -H 'Accept: application/json' \
  -d '{"url": "https://example.com", "respondWith": "content", "retainImages": "alt"}'

# SSE streaming
curl -H 'Accept: text/event-stream' 'https://r.jina.ai/https://example.com'

# Upload a PDF directly (base64 body field)
curl -X POST 'https://r.jina.ai/' \
  -H 'Content-Type: application/json' -H 'Accept: application/json' \
  -d '{"pdf": "<base64>", "page": 2}'
```

--------------------------------

### crawl

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/crawler-host.md

The crawl method is exposed as two HTTP routes: POST / and GET ::url. Both routes crawl a given URL and return markdown content. POST requests accept crawler options in the request body, while GET requests accept them via headers.

```APIDOC
## POST /

### Description
Crawls any URL into markdown. This method is used for POST requests to the root path and accepts crawler options in the request body.

### Method
POST

### Endpoint
/

### Parameters
#### Request Body
- **crawlerOptions** (CrawlerOptions) - Optional - Crawler options that can be provided in the request body.

### Request Example
```json
{
  "url": "https://example.com",
  "options": {}
}
```

### Response
#### Success Response (200)
- **RawString** - The raw markdown content.
- **FormattedPageDto** - A formatted page DTO.
- **OutputServerEventStream** - An SSE stream output.

#### Response Example
```json
{
  "content": "# Example\n\nThis is the markdown content."
}
```

### Error Handling
- **400 ParamValidationError** - Thrown when no URL is provided.
- **422 AssertionFailureError** - Thrown when no content is available.
- **451 SecurityCompromiseError** - Thrown when the domain is blocked.

## GET ::url

### Description
Crawls any URL into markdown. This method is used for GET requests to a URL path and accepts crawler options via headers only.

### Method
GET

### Endpoint
::url

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL to crawl.

#### Headers
- **crawlerOptions** (CrawlerOptionsHeaderOnly) - Optional - Crawler options provided via headers.

### Request Example
```
GET https://r.jina.ai/https://example.com
```

### Response
#### Success Response (200)
- **RawString** - The raw markdown content.
- **FormattedPageDto** - A formatted page DTO.
- **OutputServerEventStream** - An SSE stream output.

#### Response Example
```json
{
  "content": "# Example\n\nThis is the markdown content."
}
```

### Error Handling
- **400 ParamValidationError** - Thrown when no URL is provided.
- **422 AssertionFailureError** - Thrown when no content is available.
- **451 SecurityCompromiseError** - Thrown when the domain is blocked.
```

--------------------------------

### configure

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/serp-host.md

Builds ScrappingOptions for SERP fetching from CrawlerOptions.

```APIDOC
## configure

### Description
Builds `ScrappingOptions` for SERP fetching: `proxyUrl`, `cookies` (`setCookies`), `overrideUserAgent` (`userAgent`), `timeoutMs` (`timeout * 1000`), `locale`, `referer`, `viewport`, `proxyResources` (when `proxyUrl` or `proxy` ends with `+`), `allocProxy` (`proxy` minus trailing `+`), plus `Accept-Language` when `locale` set.

### Method
async

### Endpoint
`src/api/serp.ts:281`

### Parameters
- **opts** (CrawlerOptions) - Required - The crawler options.

### Returns
- **Promise<ScrappingOptions>** - The constructed scraping options.
```

--------------------------------

### walkForAssets() method

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/entry-points.md

Walks <packageRoot>/public via FsWalk.walkOut and stores WalkOutEntity per relative path in this.assets.

```ts
async walkForAssets(): Promise<void>
```

--------------------------------

### newPage

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/puppeteer.md

Creates a dedicated incognito browser context and page with stealth setup and request filtering.

```APIDOC
## newPage

### Description
Creates a dedicated incognito BrowserContext and page. Failure to create a page kills the browser and throws ServiceNodeResourceDrainError. Applies stealth setup, bypasses CSP, sets a 1280x1280 viewport, sets the effective user agent, injects scripts, and sets up a request-filter queuing mechanism capped at concurrentRequestsPerPage.

### Method
async newPage(bewareDeadLock: any = false): Promise<Page>

### Parameters
- **bewareDeadLock** (any) - Optional - Defaults to false. When true, may avoid deadlock scenarios.

### Returns
- **Promise<Page>** - A promise that resolves to a Page instance.

### Source
`src/services/puppeteer.ts:800`
```

--------------------------------

### CURL sideLoadBlob

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/curl.md

Documentation for the sideLoadBlob method of the CurlControl service, including usage examples and error handling.

```APIDOC
## [CURL] sideLoadBlob

### Description
Fetches a URL using the CurlControl service and returns the response as a FancyFile. The method supports configurable timeouts, maximum response size, and extra headers.

### Method
CURL

### Endpoint
sideLoadBlob(url: URL, opts: CURLScrappingOptions)

### Parameters
#### Path Parameters
- **url** (URL) - Required - The URL to fetch.

#### Request Body
- **timeoutMs** (number) - Optional - Timeout in milliseconds for the request.
- **maxSize** (number) - Optional - Maximum allowed response size in bytes.
- **extraHeaders** (object) - Optional - Additional HTTP headers to include in the request.

### Request Example
```ts
const opts: CURLScrappingOptions = {
    timeoutMs: 15000,
    maxSize: 10 * 1024 * 1024,
    extraHeaders: { 'Accept-Language': 'en-US,en;q=0.9' },
};
const r = await curl.sideLoadBlob(url, opts);
```

### Response
#### Success Response
- **status** (number) - HTTP status code of the response.
- **contentType** (string) - Content type of the response body.
- **finalURL** (URL) - The final URL after any redirects.
- **file** (FancyFile) - The fetched content as a FancyFile object.
- **sideLoadOpts** (object) - Options that can be fed back to the browser engine.

#### Response Example
```ts
console.log(r.status, r.contentType, r.finalURL?.href);
```

### Errors
- **ServiceBadAttemptError** (50304) - Retryable fetch failures (timeouts, 5xx, connection-level).
- **ServiceBadApproachError** (50305) - Non-retryable policy violations (unsupported scheme, invalid request).
- **TargetFileTooLargeError** (41302) - Body exceeds `maxSize`.
```

--------------------------------

### GET https://s.jina.ai/{query}

Source: https://github.com/jina-ai/reader/blob/main/README.md

Performs a web search and returns the content of the top 5 results.

```APIDOC
## [GET] https://s.jina.ai/{query}

### Description
Performs a web search for the given query, fetches the top 5 results, and returns the content of each result in an LLM-friendly format.

### Method
GET

### Endpoint
https://s.jina.ai/{query}

### Parameters
#### Path Parameters
- **query** (string) - Required - The search query. Must be URL-encoded.

#### Query Parameters
- **site** (string) - Optional - Restricts the search to the specified site(s). Can be repeated to specify multiple sites.

### Request Example
https://s.jina.ai/Who%20will%20win%202024%20US%20presidential%20election%3F

### Response
#### Success Response (200)
- **body** (string) - The fetched content from the top 5 search results.

#### Response Example
(Not provided in source)
```

--------------------------------

### DefaultBucket class implementation

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/storage.md

Defines the DefaultBucket singleton service. Initializes object storage options from environment variables (GCP_STORAGE_ENDPOINT, GCP_STORAGE_REGION, GCP_STORAGE_BUCKET, GCP_STORAGE_ACCESS_KEY, GCP_STORAGE_SECRET_KEY) with defaults. Provides readSingleFile, putBuffer, and putBufferIfNotExist methods.

```typescript
@singleton()
export class DefaultBucket extends AbstractObjectStorageService {
    logger = this.globalLogger.child({ service: this.constructor.name });
    override options!: ObjectStorageOptions;

    constructor(protected globalLogger: GlobalLogger, protected envConfig: EnvConfig) { super(...arguments); }

    override async init(): Promise<void> {
        await this.dependencyReady();
        const bucketUrl = new URL(process.env.GCP_STORAGE_ENDPOINT || 'https://storage.googleapis.com');
        this.options = {
            region: this.envConfig.GCP_STORAGE_REGION || 'us-central1',
            bucket: process.env.GCP_STORAGE_BUCKET || `${process.env.GCLOUD_PROJECT}.appspot.com`,
            endPoint: bucketUrl.hostname,
            port: Number(bucketUrl.port) || (bucketUrl.protocol === 'https:' ? 443 : 80),
            useSSL: bucketUrl.protocol === 'https:',
            accessKey: this.envConfig.GCP_STORAGE_ACCESS_KEY,
            secretKey: this.envConfig.GCP_STORAGE_SECRET_KEY,
        };
        await super.init();
        this.emit('ready');
    }

    async readSingleFile(path: string): Promise<Buffer>
    async putBuffer(path: string, buffer: Buffer, metadata?: Record<string, any>)
    async putBufferIfNotExist(path: string, buffer: Buffer, metadata?: Record<string, any>)
}
```

--------------------------------

### Usage: AsyncLocalContext set/get

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/registry.md

Shows how to use AsyncLocalContext to set and get context values like 'uid' and 'traceId'.

```ts
import { AsyncLocalContext } from './src/services/async-context';

const ctx = container.resolve(AsyncLocalContext);
ctx.set('uid', 'u_123');
ctx.set('traceId', crypto.randomUUID());
console.log(ctx.get('uid'));
```

--------------------------------

### GET/POST / and /{q}

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/endpoints.md

SERP API endpoints for search queries.

```APIDOC
## [GET, POST] /

### Description
Exposes the SERP (Search Engine Results Page) API. Runs cached searches across providers, caches results, and returns search metadata (title, url, description, images, date, sitelinks). No page content fetching is performed.

### Method
- GET
- POST

### Endpoint
`/` and `/{q}`

### Parameters
#### Query Parameters
- **q** (string) - Required - The search query.
- **type** (string) - Optional - Type of search: `web`, `images`, or `news`. Defaults to `web`.
- **provider** / **engine** (string) - Optional - Search provider: `google` or `bing`.
- **num** (integer) - Optional - Number of results, 0–20.
- **gl** (string) - Optional - Geographic location.
- **hl** (string) - Optional - Language.
- **location** (string) - Optional - Location.
- **page** (integer) - Optional - Page number.
- **fallback** (boolean) - Optional - Enables retries with progressively trimmed queries.
- **nfpr** (string) - Optional - Filter parameter.

#### Path Parameters
- **q** (string) - Optional - The search query if provided as a path parameter.

#### Request Body (POST)
- **CrawlerOptions** (object) - Optional - JSON body with crawler options.

### Request Example
```bash
curl 'https://serp.jina.ai/?q=large+language+models&type=web&num=10&hl=en'
```

### Response
#### Success Response (200)
Returns a JSON array of result entries with the following fields:
- **title** (string) - Title of the result.
- **url** (string) - URL of the result.
- **description** (string) - Description of the result.
- **imageUrl** (string) - URL of the result image.
- **imageWidth** (integer) - Width of the result image.
- **imageHeight** (integer) - Height of the result image.
- **source** (string) - Source of the result.
- **date** (string) - Date of the result.
- **siteLinks** (array) - Sitelinks associated with the result.

Response meta includes `usage.tokens` metadata. Also available as `text/plain` and SSE formats.

#### Response Example
```json
[
  {
    "title": "Example Title",
    "url": "https://example.com",
    "description": "Example description",
    "imageUrl": "https://example.com/image.jpg",
    "imageWidth": 100,
    "imageHeight": 100,
    "source": "google",
    "date": "2025-01-01",
    "siteLinks": []
  }
]
```

### Error Handling
- **400** `ParamValidationError` when `q` is missing (`Required but not provided`).
- **422** `AssertionFailureError` when no provider can serve.

```

--------------------------------

### Format snapshot with SnapshotFormatter

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/snapshot-formatter.md

Example of resolving SnapshotFormatter via tsyringe and calling formatSnapshot with mode 'content'. Logs the text and frontmatter representations.

```ts
import { container } from 'tsyringe';
import { SnapshotFormatter } from './src/services/snapshot-formatter';

async function format(snapshot, nominalUrl: URL) {
    const formatter = container.resolve(SnapshotFormatter);
    const out = await formatter.formatSnapshot('content', snapshot, nominalUrl);
    console.log(out.textRepresentation);
    console.log(out.frontmatterRepresentation);
}
```

--------------------------------

### Create and store a SERPResult

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/db-models.md

Demonstrates creating a SERPResult and storing it via storage.storeSERPResult. Uses objHashMd5B64Of to compute the queryDigest from the query object.

```ts
import { SERPResult } from './src/db/models';
import { objHashMd5B64Of } from 'civkit/hash';

const query = { q: 'jina', variant: 'web' };
const cached = SERPResult.from({
    query,
    queryDigest: objHashMd5B64Of(query),
    response: [{ link: 'https://example.com', title: 'Example' }],
    createdAt: new Date(),
    expireAt: new Date(Date.now() + 3.6e6),
});
await storage.storeSERPResult(cached);
```

--------------------------------

### Run tests and coverage with npm

Source: https://github.com/jina-ai/reader/blob/main/CONTRIBUTING.md

Commands for running unit, end-to-end, and combined tests, plus coverage reports using the Node.js built-in test runner and c8.

```bash
npm run test:unit       # unit tests
npm run test:e2e        # end-to-end tests (slower, hits docker services)
npm test                # both

npm run test:unit:coverage
npm run test:e2e:coverage
npm run test:coverage   # combined coverage report (c8)
```

--------------------------------

### SerpHost.iterProviders

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/serp-host.md

Shows how to iterate over search providers using the SerpHost class.

```APIDOC
## [TS] SerpHost.iterProviders

### Description
Iterates over the search providers in the order determined by the configuration, allowing inspection of provider classes.

### Method
iterProviders

### Endpoint
SerpHost.iterProviders(preference: string)

### Parameters
- **preference** (string) - Required - A string indicating provider preference (e.g., 'bing').

### Request Example
```ts
const host = container.resolve(SerpHost);
for (const provider of host.iterProviders('bing')) {
    console.log(provider.constructor.name);
}
```

### Response
#### Success Response
- **provider** (object) - An iterable of provider instances, each with a `constructor.name` property.
```

--------------------------------

### new MarkifyService(options)

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/markify.md

Creates a new MarkifyService instance with configuration options for heading style, GFM support, link styles, and base URL.

```APIDOC
## new MarkifyService(options)

### Description
Creates a new MarkifyService instance with configuration options for heading style, GFM support, link styles, and base URL.

### Constructor
`new MarkifyService(options)`

### Parameters
- **options** (object) - Required - Configuration options for the service.
  - **headingStyle** (string) - Optional - Heading style, e.g., 'atx'.
  - **gfm** (boolean) - Optional - Enable GitHub Flavored Markdown.
  - **linkStyle** (string) - Optional - Link style, e.g., 'referenced' or 'discarded'.
  - **linkReferenceStyle** (string) - Optional - Link reference style, e.g., 'full'.
  - **baseUrl** (string) - Optional - Base URL for resolving relative links.

### Example
```ts
const service = new MarkifyService({
    headingStyle: 'atx',
    gfm: true,
    linkStyle: 'referenced',
    linkReferenceStyle: 'full',
    baseUrl: 'https://example.com',
});
```
```

--------------------------------

### GET https://r.jina.ai/{url}

Source: https://github.com/jina-ai/reader/blob/main/cookbooks.md

Geo- and locale-sensitive scraping via the Reader API. Use headers to control proxy country, locale, and cookies.

```APIDOC
## [GET] https://r.jina.ai/{url}

### Description
Fetches a web page with geo- and locale-sensitive scraping options. Allows specifying a proxy country, locale, and cookies to control how the page is rendered and served.

### Method
GET

### Endpoint
https://r.jina.ai/{url}

### Parameters
#### Headers
- **x-proxy** (string) - Optional - Country code for the exit node (e.g., `de`). Requires a premium key.
- **x-locale** (string) - Optional - Locale string (e.g., `de-DE`) that sets `navigator.language` and `Accept-Language`.
- **x-set-cookie** (string) - Optional - Cookie string to set (e.g., `country=DE; Path=/`). Requests with this header skip the cache.

### Request Example
```bash
curl https://r.jina.ai/https://shop.example.com/product/123 \
  -H 'x-proxy: de' \
  -H 'x-locale: de-DE' \
  -H 'x-set-cookie: country=DE; Path=/'
```

### Response
#### Success Response (200)
Returns the scraped content in the requested format (default HTML, or JSON if `Accept: application/json`).

#### Response Example
(Not provided in source)

### Error Handling
- Requires a premium key for `x-proxy` and `x-locale` headers.
- Requests with `x-set-cookie` skip the cache.
```

--------------------------------

### getIndex

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/crawler-host.md

The getIndex method returns the service index. It is exposed as a GET route on the root path. If URL-related parameters are supplied, it delegates to the crawl method.

```APIDOC
## GET /

### Description
Returns the index of the service. If URL, file, PDF, or HTML parameters are supplied, it delegates to the crawl method; otherwise, it returns the service index with usage instructions.

### Method
GET

### Endpoint
/

### Parameters
#### Query Parameters
- **url** (string) - Optional - If provided, delegates to crawl.
- **file** (string) - Optional - If provided, delegates to crawl.
- **pdf** (string) - Optional - If provided, delegates to crawl.
- **html** (string) - Optional - If provided, delegates to crawl.

### Request Example
```
GET https://r.jina.ai/
```

### Response
#### Success Response (200)
- **String** - The index as plain text or JSON depending on content negotiation.
- **Object** - The index object with usage instructions.

#### Response Example
```json
{
  "usage1": "https://r.jina.ai/YOUR_URL",
  "usage2": "https://s.jina.ai/YOUR_SEARCH_QUERY",
  "homepage": "https://jina.ai/reader"
}
```
```

--------------------------------

### SerpHost.cachedSearch

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/serp-host.md

Demonstrates how to use the SerpHost class to perform a cached search with options.

```APIDOC
## [TS] SerpHost.cachedSearch

### Description
Performs a cached search using the specified variant and options. The method resolves the service, waits until it is ready, and returns an array of search results.

### Method
cachedSearch

### Endpoint
SerpHost.cachedSearch(variant: string, query: object, options?: CrawlerOptions)

### Parameters
- **variant** (string) - Required - The search variant (e.g., 'web').
- **query** (object) - Required - The search query object, containing fields like `q` and `num`.
- **options** (CrawlerOptions) - Optional - Additional crawler options, e.g., `{ noCache: true }`.

### Request Example
```ts
const opts = CrawlerOptions.from({ noCache: true });
const results = await host.cachedSearch('web', { q: 'jina reader', num: 10 }, opts);
```

### Response
#### Success Response
- **results** (Array) - An array of search result objects, each containing `title` and `link` properties.

#### Response Example
```ts
for (const r of results) {
    console.log(r.title, r.link);
}
```

### Errors
- `ParamValidationError` (400): `q` required on `/`-relative paths.
- `AssertionFailureError` (422): no provider capable of the variant.
- Inherits the error catalog of errors.md for upstream provider failures.
```

--------------------------------

### getFilteredTextStream usage

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/streams-and-helpers.md

Consumes an OpenAI-style SSE body and extracts delta content using a JSON path. The example fetches from the OpenAI API and accumulates the text chunks.

```typescript
import { getFilteredTextStream } from './src/lib/filtered-stream';

// Consume an OpenAI-style SSE body, extracting delta content
const resp = await fetch('https://api.openai.com/v1/chat/completions', { headers: {...}, body: … });
const stream = resp.body!.pipeThrough(getFilteredTextStream('data.choices[0].delta.content'));
let text = '';
for await (const chunk of stream) text += chunk;
```

--------------------------------

### Run Reader with S3 caching

Source: https://github.com/jina-ai/reader/blob/main/README.md

Run the container with S3-compatible storage environment variables to enable caching of fetched pages. Replace the endpoint, bucket, and credentials with your own values.

```bash
docker run --rm -p 3000:8081 \
  -e GCP_STORAGE_ENDPOINT=https://s3.example.com \
  -e GCP_STORAGE_BUCKET=reader-cache \
  -e GCP_STORAGE_ACCESS_KEY=... \
  -e GCP_STORAGE_SECRET_KEY=... \
  ghcr.io/jina-ai/reader:oss
```

--------------------------------

### Guarded function using assertTier

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/base-auth.md

Example of a guarded function that checks if the caller is internal, then calls assertTier with a feature name. It catches AuthenticationRequiredError and logs a message, otherwise rethrows the error.

```typescript
import { BaseAuthDTO } from './src/dto/base-auth';
import { AuthenticationRequiredError } from 'civkit/civ-rpc';

async function guarded(auth: BaseAuthDTO) {
    if (auth.isInternal) {
        return; // trusted internal caller
    }
    try {
        await auth.assertTier(2, 'Proxy allocation');
    } catch (err) {
        if (err instanceof AuthenticationRequiredError) {
            console.error('login required:', err.message);
        } else {
            throw err; // TierFeatureConstraintError etc.
        }
    }
}
```

--------------------------------

### registerRoutes() method

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/entry-points.md

Middleware order: koaCompress (compresses text/*, application/json, *+json, *+xml, application/x-ndjson) → asset controller → registry.makeShimController() (the RPC router + envelope).

```ts
registerRoutes(): void
```

--------------------------------

### Direct crawl with CrawlerHost

Source: https://github.com/jina-ai/reader/blob/main/_autodocs/api-reference/crawler-host.md

Shows how to resolve the CrawlerHost from the DI container and perform a direct crawl, equivalent to a GET request to r.jina.ai. The crawl method returns a formatted response with text representation.

```ts
import { container } from 'tsyringe';
import 'reflect-metadata';
import { CrawlerHost } from './src/api/crawler';
import { CrawlerOptions } from './src/dto/crawler-options';

async function main() {
    const host = container.resolve(CrawlerHost);
    await host.serviceReady();

    // Direct crawl (equivalent of GET https://r.jina.ai/https://example.com)
    const rpcReflect = /* provided by civkit runtime */;
    const ctx = /* koa Context */;
    const auth = /* BaseAuthDTO */;
    const opts = CrawlerOptions.from({ url: 'https://example.com', respondWith: 'markdown' });
    const formatted = await host.crawl(rpcReflect, ctx, auth, opts, opts);
    console.log(formatted.textRepresentation); // "Title: …\nURL Source: …\nMarkdown Content: …"
}
```
Tokens:
Raw