ZPI SDK

https://github.com/zeative/zpi-sdk
Admin
ZPI SDK is a universal, zero-dependency TypeScript SDK for the Zest API (Zapi) scraper platform.
Tokens:4,516
Snippets:51
Trust Score:7.3
License:MIT
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install ZPI SDK

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Install the ZPI SDK using npm, pnpm, yarn, or bun. Requires Node.js v20+.

```bash
npm i zpi-sdk      # or: pnpm add zpi-sdk  •  yarn add zpi-sdk  •  bun add zpi-sdk
```

--------------------------------

### Import ZPI SDK in Deno

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Import the ZpiClient in Deno using the npm: specifier. No installation step is needed for Deno.

```typescript
import { ZpiClient } from 'npm:zpi-sdk'
```

--------------------------------

### ScraperMap Declaration Merge Example

Source: https://github.com/zeative/zpi-sdk/blob/main/docs/run-signature.md

An example of how to extend the `ScraperMap` interface using declaration merging to provide specific types for scraper results. This allows the `run()` method's return type to be narrowed.

```typescript
declare module "zpi-sdk" {
  interface ScraperMap {
    "social:instagram": {
      profile: { params: { username: string }; result: { id: string; name: string } };
    };
  }
}
```

--------------------------------

### Catalog Discovery

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Access public catalog information without authentication. Includes listing items, getting details, and retrieving schema or statistics for specific scrapers.

```typescript
const { items } = await client.catalog.list({ cat: 'social', limit: 20 })
const detail = await client.catalog.get('social:instagram')
const schema = await client.catalog.schema('social:instagram', 'profile')
const stats = await client.catalog.stats('social:instagram')
```

--------------------------------

### Initialize ZpiClient and Run Scraper

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Instantiate the ZpiClient with an API key and use the `run` method to execute a scraper. The SDK automatically handles HTTP methods and path parameters.

```typescript
import { ZpiClient } from 'zpi-sdk'

const client = new ZpiClient({ apiKey: 'zpi_...' })

const data = await client.run('social:instagram', 'profile', { username: 'instagram' })
console.log(data)
```

--------------------------------

### Configure ZpiClient

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Initialize the ZpiClient with optional configuration parameters such as API key, base URL, timeout, retries, and a custom fetch implementation.

```typescript
const client = new ZpiClient({
  apiKey: 'zpi_...',
  baseURL: 'https://api.zpi.web.id', // override the default base URL
  timeoutMs: 30_000,                 // per-request timeout
  maxRetries: 2,                     // retry budget for retryable failures
  fetch: globalThis.fetch,           // inject your own fetch (tests, proxies, edge runtimes)
})
```

--------------------------------

### RunOpts Interface

Source: https://github.com/zeative/zpi-sdk/blob/main/docs/run-signature.md

Defines the optional configuration for the `run` method. Key options include method, abort signal, timeout, and idempotency key for retries.

```typescript
interface RunOpts {
  method?: "GET" | "POST";        // default POST
  signal?: AbortSignal;           // composed with the per-request timeout (abort if EITHER fires)
  timeoutMs?: number;             // per-request; falls back to client timeoutMs
  idempotencyKey?: string;        // REQUIRED to make a failing POST retry-eligible; reused across attempts
  headers?: Record<string, string>;
  pathRest?: string;              // extra path segments appended after /v1/{projectKey}/{endpoint}
}
```

--------------------------------

### Create and Use MCP Client for AI Agents

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Utilize the `createMcpClient` from 'zpi-sdk/mcp' to allow AI agents to discover and call scrapers as MCP tools. This client communicates via JSON-RPC over a Streamable-HTTP server.

```typescript
import { createMcpClient } from 'zpi-sdk/mcp'

const mcp = createMcpClient({ apiKey: 'zpi_...' })

const tools = await mcp.listTools()           // handshake runs lazily on first call
const result = await mcp.callTool('run_scraper', { /* tool args */ })
```

--------------------------------

### client.run()

Source: https://github.com/zeative/zpi-sdk/blob/main/docs/run-signature.md

Executes a scraper request. It returns the unwrapped data from the backend envelope. The return type is `unknown` by default and can be narrowed using `ScraperMap`.

```APIDOC
## client.run()

### Description
Executes a scraper request. It returns the unwrapped data from the backend envelope `{ project, data, timestamp }` (i.e. only `.data`). Typed `unknown` by default; narrowed via `ScraperMap` when codegen has emitted entries.

### Method
`run<T = unknown>(
  projectKey: string,        // "category:scraper" (preferred) or bare "scraper" (legacy)
  endpoint: string,          // endpoint slug
  params?: Record<string, unknown>, // merged into query (GET) or JSON body (POST)
  opts?: RunOpts
): Promise<T>;

### Parameters
#### Path Parameters
- **projectKey** (string) - Required - The project key, typically in the format "category:scraper" or a legacy bare "scraper".
- **endpoint** (string) - Required - The endpoint slug for the scraper.

#### Query Parameters
- **params** (Record<string, unknown>) - Optional - Parameters merged into the query for GET requests or the JSON body for POST requests.

#### Request Body
(Implicitly handled by `params` based on method)

### Options (`RunOpts`)
#### `RunOpts` Interface
```ts
interface RunOpts {
  method?: "GET" | "POST";        // default POST
  signal?: AbortSignal;           // composed with the per-request timeout (abort if EITHER fires)
  timeoutMs?: number;             // per-request; falls back to client timeoutMs
  idempotencyKey?: string;        // REQUIRED to make a failing POST retry-eligible; reused across attempts
  headers?: Record<string, string>;
  pathRest?: string;              // extra path segments appended after /v1/{projectKey}/{endpoint}
}
```

### Behaviors
- **Auto Method**: When `opts.method` is omitted, a 405 response will cause the client to flip between GET and POST once, and the learned verb is memoized per `projectKey/endpoint`. Explicitly setting `opts.method` disables this behavior.
- **Forgiving Endpoint**: The `endpoint` parameter can include extra segments. Placeholders like `":param"` are stripped, and literal segments are appended to `pathRest`. For example, `"resolve"`, `"resolve/:url"`, and `"/resolve/"` are treated equivalently.

### Method / Retry / Timeout Contract
- **Default Method**: POST (subject to auto-method behavior).
- **Timeout**: Per-request `timeoutMs` is composed with the external `opts.signal`. A timeout results in `ZpiTimeoutError`, an external abort in `ZpiAbortError`, and a raw network failure in `ZpiNetworkError` (with a `.cause`).
- **Retry**: Retries are performed for network errors and specific HTTP status codes (429, 502, 503, 504) with exponential backoff and jitter, capped by `maxRetries`. The `Retry-After` header and the `content.retryAfterSec` field in the 429 response body are honored. Other 4xx errors are not retried. POST requests are only retried if an `idempotencyKey` is provided; the same key is reused across all retry attempts.

### `ScraperMap` Declaration-Merge Contract
`ScraperMap` is an empty interface by default. Codegen merges concrete entries via `declare module`. Example:
```ts
declare module "zpi-sdk" {
  interface ScraperMap {
    "social:instagram": {
      profile: { params: { username: string }; result: { id: string; name: string } };
    };
  }
}
```
The `ScraperResult<K, E>` helper resolves to `ScraperMap[K][E]["result"]` if a match exists, otherwise it defaults to `unknown`.
```

--------------------------------

### Run Scrapers

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Execute various scrapers using the client.run method, specifying the scraper category, name, and parameters.

```typescript
const profile = await client.run('social:instagram', 'profile', { username: 'instagram' })
const video = await client.run('downloader:tiktok', 'video', { url: 'https://tiktok.com/...' })
const gold = await client.run('finance:goldprice', 'latest', {})
```

--------------------------------

### Generate Typed Bindings

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Generate per-scraper type definitions using the 'zpi codegen' command. This creates types-only files for autocompletion of 'run()' parameters. Supports filtering by scraper category.

```bash
npx zpi codegen                 # → ./zpi-sdk.gen.d.ts
npx zpi codegen --out ./types/zpi.gen.d.ts --filter social
```

--------------------------------

### Client Run Signature

Source: https://github.com/zeative/zpi-sdk/blob/main/docs/run-signature.md

The `run` method signature for the client. It's used to execute scraper projects and returns a promise that resolves to the unwrapped data. The return type `T` defaults to `unknown` and can be narrowed by `ScraperMap`.

```typescript
client.run<T = unknown>(
  projectKey: string,        // "category:scraper" (preferred) or bare "scraper" (legacy)
  endpoint: string,          // endpoint slug
  params?: Record<string, unknown>, // merged into query (GET) or JSON body (POST)
  opts?: RunOpts
): Promise<T>;
```

--------------------------------

### Parse Webhooks

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Verify and parse incoming webhook deliveries using the 'zpi-sdk/webhooks' helper. Requires the raw request body and signature, and uses a webhook secret for verification. Throws ZpiWebhookVerifyError on invalid signatures or payloads.

```typescript
import { parseWebhook } from 'zpi-sdk/webhooks'

// In your HTTP handler — pass the RAW body string, not the parsed JSON:
const event = await parseWebhook(rawBody, {
  signature: req.headers['x-zpi-signature'],
  secret: process.env.ZPI_WEBHOOK_SECRET,
})
// → { id, event: 'bulk.completed' | …, data, deliveredAt }
// Throws ZpiWebhookVerifyError on a bad signature or malformed payload.
```

--------------------------------

### Handle ZPI Errors in TypeScript

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Catch and differentiate between various ZPI errors by checking their instance types. Import specific error classes for granular handling.

```typescript
import { ZpiClient, ZpiPlanGateError, ZpiRateLimitError, ZpiError } from 'zpi-sdk'

try {
  await client.run('social:instagram', 'profile', { username: 'instagram' })
} catch (e) {
  if (e instanceof ZpiPlanGateError) console.log('upgrade required:', e.requiredPlan, e.upgradeUrl)
  else if (e instanceof ZpiRateLimitError) console.log('retry after', e.retryAfterSec, 's')
  else if (e instanceof ZpiError) console.log('zpi error:', e.status, e.code, e.raw)
}
```

--------------------------------

### Submit and Wait for Bulk Jobs

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Submit multiple items for processing in a bulk job and wait for completion, with optional progress reporting. Iterates through results, logging status and data or errors.

```typescript
const job = await client.bulk.submit('social:instagram', 'profile', [
  { url: 'https://instagram.com/a' },
  { url: 'https://instagram.com/b' },
])

const result = await job.wait({
  onProgress: (j) => console.log(j.succeeded, '/', j.total),
})

for (const item of result.items ?? []) {
  console.log(item.status, item.data ?? item.error)
}
```

--------------------------------

### Stream Data

Source: https://github.com/zeative/zpi-sdk/blob/main/README.md

Process chunked or SSE endpoints by iterating over the async iterable returned by client.stream. Handles SseEvent or string data types.

```typescript
for await (const event of client.stream('ai:chat', 'completions', { prompt: 'hi' })) {
  // StreamEvent = SseEvent ({ event?, data, id? }) | string
  console.log(typeof event === 'string' ? event : event.data)
}
```
Tokens:
Raw