### Build Crawler on Apify Platform (SDK v3 + Crawlee + Playwright)

Source: https://docs.apify.com/sdk/js/

Example demonstrating how to build an SDK-v2-like crawler on the Apify platform using Apify SDK v3, Crawlee, and Playwright. This setup requires installing all three libraries.

```bash
npm install apify crawlee playwright
```

```javascript
// Ensure you have "type": "module" in package.json or use .mjs suffix
import { Actor } from 'apify';
import { PlaywrightCrawler } from 'crawlee';

await Actor.init();

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks }) {
        const title = await page.title();
        console.log(`Title of ${request.loadedUrl} is '${title}'`);
        await Actor.pushData({ title, url: request.loadedUrl });
        await enqueueLinks();
    }
});

await crawler.run(['https://crawlee.dev']);
await Actor.exit();
```

--------------------------------

### Python API Client: Run Actor and Get Dataset Items

Source: https://docs.apify.com/api

This Python code snippet demonstrates using the Apify client library to execute an actor and fetch its output. It covers installation and provides examples for initiating an actor run and listing items from its dataset.

```python
# Installation: pip install apify-client

from apify_client import ApifyClient

# Initialize the client
client = ApifyClient('YOUR_API_TOKEN')

# Run an actor and wait for it to finish
# The call() method returns the run object, which contains defaultDatasetId
run = client.actor("apify/hello-world").call()

# Get the dataset ID from the run object
dataset_id = run['defaultDatasetId']

# Fetch items from the dataset
# You can also use dataset_client.iterate_items() for large datasets
dataset_client = client.dataset(dataset_id)
items = dataset_client.list_items().items

print(items)
```

--------------------------------

### Apify SDK and Crawlee Examples

Source: https://apify.com/change-log

Information on accessing and running examples provided within the Apify SDK (JavaScript) and Crawlee documentation.

```APIDOC
Running Documentation Examples:
  - Apify SDK (JavaScript): All examples from the documentation can be run directly.
    - Source: https://docs.apify.com/sdk/js/docs/examples
  - Crawlee: All examples from the documentation can be run directly.
    - Source: https://crawlee.dev/docs/examples
```

--------------------------------

### API Request Examples: REST and GraphQL

Source: https://docs.apify.com/academy/api-scraping

Illustrates common request formats for REST APIs, including GET and POST methods with example endpoints and parameters. Also shows a GraphQL query structure for fetching data.

```APIDOC
REST API Examples:

GET https://api.example.com/users/123
  - Retrieves user data for a specific user ID.

GET https://api.example.com/comments/abc123?limit=100
  - Fetches comments with a limit on the number of results.

POST https://api.example.com/orders
  - Submits new order data to the server.

GraphQL API Example:

query($number_of_repos: Int!) {
  viewer {
    name
    repositories(last: $number_of_repos) {
      nodes {
        name
      }
    }
  }
}
  - A GraphQL query to fetch the viewer's name and a list of their repositories, with a variable to control the number of repositories.
```

--------------------------------

### Install Crawlee for External Use

Source: https://docs.apify.com/sdk/js/

Instructions for installing the Crawlee library, which retains the crawling and scraping-related tools from the original Apify SDK, for use outside the Apify platform.

```bash
npm install crawlee
```

--------------------------------

### Apify CLI Installation

Source: https://docs.apify.com/platform/actors/development

Command to install the Apify CLI globally using npm. The Apify CLI is used for managing actors, running them locally, and interacting with the Apify platform.

```bash
npm install -g apify-cli
```

--------------------------------

### Create Actors via GitHub Integration Guide

Source: https://apify.com/change-log

Guide on streamlining actor creation using GitHub or other Git providers. It highlights improvements to the developer experience and the flexibility of integrating with various Git services.

```markdown
Create Actors through GitHub or other Git providers

Create actors through GitHub easier with our newly redesigned flow! One of our goals is to make developer experience as smooth as possible and GitHub is a critical part of that. 

You can either create actor through GitHub or you can choose other Git providers, we cover them all!

![GitHub GIF.gif](https://cdn-cms.apify.com/Git_Hub_GIF_cb00851aa4.gif)
```

--------------------------------

### Install Apify CLI

Source: https://docs.apify.com/cli/

Installs the Apify CLI globally using npm. This command is essential for accessing Apify's command-line functionalities.

```bash
npm i -g apify-cli
```

--------------------------------

### Install Apify CLI with Homebrew

Source: https://apify.com/change-log

Installs the Apify command-line interface (CLI) using the Homebrew package manager. This is a convenient way to get started with Apify tools on macOS and Linux.

```shell
brew install apify-cli
```

--------------------------------

### Apify Deployment Guide

Source: https://apify.com/resources/startups

Information on how to deploy your web scraping or automation projects to the Apify platform, enabling scalable execution and management.

```APIDOC
Deploy to Apify:
  Learn how to package and deploy your web scraping projects (actors) to the Apify platform for scalable execution.
  URL: https://docs.apify.com/platform/actors/development/deployment

Deployment Steps:
  1. Prepare your Actor:
     - Ensure your project is structured correctly (e.g., using Crawlee).
     - Define an `actor.json` file with metadata, entry point, and dependencies.
  2. Package your Actor:
     - Use the Apify CLI (`npm install -g apify-cli`) to package your project.
     - Command: `apify init` (to create `actor.json` if needed), then `apify package`.
  3. Push your Actor:
     - Authenticate with Apify CLI: `apify login`.
     - Push the packaged actor: `apify push`.

Key Configuration (`actor.json`):
  - `name`: Unique name for your actor.
  - `version`: Version number.
  - `template`: Base template used (e.g., 'node-basic', 'node-puppeteer').
  - `entryPoint`: The main script file (e.g., 'main.js').
  - `environmentVariables`: Environment variables required by your actor.
  - `cpu`, `memoryMbytes`: Resource allocation.
```

--------------------------------

### Apify CLI Basic Workflow

Source: https://docs.apify.com/cli/

Demonstrates the fundamental commands for creating, running, and deploying an Apify Actor using the Apify CLI. This workflow covers the lifecycle from project creation to cloud deployment.

```bash
# Create your first Actor
apify create my-actor

# Go into the project directory
cd my-actor

# Run it locally
apify run

# Log into your Apify account and deploy it to Apify Platform
apify login
apify push
```

--------------------------------

### JavaScript API Client: Run Actor and Get Dataset Items

Source: https://docs.apify.com/api

This JavaScript code snippet shows how to use the Apify client library to run an actor and retrieve its results. It includes installation instructions and examples for calling an actor and listing its dataset items.

```javascript
npm install apify-client

const { ApifyClient } = require('apify-client');

const client = new ApifyClient({
    token: 'MY-APIFY-TOKEN',
});

// Starts an actor and waits for it to finish.
const { defaultDatasetId } = await client.actor('john-doe/my-cool-actor').call();

// Fetches results from the actor's dataset.
const { items } = await client.dataset(defaultDatasetId).listItems();

console.log(items);
```

--------------------------------

### Install and Use ApifyClient in JavaScript

Source: https://docs.apify.com/api/client/js/

Demonstrates how to install the Apify client library and use it to interact with the Apify API. This includes initializing the client with an API token, calling an actor, and retrieving data from its default dataset.

```bash
npm install apify-client
```

```javascript
const { ApifyClient } = require('apify-client');

const client = new ApifyClient({
    token: 'MY-APIFY-TOKEN',
});

// Starts an actor and waits for it to finish.
const { defaultDatasetId } = await client.actor('john-doe/my-cool-actor').call();

// Fetches results from the actor's dataset.
const { items } = await client.dataset(defaultDatasetId).listItems();

console.log(items);
```

--------------------------------

### TypeScript CheerioCrawler Example

Source: https://apify.com/templates/ts-crawlee-cheerio

This snippet demonstrates a basic CheerioCrawler setup using the Apify SDK. It initializes the actor, retrieves input parameters like start URLs and maximum requests, configures a proxy, and defines a request handler to extract page titles and save them to a dataset. It uses Cheerio for HTML parsing and Crawlee for the crawling logic.

```typescript
import { Actor } from 'apify';
import { CheerioCrawler, Dataset } from 'crawlee';

interface Input {
    startUrls: {
        url: string;
        method?: 'GET' | 'HEAD' | 'POST' | 'PUT' | 'DELETE' | 'TRACE' | 'OPTIONS' | 'CONNECT' | 'PATCH';
        headers?: Record<string, string>;
        userData: Record<string, unknown>;
    }[];
    maxRequestsPerCrawl: number;
}

await Actor.init();

const { startUrls = ['https://apify.com'], maxRequestsPerCrawl = 100 } =
    (await Actor.getInput<Input>()) ?? ({} as Input);

const proxyConfiguration = await Actor.createProxyConfiguration();

const crawler = new CheerioCrawler({
    proxyConfiguration,
    maxRequestsPerCrawl,
    requestHandler: async ({ enqueueLinks, request, $, log }) => {
        log.info('enqueueing new URLs');
        await enqueueLinks();

        const title = $('title').text();
        log.info(`${title}`, { url: request.loadedUrl });

        await Dataset.pushData({ url: request.loadedUrl, title });
    },
});

await crawler.run(startUrls);

await Actor.exit();
```

--------------------------------

### Python Starter Actor Template

Source: https://apify.com/partners/actor-developers

A starter template for building Python Actors on Apify. It uses HTTPX for fetching web pages and Beautiful Soup for parsing HTML. Includes instructions for installing Python packages via virtual environments.

```python
Scrape single page with provided URL with HTTPX and extract data from page's HTML with Beautiful Soup.
Messages: To install additional Python packages, you need to activate the virtual environment in the ".venv" folder in the actor directory.
Default Run Options: build: latest, memoryMbytes: 1024, timeoutSecs: 3600
Showcase Files: src/main.py, src/__main__.py
Use Cases: STARTER, WEB_SCRAPING
```

--------------------------------

### Actor Configuration Example

Source: https://apify.com/change-log

Illustrates a basic configuration structure for an Apify Actor, specifying related files like README and Dockerfile.

```json
{
  "actor": {
    "readme": "../../shared/README.md",
    "dockerfile": "./Dockerfile"
  }
}
```

--------------------------------

### Code Templates Navigation Item

Source: https://apify.com/integrations

Configuration for a navigation link offering code templates for Python, JavaScript, and TypeScript, categorized under 'Get started'.

```json
{
  "href": "/templates",
  "label": "Code templates",
  "description": "Python, JavaScript, and TypeScript",
  "tag": null,
  "icon": {
    "name": "Item=Template.svg",
    "alternativeText": null,
    "caption": null,
    "width": 24,
    "height": 24,
    "formats": null,
    "hash": "Item_Template_67a5124812",
    "ext": ".svg",
    "mime": "image/svg+xml",
    "size": 2.34,
    "url": "https://cdn-cms.apify.com/Item_Template_...",
    "previewUrl": null,
    "provider": "aws-s3",
    "provider_metadata": null,
    "createdAt": "...",
    "updatedAt": "...",
    "documentId": "...",
    "publishedAt": "...",
    "isUrlSigned": true
  }
}
```

--------------------------------

### Apify Actor Setup with PuppeteerCrawler

Source: https://apify.com/templates/ts-crawlee-puppeteer-chrome

This snippet shows the core setup for an Apify Actor using TypeScript and the Crawlee library. It configures a PuppeteerCrawler with proxy settings, launch options for headless Chrome, and a request handler defined in a separate routes file. It then runs the crawler with specified start URLs and gracefully exits the Actor.

```typescript
import { Actor } from '@apify/actor';
import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes';

// Define the input schema for the Actor.
interface Input {
  startUrls?: string[];
}

// Define the URLs to start the crawler with - get them from the input of the Actor or use a default list.
const { startUrls = ['https://apify.com'] } = (await Actor.getInput<Input>()) ?? {};

// Create a proxy configuration that will rotate proxies from Apify Proxy.
const proxyConfiguration = await Actor.createProxyConfiguration();

// Create a PuppeteerCrawler that will use the proxy configuration and and handle requests with the router from routes.ts file.
const crawler = new PuppeteerCrawler({
  proxyConfiguration,
  requestHandler: router,
  launchContext: {
    launchOptions: {
      args: [
        '--disable-gpu', // Mitigates the "crashing GPU process" issue in Docker containers
        '--no-sandbox', // Mitigates the "sandboxed" process issue in Docker containers
      ],
    },
  },
});

// Run the crawler with the start URLs and wait for it to finish.
await crawler.run(startUrls);

// Gracefully exit the Actor process. It's recommended to quit all Actors with an exit().
await Actor.exit();

```

--------------------------------

### Integrate FirstPromoter Referral Program

Source: https://apify.com/pricing

Dynamically loads and initializes the FirstPromoter JavaScript SDK for referral program tracking. It includes error handling for script loading.

```javascript
(function(){
  var t=document.createElement("script");
  t.type="text/javascript",t.async=!0,t.src='https://cdn.firstpromoter.com/fprom.js',
  t.onload=t.onreadystatechange=function(){
    var t=this.readyState;
    if(!t||"complete"==t||"loaded"==t) try {
      $FPROM.init("35gj8m4w",".apify.com")
    } catch(t) {}
  };
  var e=document.getElementsByTagName("script")[0];
  e.parentNode.insertBefore(t,e)
})();
```

--------------------------------

### Load FirstPromoter Script

Source: https://apify.com/use-cases/lead-generation

This snippet initializes the FirstPromoter referral program script. It dynamically creates a script element, sets its source to the FirstPromoter CDN, and attaches an onload handler to call `FPROM.init` with the provided campaign code and domain.

```javascript
(function(){
  var t = document.createElement("script");
  t.type = "text/javascript", t.async = !0, t.src = 'https://cdn.firstpromoter.com/fprom.js';
  t.onload = t.onreadystatechange = function() {
    var t = this.readyState;
    if (!t || "complete" == t || "loaded" == t) try {
      $FPROM.init("35gj8m4w", ".apify.com")
    } catch (t) {}
  };
  var e = document.getElementsByTagName("script")[0];
  e.parentNode.insertBefore(t, e)
})();
```

--------------------------------

### Apify SDK Initialization and PlaywrightCrawler Setup

Source: https://apify.com/templates/ts-crawlee-playwright-chrome

This snippet demonstrates the core setup for an Apify Actor using PlaywrightCrawler. It initializes the Apify SDK, retrieves crawler configuration from input, sets up proxy configuration, instantiates PlaywrightCrawler with request handlers, and starts the crawl. It also includes essential lifecycle methods like Actor.init() and Actor.exit().

```typescript
import { Actor } from 'apify';
import { PlaywrightCrawler } from 'crawlee';
import { router } from './routes.js';

interface Input {
 startUrls: {
 url: string;
 method?: 'GET' | 'HEAD' | 'POST' | 'PUT' | 'DELETE' | 'TRACE' | 'OPTIONS' | 'CONNECT' | 'PATCH';
 headers?: Record<string, string>;
 userData: Record<string, unknown>;
 }[];
 maxRequestsPerCrawl: number;
}

// Initialize the Apify SDK
await Actor.init();

// Structure of input is defined in input_schema.json
const { startUrls = ['https://apify.com'], maxRequestsPerCrawl = 100 } =
 (await Actor.getInput<Input>()) ?? ({} as Input);

const proxyConfiguration = await Actor.createProxyConfiguration();

const crawler = new PlaywrightCrawler({
 proxyConfiguration,
 maxRequestsPerCrawl,
 requestHandler: router,
 launchContext: {
 launchOptions: {
 args: [
 '--disable-gpu',
 ],
 },
 },
});

await crawler.run(startUrls);

// Exit successfully
await Actor.exit();
```

--------------------------------

### Run Actor Programmatically (JavaScript)

Source: https://docs.apify.com/platform/actors/running

Execute an Apify Actor from a JavaScript application using the Apify Client library. This example shows how to start an Actor, pass input, and retrieve results.

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({
    token: 'MY-API-TOKEN',
});

// Start the Google Maps Scraper Actor and wait for it to finish.
const actorRun = await client.actor('compass/crawler-google-places').call({
    queries: 'apify',
});

// Fetch scraped results from the Actor's dataset.
const { items } = await client.dataset(actorRun.defaultDatasetId).listItems();

console.dir(items);
```

--------------------------------

### Crawlee + Puppeteer + Chrome Scraper

Source: https://apify.com/templates/ts-crawlee-playwright-chrome

An example of a web scraper using Crawlee with Puppeteer and headless Chrome. This setup allows for JavaScript rendering and is more robust against anti-scraping techniques, though it may be slower.

```typescript
/*
  Crawlee + Puppeteer + Chrome Scraper Example
  Uses Puppeteer and headless Chrome to render JavaScript. Harder to block, but slower than plain HTTP.
  Requires Node.js, Crawlee, and Puppeteer.
*/

// Example usage (conceptual):
// import { PlaywrightCrawler } from 'crawlee';

// const crawler = new PlaywrightCrawler({
//     async requestHandler({ page, request }) {
//         // Page is already rendered by Playwright (which uses Chromium by default)
//         const html = await page.content();
//         // Process HTML or interact with the page further
//         console.log(`Scraped ${request.url} with Playwright`);
//     },
// });

// await crawler.run(['https://example.com']);
```

--------------------------------

### Crawlee Documentation

Source: https://apify.com/resources/startups

Links to the official documentation for Crawlee, a powerful Node.js library for building web scrapers and browser automation solutions.

```APIDOC
Crawlee Documentation:
  Crawlee is an open-source framework for web scraping and crawling, built on Node.js. It provides tools for managing requests, handling browser automation (Puppeteer, Playwright), processing data, and more.
  URL: https://crawlee.dev/

Key Features:
  - Request Queue: Efficiently manage URLs to be crawled.
  - Cheerio/Puppeteer/Playwright Integration: Flexible scraping capabilities for static and dynamic websites.
  - Data Processing: Tools for transforming and storing scraped data.
  - Actor Integration: Seamless deployment and execution on the Apify platform.

Getting Started:
  1. Install Crawlee: `npm install crawlee`
  2. Create a crawler: Use the provided templates or build custom crawlers.
  3. Run your crawler: Execute locally or deploy to Apify.
```

--------------------------------

### Python Starter Template

Source: https://apify.com/anti-blocking

A starter template for scraping a single page using Python with HTTPX and Beautiful Soup. It provides a basic setup for fetching HTML content and parsing it.

```python
# Example usage of Python starter template
# Requires installation of httpx and beautifulsoup4

import httpx
from bs4 import BeautifulSoup

url = "https://www.example.com"

try:
    response = httpx.get(url)
    response.raise_for_status() # Raise an exception for bad status codes
    
    soup = BeautifulSoup(response.text, 'html.parser')
    
    # Example: Extracting the title of the page
    title = soup.title.string
    print(f"Page Title: {title}")
    
    # Add your specific scraping logic here

except httpx.HTTPStatusError as e:
    print(f"HTTP error occurred: {e}")
except httpx.RequestError as e:
    print(f"An error occurred while requesting {e.request.url!r}: {e}")

```

--------------------------------

### Helpful tips to create Actors faster

Source: https://apify.com/change-log

This article provides guidance on creating Actors faster within the Apify Console's web IDE. It outlines a 6-step process and points out common pitfalls to avoid, aiming to improve developer efficiency.

```text
If you're struggling with creating Actors in Apify Console, we have you covered. 

We've introduced Tips to our web IDE that explain how to create Actors faster in 6 steps and what to watch out for. 

Do you think it helps? Check it out and let us know what you think! 

![Guiding users through tips.gif](https://cdn-cms.apify.com/Guiding_users_through_tips_9b3d636d9e.gif)
```

--------------------------------

### TypeScript PuppeteerCrawler Actor Setup

Source: https://apify.com/templates/ts-crawlee-puppeteer-chrome

Initializes an Apify Actor, configures a PuppeteerCrawler with proxy support and request handlers, and runs it with specified start URLs. Handles JavaScript rendering and includes options for Docker compatibility.

```typescript
1// Apify SDK - toolkit for building Apify Actors (Read more at https://docs.apify.com/sdk/js/).
2import { Actor } from 'apify';
3// Web scraping and browser automation library (Read more at https://crawlee.dev)
4import { PuppeteerCrawler } from 'crawlee';
5
6import { router } from './routes.js';
7
8// The init() call configures the Actor for its environment. It's recommended to start every Actor with an init().
9await Actor.init();
10
11interface Input {
12    startUrls: {
13        url: string;
14        method?: 'GET' | 'HEAD' | 'POST' | 'PUT' | 'DELETE' | 'TRACE' | 'OPTIONS' | 'CONNECT' | 'PATCH';
15        headers?: Record<string, string>;
16        userData: Record<string, unknown>;
17    }[];
18}
19// Define the URLs to start the crawler with - get them from the input of the Actor or use a default list.
20const { startUrls = ['https://apify.com'] } = (await Actor.getInput<Input>()) ?? {};
21
22// Create a proxy configuration that will rotate proxies from Apify Proxy.
23const proxyConfiguration = await Actor.createProxyConfiguration();
24
25// Create a PuppeteerCrawler that will use the proxy configuration and and handle requests with the router from routes.ts file.
26const crawler = new PuppeteerCrawler({
27    proxyConfiguration,
28    requestHandler: router,
29    launchContext: {
30        launchOptions: {
31            args: [
32                '--disable-gpu', // Mitigates the "crashing GPU process" issue in Docker containers
33                '--no-sandbox', // Mitigates the "sandboxed" process issue in Docker containers
34            ],
35        },
36    },
37});
38
39// Run the crawler with the start URLs and wait for it to finish.
40await crawler.run(startUrls);
41
42// Gracefully exit the Actor process. It's recommended to quit all Actors with an exit().
43await Actor.exit();

```

--------------------------------

### Crawlee + Puppeteer + Chrome Scraper

Source: https://apify.com/templates

An example of a web scraper using Crawlee with Puppeteer and headless Chrome. This setup renders JavaScript, making it suitable for dynamic websites, though it is slower than plain HTTP requests.

```javascript
/*
  Example of a Puppeteer and headless Chrome web scraper. Headless browsers render JavaScript and are harder to block, but they're slower than plain HTTP.
*/
// Placeholder for actual code implementation
```

--------------------------------

### Create Apify Python Actor

Source: https://docs.apify.com/sdk/python/

Demonstrates the command-line interface command to create a new Apify Actor project using the Python SDK. This is the initial step for setting up a new Python-based Actor.

```bash
apify create my-python-actor
```