Botasaurus

https://www.omkar.cloud/botasaurus/docs/
Admin
Botasaurus is a web scraping framework that enables developers to build reliable and efficient bots...
Tokens:38,229
Snippets:377
Trust Score:7.5
Update:8 months ago
Show doc for...
Tokens:
Raw
### Install Project Dependencies

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/quick-start

Installs all the necessary Node.js packages required for the Botasaurus desktop starter project. This command should be run after cloning the repository.

```bash
npm install
```

--------------------------------

### Disable API Autostart with disableApiAutostart

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

This code example shows how to prevent the Botasaurus API server from starting automatically when the application launches using `disableApiAutostart()`. The API will then need to be manually started from the 'API' tab.

```typescript
// API will not run until manually started from the desktop GUI  
ApiConfig.disableApiAutostart();  

```

--------------------------------

### Install Botasaurus CLI and Apache Web Server

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-aws

This command installs the Botasaurus CLI and the Apache web server on your EC2 instance. It fetches a script from a GitHub repository and executes it.

```bash
curl -sL https://raw.githubusercontent.com/omkarcloud/botasaurus/master/vm-scripts/install-bota-desktop.sh | bash  
```

--------------------------------

### Install Botasaurus Desktop App

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-google-cloud

Installs the Botasaurus Desktop application on the VM using a Debian installer URL. The `--port` and `--api-base-path` options can be used to configure the app and host multiple APIs on the same VM.

```python
python3 -m bota install-desktop-app --debian-installer-url https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb
```

```python
python3 -m bota install-desktop-app \
  --debian-installer-url https://amazon-invoice-extractor.s3.us-east-1.amazonaws.com/Amazon+Invoice+Extractor-amd64.deb \
  --port 8001 \
  --api-base-path /amazon-invoices
```

--------------------------------

### Install Botasaurus Desktop App

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-aws

Installs the Botasaurus Desktop application on an EC2 instance using a provided Debian installer URL. This command can be customized with options to set the port and API base path for the application.

```python
python3 -m bota install-desktop-app --debian-installer-url https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb  
```

--------------------------------

### Enable Botasaurus API in Server Configuration

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

This snippet shows how to enable the Botasaurus API by importing and calling `ApiConfig.enableApi()` in your server configuration file. Enabling the API automatically generates documentation and starts an API server.

```typescript
import ApiConfig from "botasaurus-server/api-config";  
  
// Enable the API  
ApiConfig.enableApi();  

```

--------------------------------

### Install Botasaurus CLI and Apache Web Server

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-google-cloud

Installs the Botasaurus CLI and Apache web server on the VM. This is a prerequisite for managing Botasaurus Desktop API requests.

```bash
curl -sL https://raw.githubusercontent.com/omkarcloud/botasaurus/master/vm-scripts/install-bota-desktop.sh | bash
```

--------------------------------

### Installer Download URL Format (AWS S3)

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/packaging-publishing

This is the format for accessing publicly available installer files stored in an AWS S3 bucket. Replace '<your-bucket-name>' with your bucket's name and '<your-product-name>.<extension>' with the specific installer file's name and extension.

```bash
https://<your-bucket-name>.s3.amazonaws.com/<your-product-name>.<extension>
```

--------------------------------

### Add Custom API Routes with addCustomRoutes

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

This example demonstrates extending the Botasaurus API with custom endpoints and middleware using `addCustomRoutes()`. It accepts a Fastify instance, allowing for flexible route definitions and the implementation of features like authentication or health checks.

```typescript
ApiConfig.addCustomRoutes((server) => {  
  server.get('/health', (request, reply) => {  
    return reply.send({ status: 'OK'});  
  });  
});  

```

```typescript
ApiConfig.addCustomRoutes((server) => {  
  server.addHook('onRequest', (request, reply, done) => {  
    // Check for secret  
    const secret = request.headers['x-secret'] as string;  
  
    if (secret === '49cb1de3-419b-4647-bf06-22c9e1110313') {  
      // Valid secret, proceed  
      return done();   
    } else {  
      return reply.status(401).send({  
        message: 'Unauthorized: Invalid secret.',  
      });  
    }  
  });  
});  

```

--------------------------------

### Install Certbot for Apache

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-domain-and-ssl

Installs Certbot and its Apache plugin on Debian/Ubuntu systems to manage SSL certificates. This is a prerequisite for obtaining and installing Let's Encrypt certificates for Apache web servers.

```bash
sudo apt install certbot python3-certbot-apache -y
```

--------------------------------

### Install Botasaurus using pip

Source: https://www.omkar.cloud/botasaurus/docs/what-is-botasaurus

This command installs or upgrades the Botasaurus package using pip. Ensure you have Python and pip installed on your system.

```bash
python -m pip install --upgrade botasaurus
```

--------------------------------

### Launch Botasaurus Desktop App

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/quick-start

Launches the Botasaurus desktop starter application in development mode. This command starts the development server and opens the application window.

```bash
npm run dev
```

--------------------------------

### Install Multiple Botasaurus Desktop Apps

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-aws

Installs additional Botasaurus Desktop applications on the same EC2 instance. It requires specifying unique `--port` and `--api-base-path` for each application to avoid conflicts.

```python
python3 -m bota install-desktop-app \
  --debian-installer-url https://amazon-invoice-extractor.s3.us-east-1.amazonaws.com/Amazon+Invoice+Extractor-amd64.deb \
  --port 8001 \
  --api-base-path /amazon-invoices  
```

--------------------------------

### Set up Botasaurus project directory

Source: https://www.omkar.cloud/botasaurus/docs/what-is-botasaurus

These commands create a new directory for your Botasaurus project and navigate into it. The `code .` command opens the current directory in VSCode if installed.

```bash
mkdir my-botasaurus-project
cd my-botasaurus-project
code .
```

--------------------------------

### Obtain and Install SSL Certificate with Certbot

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-domain-and-ssl

Uses Certbot to automatically obtain and install a free SSL certificate from Let's Encrypt for Apache. It configures Apache to use the certificate and skips email registration for convenience.

```bash
sudo certbot --apache --agree-tos --register-unsafely-without-email
```

--------------------------------

### GET /v1/health

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

Provides a health check status for the API server.

```APIDOC
## GET /v1/health

### Description
This endpoint provides a simple health check for the API server, indicating its operational status.

### Method
`GET`

### Endpoint
`/v1/health`

### Parameters
*   None

### Request Example
*(No request body for GET requests)*

### Response
#### Success Response (200)
*   **status** (string) - The operational status of the API (e.g., "OK").

#### Response Example
```json
{
  "status": "OK"
}
```
```

--------------------------------

### Install Botasaurus CLI and Create Static IP Address (Python)

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-google-cloud

Installs the Botasaurus CLI and creates a static IP address for your Google Cloud VM. This ensures your VM is consistently reachable at the same IP. The CLI will prompt for a name and region for the IP address.

```python
python -m pip install bota --upgrade
python -m bota create-ip # Create a static IP address for your VM
```

--------------------------------

### Uninstall Botasaurus Desktop App using Debian URL

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-aws

Uninstalls a Botasaurus Desktop application from an EC2 instance using its Debian installer URL. Ensure you replace the example URL with your application's actual URL.

```python
python3 -m bota uninstall-desktop-app --debian-installer-url https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb  
```

--------------------------------

### Package Desktop Extractor for Current OS (npm)

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/packaging-publishing

This command builds an installer for the operating system where it is executed. The resulting installer can be found in the 'release/build' folder after the command completes. Ensure you are in the project's root directory before running.

```bash
npm run package
```

--------------------------------

### Run Botasaurus in Docker

Source: https://www.omkar.cloud/botasaurus/docs/what-is-botasaurus

This example provides the command-line steps to set up and run Botasaurus projects within a Docker environment. It involves cloning the Botasaurus Starter Template, navigating into the project directory, and then building and running the Docker containers using Docker Compose.

```bash
git clone https://github.com/omkarcloud/botasaurus-starter my-botasaurus-project
cd my-botasaurus-project
docker-compose build && docker-compose up

```

--------------------------------

### True First Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Shows an example of sorting boolean fields to prioritize `true` values. This is useful for filtering or highlighting items that meet a specific true condition.

```typescript
new sorts.TrueFirstSort("is_available")

```

--------------------------------

### Install PDF Parsing Package with npm

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/your-first-extractors/amazon-pdf-invoice-extractor

Installs the 'electron-pdf-parse' package, a fork of 'pdf-parse' optimized for Electron, which is used for reading PDF file content.

```bash
npm install electron-pdf-parse
```

--------------------------------

### API Configuration

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

This section covers the core configuration options for enabling and customizing the Botasaurus server API.

```APIDOC
## API Configuration Options

### Description
This section outlines the various methods available for configuring the Botasaurus server API, including enabling the API, setting the port, defining the base path, adding scraper aliases, and implementing custom routes.

### Methods

#### `ApiConfig.enableApi()`

*   **Description**: Enables the API functionality for the Botasaurus server.
*   **Method**: `POST` (conceptual, as this is a configuration method)
*   **Endpoint**: N/A

#### `ApiConfig.setApiPort(port: number)`

*   **Description**: Sets the port number on which the API server will listen.
*   **Parameters**:
    *   `port` (number) - Required - The port number for the API server.
*   **Method**: `POST` (conceptual)
*   **Endpoint**: N/A

#### `ApiConfig.disableApiAutostart()`

*   **Description**: Disables the automatic starting of the API server when the application initializes.
*   **Method**: `POST` (conceptual)
*   **Endpoint**: N/A

#### `ApiConfig.setApiBasePath(path: string)`

*   **Description**: Sets a base path prefix for all API routes.
*   **Parameters**:
    *   `path` (string) - Required - The base path for API routes (e.g., `/v1`).
*   **Method**: `POST` (conceptual)
*   **Endpoint**: N/A

#### `ApiConfig.addScraperAlias(scraper: Scraper, endpoint: string)`

*   **Description**: Adds a custom endpoint alias for a given scraper.
*   **Parameters**:
    *   `scraper` (Scraper) - Required - The scraper function to alias.
    *   `endpoint` (string) - Required - The desired API endpoint path for the scraper (e.g., `/hotels/search`).
*   **Method**: `POST` (conceptual)
*   **Endpoint**: N/A

#### `ApiConfig.addCustomRoutes(callback: (server: FastifyInstance) => void)`

*   **Description**: Allows the addition of custom routes and middleware to the API server.
*   **Parameters**:
    *   `callback` (function) - Required - A function that receives the Fastify server instance to define custom routes and hooks.
*   **Method**: `POST` (conceptual)
*   **Endpoint**: N/A

### Example Usage (Complex Configuration)
```typescript
import ApiConfig from 'botasaurus-server/api-config';
import { hotelsSearchScraper } from "../src/scrapers";

// Enable API functionality
ApiConfig.enableApi();

// Production configuration
ApiConfig.setApiPort(3000);
ApiConfig.setApiBasePath("/v1");

// Add scraper aliases for direct access
ApiConfig.addScraperAlias(hotelsSearchScraper, '/hotels/search');

// Add custom routes
ApiConfig.addCustomRoutes((server) => {
  // Health check for monitoring
  server.get('/health', (request, reply) => {
    return reply.send({ status: 'OK' });
  });

  // Authentication middleware
  server.addHook('onRequest', (request, reply, done) => {
    // Check for secret
    const secret = request.headers['x-secret'] as string;

    if (secret === '49cb1de3-419b-4647-bf06-22c9e1110313') {
      // Valid secret, proceed
      return done();
    } else {
      return reply.status(401).send({
        message: 'Unauthorized: Invalid secret.',
      });
    }
  });
});
```
```

--------------------------------

### Truthy First Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Provides an example for sorting fields that prioritize any 'truthy' value. This is more general than `TrueFirstSort` and includes non-boolean truthy values.

```typescript
new sorts.TruthyFirstSort("description")

```

--------------------------------

### Newest Date First Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Provides an example for sorting by date in descending order (newest first). This is useful for time-series data where the most recent entries are prioritized.

```typescript
new sorts.NewestDateFirstSort("created_at")

```

--------------------------------

### Authentication Middleware

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

All requests to the API require authentication using the 'x-secret' header.

```APIDOC
## Authentication Middleware

### Description
This middleware enforces authentication for all incoming API requests. It checks for a valid secret provided in the `x-secret` request header.

### Method
Applies to all HTTP methods (GET, POST, PUT, DELETE, etc.)

### Endpoint
All API endpoints under the base path (`/v1` in the example)

### Parameters

#### Request Headers
*   **x-secret** (string) - Required - The secret key for authentication.

### Request Example
```http
GET /v1/hotels/search HTTP/1.1
Host: example.com
X-Secret: 49cb1de3-419b-4647-bf06-22c9e1110313
```

### Response
#### Error Response (401)
*   **message** (string) - Description of the authentication error.

#### Response Example (Unauthorized)
```json
{
  "message": "Unauthorized: Invalid secret."
}
```
```

--------------------------------

### Nulls First Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Shows how to configure a sort to place `null` values at the beginning of the results. This is useful when you want to address or display records with missing data first.

```typescript
new sorts.NullsFirstSort("end_date")

```

--------------------------------

### False First Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Demonstrates how to sort boolean fields to prioritize `false` values. This can be used to bring items that do not meet a certain condition to the top.

```typescript
new sorts.FalseFirstSort("discounted")

```

--------------------------------

### Configure Electron Builder for S3 Publishing (JSON)

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/packaging-publishing

This configuration snippet is added to the 'package.json' file to set up Electron Builder for publishing installers to an AWS S3 bucket. Replace 'your-bucket-name' with the actual name of your S3 bucket.

```json
"build": {
  "publish": {
    "provider": "s3",
    "bucket": "your-bucket-name"
  }
}
```

--------------------------------

### Uninstall Botasaurus Desktop App

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-google-cloud

Uninstalls the Botasaurus Desktop application from the VM. This can be done using either the Debian installer URL or the package name.

```python
python3 -m bota uninstall-desktop-app --debian-installer-url https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb
```

```python
python3 -m bota uninstall-desktop-app --package-name yahoo-finance-extractor
```

--------------------------------

### Create AntiDetectRequests Instance in Botasaurus

Source: https://www.omkar.cloud/botasaurus/docs/what-is-botasaurus

This example demonstrates how to create an instance of `AntiDetectRequests`, a tool for making HTTP requests that mimic real browser behavior and avoid detection. It shows how to use the instance to parse HTML content with BeautifulSoup.

```python
anti_detect_request = bt.create_request()
soup = anti_detect_request.bs4("https://www.omkar.cloud/")
# ... Additional code

```

--------------------------------

### Set API Base Path with setApiBasePath

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

Illustrates how to add a prefix to all API routes using `setApiBasePath()`. This is beneficial for organizing APIs under specific subpaths like `/v1` or `/api`, aiding in versioning or multi-API hosting.

```typescript
// Prefixes all routes with /v1  
ApiConfig.setApiBasePath("/v1");  

```

--------------------------------

### Pause Scraper Execution with bt.prompt()

Source: https://www.omkar.cloud/botasaurus/docs/what-is-botasaurus

This example illustrates how to pause a Botasaurus scraper during development to allow for manual inspection. The `bt.prompt()` function halts execution until the user provides input, which is helpful for debugging.

```python
bt.prompt()

```

--------------------------------

### Get VM External IP Address

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-domain-and-ssl

Retrieves the external IP address of the current virtual machine. This command is useful for verifying DNS A record configurations and troubleshooting Certbot challenges.

```bash
curl icanhazip.com
```

--------------------------------

### Falsy First Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Illustrates sorting fields to prioritize any 'falsy' value. Similar to `FalseFirstSort`, but encompasses all falsy types (e.g., 0, '', null, undefined).

```typescript
new sorts.FalsyFirstSort("description")

```

--------------------------------

### Combined Multi-Level Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Illustrates creating a complex sorting option by combining multiple individual sort criteria. This allows for sophisticated data organization based on several fields and conditions.

```typescript
new sorts.Sort({
  label: "Top Products",
  sorts: [
      new sorts.AlphabeticAscendingSort("name"),
      new sorts.NumericDescendingSort("reviews"),
      new sorts.TrueFirstSort("is_available")
  ]
})

```

--------------------------------

### Scrape Heading using @request with Botasaurus

Source: https://www.omkar.cloud/botasaurus/docs/what-is-botasaurus

This example showcases the `@request` decorator in Botasaurus, which is similar to `@browser` but designed for making HTTP requests. It demonstrates parallel processing, caching, proxy usage, and BeautifulSoup integration for parsing HTML content to extract a heading.

```python
@request(parallel=40, cache=True, proxy="http://your_proxy_address:your_proxy_port", data=["https://www.omkar.cloud/", ...])
def scrape_heading_task(request: AntiDetectDriver, link):
  soup = request.bs4(link)
  heading = soup.find('h1').get_text()
  return {"heading": heading}

```

--------------------------------

### Clone Botasaurus Desktop Starter Project

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/quick-start

Clones the Botasaurus desktop starter project from GitHub and navigates into the project directory. This is the first step to setting up the development environment.

```bash
git clone https://github.com/omkarcloud/botasaurus-desktop-starter my-botasaurus-app
cd my-botasaurus-app
```

--------------------------------

### GET /v1/hotels/search

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

Retrieves hotel search results. This endpoint is configured via a scraper alias.

```APIDOC
## GET /v1/hotels/search

### Description
This endpoint allows users to perform hotel searches. It is directly mapped to the `hotelsSearchScraper` function using an alias.

### Method
`GET`

### Endpoint
`/v1/hotels/search`

### Parameters

#### Query Parameters
*   *(Specific query parameters depend on the `hotelsSearchScraper` implementation)*

### Request Example
*(No specific request body for GET requests, parameters are typically in query string)*

### Response
#### Success Response (200)
*   *(Response structure depends on the `hotelsSearchScraper` implementation)*

#### Response Example
```json
{
  "hotels": [
    {
      "name": "Example Hotel",
      "location": "Example City"
    }
  ]
}
```
```

--------------------------------

### Add Scraper Alias with addScraperAlias

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

Shows how to create a direct GET endpoint for a scraper using `addScraperAlias()`. This bypasses the usual task overhead, allowing immediate execution via a custom URL and enabling features like input validation and caching.

```typescript
import { hotelsSearchScraper } from "../src/scrapers";  
  
// Creates direct GET endpoint at /hotels/search  
ApiConfig.addScraperAlias(hotelsSearchScraper, "/hotels/search");  

```

--------------------------------

### Run Botasaurus in Gitpod

Source: https://www.omkar.cloud/botasaurus/docs/what-is-botasaurus

This example outlines how to run Botasaurus projects using Gitpod, a browser-based IDE. It involves cloning the starter template, selecting a suitable machine, and executing the main Python script. It also notes the fast internet speed and the need for user interaction to prevent session timeouts.

```bash
python main.py

```

--------------------------------

### Create a Simple 'Overview' View with Fields in Botasaurus Server

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/custom-views

This code snippet demonstrates how to create a basic view named 'Overview' that displays 'name' and 'price' fields. It utilizes the Botasaurus Server library to define the view and register it with a scraper. Dependencies include 'botasaurus-server/server' and 'botasaurus-server/ui'.

```typescript
import { Server } from 'botasaurus-server/server';
import { View, Field } from 'botasaurus-server/ui';
import { yourScraper } from '../src/yourScraper'; // your scraper function

/* 1. Define the view */
const overviewView = new View('Overview', [
  new Field('name'),
  new Field('price'),
]);
/* 2. Register scraper + view */
Server.addScraper(yourScraper, { views: [overviewView] });

```

--------------------------------

### Configure Field Options: outputKey, map, and showIf in Botasaurus Server

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/custom-views

This example shows how to configure a 'Field' in Botasaurus Server with advanced options. It includes `outputKey` to rename the column, `map` to transform the value using a function, and `showIf` to conditionally display the column based on input data. This provides flexibility in data presentation and manipulation.

```typescript
new Field("reviews_per_rating", {
    outputKey: "average_rating",
    map: (value, record) => {
        // ... Logic to calculate average rating from the 'value' object
    },
    // Only show this column if the user checked "scrape_prices"
    showIf: (inputData) => inputData.scrape_prices === true
})

```

--------------------------------

### Fix Electron Sandbox Permissions on Ubuntu

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/quick-start

Resolves an issue on Ubuntu 24.04 and later related to Electron sandbox permissions. This involves changing the ownership and permissions of the Electron sandbox executable.

```bash
sudo chown root:root ./node_modules/electron/dist/chrome-sandbox
sudo chmod 4755 ./node_modules/electron/dist/chrome-sandbox
```

--------------------------------

### Scrape Blog Articles and Links with Botasaurus

Source: https://www.omkar.cloud/botasaurus/docs/what-is-botasaurus

This example demonstrates a complete web scraping project using Botasaurus. It first scrapes a list of article links from a blog and then visits each link to extract the article's heading and publication date. It utilizes the @browser decorator for managing browser instances and includes caching and image blocking.

```python
@browser(block_images=True,  
         cache=True,   
         parallel=bt.calc_max_parallel_browsers,   
         reuse_driver=True)  
def scrape_articles(driver: AntiDetectDriver, link):
    driver.get(link)

    heading = driver.text("h1")
    date = driver.text("time")

    return {
        "heading": heading,   
        "date": date,   
        "link": link,   
    }

@browser(block_images=True, cache=True)  
def scrape_article_links(driver: AntiDetectDriver, data):
    # Visit the Omkar Cloud website
    driver.get("https://www.omkar.cloud/blog/")
      
    links = driver.links("h3 a")

    filename = "links"
    return filename, links

if __name__ == "__main__":
    # Launch the web scraping task
    links = scrape_article_links()
    scrape_articles(links)

```

--------------------------------

### Alphabetic Ascending Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Provides an example of how to implement an alphabetic ascending sort on a specified field. This sort type is useful for ordering text-based data alphabetically from A to Z.

```typescript
new sorts.AlphabeticAscendingSort("name")

```

--------------------------------

### Create Product Overview and Featured Reviews Views with Botasaurus Server

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/custom-views

This TypeScript code sets up a Botasaurus server to display scraped product data. It defines an 'Overview' view that calculates the average rating and expands the rating distribution, and a 'Featured Reviews' view that lists individual reviews. The server is configured to use these views with a specified scraper.

```typescript
import { Server } from 'botasaurus-server/server';
import { View, Field, ExpandDictField, ExpandListField, CustomField } from 'botasaurus-server/ui';
import { scrapeProductData } from '../src/scrapeProductData';   // <-- your scraper

/**
 * Calculate the average product-rating.
 *
 * @param value - The `reviews_per_rating` dictionary.
 * @param record - The entire product record.
 */
function calculateAverageRating(value: Record<string, number>): number {
  const totalReviews = Object.values(value || {}).reduce((a, b) => a + b, 0);
  if (totalReviews === 0) return 0;

  let ratingSum = 0;
  for (const [rating, count] of Object.entries(value)) {
    ratingSum += Number(rating) * count;
  }
  return parseFloat((ratingSum / totalReviews).toFixed(2));
}


const overviewView = new View('Overview', [
  new Field('id'),
  new Field('name'),
  new Field('price'),
  new Field('reviews'),

  // Create a new column 'average_rating' by transforming 'reviews_per_rating'
  new Field('reviews_per_rating', {
    outputKey: 'average_rating',
    map: calculateAverageRating,
  }),

  // Expand the 'reviews_per_rating' dictionary into individual rating columns
  new ExpandDictField('reviews_per_rating', [
    new Field('1', { outputKey: 'rating_1' }),
    new Field('2', { outputKey: 'rating_2' }),
    new Field('3', { outputKey: 'rating_3' }),
    new Field('4', { outputKey: 'rating_4' }),
    new Field('5', { outputKey: 'rating_5' }),
  ]),
]);

const featuredReviewsView = new View('Featured Reviews', [
  // Alias parent data for clarity in the reviews table
  new Field('id',   { outputKey: 'product_id'   }),
  new Field('name', { outputKey: 'product_name' }),

  // Expand the list of featured reviews into separate rows
  new ExpandListField('featured_reviews', [
    new Field('rating'),
    new Field('content'),
  ]),
]);

Server.addScraper(
  scrapeProductData,
  {
    views: [
      overviewView,
      featuredReviewsView,
    ],
  }
);
```

--------------------------------

### Numeric Descending Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Demonstrates the usage of a numeric descending sort. This type of sort arranges numerical data from largest to smallest.

```typescript
new sorts.NumericDescendingSort("price")

```

--------------------------------

### Dynamically Fetch Options with searchMethod in Botasaurus Select

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/input-controls

Demonstrates how to dynamically fetch options for a select dropdown using the `searchMethod` property. This involves defining an asynchronous handler function on the server and registering it with `Server.addSearchOptionsEndpoints()`. The handler must accept a query and form data, returning an array of objects with `value` and `label` properties.

```javascript
.select("city", {  
  searchMethod: "searchCityOptions",  // Backend method that fetches options  
  canCreateOptions: true,          // Allow users to create custom options  
  defaultValue: {value: 'US__CA__SF', label: "San Francisco"},
})

```

```typescript
import { Server } from 'botasaurus-server/server';  
import axios from 'axios';  
import { config } from '../../main/config';  
  
const baseUrl = config.isDev ? "http://0.0.0.0:3000" : "https://api.my-app.com";  
  
async function searchCityOptions(query: string, data: any) {  
  const params = {  
    query,  
  };
  
  try {  
    const response = await axios.get(`${baseUrl}/search/cities`, { params });  
    return response.data; // Must return array of { value, label }  
  } catch (error) {  
    throw error; // Propagate errors for proper handling  
  }  
}

// Register the search endpoint  
Server.addSearchOptionsEndpoints({
  searchCityOptions,
});

```

--------------------------------

### Nulls Last Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Demonstrates a sort configuration that places `null` values at the end of the results. This is helpful for focusing on records that have complete data.

```typescript
new sorts.NullsLastSort("end_date")

```

--------------------------------

### Alphabetic Descending Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Illustrates the implementation of an alphabetic descending sort. This is used to sort text data in reverse alphabetical order (Z to A).

```typescript
new sorts.AlphabeticDescendingSort("name")

```

--------------------------------

### Numeric Ascending Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Shows how to create a numeric ascending sort for a given field. This is commonly used for sorting numerical data from smallest to largest.

```typescript
new sorts.NumericAscendingSort("price")

```

--------------------------------

### Combine Split Tasks with createAllTask

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/task-configuration

Automatically generates a combined view of all split task results into a single 'All' task. Set `createAllTask` to `true` when using `splitTask` to improve usability by consolidating results.

```typescript
import { Server } from "botasaurus-server/server"
import { stockPriceScraper } from "../src/scraper"

function splitTask(data) {
    ...
}

Server.addScraper(
    stockPriceScraper,
    {
        splitTask: splitTask,
        createAllTask: true // Add this line
    }
);

```

--------------------------------

### Oldest Date First Sort Example

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Illustrates how to sort by date in ascending order (oldest first). This is beneficial for chronological data where the earliest entries are of primary interest.

```typescript
new sorts.OldestDateFirstSort("created_at")

```

--------------------------------

### Displaying Help Text in JavaScript

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/input-controls

Adds a help text tooltip to a form control. The help text is displayed when a user hovers over a question mark icon next to the control. This is useful for providing additional context or instructions.

```javascript
.text("api_key", {  
   label: "API Key",  
   helpText: "Find API key in Dashboard → Settings → API"  
})
```

--------------------------------

### Practical Application of Async Queue for Web Scraping Titles

Source: https://www.omkar.cloud/botasaurus/docs/what-is-botasaurus

Demonstrates using `async_queue` to scrape webpage titles from a list of links. It involves initializing the queue, adding links to it, and then retrieving all scraped titles. The `async_queue` ensures that only unique links are processed.

```python
from botasaurus import browser, AntiDetectDriver

@browser(async_queue=True)
def scrape_title(driver: AntiDetectDriver, link):
    driver.get(link)  # Navigate to the link
    return driver.title  # Scrape the title of the webpage

@browser()
def scrape_all_titles(driver: AntiDetectDriver):
    # ... Your code to visit the initial page ...

    title_queue = scrape_title()  # Initialize the asynchronous queue

    while not end_of_page_detected(driver):  # Replace with your end-of-list condition
        title_queue.put(driver.links('a'))  # Add each link to the queue
        driver.scroll(".scrollable-element")


    return title_queue.get()  # Get all the scraped titles at once

if __name__ == "__main__":
    all_titles = scrape_all_titles()  # Call the function to start the scraping process

```

--------------------------------

### Create Single Select Dropdown Filter (TypeScript)

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting

Example of creating a single select dropdown filter for a 'category' field with predefined options using Botasaurus UI filters. It supports case-insensitive matching.

```typescript
new filters.SingleSelectDropdown("category", { 
    options: [ 
        { value: "apparel",     label: "Apparel" }, 
        { value: "electronics", label: "Electronics" } 
    ]
})

```

--------------------------------

### Configure API Port with setApiPort

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

Demonstrates how to change the default port (8000) for the Botasaurus API server using the `setApiPort()` method. This is useful when the default port is in use or a specific port is required.

```typescript
// Set API to run on port 8080  
ApiConfig.setApiPort(8080);  

```

--------------------------------

### Select Input Control with Dynamic Options - JavaScript

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/enhancing-scrapers/input-controls

Configures a dropdown menu where options are fetched dynamically from a backend `searchMethod`. It supports allowing users to create custom options and specifies the format for the default value, which should be an object with `value` and `label` properties.

```javascript
.select("city", {
  searchMethod: "searchCityOptions",
  canCreateOptions: true,
  defaultValue: {value: 'US__CA__SF', label: "San Francisco"},
})
```

--------------------------------

### Configure Botasaurus Server API with Custom Routes and Auth (TypeScript)

Source: https://www.omkar.cloud/botasaurus/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api

This TypeScript code configures the Botasaurus server to enable API functionality. It sets the API port to 3000, prefixes all routes with '/v1', and maps the 'hotelsSearchScraper' to the '/hotels/search' endpoint. It also adds a '/health' route and implements an 'onRequest' hook to enforce authentication using a secret in the 'x-secret' header.

```typescript
import ApiConfig from 'botasaurus-server/api-config';
import { hotelsSearchScraper } from "../src/scrapers";

// Enable API functionality
ApiConfig.enableApi();

// Production configuration
ApiConfig.setApiPort(3000);
ApiConfig.setApiBasePath("/v1");

// Add scraper aliases for direct access
ApiConfig.addScraperAlias(hotelsSearchScraper, '/hotels/search');

// Add custom routes
ApiConfig.addCustomRoutes((server) => {
  // Health check for monitoring
  server.get('/health', (request, reply) => {
    return reply.send({ status: 'OK'});
  });

  // Authentication middleware  
  server.addHook('onRequest', (request, reply, done) => {
    // Check for secret
    const secret = request.headers['x-secret'] as string;

    if (secret === '49cb1de3-419b-4647-bf06-22c9e1110313') {
      // Valid secret, proceed
      return done(); 
    } else {
      return reply.status(401).send({
        message: 'Unauthorized: Invalid secret.',
      });
    }
  });
});

```
Tokens:
Raw