### Extract Links Using Each Loop

Source: https://github.com/jdecampos/doc-bright-data/blob/main/10 - best-practices-for-web-scraper-ide.md

This example demonstrates extracting link URLs using a traditional `each()` loop. While functional, array methods like `map()` are generally preferred for conciseness.

```javascript
const links = [];
$('.card.product-wrapper').each(function(i, el) {
  links.push({url: $(this).find('h4 a').attr('href')});
})
return links;
```

--------------------------------

### Track Browser Event Listeners

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Starts tracking event listeners created by the browser. This is a prerequisite for later disabling them.

```javascript
track_event_listeners();
```

--------------------------------

### Example Interaction Code for Web Scraper IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/05 - develop-a-self-managed-scraper-with-the-ide.md

Use this JavaScript code to navigate a website, wait for specific elements, parse data, and collect it within the Web Scraper IDE. Ensure the target website structure matches the selectors used.

```javascript
// Example interaction code
navigate(input.url);
wait('.product-list');
let data = parse();
collect(data);
```

--------------------------------

### Capture and Replay GraphQL Requests

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part1.md

The `capture_graphql()` function captures and replays GraphQL requests. It can be configured with options for payload and URL. Use `wait_captured()` to get captured requests and `replay()` to replay them with new variables.

```javascript
let q = capture_graphql({
    payload: {id: 'ProfileQuery'},
    // you may need to pass url opt as RegExp in case when 
    // graphql endpoint is not "*/graphql" which is default value
    // url: /\bgraphql\b/ // default
});

navigate('https://example.com');

let [first_query, first_response] = q.wait_captured();

collect(first_response.data.profile);

let second = q.replay({
    variables: {other_id: 2},
});

collect(second.data.profile);
```

```javascript
let q = capture_graphql({
    payload: {id: 'ProfileQuery'},
    // you may need to pass url opt as RegExp in case when 
    // graphql endpoint is not "*/graphql" which is default value
    // url: /\bgraphql\b/ // default
});

navigate('https://example.com');

if (!q.is_captured())
    click('#load_more');

let [first_query, first_response] = q.wait_captured();

collect(first_response.data.profile);

let second = q.replay({
    variables: {other_id: 2},
});

collect(second.data.profile);


```

--------------------------------

### Example Parser Code for Web Scraper IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/05 - develop-a-self-managed-scraper-with-the-ide.md

This JavaScript code parses HTML content to extract product details like title, price, and URL. It uses jQuery-like selectors to target elements within the '.product-list .item' class. The extracted URLs are converted to absolute URLs.

```javascript
// Example parser code
return {
  products: $('.product-list .item').map(function() {
    return {
      title: $(this).find('.title').text().trim(),
      price: $(this).find('.price').text().trim(),
      url: new URL($(this).find('a').attr('href'), location.href).href
    };
  }).get()
};
```

--------------------------------

### track_event_listeners

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Starts tracking event listeners created by the browser, which is necessary for later use of `disable_event_listeners()`.

```APIDOC
## track_event_listeners

### Description
Starts tracking the event listeners that the browser creates. This function is a prerequisite for using `disable_event_listeners()` later.

### Syntax
TRACK_EVENT_LISTENERS();

### Examples
```javascript
track_event_listeners();
```
```

--------------------------------

### Get Browser Window Size

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part1.md

The `browser_size()` function returns the current dimensions of the browser window. This is a placeholder function.

```javascript
TBD
```

--------------------------------

### Get Response Headers in JavaScript

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Retrieve the response headers from the last page load. Access specific headers like 'content-type' using object notation.

```javascript
let headers = response_headers();
console.log('content-type', headers['content-type']);
```

--------------------------------

### Navigate with Options

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

Customize navigation behavior using options like `wait_until`, `referer`, `timeout`, `header`, and `fingerprint`.

```javascript
// waits until DOM content loaded event is fired in the browser
navigate([url], {wait_until: 'domcontentloaded'});
```

```javascript
// adds a referer to the navigation
navigate([url], {referer: [url]});
```

```javascript
// the number of milliseconds to wait for. Default is 30000 ms
navigate([url], {timeout: 45000});
```

```javascript
// add headers to the navigation
navigate([url], {header : 'accept: text/html'});
```

```javascript
// specify browser width/height
navigate([url], {fingerprint: {screen: {width: 400, height: 400}}});
```

--------------------------------

### Get Redirect History

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Retrieves a list of URLs that the browser was redirected through since the last 'navigate' call.

```javascript
navigate('http://google.com');
let redirects = redirect_history();
// returns:
// [
//   'http://google.com',
//   'http://www.google.com',
//   'https://www.google.com/',
// ]
```

--------------------------------

### Process Product Listings

Source: https://github.com/jdecampos/doc-bright-data/blob/main/02 - basic.md

Navigates to individual product pages based on URLs extracted from search results. Use this to process each product found in a search.

```javascript
navigate(input.search_page)
let listings = parse().listings
for (let listing_url of listings)
     next_stage({listing_url})
```

--------------------------------

### Navigate to URL with Options

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Navigates the browser to a specified URL. Supports options for waiting, referer, timeout, status code handling, and screen dimensions.

```javascript
navigate(input.url);
navigate('https://example.com');
```

```javascript
navigate(`url`, {wait_until: 'domcontentloaded'});
```

```javascript
navigate(`url`, {referer: `url`});
```

```javascript
navigate(`url`, {timeout: 45000});
```

```javascript
navigate(`url`, {allow_status: [404]});
```

```javascript
navigate(`url`, {
    fingerprint: {screen: {width: 400, height: 400}},
});
```

--------------------------------

### Navigating to a URL in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

Use the `navigate` function to direct the browser session to a specified URL. It accepts a URL string or an object with navigation options.

```javascript
navigate([url]);
```

```javascript
navigate(input.url);
```

```javascript
navigate('https://example.com');
```

--------------------------------

### Resolve URL in JavaScript

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Use this function to get the final URL after redirects. Ensure 'parse' and 'resolve_url' are available in the scope.

```javascript
let {href} = parse().anchor_elem_data;
collect({final_url: resolve_url(href)});
```

--------------------------------

### Collect Product Data

Source: https://github.com/jdecampos/doc-bright-data/blob/main/02 - basic.md

Navigates to a product page and collects the parsed product data. Use this after navigating to a specific product's page.

```javascript
navigate(input.listing_url)
collect(parse())
```

--------------------------------

### Get status code of last page load

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Retrieves the HTTP status code from the most recent page load operation.

```javascript
collect({status_code: status_code()});
```

--------------------------------

### Get Element Bounding Box

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part1.md

The `bounding_box()` function returns the coordinates of the first matched element relative to the page. It requires a CSS selector.

```javascript
let box = bounding_box('.product-list');
// box == {
//   top: 10,
//   right: 800,
//   bottom: 210,
//   left: 200,
//   x: 200,
//   y: 10,
//   width: 600,
//   height: 200,
// }
```

--------------------------------

### Standard Navigation and Waiting

Source: https://github.com/jdecampos/doc-bright-data/blob/main/10 - best-practices-for-web-scraper-ide.md

For straightforward scraping tasks, use basic navigation and waiting functions. This approach is clear and avoids unnecessary complexity.

```javascript
navigate("https://example.com");
wait('h1');
```

--------------------------------

### Moving to the Next Stage in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

The `next_stage` function initiates the next stage of the crawler, passing a specified input object.

```javascript
next_stage({url: 'http://example.com', page: 1});
```

--------------------------------

### Get Text Content Safely

Source: https://github.com/jdecampos/doc-bright-data/blob/main/10 - best-practices-for-web-scraper-ide.md

Use `text_sane()` to safely extract text content, or apply string manipulation like `replace()` with regex to isolate numerical values.

```javascript
let name = $('a').text_sane();
or if you need only numbers
let value= +$('a').text().replace(/\D+/g, '');
```

--------------------------------

### load_html

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Loads an HTML string and returns a Cheerio instance, allowing for easy parsing and manipulation of the HTML content.

```APIDOC
## load_html

### Description
Loads an HTML string and returns a Cheerio instance, allowing for easy parsing and manipulation of the HTML content.

### Parameters
#### html
- **html** (string) - Required - Any HTML string to be parsed.
```

--------------------------------

### run_stage

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Executes a specific stage of the crawler using a new browser session, passing input data to it.

```APIDOC
## run_stage

### Description
Executes a specific stage of the crawler using a new browser session, passing input data to it.

### Syntax
```javascript
run_stage(<stage>, <input>);
```

### Parameters
- **stage** (number) - Required - The stage number to run (1 is the first stage).
- **input** (object) - Required - An object containing input data to pass to the next browser session.
```

--------------------------------

### next_stage

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Initiates the next stage of the crawler with a provided input object. This is used to pass data to subsequent crawling steps.

```APIDOC
## next_stage

### Description
Initiates the next stage of the crawler with a provided input object. This is used to pass data to subsequent crawling steps.

### Parameters
#### input
- **input** (object) - Required - The input object to pass to the next browser session.
```

--------------------------------

### Load HTML and Collect Data

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Loads an HTML string, parses it with Cheerio, and collects specific data. The 'load_html' function returns a Cheerio instance.

```javascript
let $$ = load_html('<p id="p1">p1</p><p id="p2">p2</p>');
collect({data: $$('#p2').text()});
```

--------------------------------

### Collecting Money Data

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part4.md

Instantiate a 'Money' object with a value and currency, then collect it.

```javascript
let p = new Money(10, 'USD');
collect({product_price: p});
```

--------------------------------

### navigate

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Navigates the browser to a specified URL. It supports various options for controlling the navigation process, including waiting for specific events, setting referers, timeouts, and allowing specific HTTP status codes.

```APIDOC
## navigate

### Description
Navigates the browser to a specified URL. It supports various options for controlling the navigation process, including waiting for specific events, setting referers, timeouts, and allowing specific HTTP status codes.

### Syntax
navigate(<url>, [opt]);

### Parameters
#### url
- **url** (string) - Required - The URL to navigate to.
#### opt
- **opt** (object) - Optional - Navigation options.
  - **wait_until** (string) - Optional - Specifies when to consider the navigation complete (e.g., 'domcontentloaded').
  - **referer** (string) - Optional - Sets the referer header for the navigation.
  - **timeout** (number) - Optional - The time in milliseconds to wait for the navigation to complete. Defaults to 30000 ms.
  - **allow_status** (array) - Optional - An array of HTTP status codes that should not throw an error (e.g., [404]).
  - **fingerprint** (object) - Optional - Object to specify browser fingerprint properties like screen dimensions.
    - **screen** (object) - Optional - Screen properties.
      - **width** (number) - Optional - Screen width.
      - **height** (number) - Optional - Screen height.
```

--------------------------------

### Running a Specific Stage in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

The `run_stage` function executes a specified stage of the crawler with a new browser session and input.

```javascript
run_stage(2, {url: 'http://example.com', page: 1});
```

--------------------------------

### Emulate Device Viewport

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part4.md

Changes the user agent and screen parameters to emulate a specific mobile device. This allows viewing pages as they would appear on different devices.

```javascript
emulate_device('iPhone X');
emulate_device('Pixel 2');
```

--------------------------------

### Efficient Pagination with `rerun_stage()`

Source: https://github.com/jdecampos/doc-bright-data/blob/main/10 - best-practices-for-web-scraper-ide.md

For websites with pagination, call `rerun_stage()` from the root page for each page instead of from within each page. This enables parallel request processing and speeds up scraping.

```javascript
navigate(input.url);
let $ = html_load(html());
let next_page_url = $('.next_page').attr('href');
rerun_stage({url: next_page_url});
```

```javascript
let url = new URL(input.url);
if (input.page)
    url.searchParams.set('page', input.page);
navegate(url);
// The input.page does not exist except on the root page, from which we
// have already invoked rerun_stage() for each page.
if (input.page)
    return;

let $ = html_load(html());
let total_products = +$('.total_pages').text();
let total_pages = Math.ceil(total_products / 20);
total_pages = Math.min(total_pages, 50);

for (let page=2;page<=total_pages;page++)
    rerun_stage({url: input.url, page});
```

--------------------------------

### Configure HTML Capture Options

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Sets options to control how HTML is captured, such as including coordinate attributes.

```javascript
html_capture_options({
    coordinate_attributes: true,
});
```

--------------------------------

### request

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Make a direct HTTP request to a specified URL or with given options. Supports various HTTP methods and request configurations.

```APIDOC
## request

### Description
Make a direct HTTP request.

### Parameters
#### Path Parameters
- **url | options** (string | object) - Required - The URL to make the request to, or an object containing request options.

### Request Body (if options object is used)
#### Request Body
- **url** (string) - Required - The URL for the request.
- **method** (string) - Optional - The HTTP method (e.g., GET, POST). Defaults to GET.
- **headers** (object) - Optional - An object containing request headers.
- **body** (object) - Optional - An object representing the request body.
```

--------------------------------

### Make HTTP Request in JavaScript

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Perform direct HTTP requests. Supports both simple URL strings and detailed options objects for methods, headers, and bodies.

```javascript
let res = request('http://www.example.com');
```

```javascript
let res = request({url: 'http://www.example.com', method: 'POST', headers: {'Content-type': 'application/json'}, body: {hello: 'world'}})
```

--------------------------------

### Interaction Code: Using Parsed Data for Requests

Source: https://github.com/jdecampos/doc-bright-data/blob/main/11 - complete-web-scraper-ide-examples.md

Illustrates how to extract data (like a product ID) from an HTML page during interaction, and then use that data to construct a dynamic request URL.

```javascript
navigate("https://example.com/1");
tag_html("page1");
let page_html = html();
let page_html2 = wait_for_parser_value("page1"); // the same
let $ = load_html(page_html);
let req_id = $('.product-id').text();
tag_request("product_json", {url: "https://example.com/product/"+req_id});
```

--------------------------------

### Interaction Code: Tagging Multiple Pages

Source: https://github.com/jdecampos/doc-bright-data/blob/main/11 - complete-web-scraper-ide-examples.md

Demonstrates how to navigate through multiple pages and tag the HTML content of each page with a unique identifier for later parsing.

```javascript
navigate("https://example.com/1");
tag_html("page1");
navigate("https://example.com/2");
tag_html("page2");
navigate("https://example.com/3");
tag_html("page3");
```

--------------------------------

### mouse_to

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Moves the mouse cursor to a specified (x, y) position on the page. This can be used to simulate user interaction.

```APIDOC
## mouse_to

### Description
Moves the mouse cursor to a specified (x, y) position on the page. This can be used to simulate user interaction.

### Syntax
mouse_to(<x>, <y>);

### Parameters
#### x
- **x** (number) - Required - The target x-coordinate.
#### y
- **y** (number) - Required - The target y-coordinate.
```

--------------------------------

### Collecting Image Data

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part4.md

Instantiate an 'Image' object with a URL and collect it using the 'collect' function.

```javascript
let i = new Image('https://example.com/image.png');
collect({image: i});
```

--------------------------------

### Basic Product Detail Page (PDP) Scraper

Source: https://github.com/jdecampos/doc-bright-data/blob/main/11 - complete-web-scraper-ide-examples.md

Scrapes detailed information from a product page, including company info, social links, and tech stack. Handles URL redirection and checks for 'Page not found'.

```javascript
let url = new URL(input.url.replace('https://www.slintel.com','https://6sense.com'));
url = new URL(url.pathname, 'https://6sense.com');
navigate(url);
if (location.href === 'https://6sense.com/company')
  dead_page(`Page not found`);
tag_html('html');
```

```javascript
const $ = html_load(parser.html)
const nextData = JSON.parse($('#__NEXT_DATA__').html());
const pageProps = nextData.props.pageProps;
const companyInfo = pageProps.company_data.companyInfo;

const linkedin = companyInfo.linkedin ?
  (companyInfo.linkedin.includes('http') ? companyInfo.linkedin :
    'https://'+companyInfo.linkedin) : null;
const techCategories = pageProps.company_data?.technologies_mapper_view?.categories
  ? Object.values(pageProps.company_data.technologies_mapper_view.categories).map(v => Object.keys(v)).flat()
  : null;
const pageData = {
    "name": companyInfo.name,
    "about": companyInfo.company_description,
    "num_employees": companyInfo.employee_range,
    "type": companyInfo.company_type,
    "industries": companyInfo.industry_v2_ranked.filter(v=>v),
    "techstack_arr": techCategories,
    "country_code": companyInfo.country,
    "website": companyInfo.display_domain
      ? new URL(companyInfo.display_domain.includes('http') ? companyInfo.display_domain : 'https://'+companyInfo.display_domain)
      : null,
    "social_media_urls": linkedin,
    "company_news": pageProps.company_data.company_news.map(v=>({
      title: "company_news_data",
      date: "",
      link: "",
    })),
    "last_updated": new Date(companyInfo.last_updated_at*1e3),
    "url": new URL(parser.html_url),
    "logo": companyInfo.logo,
    "location": companyInfo.location,
    "region": [companyInfo.country, companyInfo.state].filter(v=>v).join(', '),
    "id": nextData.query.companyid,
    "slintel_resources": companyInfo.recommended_companies.map(v=>({
      link: v.display_domain ? new URL(v.display_domain.includes('http') ? v.display_domain : 'https://'+v.display_domain) : null,
      title: v.name,
      type: v.company_type,
    })),
    "stock_symbol": companyInfo.stock_symbol,
};
return pageData;
```

--------------------------------

### Accessing Input Data in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

The global `input` object provides access to trigger input or results from previous stages.

```javascript
navigate(input.url);
```

--------------------------------

### Clicking Elements in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

The `click` function simulates a mouse click on a specified element after it appears on the page.

```javascript
click(<selector>);
```

```javascript
click('#show-more');
```

--------------------------------

### Using Cheerio for DOM Manipulation

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part4.md

Utilize the '$' instance, which is a cheerio object, to select and manipulate elements in the HTML.

```javascript
$('#example').text();
$('$example').attr('href');
$('#example').text_sane(); /* This is like $().text() but also trims text and replace all space characters with single space "a b \t\n\n c" -> "a b c" */
```

--------------------------------

### Parse User Activity Summary

Source: https://github.com/jdecampos/doc-bright-data/blob/main/11 - complete-web-scraper-ide-examples.md

Extracts various summary statistics for user activity, including reputation, posts, and badges. Includes error handling for JSON parsing.

```javascript
'topactivity': ($) => {
        let summary_graph_data;
        try {
            summary_graph_data = JSON.parse(
              /graphDatas[^[]+([[^]]+])/gm.exec(
                $('*').first().html())?.[1]);
        } catch(e) {
            console.log('graphData not found')
        }
        let [summary_people_reached, summary_posts_edited,
          summary_helpful_flags, summary_votes_cast] = 
          $('div:not(.js-highlight-box-reputation) > h1.flex--item + div .flex--item .fs-body3')
            .toArray().map(v=>$(v).text_sane().replace(/,/gm, '') || null);
        return {
            summary_reputation: +$('#top-cards h4.fs-headline1')
              .text().replace(/D+/gm, ''),
            summary_top_overall: $('a[href*="alltime"]')
              .first().text_sane() || null,
            summary_next_tag_badge: $('#rep-card-next-tag-badge a')
              .text_sane() || null,
            summary_graph_data,
            summary_next_tag_score:
              $('div.fl-shrink1 + div.fl-grow1 .fs-fine')
                .first().text_sane().replace(/,/gm, '') || null,
            summary_next_tag_answers:
              $('div.fl-shrink1 + div.fl-grow1 .fs-fine')
                .last().text_sane().replace(/,/gm, '') || null,
            summary_badges: $('h3 +div .s-badge').toArray()
              .map(v => $(v).attr('title')).join(', '),
            summary_last_badge: $('#badge-card-last').text_sane() || null,
            summary_next_badge: {
                name: $('#js-badge-card-next').text_sane() || null,
                progress: $('h4.flex--item.ws-nowrap + span')
                  .text_sane() || null
            },
            summary_people_reached,
            summary_posts_edited,
            summary_helpful_flags,
            summary_votes_cast,
        };
    },
```

--------------------------------

### tag_download

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Allows capturing files downloaded by the browser based on a URL pattern.

```APIDOC
## tag_download

### Description
Allows capturing files downloaded by the browser based on a URL pattern.

### Syntax
```javascript
tag_download(<url>);
```

### Parameters
- **url** (RegExp|string) - Required - A pattern or string to match the download URL against.

### Usage Example
```javascript
let SEC = 1000;
let download = tag_download(/example.com\/foo\/bar/);
click('button#download');
let file1 = download.next_file({timeout: 10*SEC});
let file2 = download.next_file({timeout: 20*SEC});
collect({file1, file2});
```
```

--------------------------------

### Interaction Code: Navigate and Parse (Old vs. New)

Source: https://github.com/jdecampos/doc-bright-data/blob/main/11 - complete-web-scraper-ide-examples.md

Compares old and new syntax for navigating to a URL and parsing content in the Web Scraper IDE. The new syntax simplifies data collection.

```javascript
// Old code
navigate("https://example.com");
collect(parse());

// New code
navigate("https://example.com");

// New code alternative
navigate("https://example.com");
tag_html("html_key");
```

--------------------------------

### Using the URL Class in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

The `URL` class, part of NodeJS's standard 'url' module, can be instantiated to parse and manipulate URL strings.

```javascript
let u = new URL('https://example.com');
```

--------------------------------

### Handle Dead Pages with 'dead_page' Condition

Source: https://github.com/jdecampos/doc-bright-data/blob/main/10 - best-practices-for-web-scraper-ide.md

When using the navigate command, add a 'dead_page' condition to check for non-existent pages. This prevents unnecessary retries, especially when the website returns status codes other than 404.

```javascript
try {
    // no need to wait 30sec 'ok-selector' in case of dead_page()
    wait('ok-selector');
} catch(e) {
    // in this case we can't be sure that the page is real dead
    dead_page('Page doesn\'t exist');
}
```

```javascript
wait('ok-selector, 404-selector');
if (el_exists('404-selector'))
  dead_page();
```

--------------------------------

### Tag downloaded files

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Captures files downloaded by the browser based on a URL pattern. Allows for sequential retrieval of downloaded files with timeouts.

```javascript
let SEC = 1000;
let download = tag_download(/example.com\/foo\/bar/);
click('button#download');
let file1 = download.next_file({timeout: 10*SEC});
let file2 = download.next_file({timeout: 20*SEC});
collect({file1, file2});
```

--------------------------------

### right_click

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Simulates a right-click on a specified element. It waits for the element to appear before performing the action.

```APIDOC
## right_click

### Description
Simulates a right-click on a specified element. It waits for the element to appear before performing the action.

### Syntax
```javascript
right_click(<selector>);
```

### Parameters
- **selector** (string) - Required - The CSS selector of the element to right-click on.
```

--------------------------------

### Check Element Existence Before Proceeding

Source: https://github.com/jdecampos/doc-bright-data/blob/main/10 - best-practices-for-web-scraper-ide.md

Use `el_exists` to verify if an element is present on the page before attempting to interact with it. Throws an error if the element is not found, preventing further script execution on a malformed page.

```javascript
if (!el_exists('.product-title'))
    throw new Error('Failed to load product page');
```

--------------------------------

### Hover on Element

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Waits for an element to appear and then hovers over it. Use a CSS selector to specify the element.

```javascript
hover('#item');
```

--------------------------------

### Collecting Data Lines in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

Use the `collect` function to add a data line to the crawler's dataset. It can optionally include a validation function.

```javascript
collect([data_line] [validate_fn]);
```

```javascript
collect({ title: page_data.title, price: page_data.price });
```

```javascript
collect({ price: data.price });
```

```javascript
collect(line, l=>!l && throw new Error('Empty line'));
```

--------------------------------

### Loading More Content with Scroll in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

The `load_more` function simulates scrolling to the bottom of a list to trigger lazy-loaded content, commonly used on infinite-scroll sites.

```javascript
load_more(<selector>);
```

```javascript
load_more('.search-results');
```

--------------------------------

### html_capture_options

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Configures options for how HTML content is captured during the crawling process. This allows for fine-tuning the data extraction.

```APIDOC
## html_capture_options

### Description
Configures options for how HTML content is captured during the crawling process. This allows for fine-tuning the data extraction.

### Parameters
#### options
- **options** (object) - Required - An object which accepts options defining how HTML capturing should be processed.
  - **coordinate_attributes** (boolean) - Optional - If true, coordinate attributes will be captured.
```

--------------------------------

### Selecting Options in Dropdowns in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

The `select` function is used to choose a specific value from a select (dropdown) element.

```javascript
select(<select>, <value>);
```

```javascript
select('#country', 'Canada');
```

--------------------------------

### input

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Provides access to the input object, which is available in the interaction code. This object is passed from trigger input or next_stage() calls.

```APIDOC
## input

### Description
Provides access to the input object, which is available in the interaction code. This object is passed from trigger input or next_stage() calls.

### Usage
```javascript
navigate(input.url);
```
```

--------------------------------

### Save Video URL from Element

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Saves the video URL from a specified HTML element using a CSS selector. Supports download options.

```javascript
tag_video(field, selector);
tag_video('video', '#product-video', {download: true});
```

--------------------------------

### Typing Text into Inputs in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

Use the `type` function to enter text into input fields. It waits for the input to appear and can simulate special keys like 'Enter' or 'Backspace'.

```javascript
type(<selector>, <text>);
```

```javascript
type('#location', 'New York');
```

```javascript
type(<selector>, ['Enter']);
```

```javascript
type(<selector>, ['Backspace']);
```

--------------------------------

### Collect Video Data

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part4.md

Instantiates a Video object with a given source URL and collects its data. This is typically used in conjunction with a 'collect' function.

```javascript
let v = new Video('https://example.com/video.mp4');
collect({video: v});
```

--------------------------------

### load_sitemap

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Reads URLs from an XML sitemap, supporting sitemap indexes and compressed sitemaps. It returns a list of pages or child sitemaps.

```APIDOC
## load_sitemap

### Description
Reads URLs from an XML sitemap, supporting sitemap indexes and compressed sitemaps. It returns a list of pages or child sitemaps.

### Parameters
#### url
- **url** (string) - Required - The URL of the sitemap XML file.
```

--------------------------------

### Set data lines for dataset

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Adds an array of data lines to the dataset for the current page crawl. Subsequent calls override previous ones. Includes optional validation.

```javascript
set_lines(products_so_far);
set_lines(products_so_far, i=>{
    if (!i.price)
        throw new Error('Missing price');
});
```

--------------------------------

### Extract URLs Using Array Methods

Source: https://github.com/jdecampos/doc-bright-data/blob/main/10 - best-practices-for-web-scraper-ide.md

Leverage `toArray()` and `map()` for concise and readable extraction of data from multiple elements. This is preferred over traditional `each()` loops.

```javascript
const links = $('.card.product-wrapper').toArray().map(v=>({
  url: $(v).find('h4 a').attr('href'),
}));
```

--------------------------------

### Logging Messages in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

Use `console.log` to output messages from the interaction code, useful for debugging and monitoring.

```javascript
console.log(1, 'luminati', [1, 2], {key: value});
```

--------------------------------

### load_more

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Scrolls to the bottom of a list to trigger the loading of more items, commonly used for infinite-scroll websites. It can also specify child elements and trigger selectors.

```APIDOC
## load_more

### Description
Scrolls to the bottom of a list to trigger the loading of more items, commonly used for infinite-scroll websites. It can also specify child elements and trigger selectors.

### Parameters
#### selector
- **selector** (string) - Required - Selector for the element that contains the lazy-loaded items.
#### options
- **options** (object) - Optional - Configuration for loading more items.
  - **children** (string) - Optional - Selector for the child elements within the list.
  - **trigger_selector** (string) - Optional - Selector for the element that triggers loading more items.
  - **timeout** (number) - Optional - The time in milliseconds to wait for new items to load.
```

--------------------------------

### press_key

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Simulates pressing special keys, such as 'Enter' or 'Backspace', on the currently focused input element. This is often used after typing text into search fields.

```APIDOC
## press_key

### Description
Simulates pressing special keys, such as 'Enter' or 'Backspace', on the currently focused input element. This is often used after typing text into search fields.

### Parameters
#### key
- **key** (string) - Required - The key to press (e.g., 'Enter', 'Backspace').
```

--------------------------------

### Mark Page as Blocked

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part1.md

Use `blocked()` to indicate that a page was blocked by the website. An optional message can be provided.

```javascript
blocked();
blocked('Login page was shown');
```

--------------------------------

### Reparse Data with next_stage and rerun_stage

Source: https://github.com/jdecampos/doc-bright-data/blob/main/11 - complete-web-scraper-ide-examples.md

This feature allows re-parsing collected data without rerunning the entire interaction code. Use `load_html` to parse data in intermediate steps.

```javascript
return $('a[data-qa-action="product-click"]').toArray()
  .map(v => ({url: new URL($(v).attr('href'))}));
```

```javascript
block(['*.ico', '*.png', '*.jpg', '/images/', '*gif']);
navigate(input.url);
wait('.view-option-selector-button, .product-groups_empty-list');
if(el_exists('.product-groups_empty-list'))
  dead_page('There are no products in this category')
wait('[aria-describedby="onetrust-policy-text"]');
click('button#onetrust-reject-all-handler');
wait_hidden('[aria-describedby="onetrust-policy-text"]');
$('.view-option-selector-button').eq(2).click();
wait_page_idle(2000);
scroll_to('bottom');
wait_page_idle(2000);
```

```javascript
return $('a[data-qa-action="product-click"]').toArray()
  .map(v => ({url: new URL($(v).attr('href'))}));
```

--------------------------------

### Load Sitemap URLs

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Reads URLs from a sitemap XML file, supporting compressed files and sitemap indexes. Returns an object with 'pages' or 'children' properties.

```javascript
let {pages} = load_sitemap({url: 'https://example.com/sitemap.xml.gz'});
```

```javascript
let {children} = load_sitemap({url: 'https://example.com/sitemap-index.xml'});
```

--------------------------------

### Detect Block on Page

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part1.md

The `detect_block()` function checks for specific conditions on a page resource to detect a block. It requires resource and condition parameters, such as existence or text content.

```javascript
detect_block({selector: '.foo'}, {exists: true});
detect_block({selector: '.bar'}, {has_text: 'text'});
detect_block({selector: '.baz'}, {has_text: /regex_pattern/});
```

--------------------------------

### Minimize Browser Requests by Combining Selectors

Source: https://github.com/jdecampos/doc-bright-data/blob/main/10 - best-practices-for-web-scraper-ide.md

Combine multiple selectors into a single call for interaction commands like click, el_exists, el_is_visible, and wait. This reduces latency and improves performance by making fewer requests to the browser.

```javascript
if (!(el_exists('#price1')) || el_exists('#price2')
  || el_exists('#price3') || el_exists('#discount'))
{
    dead_page('No price found');
}
```

```javascript
if (!el_exists('#price1, #price2, #price3, #discount'))
  dead_page('No price found');
```

--------------------------------

### Waiting for Specific Text in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

Use `wait_for_text` to wait until an element contains specific text. This is useful for dynamic content loading.

```javascript
wait_for_text(<selector>, <text>);
```

```javascript
wait_for_text('.location', 'New York');
```

--------------------------------

### Scroll to an element or position

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Scrolls the page to make a specific element visible. Use 'top' or 'bottom' for page extremities. Includes an option for immediate scrolling.

```javascript
scroll_to('.author-profile');
scroll_to('top'); // scroll to the top of the page
scroll_to('bottom'); // scroll to the bottom of the page
scroll_to('top', {immediate: true}); // jump to top of page immediately
```

--------------------------------

### Synchronize Network Requests with `wait_for_parser_value()`

Source: https://github.com/jdecampos/doc-bright-data/blob/main/10 - best-practices-for-web-scraper-ide.md

When using `tag_response()` and needing to ensure a request is complete before collecting data, use `wait_for_parser_value()`. This function waits for the tagged response to be available.

```javascript
tag_response('product', /api\/product/);

// parser code
// in this case we can't be sure that the request is finished
let {product} = parser;
return product.data;
```

```javascript
tag_response('product', /api\/product/);
navegate('https://example.com');
wait_for_parser_value('product');

// parser code
let {product} = parser;
return product.data;
```

```javascript
tag_response('product', /api\/product/);
navegate('https://example.com');

// you can also get the data from the response in interaction
let product = wait_for_parser_value('product');
navegate(product.reviews_url);
tag_html('reviews_html');

// parser code
let {product, reviews_html} = parser;
let $ = html_load(reviews_html);
let reviews = $('.review').toArray().map(v=>$(v).text());
return {
  ...product.data,
  reviews,
};
```

--------------------------------

### Tag all matching responses

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Saves response data for all browser requests that match a given URL pattern. Supports JSONP parsing and error allowance.

```javascript
tag_all_responses(<field>, <pattern>, <options>);
tag_all_responses('resp', /url/, {jsonp: true});
tag_all_responses('resp', /url/, {allow_error: true});
tag_all_responses('profiles', /\/api\/profile/);
navigate('https://example.com/sports');
let profiles = parse().profiles;
for (let profile of profiles)
    collect(profile);
```

--------------------------------

### Type Special Keys

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Simulates typing special keys like 'Enter' or 'Backspace' into the currently focused input element.

```javascript
press_key('Enter');
```

```javascript
press_key('Backspace');
```

--------------------------------

### Waiting for Elements in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

The `wait` function pauses execution until a specified element appears on the page. It supports various selectors and optional timeout settings.

```javascript
wait(<selector>);
```

```javascript
wait('#welcome-splash');
```

```javascript
wait('.search-results .product');
```

```javascript
wait('[href^="/product"]');
```

```javascript
wait(<selector>, {timeout: 5000});
```

```javascript
wait(<selector>, {hidden: true});
```

--------------------------------

### tag_screenshot

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Saves a screenshot of the current page HTML. Options can be provided to customize the filename and whether to capture the full page.

```APIDOC
## tag_screenshot

### Description
Saves a screenshot of the page HTML. You can specify options like the filename and whether to capture the full page.

### Syntax
tag_screenshot(<field>, <options>);

### Parameters
- **field** (string) - The name of the tagged field.
- **options** (object) - Download options. `filename` specifies the output file name. `full_page` (boolean) defaults to true.

### Examples
```javascript
tag_screenshot('html_screenshot', {filename: 'screen'});
tag_screenshot('view', {full_page: false});
```
```

--------------------------------

### Tag JavaScript Value from Page

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Tags a JavaScript value directly from the browser page using a specified key.

```javascript
tag_window_field(field, key);
tag_window_field('initData', '__INIT_DATA__');
```

--------------------------------

### Freeze Page State

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part1.md

Forces the page to stop making any further changes. This experimental command is useful for capturing a page in a static state for consistent snapshots.

```javascript
freeze_page();
```

--------------------------------

### Scrape Multiple Tabs from User Profile

Source: https://github.com/jdecampos/doc-bright-data/blob/main/11 - complete-web-scraper-ide-examples.md

Navigates to a user profile and scrapes data from multiple tabs (e.g., 'topactivity', 'answers', 'questions'). Handles potential 404 errors for tabs.

```javascript
const tabs = [
  'topactivity',
  'answers',
  'questions',
  'tags',
  'articles',
  'badges',
  'bookmarks',
  'bounties'
];
function loadTab(url, name) {
  return new Promise(async (resolve, reject)=>{
    try {
        const tabUrl = new URL(url);
        tabUrl.searchParams.set('tab', name);
        navigate(tabUrl, { allow_status: [404] });
        const html_ = html();
        tag_html(name);
        resolve(html_);
    } catch(e) { reject(e); }
  });
}
const userIdFromUrl = (input.url && input.url.includes('https://stackoverflow.com/users/'))
    ? input.url.replace('https://stackoverflow.com/users/', '').split('/').shift()
    : null;
const userId = input.user_id || userIdFromUrl || 1;
const userUrl = `https://stackoverflow.com/users/${userId}`;
navigate(userUrl, { allow_status: [404] });
tag_html('user');
Promise.all(tabs.map( async tabName => loadTab(userUrl, tabName)));
```

--------------------------------

### Accessing Current Location Info in IDE

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

The `location` object provides information about the current browser location, including the `href` property.

```javascript
navigate('https://example.com');
```

```javascript
location.href;
```

--------------------------------

### Scroll to Element

Source: https://github.com/jdecampos/doc-bright-data/blob/main/04 - coding-environment-tutorials.md

Scrolls the page to make a specified element visible. Use with an element selector.

```javascript
scroll_to(<selector>);
scroll_to('.author-profile');
```

--------------------------------

### select

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Selects a specific value from a select (dropdown) element.

```APIDOC
## select

### Description
Selects a specific value from a select (dropdown) element.

### Syntax
```javascript
select(<selector>, <value>);
```

### Parameters
- **selector** (string) - Required - The CSS selector of the select element.
- **value** (string) - Required - The value to select from the dropdown.
```

--------------------------------

### Parse Page Data

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part2.md

Parses the current page's data and collects specific fields like title and price. Assumes 'parse()' returns an object with these properties.

```javascript
let page_data = parse();
collect({
    title: page_data.title,
    price: page_data.price,
});
```

--------------------------------

### Wait for Any Condition

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part4.md

Waits until at least one of the specified conditions (element selectors) is met. Useful for scenarios where multiple elements might appear.

```javascript
wait_any(['#title', '#notfound']);
```

--------------------------------

### Extract User Profile Data

Source: https://github.com/jdecampos/doc-bright-data/blob/main/11 - complete-web-scraper-ide-examples.md

Parses and extracts detailed user profile information from a web page, including ID, name, title, stats, and about section. Requires Cheerio or a similar DOM manipulation library.

```javascript
communities = communities.find('li.flex--item')
            .toArray().map(v=>({
              name: $(v).find('.fl-grow1').text_sane() || null,
              score: $(v).find('.fl-shrink0').text_sane()
                .replace(/,/gm, '') || null,
            }));
      }
      else
        communities = [];
      let badges = $('.flex__fl-equal > .flex--item').toArray()
        .map(v=>{
          let $v = $(v);
          return {
              badge_type: $v.find('.mr12 .fc-gold').length
                ? 'gold'
                : $v.find('.mr12 .fc-silver').length
                    ? 'silver'
                    : $v.find('.mr12 .fc-bronze').length
                        ? 'bronze'
                        : 'unknown',
              badge_name: $v.find('.fs-caption').text_sane()
                .replace(' badges', '') || null,
              number_of_badges: $v.find('.fs-title').text()
                .replace(/,/gm, '') || null,
              badges: $v.find('a.badge').toArray().map(b=>({
                  name: $(b).text_sane() || null,
                  description: $(b).attr('title') || null
              }))
          };
      });
      return {
        url: new URL(parser.user_url),
        id: parser.user_url.split('/').pop(),
        'id#': (input.url && input.url
            .includes('https://stackoverflow.com/users/'))
            ? input.url?.match(/https:\/\/stackoverflow.com\/users\/(d{1,15}).*/)?.[1]/gm)
            : input.user_id
          || 1,
        user_id: +$('[property="og:url"]').attr('content')
          .match(/https:\/\/stackoverflow.com\/users\/(d{1,15}).*/)?.[1]/)/gm
            || +input.url?.match(/https:\/\/stackoverflow.com\/users\/(d{1,15}).*/)?.[1]/gm)
            || +input.user_id
            || null,
        name: $('.fs-headline2').text_sane() || null,
        type: $('#mainbar-full > div:first-child .s-badge')
          .text_sane() || null,
        title: $('#mainbar-full > div:first-child .fs-title')
          .text_sane() || null,
        active_from: new Date(
          $('#mainbar-full > div:first-child .list-reset span[title]').attr('title')),
        last_seen: $('#mainbar-full > div:first-child .fs-title + ul li:last-child')
          .text_sane() || null,
        linkes: $('#mainbar-full > div:first-child ul.list-reset:last-child a')
          .toArray().map(v => $(v).attr('href')),
        location: $('#mainbar-full > div:first-child ul.list-reset:last-child li:last-child div[title]')
          .attr('title') || null,
        stats: $('#stats .flex--item').toArray()
          .map(v => $(v).text_sane()).filter(v => v),
        about: $('#main-content .s-prose p').toArray()
          .map(v => $(v).text_sane()) || null, //.join('\r'),
        collectives,
        communities,
        badges,
        top_tags: $('#top-tags .p12').toArray().map(v=>({
            name: $(v).find('.s-tag').text_sane() || null,
            score: $(v).find('.d-flex > .d-flex:nth-child(1)')
              .text_sane().replace(' Score', '')
              .replace(/,/gm, '') || null,
            posts: $(v).find('.d-flex > .d-flex:nth-child(2)')
              .text_sane().replace(' Posts', '')
              .replace(/,/gm, '') || null,
            posts_percent: $(v).find('.d-flex > .d-flex:nth-child(3)')
              .text_sane().replace(' Posts %', '').replace(/,/gm, ''),
        })),
        top_posts: $('#js-top-posts .p12 .d-flex').toArray().map(v=>({
            type: $(v).find('.iconQuestion').length ? 'question'
              : ($(v).find('.iconAnswer').length ? 'answer' : 'none'),
            score: $(v).find('.s-badge').text_sane()
              .replace(/,/gm, '') || null,
            text: $(v).find('a.d-table')
              .text_sane() || null,
            date: new Date($(v).find('.relativetime')
              .attr('title'))
        })),
        top_meta_posts: $('#js-top-posts + div .p12 .d-flex')
          .toArray().map(v=>({
            score: $(v).find('.s-badge').text_sane()
              .replace(/,/gm, '') || null,
            text: $(v).find('a.d-table').text_sane() || null,
        })),
        top_network_posts: $('#js-top-posts + div + div .p12 .d-flex')
          .toArray().map(v=>({
            score: $(v).find('.s-badge').text_sane()
              .replace(/,/gm, '') || null,
            text: $(v).find('a.d-table').text_sane() || null,
        }))
      };
    }
```

--------------------------------

### tag_all_responses

Source: https://github.com/jdecampos/doc-bright-data/blob/main/09 - functions-part3.md

Tags all browser request responses that match a given URL pattern.

```APIDOC
## tag_all_responses

### Description
Tags all browser request responses that match a given URL pattern.

### Syntax
```javascript
tag_all_responses(<field>, <pattern>, [options]);
```

### Parameters
- **field** (string) - Required - The name of the field to store the tagged response.
- **pattern** (RegExp|string) - Required - The URL pattern to match against.
- **options** (object) - Optional - Configuration options. `jsonp: true` can parse JSONP responses. `allow_error: true` includes error responses.
```