### Installing Herd SDK (Bash) Source: https://herd.garden/llms.txt Provides the bash command to install the Monitoro Herd Python SDK using pip. This is a prerequisite for using the Python examples. ```bash pip install monitoro-herd ``` -------------------------------- ### Google Search Trail package.json Configuration Source: https://herd.garden/llms.txt Example package.json file for a Herd trail, specifying the trail name, description, version, and dependencies like the Herd core library. ```json { "name": "google-search", "description": "Search google for webpages.", "version": "1.0.0", "dependencies": { "@monitoro/herd": "latest" } } ``` -------------------------------- ### Install Herd SDK with pnpm (Bash) Source: https://herd.garden/llms.txt Use this command to install the Herd SDK globally on your system using the pnpm package manager. ```bash pnpm add -g @monitoro/herd ``` -------------------------------- ### Install Herd SDK with yarn (Bash) Source: https://herd.garden/llms.txt Use this command to install the Herd SDK globally on your system using the yarn package manager. ```bash yarn global add @monitoro/herd ``` -------------------------------- ### Automate Browser with Herd SDK (Python) Source: https://herd.garden/llms.txt Python example demonstrating how to connect to a Herd device using the SDK, create a new page, navigate to a URL, and extract data using simple selectors. ```python from monitoro_herd import HerdClient # Connect to your Herd device client = HerdClient("your-token") await client.initialize() devices = await client.list_devices() device = devices[0] # Create a new page and navigate page = await device.new_page() await page.goto("https://example.com") # Extract data using simple selectors data = await page.extract({ "title": "h1", "description": "p", "link": "a" }) print("Extracted data:", data) ``` -------------------------------- ### Install Herd SDK with npm (Bash) Source: https://herd.garden/llms.txt Use this command to install the Herd SDK globally on your system using the npm package manager. ```bash npm install -g @monitoro/herd ``` -------------------------------- ### Installing Herd JavaScript SDK Source: https://herd.garden/llms.txt Install the Herd JavaScript SDK using npm to set up your development environment. Requires Node.js version 14 or higher. ```bash npm install @monitoro/herd ``` -------------------------------- ### Defining Trail URLs in TypeScript Source: https://herd.garden/llms.txt Example of defining an array of URL configurations in urls.ts for a Herd trail. Each entry specifies an ID, template, description, examples, and parameter definitions. ```typescript export default [{ "id": "my-url", "template": "https://example.com/path?param1={param1}¶m2={param2}", "description": "Description of what this URL represents", "examples": [ { "param1": "value1", "param2": "value2" } ], "params": { "param1": { "type": "string", "required": true, "description": "Description of param1" }, "param2": { "type": "string", "required": false, "default": "defaultValue", "description": "Description of param2" } } }]; ``` -------------------------------- ### Automate Browser with Herd SDK (JavaScript) Source: https://herd.garden/llms.txt JavaScript example demonstrating how to connect to a Herd device using the SDK, create a new page, navigate to a URL, and extract data using simple selectors. ```javascript // Connect to your Herd device const client = new HerdClient('your-token'); await client.initialize(); const devices = await client.listDevices(); const device = devices[0]; // Create a new page and navigate const page = await device.newPage(); await page.goto("https://example.com"); // Extract data using simple selectors const data = await page.extract({ title: "h1", description: "p", link: "a" }); console.log("Extracted data:", data); ``` -------------------------------- ### MCP Configuration for Herd Browser Trail (JSON) Source: https://herd.garden/llms.txt Example JSON configuration snippet to add the Herd browser trail as an MCP server, allowing AI agents to interact with it. ```json { "mcpServers": { "browser": { "command": "herd", "args": [ "trail", "server", "@herd/browser" ] } } } ``` -------------------------------- ### Implementing Trail Action in TypeScript Source: https://herd.garden/llms.txt Defines a TrailAction class with manifest, test, and run methods. It shows how to define parameters, results, examples, navigate pages, extract data using resources, and handle errors. ```typescript import type { Device, TrailAction, TrailActionManifest, TrailRunResources } from "@monitoro/herd"; export class MyTrailAction implements TrailAction { manifest: TrailActionManifest = { name: "my-action", description: "Description of what this action does", params: { param1: { type: "string", description: "Description of param1" }, param2: { type: "number", description: "Description of param2", default: 10 } }, result: { type: "array", description: "Description of the result", items: { type: "object", properties: { property1: { type: "string" }, property2: { type: "number" } } } }, examples: [ { "param1": "value1", "param2": 10 } ] } async test(device: Device, params: Record, resources: TrailRunResources) { try { const result = await this.run(device, params, resources); // Validate result if (!result || result.length === 0) { return { status: "error", message: "No results found", result: [] }; } return { status: "success", result }; } catch (e) { return { status: "error", message: `Error: ${e}`, result: null }; } } async run(device: Device, params: Record, resources: TrailRunResources) { const { param1, param2 } = params; const page = await device.newPage(); try { // Navigate to URL using the URL template from urls.ts await page.goto(resources.url('my-url', { param1, param2 })); // Extract data using selectors from selectors.ts const extracted = await page.extract(resources.selector('my-selector')); // Process extracted data const results = (extracted as any)?.dataKey || []; // Return processed results return results; } finally { await page.close(); } } } ``` -------------------------------- ### Extracting Lists of Items (`_$r`) - JavaScript Source: https://herd.garden/llms.txt Shows how to use the `_$r` (repeat) selector to find multiple elements matching a pattern and extract nested properties for each, followed by an example of iterating through the results. ```javascript const data = await page.extract({ items: { _$r: '.item', // Find all elements with class "item" title: 'h2', // For each item, get the title price: '.price', // For each item, get the price date: 'time' // For each item, get the date } }); // Access the extracted items data.items.forEach(item => { console.log(`${item.title}: ${item.price}, Posted: ${item.date}`); }); ``` -------------------------------- ### Extracting Page Metadata (Python) Source: https://herd.garden/llms.txt This example demonstrates extracting metadata about the page itself in Python, such as the page title from the `` tag, the meta description from a `<meta>` tag, and the canonical URL from a `<link>` tag's `href` attribute using the `attribute` key. ```python pageInfo = await page.extract({ "title": "title", "metaDescription": 'meta[name="description"]', "canonicalUrl": { "_$": 'link[rel="canonical"]', "attribute": "href" } }) ``` -------------------------------- ### Extracting Page Metadata (JavaScript) Source: https://herd.garden/llms.txt This example demonstrates extracting metadata about the page itself, such as the page title from the `<title>` tag, the meta description from a `<meta>` tag, and the canonical URL from a `<link>` tag's `href` attribute using the `attribute` key. ```javascript const pageInfo = await page.extract({ title: 'title', metaDescription: 'meta[name="description"]', canonicalUrl: { _$: 'link[rel="canonical"]', attribute: 'href' } }); ``` -------------------------------- ### Extracting Lists of Items (`_$r`) - Python Source: https://herd.garden/llms.txt Shows how to use the `_$r` (repeat) selector in Python to find multiple elements matching a pattern and extract nested properties for each, followed by an example of iterating through the results. ```python data = await page.extract({ "items": { "_$r": ".item", # Find all elements with class "item" "title": "h2", # For each item, get the title "price": ".price", # For each item, get the price "date": "time" # For each item, get the date } }) # Access the extracted items for item in data["items"]: print(f"{item['title']}: {item['price']}, Posted: {item['date']}") ``` -------------------------------- ### Defining Trail Selectors in TypeScript Source: https://herd.garden/llms.txt Example of defining an array of selector configurations in selectors.ts for a Herd trail. Selectors specify how to extract data from web pages using CSS selectors and data keys. ```typescript export default [{ "id": "my-selector", "value": { "dataKey": { "_$r": "#main-container", "title": { "_$": ".title" }, "description": { "_$": ".description" }, "link": { "_$": "a.link", "attribute": "href" } } }, "description": "Selector for extracting specific data", "examples": [ { "urlId": "my-url", "urlParams": { "param1": "value1", "param2": "value2" } } ] }]; ``` -------------------------------- ### Extracting News Article Lists (JavaScript) Source: https://herd.garden/llms.txt This example shows how to extract a list of news articles using `_$r` for repeating elements. It defines selectors and pipes for extracting the title, price (demonstrating reuse of `parseNumber`), and date (`parseDate`) for each article item. ```javascript const articles = await page.extract({ items: { _$r: '.item', title: { _$: 'h2', pipes: ['trim', 'toLowerCase'] }, price: { _$: '.price', pipes: ['parseNumber'] }, date: { _$: 'time', pipes: ['parseDate'] } } }); ``` -------------------------------- ### Using parseNumber for Various Formats (Python) Source: https://herd.garden/llms.txt This example shows how the `parseNumber` pipe handles different number formats in Python, including currency symbols, commas, decimals, and magnitude suffixes (M for million, T for trillion). It extracts three different price formats and converts them to numerical values. ```python data = await page.extract({ "price1": { "_$": ".price-1", # Contains "$1,234.56" "pipes": ["parseNumber"] # Result: 1234.56 }, "price2": { "_$": ".price-2", # Contains "$1.5M" "pipes": ["parseNumber"] # Result: 1500000 }, "price3": { "_$": ".price-3", # Contains "1.5T€" "pipes": ["parseNumber"] # Result: 1500000000000 } }) ``` -------------------------------- ### Using parseNumber for Various Formats (JavaScript) Source: https://herd.garden/llms.txt This example shows how the `parseNumber` pipe handles different number formats, including currency symbols, commas, decimals, and magnitude suffixes (M for million, T for trillion). It extracts three different price formats and converts them to numerical values. ```javascript const data = await page.extract({ price1: { _$: '.price-1', // Contains "$1,234.56" pipes: ['parseNumber'] // Result: 1234.56 }, price2: { _$: '.price-2', // Contains "$1.5M" pipes: ['parseNumber'] // Result: 1500000 }, price3: { _$: '.price-3', // Contains "1.5T€" pipes: ['parseNumber'] // Result: 1500000000000 } }); ``` -------------------------------- ### Extracting News Article Lists (Python) Source: https://herd.garden/llms.txt This example shows how to extract a list of news articles in Python using `_$r` for repeating elements. It defines selectors and pipes for extracting the title, price (demonstrating reuse of `parseNumber`), and date (`parseDate`) for each article item. ```python articles = await page.extract({ "items": { "_$r": ".item", "title": { "_$": "h2", "pipes": ["trim", "toLowerCase"] }, "price": { "_$": ".price", "pipes": ["parseNumber"] }, "date": { "_$": "time", "pipes": ["parseDate"] } } }) ``` -------------------------------- ### Creating a Page and Navigating Source: https://herd.garden/llms.txt Create a new browser page within the connected device and navigate to a specified URL. The `goto` method loads the URL and waits for the page to finish loading. ```javascript // Create a new page in the browser const page = await device.newPage(); // Navigate to a website await page.goto('https://example.com'); console.log('Successfully navigated to example.com'); ``` -------------------------------- ### Creating Page and Navigating (Python) Source: https://herd.garden/llms.txt Shows how to create a new browser page on a connected device and navigate to a specified URL using the Herd SDK in Python. Requires a connected device. ```python # Create a new page page = await device.new_page() # Navigate to a website await page.goto("https://example.com") print("Successfully navigated to example.com") ``` -------------------------------- ### Connecting to a Herd Device (Python) Source: https://herd.garden/llms.txt Demonstrates how to list available devices using the Herd client and connect to the first device in the list. Requires an initialized Herd client. ```python # Get available devices devices = await client.list_devices() # Connect to the first device device = devices[0] print(f"Connected to device: {device.id}") ``` -------------------------------- ### Basic Data Extraction with Herd Client (Python) Source: https://herd.garden/llms.txt Demonstrates the fundamental steps for using the monitoro_herd library to connect to a device, navigate to a URL, and extract data using CSS selectors. Includes error handling and resource cleanup. ```python import asyncio from monitoro_herd import HerdClient async def basic_extraction(): # Initialize the client client = HerdClient("your-token") try: # Initialize the connection await client.initialize() print("Client initialized successfully") # Get the first available device devices = await client.list_devices() if not devices: raise Exception("No devices available") device = devices[0] print(f"Connected to device: {device.id}") # Create a new page page = await device.new_page() print("New page created") # Navigate to a website print("Navigating to example.com...") await page.goto("https://example.com") print("Navigation complete") # Extract data using simple selectors print("Extracting content...") data = await page.extract({ "title": "h1", "description": "p", "link": "a" }) # Display the extracted data print("\nExtracted data:") print(f"Title: {data['title']}") print(f"Description: {data['description']}") print(f"Link: {data['link']}") except Exception as e: print(f"Error during automation: {e}") finally: # Always close resources print("Closing client connection...") await client.close() print("Client connection closed") # Run the async function asyncio.run(basic_extraction()) ``` -------------------------------- ### Run First Browser Trail (Bash) Source: https://herd.garden/llms.txt Execute the built-in browser trail using the Herd CLI. This command runs the trail for 'https://example.com' and outputs the result in markdown format. ```bash herd trail run @herd/browser -a markdown -p '{"url": "https://example.com"}' ``` -------------------------------- ### Complete Basic Automation Workflow - JavaScript Source: https://herd.garden/llms.txt This function demonstrates a full basic automation flow using the Monitoro Herd client. It initializes the client, lists devices, creates a new page, navigates to a URL, extracts content based on CSS selectors, and ensures the client is closed afterwards. ```javascript import { HerdClient } from '@monitoro/herd'; async function runBasicAutomation() { const client = new HerdClient('your-token'); try { // Initialize the client await client.initialize(); console.log('Client initialized successfully'); // Get the first available device const devices = await client.listDevices(); if (devices.length === 0) { throw new Error('No devices available'); } const device = devices[0]; console.log(`Connected to device: ${device.id}`); // Create a new page const page = await device.newPage(); console.log('New page created'); // Navigate to a website console.log('Navigating to example.com...'); await page.goto('https://example.com'); console.log('Navigation complete'); // Extract content console.log('Extracting content...'); const content = await page.extract({ title: 'h1', description: 'p', link: 'a' }); // Display the extracted content console.log('\nExtracted content:'); console.log(`Title: ${content.title}`); console.log(`Description: ${content.description}`); console.log(`Link: ${content.link}`); } catch (error) { console.error('Error during automation:', error); } finally { // Always close the client when done console.log('Closing client connection...'); await client.close(); console.log('Client connection closed'); } } // Run the automation runBasicAutomation(); ``` -------------------------------- ### Initializing Herd Client (Python) Source: https://herd.garden/llms.txt Shows how to import the HerdClient class and initialize it with an API token in Python. It emphasizes the need to call initialize() before use. Requires the monitoro-herd library. ```python # Import the Herd client from monitoro_herd import HerdClient # Initialize the client with your API URL and token client = HerdClient('your-token') # Always initialize the client before using it client.initialize() ``` -------------------------------- ### Basic Trail package.json Configuration Source: https://herd.garden/llms.txt Minimal package.json configuration required for a new Herd trail, defining the trail name, version, and dependency on the Herd core library. ```json { "name": "my-trail", "version": "1.0.0", "dependencies": { "@monitoro/herd": "latest" } } ``` -------------------------------- ### New Trail Directory Structure Source: https://herd.garden/llms.txt Standard directory structure for creating a new Herd trail, including placeholder files for URLs, selectors, actions, and package configuration. ```text my-trail/ urls.ts selectors.ts actions.ts package.json ``` -------------------------------- ### Scraping Hacker News with Monitoro Herd (Python) Source: https://herd.garden/llms.txt Demonstrates a complete workflow for scraping Hacker News using Monitoro Herd. It initializes the client, navigates to the site, extracts story details and metadata using a complex selector object, combines the data, and prints the results. Includes resource management by closing the page and client. ```python import asyncio from monitoro_herd import HerdClient async def scrape_hacker_news(): client = HerdClient("your-token") try: await client.initialize() devices = await client.list_devices() device = devices[0] page = await device.new_page() # Navigate to Hacker News print("Navigating to Hacker News...") await page.goto("https://news.ycombinator.com") # Extract stories and their metadata print("Extracting stories...") data = await page.extract({ # Extract the story elements "stories": { "_$r": ".athing", # Each story row "title": ".titleline > a", # Story title "site": ".sitestr", # Source website "link": { "_$": ".titleline > a", # Story link "attribute": "href" # Get the URL } }, # Extract the metadata (points, author, etc.) "metadata": { "_$r": ".subline", # Metadata rows "points": ".score", # Points count "author": ".hnuser", # Author username "time": ".age" # Submission time } }) # Combine stories with their metadata # (They're in separate lists but in the same order) combined_stories = list(zip(data["stories"], data["metadata"])) # Display the first 3 stories print(f"\nExtracted {len(combined_stories)} stories:") for i, (story, meta) in enumerate(combined_stories[:3]): print(f"\nStory {i+1}:") print(f"Title: {story['title']}") if "site" in story: print(f"Site: {story['site']}") print(f"Link: {story['link']}") if "points" in meta: print(f"Points: {meta['points']}") if "author" in meta: print(f"Author: {meta['author']}") if "time" in meta: print(f"Posted: {meta['time']}") finally: await page.close() await client.close() # Run the function asyncio.run(scrape_hacker_news()) ``` -------------------------------- ### Basic Data Extraction with Monitoro Herd (Python) Source: https://herd.garden/llms.txt Shows how to perform basic data extraction in Python using Monitoro Herd's `page.extract` method. It utilizes simple CSS selectors to retrieve text content from elements such as headings, paragraphs, and links, and then prints the resulting dictionary. ```python # Extract basic text content data = await page.extract({ "title": "h1", # Extracts the main heading "description": "p", # Extracts the first paragraph "link": "a" # Extracts the first link text }) print(data["title"]) # "Welcome to Our Website" print(data["description"]) # "This is our homepage." ``` -------------------------------- ### Extracting E-commerce Product Listings (JavaScript) Source: https://herd.garden/llms.txt This snippet demonstrates extracting a list of items, such as products from an e-commerce search results page. It uses `_$r` to select repeating elements and then defines nested selectors and pipes to extract specific data (title, price, rating, reviews) from each item. ```javascript const searchResults = await page.extract({ products: { _$r: '[data-component-type="s-search-result"]', title: { _$: 'h2 .a-link-normal', pipes: ['trim'] }, price: { _$: '.a-price .a-offscreen', pipes: ['parseNumber'] }, rating: { _$: '.a-icon-star-small .a-icon-alt', pipes: ['trim'] }, reviews: { _$: '.a-size-base.s-underline-text', pipes: ['trim'] } } }); ``` -------------------------------- ### Extracting E-commerce Product Listings (Python) Source: https://herd.garden/llms.txt This snippet demonstrates extracting a list of items in Python, such as products from an e-commerce search results page. It uses `_$r` to select repeating elements and then defines nested selectors and pipes to extract specific data (title, price, rating, reviews) from each item. ```python searchResults = await page.extract({ "products": { "_$r": '[data-component-type="s-search-result"]', "title": { "_$": "h2 .a-link-normal", "pipes": ["trim"] }, "price": { "_$": ".a-price .a-offscreen", "pipes": ["parseNumber"] }, "rating": { "_$": ".a-icon-star-small .a-icon-alt", "pipes": ["trim"] }, "reviews": { "_$": ".a-size-base.s-underline-text", "pipes": ["trim"] } } }) ``` -------------------------------- ### Initializing Herd JavaScript Client Source: https://herd.garden/llms.txt Initialize the Herd client with your API token and URL. This step is required before performing any automation tasks. Replace 'your-token' with your actual API token. ```javascript // Import the Herd client import { HerdClient } from '@monitoro/herd'; // Initialize the client with your API URL and token const client = new HerdClient('your-token'); // Always initialize the client before using it await client.initialize(); ``` -------------------------------- ### Extracting Basic Data (Python) Source: https://herd.garden/llms.txt Illustrates how to extract data from a web page using CSS selectors with the page.extract() method in Python. Shows how to access the extracted data. Requires a loaded page. ```python # Extract basic information data = await page.extract({ "title": "h1", # Main heading "description": "p", # First paragraph "link": "a" # First link text }) # Display the extracted data print("Extracted data:") print(f"Title: {data['title']}") print(f"Description: {data['description']}") print(f"Link: {data['link']}") ``` -------------------------------- ### Basic Data Extraction with Monitoro Herd (JavaScript) Source: https://herd.garden/llms.txt Illustrates the simplest form of data extraction using Monitoro Herd's `page.extract` method in JavaScript. It uses basic CSS selectors to pull text content from common elements like headings, paragraphs, and links, and then logs the extracted data. ```javascript // Extract basic text content const data = await page.extract({ title: 'h1', // Extracts the main heading description: 'p', // Extracts the first paragraph link: 'a' // Extracts the first link text }); console.log(data.title); // "Welcome to Our Website" console.log(data.description); // "This is our homepage." ``` -------------------------------- ### Connecting to a Herd Device Source: https://herd.garden/llms.txt Connect to an available browser device after initializing the client. This code retrieves the list of devices and connects to the first one. In production, you might select a device based on specific criteria. ```javascript // Get a list of available devices const devices = await client.listDevices(); // Connect to the first available device const device = devices[0]; console.log(`Connected to device: ${device.id}`); ``` -------------------------------- ### Testing Trail Action Locally with Herd CLI Source: https://herd.garden/llms.txt Demonstrates how to use the Herd CLI to test a specific trail action locally. It runs the test method defined in the action class. ```bash herd trail test --action my-trail ``` -------------------------------- ### Extracting Data with Pipes (Python) Source: https://herd.garden/llms.txt This snippet demonstrates basic data extraction using `page.extract()` in Python. It selects elements using CSS selectors (`_$:`) and applies transformations using `pipes`, such as `parseNumber` to convert currency strings to numbers and `trim` and `toLowerCase` for text manipulation. ```python data = await page.extract({ "price": { "_$": ".price", "pipes": ["parseNumber"] # Convert "$1,234.56" to 1234.56 }, "title": { "_$": "h1", "pipes": ["trim", "toLowerCase"] # Apply multiple transformations } }) ``` -------------------------------- ### Using the Root Selector (`:root`) - Python Source: https://herd.garden/llms.txt Explains and demonstrates the use of the `:root` selector in Python to target the current element being processed within a repeated extraction (`_$r`), allowing extraction of its text content or attributes. ```python data = await page.extract({ "items": { "_$r": ".item", "someElement": ":root", # Extract text of the .item element itself "classes": { "_$": ":root", "attribute": "class" # Extract class attribute of the same element } } }) ``` -------------------------------- ### Using the Root Selector (`:root`) - JavaScript Source: https://herd.garden/llms.txt Explains and demonstrates the use of the `:root` selector to target the current element being processed within a repeated extraction (`_$r`), allowing extraction of its text content or attributes. ```javascript const data = await page.extract({ items: { _$r: '.item', someElement: ':root', // Extract text of the .item element itself classes: { _$: ':root', attribute: 'class' // Extract class attribute of the same element } } }); ``` -------------------------------- ### Extracting Data with Pipes (JavaScript) Source: https://herd.garden/llms.txt This snippet demonstrates basic data extraction using `page.extract()`. It selects elements using CSS selectors (`_$:`) and applies transformations using `pipes`, such as `parseNumber` to convert currency strings to numbers and `trim` and `toLowerCase` for text manipulation. ```javascript const data = await page.extract({ price: { _$: '.price', pipes: ['parseNumber'] // Convert "$1,234.56" to 1234.56 }, title: { _$: 'h1', pipes: ['trim', 'toLowerCase'] // Apply multiple transformations } }); ``` -------------------------------- ### Closing Herd Resources (Python) Source: https://herd.garden/llms.txt Explains the importance of closing resources like pages and the client when automation is complete to free up resources. Provides the Python code for closing the page and the client. ```python # Close the page await page.close() # Close the client await client.close() ``` -------------------------------- ### Click Element and Wait for Navigation - JavaScript Source: https://herd.garden/llms.txt This snippet shows how to click an element and then wait for navigation to complete. The `waitForNavigation` option with `'networkidle2'` waits until network activity has been minimal for a specified duration after the click. ```javascript // Click and wait for navigation await page.click('input[type="submit"]', { waitForNavigation: 'networkidle2' }); console.log('Search button clicked and navigation completed'); ``` -------------------------------- ### Google Search Trail Directory Structure Source: https://herd.garden/llms.txt Defines the standard directory structure for a Herd trail, including configuration files and action implementations. ```text google-search/ urls.ts selectors.ts actions.ts package.json ``` -------------------------------- ### Testing Trail Selector Locally with Herd CLI Source: https://herd.garden/llms.txt Illustrates how to test a specific selector defined in the trail using the Herd CLI, independent of the full action run. ```bash herd trail test --selector my-selector ``` -------------------------------- ### Watching and Retesting Trail Action with Herd CLI Source: https://herd.garden/llms.txt Shows how to use the Herd CLI to automatically re-run trail action tests when changes are detected in the trail files. ```bash herd trail test --action my-trail --watch # or herd trail test -a my-trail -w ``` -------------------------------- ### Automating Google Search with Herd SDK (JavaScript) Source: https://herd.garden/llms.txt Demonstrates a complete browser automation flow using the Herd SDK in JavaScript to navigate to Google, perform a search, wait for results, and extract search result titles. Requires a Herd API token. ```javascript async function searchExample() { const client = new HerdClient('your-token'); try { await client.initialize(); const devices = await client.listDevices(); const device = devices[0]; const page = await device.newPage(); // Navigate to a search engine console.log('Navigating to Google...'); await page.goto('https://www.google.com'); // Type in the search box console.log('Entering search query...'); await page.type('input[name="q"]', 'Monitoro Herd automation'); // Submit the search form and wait for results console.log('Submitting search...'); await page.click('input[type="submit"]', { waitForNavigation: 'networkidle2' }); // Wait for results to load completely console.log('Waiting for search results...'); await page.waitForSelector('#search'); // Extract search result titles console.log('Extracting search results...'); const searchResults = await page.extract({ titles: { _$r: '#search .g h3', // _$r extracts multiple elements text: ':root' // For each match, get its text } }); // Display the search result titles console.log('\nSearch Results:'); searchResults.titles.forEach((result, index) => { console.log(`${index + 1}. ${result.text}`); }); } catch (error) { console.error('Error:', error); } finally { await client.close(); } } ``` -------------------------------- ### Watching and Retesting Trail Selector with Herd CLI Source: https://herd.garden/llms.txt Explains how to use the Herd CLI to automatically re-run selector tests when changes are detected in the trail files. ```bash herd trail test --selector my-selector --watch # or herd trail test -s my-selector -w ``` -------------------------------- ### Extracting Lists with _$r Selector (Python) Source: https://herd.garden/llms.txt Shows how to use the _$r selector within the page.extract method to find multiple elements matching a selector and extract nested properties for each element. Useful for lists like search results. ```python # Extract a list of items data = await page.extract({ "items": { "_$r": ".item", # Find all elements with class "item" "name": ".item-name", # For each item, get the name "price": ".price" # For each item, get the price } }) # Access the extracted items for item in data["items"]: print(f"Name: {item['name']}, Price: {item['price']}") ``` -------------------------------- ### Extracting Complex Data (Object Syntax) - Python Source: https://herd.garden/llms.txt Demonstrates using the expanded object syntax for complex data extraction in Python, allowing selection by CSS selector, extraction of specific attributes, and application of transformation pipes. ```python data = await page.extract({ "title": { "_$": "h1", # CSS selector "attribute": "id" # Extract the ID attribute instead of text }, "price": { "_$": ".price", # Target price element "pipes": ["parseNumber"] # Apply transformation } }) ``` -------------------------------- ### Handling Dynamic Content Before Extraction (Python) Source: https://herd.garden/llms.txt When content is loaded dynamically after the initial page load, it's necessary to wait for the specific element to appear before attempting extraction. This snippet uses `page.waitForElement()` in Python to ensure the dynamic content is present before calling `page.extract()`. ```python # Wait for dynamic content to load await page.waitForElement('#dynamic span') # Then extract the content data = await page.extract({ "content": "#dynamic span" }) ``` -------------------------------- ### Extracting Complex Data (Object Syntax) - JavaScript Source: https://herd.garden/llms.txt Demonstrates using the expanded object syntax for complex data extraction, allowing selection by CSS selector, extraction of specific attributes, and application of transformation pipes. ```javascript const data = await page.extract({ title: { _$: 'h1', // CSS selector attribute: 'id' // Extract the ID attribute instead of text }, price: { _$: '.price', // Target price element pipes: ['parseNumber'] // Apply transformation } }); ``` -------------------------------- ### Extracting Attributes with attribute Key (Python) Source: https://herd.garden/llms.txt Illustrates how to extract specific HTML attributes (like href) from elements using the attribute key in the extraction configuration, often combined with the _$r selector for lists. ```python # Extract links and their href attributes data = await page.extract({ "links": { "_$r": "a", # Find all links "text": ":root", # Get the link text "url": { "_$": ":root", # Reference the same element "attribute": "href" # Get its href attribute } } }) # Display the links for link in data["links"]: print(f"Link: {link['text']} -> {link['url']}") ``` -------------------------------- ### Extracting Element Properties - JavaScript Source: https://herd.garden/llms.txt Shows how to use the `property` key within an extraction definition to retrieve JavaScript properties (like `getBoundingClientRect` or `innerHTML`) directly from a selected element. ```javascript const data = await page.extract({ dimensions: { _$: '.box', property: 'getBoundingClientRect' // Get element dimensions }, html: { _$: '.content', property: 'innerHTML' // Get inner HTML } }); ``` -------------------------------- ### Extracting Element Properties - Python Source: https://herd.garden/llms.txt Shows how to use the `property` key within an extraction definition in Python to retrieve JavaScript properties (like `getBoundingClientRect` or `innerHTML`) directly from a selected element. ```python data = await page.extract({ "dimensions": { "_$": ".box", "property": "getBoundingClientRect" # Get element dimensions }, "html": { "_$": ".content", "property": "innerHTML" # Get inner HTML } }) ``` -------------------------------- ### Handling Dynamic Content Before Extraction (JavaScript) Source: https://herd.garden/llms.txt When content is loaded dynamically after the initial page load, it's necessary to wait for the specific element to appear before attempting extraction. This snippet uses `page.waitForElement()` to ensure the dynamic content is present before calling `page.extract()`. ```javascript // Wait for dynamic content to load await page.waitForElement('#dynamic span'); // Then extract the content const data = await page.extract({ content: '#dynamic span' }); ``` -------------------------------- ### Find Element by CSS Selector - JavaScript Source: https://herd.garden/llms.txt This snippet demonstrates how to find the first element on a page that matches a given CSS selector using the `page.$` method. It checks if the element was found and logs the result. ```javascript // Find an element using a CSS selector const searchBox = await page.$('input[name="q"]'); // Check if the element was found if (searchBox) { console.log('Search box found'); } else { console.log('Search box not found'); } ``` -------------------------------- ### Type Text into Input Field - JavaScript Source: https://herd.garden/llms.txt This snippet shows how to simulate typing text into an input field. The `page.type` method finds the element using the provided CSS selector and enters the specified text. ```javascript // Type text into an input field await page.type('input[name="q"]', 'Monitoro Herd automation'); console.log('Text entered into search box'); ``` -------------------------------- ### Click Element by CSS Selector - JavaScript Source: https://herd.garden/llms.txt This snippet demonstrates how to click an element, such as a button or link, using its CSS selector with the `page.click` method. ```javascript // Click a button await page.click('input[type="submit"]'); console.log('Search button clicked'); ``` -------------------------------- ### Extracting Information with CSS Selectors Source: https://herd.garden/llms.txt Extract data from the current page using CSS selectors. The `extract` method finds elements matching the selectors and returns their text content. ```javascript // Extract content using CSS selectors const content = await page.extract({ title: 'h1', // Extracts the main heading description: 'p', // Extracts the first paragraph link: 'a' // Extracts the first link text }); // Display the extracted content console.log('Extracted content:'); console.log(`Title: ${content.title}`); console.log(`Description: ${content.description}`); console.log(`Link: ${content.link}`); ``` -------------------------------- ### Wait for Element to Appear - JavaScript Source: https://herd.garden/llms.txt This snippet demonstrates how to pause execution until an element matching a specific CSS selector appears on the page using `page.waitForSelector`. This is useful for handling dynamically loaded content. ```javascript // Wait for an element to appear await page.waitForSelector('#search'); console.log('Search results have loaded'); ``` -------------------------------- ### Performing Nested Extraction - Python Source: https://herd.garden/llms.txt Illustrates how to define nested selectors within the extraction object in Python to capture hierarchical data structures from the page, allowing extraction of data within specific parent elements. ```python data = await page.extract({ "product": { "name": ".product-name", "details": { "_$": ".product-details", "specs": { "_$r": ".spec-item", "label": ".spec-label", "value": ".spec-value" } } } }) ``` -------------------------------- ### Performing Nested Extraction - JavaScript Source: https://herd.garden/llms.txt Illustrates how to define nested selectors within the extraction object to capture hierarchical data structures from the page, allowing extraction of data within specific parent elements. ```javascript const data = await page.extract({ product: { name: '.product-name', details: { _$: '.product-details', specs: { _$r: '.spec-item', label: '.spec-label', value: '.spec-value' } } } }); ``` -------------------------------- ### Closing Herd Resources Source: https://herd.garden/llms.txt Properly close the page and the client connection when automation tasks are complete. This releases resources and prevents memory leaks. ```javascript // Close the page when done await page.close(); // Close the client connection await client.close(); ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.