### Working with Nested iframes Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/03-iframe-classes.md Example showing how to navigate and interact with nested iframes, starting from the main page and drilling down. ```APIDOC ### Working with Nested iframes ```python # Main page driver.get("https://example.com/page-with-iframes") # Get first level iframe outer_iframe = driver.select_iframe("#outer-iframe") # Get nested iframe within first iframe inner_iframe = outer_iframe.get_iframe_by_link("example.com/nested") # Interact with content in nested iframe element = inner_iframe.select(".content").text print(element) ``` ``` -------------------------------- ### Install Botasaurus Driver Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/README.md Install the Botasaurus Driver library using pip. ```bash pip install botasaurus-driver ``` -------------------------------- ### Response Example Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/05-types-and-constants.md An example demonstrating how to use the Response object to collect and process HTTP responses. ```APIDOC ### Example ```python def on_response(request_id, response, event): # Collect response ID for later retrieval driver.responses.append(request_id) driver.after_response_received(on_response) driver.get("https://api.example.com/data") # Collect all responses responses = driver.responses.collect() for resp in responses: print(f"Request ID: {resp.request_id}") print(f"Content type: {'Base64' if resp.is_base_64 else 'Text'}") if resp.content: if resp.request_id in json_requests: data = resp.get_json_content() print(f"Data: {data}") else: text = resp.get_decoded_content() print(f"Body length: {len(text)}") ``` ``` -------------------------------- ### Collection Example Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/07-network-monitoring.md An example demonstrating how to use the `responses` collector to capture and process JSON API responses after a request is made. ```APIDOC ## Collection Example ```python def on_response(request_id, response, event): # Collect JSON API responses if "application/json" in response.headers.get("content-type", ""): driver.responses.append(request_id) driver.after_response_received(on_response) driver.get("https://api.example.com/users") # Retrieve collected responses responses = driver.responses.collect() for resp in responses: data = resp.get_json_content() print(f"Response: {data}") print(f"Request ID: {resp.request_id}") ``` ``` -------------------------------- ### Accessing YouTube Video in iframe Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/03-iframe-classes.md Example demonstrating how to get an iframe by its link and access content within it, specifically a YouTube video. ```APIDOC ## Usage Examples ### Accessing YouTube Video in iframe ```python from botasaurus_driver import Driver driver = Driver() driver.get("https://www.freecodecamp.org/news/using-entity-framework-core-with-mongodb/") # Get iframe by matching YouTube URL iframe = driver.get_iframe_by_link("www.youtube.com/embed") # Now use iframe like a page subscribers = iframe.select(".ytp-title-expanded-subtitle").text print(f"Channel subscribers: {subscribers}") driver.close() ``` ``` -------------------------------- ### Open and Get First Tab Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/11-advanced-features.md Initializes a driver and navigates to the first URL, capturing the handle for the first tab. ```python driver = Driver() # First page driver.get("https://example.com/page1") tab1 = driver._tab ``` -------------------------------- ### Retrieve Element Properties with Botasaurus Driver Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/README.md Examples of getting text content, finding elements by text, and retrieving attribute values like 'src' from images. ```python header_text = driver.get_text("h1") # Get text content ``` ```python error_message = driver.get_element_containing_text("Error: Invalid input") ``` ```python image_url = driver.select("img.logo").get_attribute("src") # Get attribute value ``` -------------------------------- ### Profile Management Example Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/05-types-and-constants.md Demonstrates how to initialize the Driver with a persistent profile and access its properties. The profile data is automatically synced when the driver is closed. ```python driver = Driver(profile="my_profile") driver.get("https://example.com") # Get profile object profile = driver.profile # Profile provides access to stored state (cookies, extensions, etc) # Automatically synced when driver.close() is called driver.close() ``` -------------------------------- ### Cloudflare Turnstile Shadow DOM Example Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/11-advanced-features.md A real-world example demonstrating how to interact with the Cloudflare Turnstile widget, which uses Shadow DOM. This involves finding an iframe, accessing its shadow root, and clicking the checkbox. ```python driver.get("https://nopecha.com/demo/cloudflare") # Find the iframe element iframe_elem = driver.select('[name="cf-turnstile-response"]').parent # Get its shadow root (the actual turnstile UI) turnstile_shadow = iframe_elem.get_shadow_root() # Find the checkbox inside checkbox = turnstile_shadow.select("input[type='checkbox']") checkbox.click() ``` -------------------------------- ### Initialize Botasaurus Driver Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/09-quick-reference.md Demonstrates various ways to initialize the Botasaurus Driver, including basic setup, headless mode with a profile, using a proxy, and configuring multiple options. ```python from botasaurus_driver import Driver, Wait # Basic driver = Driver() ``` ```python # Headless with profile driver = Driver(headless=True, profile="my_profile") ``` ```python # With proxy driver = Driver(proxy="socks5://proxy.com:1080") ``` ```python # All options driver = Driver( headless=False, profile="name", user_agent="Mozilla/5.0...", window_size="1920x1080", block_images=True, wait_for_complete_page_load=True ) ``` -------------------------------- ### Use SOCKS5 Proxy with Authentication Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/10-usage-examples.md This example shows how to configure and use a SOCKS5 proxy with username and password authentication. It verifies the proxy by fetching the IP address. ```python # Using SOCKS5 proxy with auth driver2 = Driver( headless=True, proxy="socks5://user:pass@proxy.example.com:1080" ) driver2.get("https://httpbin.org/ip") print(f"Proxy IP: {driver2.page_text}") driver2.close() ``` -------------------------------- ### Anti-Detection Setup Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/06-configuration-options.md Configure the driver for anti-detection by setting `headless` to `False`, using a specific `profile`, and employing hashed `user_agent` and `window_size`. Ensures the page load is complete before proceeding. ```python from botasaurus_driver import Driver, UserAgent, WindowSize driver = Driver( headless=False, profile="antidos_profile", user_agent=UserAgent.HASHED, window_size=WindowSize.HASHED, wait_for_complete_page_load=True ) ``` -------------------------------- ### CDP Page Navigation Example Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/08-cdp-and-javascript.md Demonstrates navigating to a URL using the CDP `page.navigate` command instead of the standard `driver.get()` method. This approach returns detailed information about the navigation event. ```python from botasaurus_driver import Driver, cdp driver = Driver() # Navigate via CDP instead of driver.get() result = driver.run_cdp_command( cdp.page.navigate(url='https://example.com') ) print(f"Frame ID: {result.frame_id}") print(f"Loader ID: {result.loader_id}") ``` -------------------------------- ### Interact with Elements using Botasaurus Driver Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/README.md Code examples for typing into input fields, clicking elements, and selecting options from dropdowns. ```python driver.type("input[name='username']", "john_doe") # Type into an input field ``` ```python driver.click("button.submit") # Click an element ``` ```python element = driver.select("button.submit") element.click() # Click on an element ``` ```python element.select_option("select#fruits", index=2) # Select an option ``` -------------------------------- ### Practical Element Traversal Example in Python Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/11-advanced-features.md Demonstrates finding a button within a specific card by first selecting all cards, then searching within each card for price and button elements, and accessing parent containers. ```python # Find a button within a specific card cards = driver.select_all(".card") for card in cards: # Get price from this card price = card.select(".price").text # Get parent container (2 levels up) product = card.get_parent_which_is("div") # Find button within this card button = card.select("button.add-to-cart") if button: print(f"Product {price} has add button") ``` -------------------------------- ### Example: Handling HTTP Responses Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/05-types-and-constants.md Demonstrates collecting and processing HTTP responses, including retrieving request IDs, checking content encoding, and printing decoded text or parsed JSON data. ```python def on_response(request_id, response, event): # Collect response ID for later retrieval driver.responses.append(request_id) driver.after_response_received(on_response) driver.get("https://api.example.com/data") # Collect all responses responses = driver.responses.collect() for resp in responses: print(f"Request ID: {resp.request_id}") print(f"Content type: {'Base64' if resp.is_base_64 else 'Text'}") if resp.content: if resp.request_id in json_requests: data = resp.get_json_content() print(f"Data: {data}") else: text = resp.get_decoded_content() print(f"Body length: {len(text)}") ``` -------------------------------- ### Enable Human Mode Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/01-driver-class.md Activates human-like mouse movements and behavior. Ensure the `botasaurus-humancursor` package is installed. ```python def enable_human_mode(self) -> None ``` -------------------------------- ### Get User Agent String Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/01-driver-class.md Retrieves the User-Agent string of the current browser. ```python @property def user_agent(self) -> str ``` -------------------------------- ### Miscellaneous Operations Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/README.md Provides examples of various utility functions for interacting with web elements and the browser. ```APIDOC ## Miscellaneous Operations ### Description This section covers common utility operations like typing into form fields, checking element presence, accessing page HTML, scrolling elements into view, and closing the browser. ### Method ```python form.type("input[name='password']", "secret_password") # Type into a form field container.is_element_present(".button") # Check element presence page_html = driver.page_html # Current page HTML driver.select(".footer").scroll_into_view() # Scroll element into view driver.close() # Close the browser ``` ``` -------------------------------- ### Work with iframes in Python Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/README.md This example demonstrates how to select an iframe using its link or CSS selector and then extract text from an element within that iframe. Ensure the driver is initialized. ```python driver.get("https://www.freecodecamp.org/news/using-entity-framework-core-with-mongodb/") iframe = driver.get_iframe_by_link("www.youtube.com/embed") # OR the following works as well # iframe = driver.select_iframe(".embed-wrapper iframe") freecodecamp_youtube_subscribers_count = iframe.select(".ytp-title-expanded-subtitle").text print(freecodecamp_youtube_subscribers_count) ``` -------------------------------- ### Execute JavaScript with Botasaurus Driver Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/README.md Examples of running JavaScript code, including executing external JS files, returning values, passing arguments, and executing JS on specific elements. ```python result = driver.run_js("script.js") # Run a JavaScript file located in the current working directory. ``` ```python result = driver.run_js("return document.title") ``` ```python pikachu = driver.run_js("return args.pokemon", {"pokemon": 'pikachu'}) # args can be a dictionary, list, string, etc. ``` ```python text_content = driver.select("body").run_js("(el) => el.textContent") ``` -------------------------------- ### Anti-Bot Detection Setup Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/10-usage-examples.md Configures the driver for enhanced anti-bot detection by using a persistent profile, a consistent user agent, and a consistent window size. It includes logic to detect and bypass Cloudflare if necessary. ```python from botasaurus_driver import Driver, UserAgent, WindowSize driver = Driver( headless=False, # Visible window profile="bot_detection_profile", # Persistent profile user_agent=UserAgent.HASHED, # Consistent UA window_size=WindowSize.HASHED, # Consistent size wait_for_complete_page_load=True ) driver.get("https://heavily-protected-site.com") # Try to bypass if needed if driver.is_bot_detected(): driver.detect_and_bypass_cloudflare() driver.close() ``` -------------------------------- ### Accessing YouTube Video in iframe Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/03-iframe-classes.md Demonstrates how to get an iframe by its URL and interact with its content. Ensure the Driver is initialized and the page is loaded before calling get_iframe_by_link. ```python from botasaurus_driver import Driver driver = Driver() driver.get("https://www.freecodecamp.org/news/using-entity-framework-core-with-mongodb/") # Get iframe by matching YouTube URL iframe = driver.get_iframe_by_link("www.youtube.com/embed") # Now use iframe like a page subscribers = iframe.select(".ytp-title-expanded-subtitle").text print(f"Channel subscribers: {subscribers}") driver.close() ``` -------------------------------- ### Get Complete Page HTML Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/01-driver-class.md Retrieves the entire HTML source code of the current page. ```python @property def page_html(self) -> str ``` -------------------------------- ### Proxy with Authentication Configuration Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/06-configuration-options.md Configure the driver to use a proxy with authentication. This setup specifies the proxy type, username, password, and host, along with a profile and waiting for complete page load. ```python driver = Driver( proxy="socks5://username:password@proxy.service.com:1080", profile="proxy_profile", wait_for_complete_page_load=True ) ``` -------------------------------- ### get Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/01-driver-class.md Navigates the driver to a specified URL. Supports bypassing Cloudflare, running custom JavaScript before page load, and setting wait times and timeouts. ```APIDOC ## get ### Description Navigate to a URL. ### Method Signature ```python def get( self, link: str, bypass_cloudflare: bool = False, js_to_run_before_new_document: Optional[str] = None, wait: Optional[int] = None, timeout: int = 60 ) -> Tab ``` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters Table | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | link | str | — | Target URL to navigate to | | bypass_cloudflare | bool | False | Automatically bypass Cloudflare challenges | | js_to_run_before_new_document | str | None | JavaScript code or file to run before page loads | | wait | int | None | Additional wait time in seconds after page loads | | timeout | int | 60 | Maximum wait time for page load in seconds | ### Returns `Tab` instance ### Example ```python driver.get("https://example.com") driver.get("https://example.com", bypass_cloudflare=True) driver.get("https://example.com", wait=3, timeout=30) ``` ``` -------------------------------- ### Initialize Botasaurus Driver with All Options Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/06-configuration-options.md Demonstrates the initialization of the Driver with all available optional parameters. All parameters are optional and can be used to customize browser behavior. ```python from botasaurus_driver import Driver driver = Driver( headless=False, enable_xvfb_virtual_display=False, proxy=None, profile=None, tiny_profile=False, block_images=False, block_images_and_css=False, wait_for_complete_page_load=True, extensions=[], arguments=[], remove_default_browser_check_argument=False, user_agent=None, window_size=None, lang=None, beep=False, host=None, port=None, browser_executable_path=None ) ``` -------------------------------- ### Get All Ancestor Elements Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/02-element-class.md Retrieve a list of all ancestor elements, starting from the parent and going up to the root of the DOM. This is useful for traversing up the DOM tree. ```python @property def all_parents(self) -> List[Element] ``` -------------------------------- ### Manage Cookies Across Sessions Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/10-usage-examples.md This example shows how to save cookies from one session and apply them to another. This is useful for maintaining login states or user preferences across different browser instances. ```python driver1 = Driver() driver1.get("https://example.com") # Simulate login driver1.type("input#user", "user@example.com") driver1.type("input#pass", "password") driver1.click("button.login") driver1.sleep(2) # Save cookies cookies = driver1.get_cookies() print(f"Saved {len(cookies)} cookies") driver1.close() # Use cookies in new session driver2 = Driver() driver2.get("https://example.com") driver2.add_cookies(cookies) driver2.reload() # Page might show logged-in state print(f"Loaded cookies, now at: {driver2.current_url}") driver2.close() ``` -------------------------------- ### Navigate to URL with Botasaurus Driver Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/01-driver-class.md Use the `get` method to navigate to a specified URL. Optional parameters allow bypassing Cloudflare, running JavaScript before the document loads, and setting wait times or timeouts. ```python def get( self, link: str, bypass_cloudflare: bool = False, js_to_run_before_new_document: Optional[str] = None, wait: Optional[int] = None, timeout: int = 60 ) -> Tab ``` ```python driver.get("https://example.com") driver.get("https://example.com", bypass_cloudflare=True) driver.get("https://example.com", wait=3, timeout=30) ``` -------------------------------- ### Monitor Network Responses with Botasaurus Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/00-index.md Set up a callback function to monitor network responses. This example registers a function that is executed after each response is received, allowing you to filter and process specific responses, such as those from 'api.example.com'. The collected response IDs can then be retrieved. ```python def on_response(request_id, response, event): if "api.example.com" in response.url: driver.responses.append(request_id) driver.after_response_received(on_response) driver.get("https://example.com") responses = driver.responses.collect() ``` -------------------------------- ### Server Environment Configuration Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/06-configuration-options.md Set up the driver for a server environment by enabling `headless` mode and `enable_xvfb_virtual_display`. It also blocks images and disables waiting for complete page load, while configuring a proxy. ```python driver = Driver( headless=True, enable_xvfb_virtual_display=True, block_images=True, wait_for_complete_page_load=False, proxy="http://proxy.company.com:8080" ) ``` -------------------------------- ### Interact with iFrame Content Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/10-usage-examples.md Access and interact with elements inside an iframe by first selecting the iframe using its CSS selector. This example demonstrates getting text from a header and counting links within the iframe. ```python driver = Driver() driver.get("https://example.com/embedded") # Access iframe by selector iframe = driver.select_iframe("iframe#content") # Interact with iframe content title = iframe.select("h1").text links = iframe.select_all("a") print(f"Iframe title: {title}") print(f"Links in iframe: {len(links)}") driver.close() ``` -------------------------------- ### Login Form Automation with Botasaurus Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/00-index.md Automate the process of filling out and submitting a login form. This example navigates to a login page, types credentials into email and password fields, clicks the submit button, and waits for the dashboard page to load. ```python driver.get("https://example.com/login") driver.type("input#email", "user@example.com") driver.type("input#password", "password") driver.click("button.submit") driver.wait_for_page_to_be("dashboard") ``` -------------------------------- ### Enable Human-Like Cursor Control Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/11-advanced-features.md Activates human-like mouse movements that mimic realistic user behavior. Requires the `botasaurus-humancursor` package to be installed. ```APIDOC ## enable_human_mode() ### Description Enables human-like mouse movements for realistic user interactions. ### Method ```python driver.enable_human_mode() ``` ### Requirements - `botasaurus-humancursor` package must be installed. ``` -------------------------------- ### Basic Web Scraping with Botasaurus Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/10-usage-examples.md Demonstrates how to initialize the driver, navigate to a URL, extract text from an element, and find all links matching a CSS selector. Ensure to close the driver when done. ```python from botasaurus_driver import Driver, Wait driver = Driver() driver.get("https://example.com") # Find elements title = driver.get_text("h1") links = driver.get_all_links("a.article") for link in links: print(f"Article: {link}") driver.close() ``` -------------------------------- ### Cookie Manipulation Example Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/05-types-and-constants.md Shows how to retrieve all cookies, filter them based on specific criteria, save selected cookie data, and restore cookies in a new browser session. ```python # Get all cookies cookies = driver.get_cookies() # Filter cookies session_cookies = [c for c in cookies if "session" in c["name"]] # Save for later saved_cookies = [ {"name": c["name"], "value": c["value"]} for c in cookies ] # Restore in new session new_driver = Driver() new_driver.add_cookies(saved_cookies) ``` -------------------------------- ### Enable Human-Like Cursor Control Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/11-advanced-features.md Activates human-like mouse movements. Requires the 'botasaurus-humancursor' package to be installed. All subsequent mouse movements will be humanized. ```python from botasaurus_driver import Driver driver = Driver() driver.enable_human_mode() # Now all mouse movements are humanized driver.click(".button") driver.move_mouse_to_element(".link") ``` -------------------------------- ### Quick Testing Configuration Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/06-configuration-options.md A simplified configuration for quick testing, disabling `headless` mode, using a test profile, blocking images, and disabling waiting for complete page load. ```python driver = Driver( headless=False, profile="test_profile", block_images=True, wait_for_complete_page_load=False ) ``` -------------------------------- ### __repr__ Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/02-element-class.md Get string representation of element. This method provides a convenient way to get a string representation of the element, useful for debugging or logging. ```APIDOC ## __repr__ ### Description Get string representation of element. ### Method Signature ```python def __repr__(self) -> str ``` ``` -------------------------------- ### Get Anchor Element's Href Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/02-element-class.md Retrieve the 'href' attribute specifically from an anchor () element. This is a convenient shortcut for getting link destinations. ```python @property def href(self) -> str ``` -------------------------------- ### Get Image/Script Element's Src Source: https://github.com/omkarcloud/botasaurus-driver/blob/master/_autodocs/02-element-class.md Retrieve the 'src' attribute from image () or script (