### Perform a Google Search with yagooglesearch Source: https://github.com/opsdisk/yagooglesearch/blob/master/README.md This example demonstrates how to initialize the SearchClient, configure search parameters, and retrieve search results. Ensure you have the library installed and imported. ```python import yagooglesearch query = "site:github.com" client = yagooglesearch.SearchClient( query, tbs="li:1", max_search_result_urls_to_return=100, http_429_cool_off_time_in_minutes=45, http_429_cool_off_factor=1.5, # proxy="socks5h://127.0.0.1:9050", verbosity=5, verbose_output=True, # False (only URLs) or True (rank, title, description, and URL) ) client.assign_random_user_agent() urls = client.search() len(urls) for url in urls: print(url) ``` -------------------------------- ### Install yagooglesearch from source Source: https://github.com/opsdisk/yagooglesearch/blob/master/README.md Follow these steps to clone the repository, set up a virtual environment, and install the library from its source code. ```bash git clone https://github.com/opsdisk/yagooglesearch cd yagooglesearch virtualenv -p python3 .venv # If using a virtual environment. source .venv/bin/activate # If using a virtual environment. pip install . # Reads from pyproject.toml ``` -------------------------------- ### Install yagooglesearch with pip Source: https://github.com/opsdisk/yagooglesearch/blob/master/README.md Use this command to install the library using pip. ```bash pip install yagooglesearch ``` -------------------------------- ### SearchClient.__init__() Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Initializes a configured Google search client. It handles query encoding, result language validation, proxy setup, SSL verification, and builds search URL templates. A random user agent is assigned by default. ```APIDOC ## SearchClient.__init__() — Initialize a Search Client Creates a configured Google search client. The constructor URL-encodes the query, validates the result language, sets up proxy dictionaries, configures SSL verification, and calls `update_urls()` to build all search URL templates. A random user agent is automatically assigned unless one is explicitly provided. ```python import yagooglesearch # Basic search returning up to 50 URLs client = yagooglesearch.SearchClient( query="site:github.com python security tools", tld="com", # Google TLD: com, co.uk, de, etc. lang_html_ui="en", # UI language lang_result="lang_en", # Filter results to English tbs="0", # No time/verbatim filter safe="off", # Safe search off num=100, # Results per page (max 100, capped by Google) max_search_result_urls_to_return=50, minimum_delay_between_paged_results_in_seconds=7, # Random delay: 7–18s http_429_cool_off_time_in_minutes=45, http_429_cool_off_factor=1.5, # Each 429 multiplies wait by 1.5x verbosity=5, # 1=critical .. 5=debug verbose_output=False, # True = dicts; False = plain URL strings ) urls = client.search() print(f"Found {len(urls)} URLs") for url in urls: print(url) # Expected output (plain URL mode): # Found 50 URLs # https://github.com/PyCQA/bandit # https://github.com/sqlmapproject/sqlmap # ... ``` ``` -------------------------------- ### Fetch Raw URL HTML with SearchClient.get_page() Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Use `get_page()` to perform an HTTP GET request for a given URL using the client's configured session, including headers, cookies, proxy, and SSL settings. It automatically handles the EU GDPR CONSENT cookie bypass. ```python import yagooglesearch client = yagooglesearch.SearchClient( query="placeholder", verbosity=5, proxy="socks5h://127.0.0.1:9050", ) # Fetch an arbitrary page using the client's configured session html = client.get_page("https://www.google.com/") if html == "HTTP_429_DETECTED": print("Blocked by Google") elif html: print(f"Retrieved {len(html)} bytes") print(html[:200]) else: print("Empty or error response") ``` -------------------------------- ### SearchClient.get_page() Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Performs an HTTP GET request to a specified URL using the client's configured session, including headers, cookies, proxy, and SSL settings. It automatically handles the EU GDPR CONSENT cookie bypass. The method returns the HTML response body as a string, or specific indicators for rate limiting (`"HTTP_429_DETECTED"`) or other non-200 responses. ```APIDOC ## `SearchClient.get_page()` — Fetch a Raw URL Performs an HTTP GET request with the configured headers, cookies, proxy, and SSL settings. Handles the EU GDPR CONSENT cookie bypass automatically. Returns the HTML response body as a string, `"HTTP_429_DETECTED"` on a 429 response (when self-managed), or `""` on other non-200 responses. ```python import yagooglesearch client = yagooglesearch.SearchClient( query="placeholder", verbosity=5, proxy="socks5h://127.0.0.1:9050", ) # Fetch an arbitrary page using the client's configured session html = client.get_page("https://www.google.com/") if html == "HTTP_429_DETECTED": print("Blocked by Google") elif html: print(f"Retrieved {len(html)} bytes") print(html[:200]) else: print("Empty or error response") ``` ``` -------------------------------- ### SearchClient.update_urls() Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Rebuilds internal URL templates based on current instance attributes. This method should be called after modifying attributes like query, tbs, lang_result, country, or start between searches to ensure URL consistency. ```APIDOC ## `SearchClient.update_urls()` — Rebuild Search URL Templates Rebuilds all four internal URL templates (`url_home`, `url_search`, `url_search_num`, `url_next_page`, `url_next_page_num`) based on the current instance attributes. Must be called after modifying attributes like `query`, `tbs`, `lang_result`, `country`, or `start` between searches. ```python import yagooglesearch client = yagooglesearch.SearchClient( query="cybersecurity news", tbs="qdr:d", # Past 24 hours max_search_result_urls_to_return=20, ) urls_today = client.search() print(f"Today: {len(urls_today)} results") # Switch to past-week filter without creating a new client client.tbs = "qdr:w" client.start = 0 # Reset pagination client.update_urls() urls_week = client.search() print(f"Past week: {len(urls_week)} results") ``` ``` -------------------------------- ### Update Search URL Templates with SearchClient.update_urls() Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Call `update_urls()` after modifying client attributes like `query`, `tbs`, `lang_result`, `country`, or `start` to rebuild internal URL templates. This ensures subsequent searches use the updated parameters. ```python import yagooglesearch client = yagooglesearch.SearchClient( query="cybersecurity news", tbs="qdr:d", # Past 24 hours max_search_result_urls_to_return=20, ) urls_today = client.search() print(f"Today: {len(urls_today)} results") # Switch to past-week filter without creating a new client client.tbs = "qdr:w" client.start = 0 # Reset pagination client.update_urls() urls_week = client.search() print(f"Past week: {len(urls_week)} results") ``` -------------------------------- ### Initialize SearchClient and Perform Basic Search Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Demonstrates basic initialization of `SearchClient` for a Google search, returning up to 50 plain URL strings. Configure parameters like TLD, language, time filters, and result limits. ```python import yagooglesearch # Basic search returning up to 50 URLs client = yagooglesearch.SearchClient( query="site:github.com python security tools", tld="com", # Google TLD: com, co.uk, de, etc. lang_html_ui="en", # UI language lang_result="lang_en", # Filter results to English tbs="0", # No time/verbatim filter safe="off", # Safe search off num=100, # Results per page (max 100, capped by Google) max_search_result_urls_to_return=50, minimum_delay_between_paged_results_in_seconds=7, # Random delay: 7–18s http_429_cool_off_time_in_minutes=45, http_429_cool_off_factor=1.5, # Each 429 multiplies wait by 1.5x verbosity=5, # 1=critical .. 5=debug verbose_output=False, # True = dicts; False = plain URL strings ) urls = client.search() print(f"Found {len(urls)} URLs") for url in urls: print(url) ``` -------------------------------- ### Configure SOCKS5 Proxy for SearchClient in Python Source: https://github.com/opsdisk/yagooglesearch/blob/master/README.md Initialize SearchClient with a SOCKS5h proxy for enhanced anonymity. This ensures DNS resolution occurs on the proxy server, making traffic appear to originate from the proxy's IP. ```python client = yagooglesearch.SearchClient( "site:github.com", proxy="socks5h://127.0.0.1:9050", ) ``` -------------------------------- ### Execute Search with Verbose Output Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Shows how to perform a search and retrieve detailed results including rank, title, URL, and description for each item. Set `verbose_output=True` for this format. ```python import yagooglesearch # Verbose output: returns rank, title, description, and URL for each result client = yagooglesearch.SearchClient( query="open source intrusion detection system", max_search_result_urls_to_return=10, verbose_output=True, verbosity=2, # Minimal console noise ) results = client.search() for item in results: print(f"Rank #{item['rank']}: {item['title']}") print(f" URL: {item['url']}") print(f" Desc: {item['description'][:80]}") ``` -------------------------------- ### Manually Rotate User Agent Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Demonstrates how to manually assign a new random user agent to the `SearchClient` instance between searches. This is useful for rotating user agents during a loop of multiple queries. ```python import yagooglesearch client = yagooglesearch.SearchClient( query="machine learning datasets", max_search_result_urls_to_return=20, ) # Manually rotate user agent before each search in a loop queries = ["python web scraping", "rust systems programming", "go concurrency patterns"] for query in queries: client.query = yagooglesearch.urllib.parse.quote_plus(query) new_ua = client.assign_random_user_agent() print(f"Using UA: {new_ua[:60]}...") client.update_urls() urls = client.search() print(f" → {len(urls)} results\n") ``` -------------------------------- ### Rotate Through Multiple Proxies in Python Source: https://github.com/opsdisk/yagooglesearch/blob/master/README.md Implement proxy rotation by iterating through a list of proxy URLs and creating a new SearchClient instance for each search query. SSL verification is conditionally disabled for HTTP proxies. ```python import yagooglesearch proxies = [ "socks5h://127.0.0.1:9050", "socks5h://127.0.0.1:9051", "http://127.0.0.1:9052", # HTTPS proxy with a self-signed SSL/TLS certificate. ] search_queries = [ "python", "site:github.com pagodo", "peanut butter toast", "are dragons real?", "ssh tunneling", ] proxy_rotation_index = 0 for search_query in search_queries: # Rotate through the list of proxies using modulus to ensure the index is in the proxies list. proxy_index = proxy_rotation_index % len(proxies) client = yagooglesearch.SearchClient( search_query, proxy=proxies[proxy_index], ) # Only disable SSL/TLS verification for the HTTPS proxy using a self-signed certificate. if proxies[proxy_index].startswith("http://"): client.verify_ssl = False urls_list = client.search() print(urls_list) proxy_rotation_index += 1 ``` -------------------------------- ### SearchClient.search() Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Executes a Google search and returns the results. This method simulates visiting the Google homepage for cookies and paginates through results until the maximum number of URLs is reached or no more results are available. It returns a list of URL strings or a list of dictionaries if `verbose_output` is enabled. ```APIDOC ## SearchClient.search() — Execute the Search and Return Results The main entry point for running a Google search. It simulates a visit to the Google homepage to collect cookies, then iterates through paginated result pages until `max_search_result_urls_to_return` is reached or results are exhausted. Returns a list of URL strings, or a list of dicts when `verbose_output=True`. ```python import yagooglesearch # Verbose output: returns rank, title, description, and URL for each result client = yagooglesearch.SearchClient( query="open source intrusion detection system", max_search_result_urls_to_return=10, verbose_output=True, verbosity=2, # Minimal console noise ) results = client.search() for item in results: print(f"Rank #{item['rank']}: {item['title']}") print(f" URL: {item['url']}") print(f" Desc: {item['description'][:80]}") # Expected output: # Rank #1: Snort - Network Intrusion Detection & Prevention System # URL: https://www.snort.org # Desc: Snort is the foremost Open Source Intrusion Prevention System (IPS)... # Rank #2: Suricata # URL: https://suricata.io # Desc: Suricata is a high performance, open source network analysis and... ``` ``` -------------------------------- ### Configure Proxy Support for Searches Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Pass a proxy string to `SearchClient` via the `proxy` parameter to route all search requests through a specified proxy. For HTTPS proxies with self-signed certificates, set `verify_ssl=False`. ```python import yagooglesearch # SOCKS5 proxy (DNS resolved on proxy server — recommended for anonymity) client = yagooglesearch.SearchClient( query="site:github.com exploit", proxy="socks5h://127.0.0.1:9050", # e.g., Tor max_search_result_urls_to_return=25, verbosity=3, ) urls = client.search() print(f"Via SOCKS5: {len(urls)} results") # HTTPS proxy with self-signed cert client2 = yagooglesearch.SearchClient( query="network scanning tools", proxy="http://127.0.0.1:8080", verify_ssl=False, verbosity=3, ) urls2 = client2.search() print(f"Via HTTPS proxy: {len(urls2)} results") # Rotating proxies across multiple queries proxies = [ "socks5h://127.0.0.1:9050", "socks5h://127.0.0.1:9051", "http://127.0.0.1:8888", ] queries = ["python exploit", "network recon tools", "ctf writeups"] for i, query in enumerate(queries): proxy = proxies[i % len(proxies)] c = yagooglesearch.SearchClient(query, proxy=proxy, max_search_result_urls_to_return=10) if proxy.startswith("http://"): c.verify_ssl = False results = c.search() print(f"Query '{query}' via {proxy}: {len(results)} results") ``` -------------------------------- ### Proxy Support Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Enables the use of proxy servers for search requests. The `proxy` parameter accepts a proxy string (HTTP, HTTPS, or SOCKS5) and is applied to all requests made by the `SearchClient` instance. SSL verification can be disabled for HTTPS proxies with self-signed certificates using `verify_ssl=False`. ```APIDOC ## Proxy Support — HTTP, HTTPS, and SOCKS5 Pass a proxy string to `SearchClient` via the `proxy` parameter. The proxy is used for the entire search lifecycle (home page visit + all paginated requests). For HTTPS proxies using self-signed certificates, disable SSL verification with `verify_ssl=False`. ```python import yagooglesearch # SOCKS5 proxy (DNS resolved on proxy server — recommended for anonymity) client = yagooglesearch.SearchClient( query="site:github.com exploit", proxy="socks5h://127.0.0.1:9050", # e.g., Tor max_search_result_urls_to_return=25, verbosity=3, ) urls = client.search() print(f"Via SOCKS5: {len(urls)} results") # HTTPS proxy with self-signed cert client2 = yagooglesearch.SearchClient( query="network scanning tools", proxy="http://127.0.0.1:8080", verify_ssl=False, verbosity=3, ) urls2 = client2.search() print(f"Via HTTPS proxy: {len(urls2)} results") # Rotating proxies across multiple queries proxies = [ "socks5h://127.0.0.1:9050", "socks5h://127.0.0.1:9051", "http://127.0.0.1:8888", ] queries = ["python exploit", "network recon tools", "ctf writeups"] for i, query in enumerate(queries): proxy = proxies[i % len(proxies)] c = yagooglesearch.SearchClient(query, proxy=proxy, max_search_result_urls_to_return=10) if proxy.startswith("http://"): c.verify_ssl = False results = c.search() print(f"Query '{query}' via {proxy}: {len(results)} results") ``` ``` -------------------------------- ### SearchClient.assign_random_user_agent() Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Rotates the user agent for the search client. It selects a random user agent from a bundled file and updates the client instance. This method is automatically called during initialization if no user agent is explicitly provided, but can also be called manually to change the user agent between searches. ```APIDOC ## SearchClient.assign_random_user_agent() — Rotate User Agent Selects a random user agent string from the bundled `user_agents.txt` file and sets it on the client instance. Called automatically during `__init__()` if no explicit `user_agent` is passed. Can be called again mid-session to rotate the user agent between separate searches. ```python import yagooglesearch client = yagooglesearch.SearchClient( query="machine learning datasets", max_search_result_urls_to_return=20, ) # Manually rotate user agent before each search in a loop queries = ["python web scraping", "rust systems programming", "go concurrency patterns"] for query in queries: client.query = yagooglesearch.urllib.parse.quote_plus(query) new_ua = client.assign_random_user_agent() print(f"Using UA: {new_ua[:60]}...") client.update_urls() urls = client.search() print(f" → {len(urls)} results\n") ``` ``` -------------------------------- ### HTTP 429 Detection and Recovery Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Configures how the library handles HTTP 429 (Too Many Requests) errors. When `yagooglesearch_manages_http_429s` is `True` (default), the library automatically retries after a cool-off period. If `False`, the string `"HTTP_429_DETECTED"` is appended to the results list, allowing manual handling. ```APIDOC ## HTTP 429 Detection and Recovery — Self-Managed Mode When `yagooglesearch_manages_http_429s=True` (default), the library automatically sleeps for `http_429_cool_off_time_in_minutes` and retries on HTTP 429. Set it to `False` to handle rate-limiting yourself: the string `"HTTP_429_DETECTED"` is appended to the results list instead. ```python import yagooglesearch client = yagooglesearch.SearchClient( query="site:pastebin.com passwords", verbosity=4, num=10, max_search_result_urls_to_return=500, minimum_delay_between_paged_results_in_seconds=1, yagooglesearch_manages_http_429s=False, # Handle 429s manually ) urls = client.search() if "HTTP_429_DETECTED" in urls: urls.remove("HTTP_429_DETECTED") print(f"Rate-limited! Collected {len(urls)} URLs before block.") print("Consider switching proxy or waiting before retrying.") else: print(f"Search complete: {len(urls)} URLs found") for url in urls: print(url) ``` ``` -------------------------------- ### Build Custom Date-Range TBS Parameter with get_tbs() Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Use the `get_tbs()` helper function to format two `datetime.date` objects into a Google `&tbs=` custom date-range string. This string can then be passed to the `tbs` argument of `SearchClient`. ```python import datetime import yagooglesearch # Search for results published between Jan 1 and Mar 31, 2024 from_date = datetime.date(2024, 1, 1) to_date = datetime.date(2024, 3, 31) tbs_value = yagooglesearch.get_tbs(from_date, to_date) print(f"tbs value: {tbs_value}") # tbs value: cdr:1,cd_min:01/01/2024,cd_max:03/31/2024 client = yagooglesearch.SearchClient( query="AI security vulnerabilities CVE", tbs=tbs_value, max_search_result_urls_to_return=30, verbosity=3, ) urls = client.search() print(f"Found {len(urls)} results from Q1 2024") for url in urls[:5]: print(f" {url}") ``` -------------------------------- ### Disable SSL Verification for HTTPS Proxy in Python Source: https://github.com/opsdisk/yagooglesearch/blob/master/README.md Instantiate SearchClient with SSL verification disabled for an HTTPS proxy, typically used with self-signed certificates. This setting can also be modified after instantiation. ```python import yagooglesearch query = "site:github.com" client = yagooglesearch.SearchClient( query, proxy="http://127.0.0.1:8080", verify_ssl=False, verbosity=5, ) ``` ```python query = "site:github.com" client = yagooglesearch.SearchClient( query, proxy="http://127.0.0.1:8080", verbosity=5, ) client.verify_ssl = False ``` -------------------------------- ### get_tbs() Source: https://context7.com/opsdisk/yagooglesearch/llms.txt A module-level helper function that formats two `datetime.date` objects into a Google `&tbs=` custom date-range string. The output is intended to be passed as the `tbs` argument to `SearchClient`. ```APIDOC ## `get_tbs()` — Build Custom Date-Range `tbs` Parameter A module-level helper function that formats two `datetime.date` objects into a Google `&tbs=` custom date-range string. The output is passed as the `tbs` argument to `SearchClient`. ```python import datetime import yagooglesearch # Search for results published between Jan 1 and Mar 31, 2024 from_date = datetime.date(2024, 1, 1) to_date = datetime.date(2024, 3, 31) tbs_value = yagooglesearch.get_tbs(from_date, to_date) print(f"tbs value: {tbs_value}") # tbs value: cdr:1,cd_min:01/01/2024,cd_max:03/31/2024 client = yagooglesearch.SearchClient( query="AI security vulnerabilities CVE", tbs=tbs_value, max_search_result_urls_to_return=30, verbosity=3, ) urls = client.search() print(f"Found {len(urls)} results from Q1 2024") for url in urls[:5]: print(f" {url}") ``` ``` -------------------------------- ### Handle HTTP 429 Rate Limiting Manually Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Set `yagooglesearch_manages_http_429s=False` to manually handle rate limiting. The string 'HTTP_429_DETECTED' will be appended to the results list when a 429 error occurs, allowing you to implement custom retry logic or logging. ```python import yagooglesearch client = yagooglesearch.SearchClient( query="site:pastebin.com passwords", verbosity=4, num=10, max_search_result_urls_to_return=500, minimum_delay_between_paged_results_in_seconds=1, yagooglesearch_manages_http_429s=False, # Handle 429s manually ) urls = client.search() if "HTTP_429_DETECTED" in urls: urls.remove("HTTP_429_DETECTED") print(f"Rate-limited! Collected {len(urls)} URLs before block.") print("Consider switching proxy or waiting before retrying.") else: print(f"Search complete: {len(urls)} URLs found") for url in urls: print(url) ``` -------------------------------- ### Manage HTTP 429s Manually in Python Source: https://github.com/opsdisk/yagooglesearch/blob/master/README.md Configure yagooglesearch to not manage HTTP 429 responses. The library will return 'HTTP_429_DETECTED' in the results list, allowing your script to handle retries or modifications. ```python import yagooglesearch query = "site:twitter.com" client = yagooglesearch.SearchClient( query, tbs="li:1", verbosity=4, num=10, max_search_result_urls_to_return=1000, minimum_delay_between_paged_results_in_seconds=1, yagooglesearch_manages_http_429s=False, # Add to manage HTTP 429s. ) client.assign_random_user_agent() urls = client.search() if "HTTP_429_DETECTED" in urls: print("HTTP 429 detected...it's up to you to modify your search.") # Remove HTTP_429_DETECTED from list. urls.remove("HTTP_429_DETECTED") print("URLs found before HTTP 429 detected...") for url in urls: print(url) ``` -------------------------------- ### Filter Search Result URLs with SearchClient Source: https://context7.com/opsdisk/yagooglesearch/llms.txt Filters raw href values from Google search results. It unwraps Google redirect URLs, excludes links without a valid network location, and removes URLs containing 'google' in the hostname. Returns the cleaned URL or None if invalid. ```python import yagooglesearch client = yagooglesearch.SearchClient(query="test", verbosity=1) test_links = [ "/url?q=https://github.com/opsdisk/yagooglesearch&sa=U", # Google redirect "https://www.example.com/page", # Valid direct URL "https://images.google.com/search?q=cats", # Google URL — excluded "/search?q=something", # No netloc — excluded None, # None input ] for raw_link in test_links: if raw_link is None: continue result = client.filter_search_result_urls(raw_link) status = result if result else "FILTERED OUT" print(f"Input: {raw_link[:60]}") print(f"Output: {status}\n") ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.