### Session Management Example (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Demonstrates how to manage sessions with cloudscraper, specifically showing how to log in to a site using a POST request and then make authenticated GET requests using the same scraper instance. ```python import cloudscraper scraper = cloudscraper.create_scraper() # Login to a site login_data = {'username': 'user', 'password': 'pass'} scraper.post("https://example.com/login", data=login_data) # Make authenticated requests response = scraper.get("https://example.com/dashboard") ``` -------------------------------- ### Basic Web Scraping Example (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md A complete example demonstrating basic web scraping using cloudscraper, including importing the library, creating a scraper, and performing both GET and POST requests with different data types. ```python import cloudscraper scraper = cloudscraper.create_scraper() # Simple GET request response = scraper.get("https://example.com") print(response.text) # POST request with data response = scraper.post("https://example.com/api", json={"key": "value"}) print(response.json()) ``` -------------------------------- ### Real-time Monitoring Setup with cloudscraper Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Demonstrates the initial setup for cloudscraper with features that enable real-time monitoring, specifically ML optimization and adaptive timing, along with debug mode enabled. This configuration is a precursor to observing performance metrics and optimization suggestions. ```python import cloudscraper scraper = cloudscraper.create_scraper( enable_ml_optimization=True, enable_adaptive_timing=True, debug=True ) ``` -------------------------------- ### Advanced Configuration Example (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Illustrates an example of advanced scraper configuration, combining multiple settings such as JavaScript interpreter, delay, stealth mode, browser selection, and debug mode for maximum compatibility and control. ```python import cloudscraper # Maximum compatibility configuration scraper = cloudscraper.create_scraper( interpreter='js2py', delay=5, enable_stealth=True, stealth_options={ 'min_delay': 2.0, 'max_delay': 5.0, 'human_like_delays': True, 'randomize_headers': True, 'browser_quirks': True }, browser='chrome', debug=True ) response = scraper.get("https://protected-site.com") ``` -------------------------------- ### Complete Enhanced Cloudscraper Configuration Example (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Demonstrates a comprehensive configuration for Cloudscraper, enabling a wide range of advanced bypass and stealth features. This example includes settings for TLS fingerprinting, anti-detection, AI challenges, adaptive timing, stealth options, proxy rotation, and CAPTCHA solving. ```python import cloudscraper # Ultimate bypass configuration scraper = cloudscraper.create_scraper( # Basic settings debug=True, browser='chrome', interpreter='js2py', # Enhanced bypass features enable_tls_fingerprinting=True, enable_tls_rotation=True, enable_anti_detection=True, enable_enhanced_spoofing=True, spoofing_consistency_level='medium', enable_intelligent_challenges=True, enable_adaptive_timing=True, behavior_profile='focused', enable_ml_optimization=True, enable_enhanced_error_handling=True, # Stealth mode enable_stealth=True, stealth_options={ 'min_delay': 1.5, 'max_delay': 4.0, 'human_like_delays': True, 'randomize_headers': True, 'browser_quirks': True, 'simulate_viewport': True, 'behavioral_patterns': True }, # Session management session_refresh_interval=3600, auto_refresh_on_403=True, max_403_retries=3, # Proxy rotation rotating_proxies=[ 'http://proxy1:8080', 'http://proxy2:8080', 'http://proxy3:8080' ], proxy_options={ 'rotation_strategy': 'smart', 'ban_time': 600 }, # CAPTCHA solving captcha={ 'provider': '2captcha', 'api_key': 'your_api_key' } ) # Monitor bypass performance stats = scraper.get_enhanced_statistics() print(f"Active bypass systems: {len(stats)}") ``` -------------------------------- ### Make requests and get statistics with CloudScraper Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Demonstrates making GET requests to multiple URLs and retrieving comprehensive bypass statistics. It also shows how to access domain-specific insights if the ML optimizer is available. ```python import cloudscraper # Create a CloudScraper instance scraper = cloudscraper.create_scraper() # Make some requests for url in ['https://site1.com', 'https://site2.com', 'https://site3.com']: response = scraper.get(url) print(f"{url}: {response.status_code}") # Get comprehensive statistics stats = scraper.get_enhanced_statistics() print("\n=== Enhanced Bypass Statistics ===") print(f"TLS Fingerprinting: {stats.get('tls_fingerprinting', 'Disabled')}") print(f"Anti-Detection: {stats.get('anti_detection', 'Disabled')}") print(f"Challenge Detection: {stats.get('intelligent_challenges', 'Disabled')}") print(f"ML Optimization: {stats.get('ml_optimization', 'Disabled')}") # Get domain-specific insights if hasattr(scraper, 'ml_optimizer'): ml_report = scraper.ml_optimizer.get_optimization_report() print(f"\nML Success Rate: {ml_report.get('global_success_rate', 0):.2%}") print(f"Tracked Domains: {ml_report.get('tracked_domains', 0)}") ``` -------------------------------- ### Install cloudscraper using pip Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md This command installs the cloudscraper library using pip, the Python package installer. Ensure you have pip installed and configured in your system's PATH. ```bash pip install cloudscraper ``` -------------------------------- ### Example scraper_config.yaml Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md A sample YAML configuration file for CloudScraper, specifying various settings like debug mode, stealth options, and proxy configurations. ```yaml debug: true interpreter: js2py enable_stealth: true stealth_options: min_delay: 0.5 max_delay: 2.0 human_like_delays: true randomize_headers: true rotating_proxies: - "http://proxy1:8080" - "http://proxy2:8080" proxy_options: rotation_strategy: "smart" ban_time: 300 enable_metrics: true ``` -------------------------------- ### Create Scraper Instance and Make Requests (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Demonstrates how to create a cloudscraper instance and use it for basic GET and POST requests, mimicking the functionality of the requests library. This is the fundamental way to interact with protected websites. ```python import cloudscraper # Create scraper instance scraper = cloudscraper.create_scraper() # Use like requests response = scraper.get("https://protected-site.com") print(response.text) # Works with all HTTP methods response = scraper.post("https://protected-site.com/api", json={"key": "value"}) ``` -------------------------------- ### Basic CloudScraper Usage Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Illustrates the fundamental usage of the CloudScraper library. It shows how to create a scraper instance and use it to make a simple GET request, similar to the standard requests library. ```python import cloudscraper # Create a CloudScraper instance scraper = cloudscraper.create_scraper() # Use it like a regular requests session response = scraper.get("https://example.com") print(response.text) ``` -------------------------------- ### Async CloudScraper Support Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Shows how to use CloudScraper asynchronously for concurrent requests. It includes examples of making single requests, batch requests, and retrieving performance statistics within an async context. ```python import asyncio import cloudscraper async def main(): async with cloudscraper.create_async_scraper( max_concurrent_requests=10, enable_stealth=True ) as scraper: # Single request response = await scraper.get('https://example.com') # Batch requests requests = [ {'method': 'GET', 'url': f'https://example.com/page{i}'} for i in range(5) ] responses = await scraper.batch_requests(requests) # Get performance stats stats = scraper.get_stats() print(f"Total requests: {stats['total_requests']}") asyncio.run(main()) ``` -------------------------------- ### HTTP Methods with Automatic Challenge Handling Source: https://context7.com/zinzied/cloudscraper/llms.txt Demonstrates using cloudscraper for standard HTTP methods (GET, POST, PUT, DELETE) with automatic Cloudflare challenge solving. It enables stealth and TLS fingerprinting. Includes examples for JSON and form data POST requests, custom headers, and closing the scraper. ```python import cloudscraper scraper = cloudscraper.create_scraper( enable_stealth=True, enable_tls_fingerprinting=True ) # GET request response = scraper.get("https://example.com/api/data") print(f"GET Status: {response.status_code}") # POST request with JSON payload data = {"username": "user123", "action": "query"} response = scraper.post( "https://example.com/api/submit", json=data, headers={'Content-Type': 'application/json'} ) print(f"POST Status: {response.status_code}") # POST with form data form_data = {"search": "cloudflare bypass", "limit": "10"} response = scraper.post( "https://example.com/search", data=form_data ) print(f"Form POST Status: {response.status_code}") # Request with custom headers headers = { 'X-API-Key': 'your-api-key', 'Accept': 'application/json' } response = scraper.get( "https://example.com/api/protected", headers=headers ) print(f"Custom Headers Status: {response.status_code}") # PUT and DELETE requests response = scraper.put("https://example.com/api/update/123", json={"status": "active"}) print(f"PUT Status: {response.status_code}") response = scraper.delete("https://example.com/api/delete/123") print(f"DELETE Status: {response.status_code}") scraper.close() ``` -------------------------------- ### Cloudscraper: Complete Enhanced Configuration Example Source: https://github.com/zinzied/cloudscraper/blob/master/README.md Provides a comprehensive example of configuring Cloudscraper with numerous advanced options, including bypass parameters, stealth mode, session management, proxy rotation, and CAPTCHA solving. This configuration aims for an ultimate bypass. ```python import cloudscraper scraper = cloudscraper.create_scraper( debug=True, browser='chrome', interpreter='js2py', enable_tls_fingerprinting=True, enable_tls_rotation=True, enable_anti_detection=True, enable_enhanced_spoofing=True, spoofing_consistency_level='medium', enable_intelligent_challenges=True, enable_adaptive_timing=True, behavior_profile='focused', enable_ml_optimization=True, enable_enhanced_error_handling=True, enable_stealth=True, stealth_options={ 'min_delay': 1.5, 'max_delay': 4.0, 'human_like_delays': True, 'randomize_headers': True, 'browser_quirks': True, 'simulate_viewport': True, 'behavioral_patterns': True }, session_refresh_interval=3600, auto_refresh_on_403=True, max_403_retries=3, rotating_proxies=[ 'http://proxy1:8080', 'http://proxy2:8080', 'http://proxy3:8080' ], proxy_options={ 'rotation_strategy': 'smart', 'ban_time': 600 }, captcha={ 'provider': '2captcha', 'api_key': 'your_api_key' } ) stats = scraper.get_enhanced_statistics() print(f"Active bypass systems: {len(stats)}") ``` -------------------------------- ### Optimize for Domain and Get Enhanced Statistics (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Optimizes the scraper for a specific domain and retrieves detailed statistics about the enhanced bypass systems currently active. This helps in understanding which features are being utilized and their status. ```python scraper.optimize_for_domain('target-site.com') stats = scraper.get_enhanced_statistics() for system, status in stats.items(): print(f"{system}: {status}") ``` -------------------------------- ### Initialize Scraper Conservatively (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md Demonstrates a gradual approach to CloudScraper configuration, starting with conservative settings for behavior profiles and spoofing consistency. This is recommended before enabling maximum stealth to avoid detection. ```python # Start conservative scraper = cloudscraper.create_scraper( behavior_profile='research', spoofing_consistency_level='low' ) ``` -------------------------------- ### Install ai-cloudscraper and Optional Dependencies Source: https://github.com/zinzied/cloudscraper/blob/master/README.md Installs the ai-cloudscraper package with optional features for AI solvers or browser automation. Users can install the base package, features for AI solvers, features for browser automation, or install from source for development. ```bash pip install ai-cloudscraper pip install ai-cloudscraper[ai] pip install ai-cloudscraper[browser] pip install -e . ``` -------------------------------- ### Configure Cloudscraper Proxy Settings Source: https://github.com/zinzied/cloudscraper/blob/master/README.md Demonstrates how to set up proxy servers for use with cloudscraper. This example shows how to configure both HTTP and HTTPS proxies for a single proxy server. ```python # Single proxy scraper = cloudscraper.create_scraper() scraper.proxies = { 'http': 'http://proxy:8080', 'https': 'http://proxy:8080' } ``` -------------------------------- ### Standard HTTP Methods with Challenge Handling Source: https://context7.com/zinzied/cloudscraper/llms.txt Utilize standard HTTP methods (GET, POST, PUT, DELETE) with CloudScraper's built-in automatic challenge handling. This ensures seamless interaction with websites protected by Cloudflare. ```APIDOC ## Standard HTTP Methods ### Description CloudScraper extends standard HTTP methods (GET, POST, PUT, DELETE, etc.) to automatically handle Cloudflare challenges (v1, v2, v3, Turnstile). Cookie persistence and circuit breaker protection are enabled by default for robust scraping. ### Method `scraper.get(url, **kwargs)` `scraper.post(url, data=None, json=None, **kwargs)` `scraper.put(url, data=None, json=None, **kwargs)` `scraper.delete(url, **kwargs)` ### Endpoint Any valid URL. ### Parameters Standard parameters for `requests` library methods, plus Cloudflare challenge handling. ### Request Example ```python import cloudscraper scraper = cloudscraper.create_scraper( enable_stealth=True, enable_tls_fingerprinting=True ) # GET request response = scraper.get("https://example.com/api/data") print(f"GET Status: {response.status_code}") # POST request with JSON payload data = {"username": "user123", "action": "query"} response = scraper.post( "https://example.com/api/submit", json=data, headers={'Content-Type': 'application/json'} ) print(f"POST Status: {response.status_code}") # POST with form data form_data = {"search": "cloudflare bypass", "limit": "10"} response = scraper.post( "https://example.com/search", data=form_data ) print(f"Form POST Status: {response.status_code}") # Request with custom headers headers = { 'X-API-Key': 'your-api-key', 'Accept': 'application/json' } response = scraper.get( "https://example.com/api/protected", headers=headers ) print(f"Custom Headers Status: {response.status_code}") # PUT and DELETE requests response = scraper.put("https://example.com/api/update/123", json={"status": "active"}) print(f"PUT Status: {response.status_code}") response = scraper.delete("https://example.com/api/delete/123") print(f"DELETE Status: {response.status_code}") scraper.close() ``` ### Response #### Success Response (200) - **status_code** (integer) - The HTTP status code of the response. - **text** (string) - The response body as text. - **json** (object) - The response body parsed as JSON (if applicable). #### Response Example ```json { "status_code": 200, "text": "..." } ``` ``` -------------------------------- ### Configure CloudScraper for Fast and Reliable Scraping Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/IMPROVEMENT_SUMMARY.md Provides an example of how to configure CloudScraper for optimal performance, focusing on speed and reliability. It shows how to enable enhanced features, use focused behavior profiles, and set conservative stealth options. ```python import cloudscraper scraper = cloudscraper.create_scraper( debug=True, browser='chrome', # Core enhanced features enable_tls_fingerprinting=True, enable_enhanced_spoofing=True, enable_enhanced_error_handling=True, # Use focused profile for faster requests enable_adaptive_timing=True, behavior_profile='focused', # Conservative stealth settings enable_stealth=True, stealth_options={ 'min_delay': 0.5, 'max_delay': 2.0, 'human_like_delays': True, 'randomize_headers': True }, # Lower recursion limits for safety solveDepth=2, max_403_retries=2 ) ``` -------------------------------- ### Configure Proxy Support (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Demonstrates how to set up proxy servers for the scraper, including single proxy configuration and rotating proxies with advanced options for rotation strategy and ban times. ```python # Single proxy scraper = cloudscraper.create_scraper() scraper.proxies = { 'http': 'http://proxy:8080', 'https': 'http://proxy:8080' } # Proxy rotation proxies = [ 'http://proxy1:8080', 'http://proxy2:8080', 'http://proxy3:8080' ] scraper = cloudscraper.create_scraper( rotating_proxies=proxies, proxy_options={ 'rotation_strategy': 'smart', 'ban_time': 300 } ) ``` -------------------------------- ### Get Enhanced System Status in Cloudscraper Source: https://github.com/zinzied/cloudscraper/blob/master/README.md Retrieves detailed statistics about the enhanced bypass systems currently active in the Cloudscraper instance. This helps in monitoring the effectiveness of the scraping setup. ```python stats = scraper.get_enhanced_statistics() for system, status in stats.items(): print(f"{system}: {status}") ``` -------------------------------- ### Hybrid Engine for Ultimate Bypass Solution Source: https://context7.com/zinzied/cloudscraper/llms.txt Utilizes the Hybrid Engine by combining TLS-Chameleon and Playwright for robust TLS fingerprinting and real browser automation. It falls back to browser automation only when necessary, offering speed and efficiency. Examples include basic usage, with AI CAPTCHA solving, and with proxy support. Ensure 'cloudscraper[hybrid]' is installed. ```python import cloudscraper # Install required dependencies first: # pip install cloudscraper[hybrid] # Basic Hybrid Engine usage scraper = cloudscraper.create_scraper( interpreter='hybrid', impersonate='chrome120', # Optional: specify browser fingerprint debug=True ) # With AI CAPTCHA solving scraper = cloudscraper.create_scraper( interpreter='hybrid', google_api_key='YOUR_GEMINI_API_KEY', debug=True ) # With proxy support (proxies are used for both HTTP and AI requests) scraper = cloudscraper.create_scraper( interpreter='hybrid', google_api_key='YOUR_GEMINI_API_KEY', rotating_proxies=['http://user:pass@proxy:port'], debug=True ) try: response = scraper.get('https://high-security-site.com') print(f"Status: {response.status_code}") if response.status_code == 200: print("✅ Successfully bypassed protection!") finally: scraper.close() ``` -------------------------------- ### Load CloudScraper Configuration from File Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Demonstrates loading CloudScraper configuration settings from external files, specifically YAML and JSON formats. This allows for cleaner and more organized configuration management. ```python import cloudscraper # Load from YAML config scraper = cloudscraper.create_scraper(config_file='scraper_config.yaml') # Or from JSON scraper = cloudscraper.create_scraper(config_file='scraper_config.json') ``` -------------------------------- ### Enhanced Bypass Troubleshooting (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Provides advanced troubleshooting steps for sites that remain blocked, including maximum stealth configuration, enabling various anti-detection features, and setting custom challenge patterns. ```python # Try maximum stealth configuration scraper = cloudscraper.create_scraper( enable_tls_fingerprinting=True, enable_anti_detection=True, enable_enhanced_spoofing=True, spoofing_consistency_level='high', enable_adaptive_timing=True, behavior_profile='research', # Slowest, most careful stealth_options={ 'min_delay': 3.0, 'max_delay': 10.0, 'human_like_delays': True } ) # Enable maximum stealth mode scraper.enable_maximum_stealth() # Add custom challenge patterns scraper.intelligent_challenge_system.add_custom_pattern( domain='problem-site.com', pattern_name='Custom Challenge', patterns=[r'custom.+challenge.+text'], challenge_type='custom', response_strategy='delay_retry' ) # Make several learning requests first for i in range(5): try: response = scraper.get('https://target-site.com/test') except Exception: pass ``` -------------------------------- ### Advanced CloudScraper Configuration Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Demonstrates how to configure the CloudScraper with advanced options, including browser emulation, stealth settings, rotating proxies, and metrics. ```python import cloudscraper # Create scraper with advanced options scraper = cloudscraper.create_scraper( browser='chrome', debug=True, enable_stealth=True, stealth_options={ 'min_delay': 1.0, 'max_delay': 3.0, 'human_like_delays': True, 'randomize_headers': True }, rotating_proxies=[ 'http://proxy1:8080', 'http://proxy2:8080' ], proxy_options={ 'rotation_strategy': 'smart', 'ban_time': 300 }, enable_metrics=True, session_refresh_interval=3600 ) response = scraper.get('https://protected-site.com') ``` -------------------------------- ### Install Optional Dependencies for AI Solvers and Browser Automation Source: https://github.com/zinzied/cloudscraper/blob/master/README.md Provides commands to install additional Python packages required for specific optional features of cloudscraper. These include libraries for AI-based CAPTCHA solving (ddddocr, ultralytics, pillow) and browser automation (playwright). ```bash pip install ddddocr ultralytics pillow pip install playwright playwright install chromium ``` -------------------------------- ### Run Demonstration Script (Shell) Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md Executes a demonstration script for enhanced bypass techniques using CloudScraper. This is typically run from the command line to showcase the capabilities of the library in real-world scenarios. ```shell python examples/enhanced_bypass_demo.py ``` -------------------------------- ### Initialize CloudScraper with Hybrid Engine (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md Demonstrates how to initialize the CloudScraper with the hybrid engine enabled, which combines HTTP requests with browser automation for bypassing complex challenges. This feature requires no API keys and automatically falls back to browser usage only when necessary. ```python scraper = cloudscraper.create_scraper( interpreter='hybrid', impersonate='chrome120' ) ``` -------------------------------- ### Troubleshoot Challenge Solving Failures (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Provides methods to resolve common challenges encountered during website scraping. It demonstrates how to instantiate a scraper with different interpreters, increase delays, or enable debug mode to identify and fix issues. ```python # Try different interpreter scraper = cloudscraper.create_scraper(interpreter='nodejs') # Increase delay scraper = cloudscraper.create_scraper(delay=10) # Enable debug mode scraper = cloudscraper.create_scraper(debug=True) ``` -------------------------------- ### Basic Enhanced CloudScraper Initialization Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md Initializes a CloudScraper instance with basic enhanced configuration options. ```python import cloudscraper ``` -------------------------------- ### Get Proxy Health Report with CloudScraper Source: https://context7.com/zinzied/cloudscraper/llms.txt Retrieves a health report for proxies managed by CloudScraper. This includes status, success rate, and total requests for each proxy. The scraper is closed after the report is generated. ```python proxy_report = scraper.get_proxy_health_report() print("\n=== Proxy Health Report ===") for proxy_info in proxy_report.get('proxies', []): print(f"Proxy: {proxy_info['proxy']}") print(f" Status: {proxy_info['status']}") print(f" Success Rate: {proxy_info['success_rate']:.2%}") print(f" Total Requests: {proxy_info['total_requests']}") scraper.close() ``` -------------------------------- ### Configure Enhanced Stealth Options (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Sets up advanced stealth options for the scraper to mimic human browsing behavior more closely. This includes customizing delays, randomizing headers, and simulating browser quirks and viewport changes for better evasion. ```python stealth_options = { 'min_delay': 1.0, # Minimum delay between requests 'max_delay': 4.0, # Maximum delay between requests 'human_like_delays': True, # Use human-like delay patterns 'randomize_headers': True, # Randomize request headers 'browser_quirks': True, # Enable browser-specific quirks 'simulate_viewport': True, # Simulate viewport changes 'behavioral_patterns': True # Use behavioral pattern simulation } ``` -------------------------------- ### Get ML Optimization Report in Cloudscraper Source: https://github.com/zinzied/cloudscraper/blob/master/README.md Retrieves the machine learning optimization report, specifically focusing on the global success rate of bypass challenges. Requires the 'ml_optimizer' to be available. ```python if hasattr(scraper, 'ml_optimizer'): report = scraper.ml_optimizer.get_optimization_report() print(f"Success rate: {report.get('global_success_rate', 0):.2%}") ``` -------------------------------- ### Configure Cloudscraper CAPTCHA Solver Source: https://github.com/zinzied/cloudscraper/blob/master/README.md Illustrates how to integrate a CAPTCHA solving service with cloudscraper. This example configures the '2captcha' provider with an API key, but the library supports several other providers as well. ```python scraper = cloudscraper.create_scraper( captcha={ 'provider': '2captcha', 'api_key': 'your_api_key' } ) ``` -------------------------------- ### Adaptive Timing Algorithm Configuration Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md Enables adaptive timing to simulate realistic human browsing behavior. Users can select different behavior profiles like 'casual', 'focused', 'research', or 'mobile'. ```python scraper = cloudscraper.create_scraper( enable_adaptive_timing=True, behavior_profile='casual' # or 'focused', 'research', 'mobile' ) ``` -------------------------------- ### Run Demonstration Script Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/ENHANCED_FEATURES.md Executes the enhanced bypass demonstration script. This is useful for quickly testing the capabilities of the CloudScraper in a simulated environment. ```python # Run the demonstration script python examples/enhanced_bypass_demo.py ``` -------------------------------- ### Get Enhanced Statistics (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md Retrieves comprehensive statistics from all CloudScraper systems, providing insights into performance and operation. The output is structured by system, detailing key metrics and configurations. This is useful for monitoring and debugging. ```python # Get enhanced statistics from all systems stats = scraper.get_enhanced_statistics() print("=== Enhanced CloudScraper Statistics ===") for system, data in stats.items(): print(f"\n{system.upper()}:") if isinstance(data, dict): for key, value in data.items(): print(f" {key}: {value}") else: print(f" {data}") ``` -------------------------------- ### Initialize CloudScraper Instance Source: https://context7.com/zinzied/cloudscraper/llms.txt Creates a CloudScraper object with configurable bypass features and stealth capabilities. Supports the Hybrid Engine, AI CAPTCHA solving, TLS fingerprinting, anti-detection systems, adaptive timing, and ML-based optimization. Cookie persistence and circuit breaker are enabled by default. ```python import cloudscraper # Basic usage with enhanced features enabled by default scraper = cloudscraper.create_scraper( debug=True, browser='chrome', # Enable cookie persistence (enabled by default) enable_cookie_persistence=True, cookie_ttl=1800, # 30 minutes # Circuit breaker (enabled by default) enable_circuit_breaker=True, circuit_failure_threshold=3, circuit_timeout=60 ) # Hybrid Engine mode (most powerful option) scraper = cloudscraper.create_scraper( interpreter='hybrid', impersonate='chrome120', google_api_key='YOUR_GEMINI_API_KEY', # Optional: for AI CAPTCHA solving debug=True ) # Enhanced bypass configuration scraper = cloudscraper.create_scraper( debug=True, browser='chrome', # Advanced TLS fingerprinting enable_tls_fingerprinting=True, enable_tls_rotation=True, # Anti-detection systems enable_anti_detection=True, # Enhanced fingerprint spoofing enable_enhanced_spoofing=True, spoofing_consistency_level='medium', # low, medium, high # Intelligent challenge detection (disabled by default to prevent recursion) enable_intelligent_challenges=False, # Adaptive timing (human behavior simulation) enable_adaptive_timing=True, behavior_profile='casual', # casual, focused, research, mobile # Machine learning optimization enable_ml_optimization=True, # Enhanced error handling enable_enhanced_error_handling=True, # Stealth mode enable_stealth=True, stealth_options={ 'min_delay': 1.0, 'max_delay': 4.0, 'human_like_delays': True, 'randomize_headers': True, 'browser_quirks': True, 'simulate_viewport': True, 'behavioral_patterns': True } ) # Make protected requests try: response = scraper.get("https://protected-cloudflare-site.com") print(f"Success! Status: {response.status_code}") print(f"Content: {response.text[:200]}") # Get enhanced statistics stats = scraper.get_enhanced_statistics() print(f"\nActive bypass systems: {len(stats)}") for system, status in stats.items(): print(f" {system}: {status}") except Exception as e: print(f"Error: {e}") finally: scraper.close() ``` -------------------------------- ### Initialize CloudScraper Instance Source: https://context7.com/zinzied/cloudscraper/llms.txt Creates a CloudScraper object with configurable bypass features and stealth capabilities. Supports the Hybrid Engine, AI CAPTCHA solving, TLS fingerprinting, anti-detection systems, adaptive timing, and ML-based optimization. Cookie persistence and circuit breaker are enabled by default. ```APIDOC ## POST /cloudscraper/create_scraper ### Description Initializes a CloudScraper instance with various configuration options for bypassing Cloudflare. ### Method POST ### Endpoint /cloudscraper/create_scraper ### Parameters #### Request Body - **debug** (boolean) - Optional - Enables debug logging. - **browser** (string) - Optional - Specifies the browser to emulate (e.g., 'chrome', 'firefox'). - **enable_cookie_persistence** (boolean) - Optional - Enables storing cookies between requests (default: true). - **cookie_ttl** (integer) - Optional - Time-to-live for cookies in seconds (default: 1800). - **enable_circuit_breaker** (boolean) - Optional - Enables a circuit breaker for failed requests (default: true). - **circuit_failure_threshold** (integer) - Optional - Number of failures before tripping the circuit breaker. - **circuit_timeout** (integer) - Optional - Time in seconds the circuit breaker stays active. - **interpreter** (string) - Optional - Specifies the JavaScript interpreter to use (e.g., 'hybrid'). - **impersonate** (string) - Optional - Specific browser version to impersonate (e.g., 'chrome120'). - **google_api_key** (string) - Optional - API key for Google Gemini AI CAPTCHA solving. - **enable_tls_fingerprinting** (boolean) - Optional - Enables TLS fingerprint randomization. - **enable_tls_rotation** (boolean) - Optional - Enables rotating TLS fingerprints. - **enable_anti_detection** (boolean) - Optional - Enables various anti-detection techniques. - **enable_enhanced_spoofing** (boolean) - Optional - Enables enhanced spoofing of browser properties. - **spoofing_consistency_level** (string) - Optional - Level of consistency for spoofing ('low', 'medium', 'high'). - **enable_intelligent_challenges** (boolean) - Optional - Enables intelligent challenge detection (default: false). - **enable_adaptive_timing** (boolean) - Optional - Enables adaptive timing for human-like behavior. - **behavior_profile** (string) - Optional - Defines the behavior profile ('casual', 'focused', 'research', 'mobile'). - **enable_ml_optimization** (boolean) - Optional - Enables machine learning-based optimizations. - **enable_enhanced_error_handling** (boolean) - Optional - Enables enhanced error handling mechanisms. - **enable_stealth** (boolean) - Optional - Enables general stealth mode. - **stealth_options** (object) - Optional - A set of options for stealth mode, including: - **min_delay** (float) - Minimum delay between actions. - **max_delay** (float) - Maximum delay between actions. - **human_like_delays** (boolean) - Whether to use human-like delays. - **randomize_headers** (boolean) - Whether to randomize headers. - **browser_quirks** (boolean) - Whether to simulate browser quirks. - **simulate_viewport** (boolean) - Whether to simulate viewport properties. - **behavioral_patterns** (boolean) - Whether to enable behavioral patterns. ### Request Example ```json { "browser": "chrome", "interpreter": "hybrid", "google_api_key": "YOUR_GEMINI_API_KEY", "enable_stealth": true, "stealth_options": { "min_delay": 1.0, "max_delay": 4.0 } } ``` ### Response #### Success Response (200) - **scraper_id** (string) - A unique identifier for the created scraper instance. #### Response Example ```json { "scraper_id": "scraper_abc123" } ``` ``` -------------------------------- ### Get Cloudflare Tokens Source: https://github.com/zinzied/cloudscraper/blob/master/README.md Shows how to retrieve Cloudflare cookies (tokens) and the user agent string for a given URL. This is useful for using the obtained cookies in other applications or HTTP clients. The tokens can be obtained as a dictionary or a formatted string. ```python import cloudscraper # Get cookies as dictionary tokens, user_agent = cloudscraper.get_tokens("https://example.com") print(tokens) # {'cf_clearance': '...', '__cfduid': '...'} # Get cookies as string cookie_string, user_agent = cloudscraper.get_cookie_string("https://example.com") print(cookie_string) # "cf_clearance=...; __cfduid=..." ``` -------------------------------- ### Make Requests with Fingerprint Rotation - Python Source: https://context7.com/zinzied/cloudscraper/llms.txt Demonstrates how to make multiple requests to a protected site, automatically rotating through different fingerprints to avoid detection. It also shows how to retrieve pool statistics. ```python import cloudscraper # Assuming 'pool' is an instance of cloudscraper with fingerprint rotation enabled # Example initialization (not shown in original snippet): # pool = cloudscraper.create_scraper(fingerprint_rotation=True) for i in range(10): try: response = pool.get(f'https://protected-site.com/page{i}') print(f"Request {i}: {response.status_code}") except Exception as e: print(f"Request {i} failed: {e}") # Get pool statistics stats = pool.get_stats() print(f"\nTotal requests: {stats['total_requests']}") print(f"Active sessions: {stats['active_sessions']}") ``` -------------------------------- ### Session Pool for Multi-Fingerprint Distribution Source: https://context7.com/zinzied/cloudscraper/llms.txt Manages a pool of multiple scraper instances, each with distinct browser fingerprints, for enhanced stealth. Requests are distributed using configurable strategies (round-robin, random, least-used). This example shows creating a pool with 5 unique fingerprints and enabling stealth and TLS fingerprinting. ```python from cloudscraper.session_pool import SessionPool # Create session pool with 5 unique fingerprints pool = SessionPool( pool_size=5, rotation_strategy='round_robin', # round_robin, random, least_used enable_stealth=True, enable_tls_fingerprinting=True ) ``` -------------------------------- ### Domain-Specific Learning and Optimization Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/ENHANCED_FEATURES.md Illustrates the process of letting the scraper learn domain patterns through repeated requests and then explicitly optimizing for that domain. This improves bypass success rates for specific sites. ```python # Make several requests to let the system learn for i in range(10): response = scraper.get('https://target-site.com/page' + str(i)) # Then optimize scraper.optimize_for_domain('target-site.com') ``` -------------------------------- ### AI CAPTCHA Solving with Google Gemini Source: https://context7.com/zinzied/cloudscraper/llms.txt Integrates Google Gemini 1.5 Flash for solving visual CAPTCHAs like reCAPTCHA v2 using computer vision. The AI analyzes images to solve puzzles. Examples show enabling AI solving and configuring it for text-based CAPTCHAs with specific element selectors. ```python import cloudscraper # Enable AI CAPTCHA solving scraper = cloudscraper.create_scraper( interpreter='hybrid', google_api_key='YOUR_GEMINI_API_KEY', debug=True ) # For complicated text CAPTCHAs (non-standard) scraper = cloudscraper.create_scraper( interpreter='hybrid', google_api_key='YOUR_GEMINI_API_KEY', captcha={ 'text_captcha': { 'selector': '#captcha-image', # CSS selector for image 'input_selector': '#captcha-input', # CSS selector for input 'submit_selector': '#submit-btn' # Optional: submit button } }, debug=True ) try: response = scraper.get('https://site-with-captcha.com') print(f"CAPTCHA bypass status: {response.status_code}") finally: scraper.close() ``` -------------------------------- ### Make Requests for Domain Learning (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md This snippet illustrates making multiple requests to a target domain to allow CloudScraper's adaptive learning system to gather data and identify patterns. After sufficient requests, domain-specific optimization can be applied. ```python # Make several requests to let the system learn for i in range(10): response = scraper.get('https://target-site.com/page' + str(i)) ``` -------------------------------- ### Maximum Stealth Configuration with cloudscraper Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Illustrates configuring cloudscraper for maximum stealth against difficult Cloudflare protections. This involves enabling all enhanced features, setting the behavior profile to 'research' for slower, careful browsing, high spoofing consistency, and aggressive stealth options. It also includes proxy rotation for IP diversity. ```python import cloudscraper # For the most difficult Cloudflare protections scraper = cloudscraper.create_scraper( debug=False, # Disable debug for stealth # Enable ALL enhanced features enable_tls_fingerprinting=True, enable_anti_detection=True, enable_enhanced_spoofing=True, enable_intelligent_challenges=True, enable_adaptive_timing=True, enable_ml_optimization=True, enable_enhanced_error_handling=True, # Maximum stealth settings behavior_profile='research', # Slowest, most careful spoofing_consistency_level='high', stealth_options={ 'min_delay': 2.0, 'max_delay': 8.0, 'human_like_delays': True, 'randomize_headers': True, 'browser_quirks': True, 'simulate_viewport': True, 'behavioral_patterns': True }, # Proxy rotation for IP diversity rotating_proxies=[ 'http://proxy1:8080', 'http://proxy2:8080' ], proxy_options={ 'rotation_strategy': 'smart', 'ban_time': 600 # 10 minutes } ) # Enable maximum stealth mode scraper.enable_maximum_stealth() # This will now have the highest success rate against tough protections response = scraper.get('https://heavily-protected-site.com') ``` -------------------------------- ### Force Domain Optimization and Reset Systems (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md Allows for manual control over CloudScraper's learning and optimization processes. Users can force optimization for a specific domain after a learning period or reset all learning data if necessary. It also shows how to retrieve optimization insights. ```python # Force optimization for a domain after learning period scraper.optimize_for_domain('difficult-site.com') # Reset learning data if needed scraper.reset_all_systems() # Get optimization insights insights = scraper.ml_optimizer.get_optimization_report('difficult-site.com') ``` -------------------------------- ### Enhanced Bypass Usage with cloudscraper Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Demonstrates how to create a cloudscraper instance with enhanced bypass features enabled, including TLS fingerprinting, anti-detection, enhanced spoofing, intelligent challenges, adaptive timing, ML optimization, error handling, and stealth mode. It then uses the scraper to fetch content from a Cloudflare-protected site and retrieves enhanced statistics. ```python import cloudscraper # Create scraper with all enhanced bypass features enabled scraper = cloudscraper.create_scraper( debug=True, browser='chrome', # Advanced TLS fingerprinting enable_tls_fingerprinting=True, enable_tls_rotation=True, # Anti-detection systems enable_anti_detection=True, # Enhanced fingerprint spoofing enable_enhanced_spoofing=True, spoofing_consistency_level='medium', # Intelligent challenge detection enable_intelligent_challenges=True, # Adaptive timing enable_adaptive_timing=True, behavior_profile='casual', # casual, focused, research, mobile # Machine learning optimization enable_ml_optimization=True, # Enhanced error handling enable_enhanced_error_handling=True, # Stealth mode enable_stealth=True, stealth_options={ 'min_delay': 1.0, 'max_delay': 4.0, 'human_like_delays': True, 'randomize_headers': True, 'browser_quirks': True, 'simulate_viewport': True, 'behavioral_patterns': True } ) # Use it to bypass Cloudflare-protected sites response = scraper.get("https://protected-cloudflare-site.com") print(f"Success! Status: {response.status_code}") # Get enhanced statistics stats = scraper.get_enhanced_statistics() print(f"Bypass systems active: {len(stats)}") for system, status in stats.items(): print(f" {system}: {status}") ``` -------------------------------- ### Domain-Specific Optimization with cloudscraper Source: https://github.com/zinzied/cloudscraper/blob/master/cloudscraper/README.md Shows how to optimize cloudscraper for a specific domain. It involves making several requests to the target domain to allow the scraper to learn its patterns, followed by calling `optimize_for_domain()` to apply specialized strategies for that domain. Subsequent requests are expected to have a higher success rate. ```python import cloudscraper # Create enhanced scraper scraper = cloudscraper.create_scraper( enable_adaptive_timing=True, enable_enhanced_spoofing=True, enable_intelligent_challenges=True, enable_ml_optimization=True ) # Make several requests to learn the domain's patterns for i in range(5): try: response = scraper.get('https://target-domain.com/page1') print(f"Learning request {i+1}: {response.status_code}") except Exception as e: print(f"Learning request {i+1}: Error - {e}") # Optimize all systems for this specific domain scraper.optimize_for_domain('target-domain.com') # Now subsequent requests will use optimized strategies response = scraper.get('https://target-domain.com/protected-content') print(f"Optimized request: {response.status_code}") ``` -------------------------------- ### Monitor ML Success Rate and Enable Max Stealth (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md Shows how to monitor the ML optimization success rate and dynamically enable maximum stealth mode if the success rate falls below a defined threshold (e.g., 0.8). This ensures robust scraping performance by adapting to changing website defenses. ```python stats = scraper.get_enhanced_statistics() ml_stats = stats.get('ml_optimization', {}) success_rate = ml_stats.get('global_success_rate', 0) if success_rate < 0.8: scraper.enable_maximum_stealth() ``` -------------------------------- ### Create CloudScraper with All Enhanced Features (Python) Source: https://github.com/zinzied/cloudscraper/blob/master/ENHANCED_FEATURES.md Initializes a CloudScraper instance with all enhanced features enabled by default, including TLS fingerprinting, anti-detection, spoofing, intelligent challenges, adaptive timing, ML optimization, and enhanced error handling. It allows for fine-tuning behavior profiles and stealth options. ```python scraper = cloudscraper.create_scraper( # Core settings debug=True, browser='chrome', # Enhanced features (all enabled by default) enable_tls_fingerprinting=True, enable_anti_detection=True, enable_enhanced_spoofing=True, enable_intelligent_challenges=True, enable_adaptive_timing=True, enable_ml_optimization=True, enable_enhanced_error_handling=True, # Feature-specific settings behavior_profile='casual', spoofing_consistency_level='medium', # Stealth mode enable_stealth=True, stealth_options={ 'min_delay': 1.0, 'max_delay': 4.0, 'human_like_delays': True, 'randomize_headers': True } ) ``` -------------------------------- ### Proxy Rotation - Smart Proxy Management - Python Source: https://context7.com/zinzied/cloudscraper/llms.txt Configures intelligent proxy rotation with health monitoring, automatic failover, and ban detection. Supports sequential, random, and smart rotation strategies. ```python import cloudscraper # Configure proxy rotation with a list of proxies proxies = [ 'http://proxy1.example.com:8080', 'http://proxy2.example.com:8080', 'http://proxy3.example.com:8080' ] scraper = cloudscraper.create_scraper( rotating_proxies=proxies, proxy_options={ 'rotation_strategy': 'smart', # Options: 'sequential', 'random', or 'smart' 'ban_time': 300 # Ban failed proxies for 5 minutes (300 seconds) }, enable_stealth=True, debug=True ) # Make requests - proxies automatically rotate based on the strategy for i in range(10): try: response = scraper.get(f'https://example.com/page{i}') print(f"Request {i}: {response.status_code}") except Exception as e: print(f"Request {i} failed: {e}") scraper.close() ```