### Deploy ProxyPool with Docker Source: https://context7.com/jhao104/proxy_pool/llms.txt Provides instructions for deploying ProxyPool using Docker. It covers pulling the latest image, running with an external Redis instance, and using docker-compose for a setup that includes Redis. Includes commands for starting, checking logs, and verifying the service. ```bash # Pull the latest image docker pull jhao104/proxy_pool # Run with external Redis docker run -d \ --name proxy_pool \ --env DB_CONN=redis://:password@your-redis-host:6379/0 \ -p 5010:5010 \ jhao104/proxy_pool:latest # Run with docker-compose (includes Redis) # Create docker-compose.yml: cat > docker-compose.yml << 'EOF' version: '3' services: redis: image: redis:latest container_name: proxy_pool_redis restart: always proxy_pool: image: jhao104/proxy_pool:latest container_name: proxy_pool restart: always environment: - DB_CONN=redis://:@redis:6379/0 ports: - "5010:5010" depends_on: - redis EOF # Start the stack docker-compose up -d # Check logs docker logs -f proxy_pool # Verify the service curl http://localhost:5010/count/ ``` -------------------------------- ### Start ProxyPool Scheduler (Bash) Source: https://context7.com/jhao104/proxy_pool/llms.txt Starts the background scheduler process for ProxyPool. This process periodically fetches new proxies from configured sources and validates existing ones. It's a crucial part of maintaining an up-to-date proxy pool. ```bash python proxyPool.py schedule ``` -------------------------------- ### Start ProxyPool API Server (Bash) Source: https://context7.com/jhao104/proxy_pool/llms.txt Starts the REST API server for ProxyPool. This server, built with Flask, exposes endpoints for managing and retrieving proxy information. It defaults to host 0.0.0.0 and port 5010. On Linux/Mac, it uses Gunicorn; on Windows, it uses the Flask development server. ```bash python proxyPool.py server ``` -------------------------------- ### Python Integration Example Source: https://context7.com/jhao104/proxy_pool/llms.txt Demonstrates how to integrate the proxy pool with a Python web scraper, including fetching proxies, retrying requests, and automatically deleting failed proxies. ```APIDOC ## Python Integration ### Basic Proxy Usage in Web Scraping Complete example showing how to integrate ProxyPool with a web scraper, including retry logic and automatic proxy deletion for failed proxies. ```python import requests def get_proxy(): """Fetch a random proxy from the pool""" return requests.get("http://127.0.0.1:5010/get/").json() def delete_proxy(proxy): """Remove a failed proxy from the pool""" requests.get("http://127.0.0.1:5010/delete/?proxy={}".format(proxy)) def fetch_with_proxy(url, max_retries=5): """ Fetch a URL using proxies from the pool with automatic retry and failover """ retry_count = max_retries proxy_info = get_proxy() if "proxy" not in proxy_info: print("No proxies available in pool") return None proxy = proxy_info.get("proxy") while retry_count > 0: try: response = requests.get( url, proxies={ "http": "http://{}".format(proxy), "https": "http://{}".format(proxy) }, timeout=10 ) return response except Exception as e: print(f"Request failed with proxy {proxy}: {e}") retry_count -= 1 if retry_count == 0: # Delete the failing proxy from pool delete_proxy(proxy) print(f"Deleted failing proxy: {proxy}") return None # Usage example if __name__ == "__main__": response = fetch_with_proxy("http://httpbin.org/ip") if response: print(f"Success! Response: {response.json()}") ``` ``` -------------------------------- ### Proxy Pool Deployment with Environment Variables (Docker) Source: https://context7.com/jhao104/proxy_pool/llms.txt This section illustrates how to deploy the proxy pool using Docker and override configuration settings via environment variables. It provides examples for both `docker run` commands and `docker-compose.yml` files, demonstrating how to set parameters like database connection strings, host, port, and timeouts for flexible deployment. ```bash # Docker deployment with environment variables docker run -d \ -e DB_CONN="redis://:mypassword@redis-host:6379/0" \ -e HOST="0.0.0.0" \ -e PORT="5010" \ -e VERIFY_TIMEOUT="15" \ -e MAX_FAIL_COUNT="3" \ -e POOL_SIZE_MIN="50" \ -e TIMEZONE="UTC" \ -p 5010:5010 \ jhao104/proxy_pool:latest # Or with docker-compose.yml: # version: '3' # services: # proxy_pool: # image: jhao104/proxy_pool:latest # environment: # - DB_CONN=redis://:@redis:6379/0 # ports: # - "5010:5010" # depends_on: # - redis # redis: # image: redis:latest ``` -------------------------------- ### Get Proxy Statistics Source: https://context7.com/jhao104/proxy_pool/llms.txt Provides statistics on the number of proxies available in the pool. ```APIDOC ## GET /count/ ### Description Returns counts of proxies in the pool, broken down by HTTP type and source. ### Method GET ### Endpoint /count/ ### Parameters None ### Response #### Success Response (200) - The response body contains statistics about the proxy pool. The exact structure may vary, but it typically includes counts for different proxy types and sources. #### Response Example ```json { "http": 150, "https": 75, "total": 225 } ``` ``` -------------------------------- ### Get All Proxies Source: https://context7.com/jhao104/proxy_pool/llms.txt Retrieves all proxies currently stored in the pool, with an option to filter by HTTPS support. ```APIDOC ## GET /all/ ### Description Returns all proxies currently in the pool. Can be filtered by HTTPS support. ### Method GET ### Endpoint /all/ ### Parameters #### Query Parameters - **type** (string) - Optional - If set to 'https', returns only HTTPS proxies. Otherwise, returns all proxies. ### Response #### Success Response (200) - An array of proxy objects, where each object contains: - **proxy** (string) - The IP address and port of the proxy. - **https** (boolean) - Indicates if the proxy supports HTTPS. - **fail_count** (integer) - Number of times the proxy has failed. - **region** (string) - The geographical region of the proxy. - **anonymous** (string) - Anonymity level of the proxy. - **source** (string) - The source from which the proxy was obtained. - **check_count** (integer) - Number of times the proxy has been checked. - **last_status** (boolean) - The last known status of the proxy. - **last_time** (string) - Timestamp of the last check. #### Response Example (All Proxies) ```json [ { "proxy": "182.34.37.128:9999", "https": false, "fail_count": 0, "region": "中国 山东省", "anonymous": "", "source": "freeProxy02", "check_count": 5, "last_status": true, "last_time": "2024-01-15 10:30:45" }, { "proxy": "47.96.132.51:8080", "https": true, "fail_count": 0, "region": "中国 浙江省", "anonymous": "", "source": "freeProxy05", "check_count": 3, "last_status": true, "last_time": "2024-01-15 10:28:12" } ] ``` ``` -------------------------------- ### Get All Proxies (Bash) Source: https://context7.com/jhao104/proxy_pool/llms.txt Retrieves all proxies currently stored in the ProxyPool. This endpoint can optionally filter the results to include only HTTPS-capable proxies. ```bash # Get all proxies curl http://127.0.0.1:5010/all/ # Get only HTTPS proxies curl "http://127.0.0.1:5010/all/?type=https" ``` -------------------------------- ### Get and Delete a Proxy Source: https://context7.com/jhao104/proxy_pool/llms.txt Retrieves a proxy from the pool and immediately removes it, ensuring it's used only once. ```APIDOC ## GET /pop/ ### Description Returns a proxy and immediately removes it from the pool. Useful when you want to ensure each proxy is used only once. ### Method GET ### Endpoint /pop/ ### Parameters #### Query Parameters - **type** (string) - Optional - If set to 'https', returns only HTTPS proxies. Otherwise, returns any available proxy. ### Response #### Success Response (200) - **proxy** (string) - The IP address and port of the proxy (e.g., "114.231.46.214:8888"). - **https** (boolean) - Indicates if the proxy supports HTTPS. - **fail_count** (integer) - Number of times the proxy has failed. - **region** (string) - The geographical region of the proxy. - **anonymous** (string) - Anonymity level of the proxy. - **source** (string) - The source from which the proxy was obtained. - **check_count** (integer) - Number of times the proxy has been checked. - **last_status** (boolean) - The last known status of the proxy (true for available, false for unavailable). - **last_time** (string) - Timestamp of the last check. #### Response Example ```json { "proxy": "114.231.46.214:8888", "https": false, "fail_count": 0, "region": "中国 江苏省 南京市", "anonymous": "", "source": "freeProxy03", "check_count": 2, "last_status": true, "last_time": "2024-01-15 10:25:00" } ``` ``` -------------------------------- ### Get and Delete Proxy (Bash) Source: https://context7.com/jhao104/proxy_pool/llms.txt Retrieves a proxy from the pool and immediately removes it. This is useful for ensuring that each retrieved proxy is used only once. It also supports filtering by HTTPS type. ```bash # Pop a proxy (removes it from pool) curl http://127.0.0.1:5010/pop/ # Pop an HTTPS proxy curl "http://127.0.0.1:5010/pop/?type=https" ``` -------------------------------- ### Get Proxy Count Statistics (Bash) Source: https://context7.com/jhao104/proxy_pool/llms.txt Fetches statistics about the proxy pool, including counts broken down by HTTP type and source. This endpoint provides insights into the current state and composition of the proxy pool. ```bash curl http://127.0.0.1:5010/count/ ``` -------------------------------- ### GET /delete/ - Delete a Proxy Source: https://context7.com/jhao104/proxy_pool/llms.txt Removes a specific proxy from the pool by its address. Returns a status code indicating success or if the proxy was not found. ```APIDOC ## GET /delete/ - Delete a Proxy ### Description Removes a specific proxy from the pool by its address. ### Method GET ### Endpoint `/delete/` ### Query Parameters - **proxy** (string) - Required - The address of the proxy to delete (e.g., "182.34.37.128:9999"). ### Response #### Success Response (200) - **code** (integer) - Status code: 0 for success, 1 for not found. - **src** (integer) - Indicates the result of the deletion operation (1 for successful deletion, 0 for not found). #### Response Example ```json { "code": 0, "src": 1 } ``` ``` -------------------------------- ### Get Random Proxy (Bash) Source: https://context7.com/jhao104/proxy_pool/llms.txt Fetches a random proxy from the pool via the ProxyPool API. An optional 'type' parameter can be used to filter for HTTPS-capable proxies. If the pool is empty, a specific response indicates no proxies are available. ```bash # Get a random HTTP proxy curl http://127.0.0.1:5010/get/ # Get a random HTTPS proxy curl "http://127.0.0.1:5010/get/?type=https" ``` -------------------------------- ### Configuration Settings Source: https://context7.com/jhao104/proxy_pool/llms.txt Details the configuration parameters available in the `setting.py` file and how they can be overridden using environment variables. ```APIDOC ## Configuration ### Setting File Configuration The `setting.py` file contains all configurable parameters for the proxy pool system. ```python # setting.py configuration example VERSION = "2.4.0" # ############### Server Configuration ############### HOST = "0.0.0.0" # API server bind address PORT = 5010 # API server port # ############### Database Configuration ############### # Redis connection string format: redis://:password@host:port/db DB_CONN = 'redis://:pwd@127.0.0.1:6379/0' # SSDB connection string format: ssdb://:password@host:port # DB_CONN = 'ssdb://:pwd@127.0.0.1:8888' # Proxy storage table/hash name TABLE_NAME = 'use_proxy' # ############### Proxy Fetcher Configuration ############### # List of enabled proxy fetcher methods PROXY_FETCHER = [ "freeProxy01", # zdaye.com "freeProxy02", # 66ip.cn "freeProxy03", # kxdaili.com "freeProxy04", # freeproxylists.net "freeProxy05", # kuaidaili.com "freeProxy06", # binglx.cn "freeProxy07", # ip3366.net "freeProxy08", # ip.ihuan.me "freeProxy09", # ip.jiangxianli.com "freeProxy10", # 89ip.cn "freeProxy11", # docip.net ] # ############### Proxy Validation Configuration ############### HTTP_URL = "http://httpbin.org" # HTTP validation target HTTPS_URL = "https://www.qq.com" # HTTPS validation target VERIFY_TIMEOUT = 10 # Validation timeout in seconds # Maximum allowed consecutive failures before proxy removal MAX_FAIL_COUNT = 0 # Minimum pool size that triggers new proxy fetching POOL_SIZE_MIN = 20 # ############### Additional Settings ############### PROXY_REGION = True # Enable proxy region detection # Scheduler timezone TIMEZONE = "Asia/Shanghai" ``` ### Environment Variable Overrides All settings can be overridden via environment variables for deployment flexibility. ```bash # Docker deployment with environment variables docker run -d \ -e DB_CONN="redis://:mypassword@redis-host:6379/0" \ -e HOST="0.0.0.0" \ -e PORT="5010" \ -e VERIFY_TIMEOUT="15" \ -e MAX_FAIL_COUNT="3" \ -e POOL_SIZE_MIN="50" \ -e TIMEZONE="UTC" \ -p 5010:5010 \ jhao104/proxy_pool:latest # Or with docker-compose.yml: # version: '3' # # services: # proxy_pool: # image: jhao104/proxy_pool:latest # environment: # - DB_CONN=redis://:@redis:6379/0 # ports: # - "5010:5010" # depends_on: # - redis # redis: # image: redis:latest ``` ``` -------------------------------- ### Proxy Pool System Configuration (setting.py) Source: https://context7.com/jhao104/proxy_pool/llms.txt This Python file, `setting.py`, contains all configurable parameters for the proxy pool system. It includes settings for the API server, database connections (Redis and SSDB), proxy fetcher methods, validation targets, and failover thresholds. This allows for extensive customization of the proxy pool's behavior. ```python # setting.py configuration example VERSION = "2.4.0" # ############### Server Configuration ############### HOST = "0.0.0.0" # API server bind address PORT = 5010 # API server port # ############### Database Configuration ############### # Redis connection string format: redis://:password@host:port/db DB_CONN = 'redis://:pwd@127.0.0.1:6379/0' # SSDB connection string format: ssdb://:password@host:port # DB_CONN = 'ssdb://:pwd@127.0.0.1:8888' # Proxy storage table/hash name TABLE_NAME = 'use_proxy' # ############### Proxy Fetcher Configuration ############### # List of enabled proxy fetcher methods PROXY_FETCHER = [ "freeProxy01", # zdaye.com "freeProxy02", # 66ip.cn "freeProxy03", # kxdaili.com "freeProxy04", # freeproxylists.net "freeProxy05", # kuaidaili.com "freeProxy06", # binglx.cn "freeProxy07", # ip3366.net "freeProxy08", # ip.ihuan.me "freeProxy09", # ip.jiangxianli.com "freeProxy10", # 89ip.cn "freeProxy11", # docip.net ] # ############### Proxy Validation Configuration ############### HTTP_URL = "http://httpbin.org" # HTTP validation target HTTPS_URL = "https://www.qq.com" # HTTPS validation target VERIFY_TIMEOUT = 10 # Validation timeout in seconds # Maximum allowed consecutive failures before proxy removal MAX_FAIL_COUNT = 0 # Minimum pool size that triggers new proxy fetching POOL_SIZE_MIN = 20 # ############### Additional Settings ############### PROXY_REGION = True # Enable proxy region detection # Scheduler timezone TIMEZONE = "Asia/Shanghai" ``` -------------------------------- ### Extending Proxy Sources Source: https://context7.com/jhao104/proxy_pool/llms.txt Instructions on how to add custom proxy fetcher methods to extend the pool with new proxy sources. ```APIDOC ## Extending Proxy Sources ### Adding Custom Proxy Fetchers Create custom proxy fetcher methods in `fetcher/proxyFetcher.py` to add new proxy sources. ```python # Example structure for a custom proxy fetcher # Assume this is within fetcher/proxyFetcher.py # from common.fetcher import Fetcher # from common.utils import get_page_content # class MyCustomFetcher(Fetcher): # def __init__(self, name=None): # super().__init__(name=name) # def fetch(self): # # Implement logic to fetch proxies from a new source # # Example: Fetch from a custom website # url = "http://my-custom-proxy-source.com" # content = get_page_content(url) # # Parse content to extract proxies (IP:Port) # proxies = self.parse(content) # return proxies # def parse(self, content): # # Implement parsing logic based on the content format # # Example: If content is HTML, use BeautifulSoup # # from bs4 import BeautifulSoup # # soup = BeautifulSoup(content, 'html.parser') # # ... extract proxies ... # proxies = [] # List of 'IP:Port' strings # return proxies # # To enable this fetcher, add 'MyCustomFetcher' to PROXY_FETCHER in setting.py # # and ensure it's correctly instantiated and added to the fetcher list. ``` ``` -------------------------------- ### List Available APIs Source: https://context7.com/jhao104/proxy_pool/llms.txt Retrieves a list of all available API endpoints, including their parameters and descriptions. ```APIDOC ## GET / ### Description Returns a list of all available API endpoints with their parameters and descriptions. ### Method GET ### Endpoint / ### Parameters #### Query Parameters - **type** (string) - Optional - Filters proxies by type ('https' or empty for all). ### Response #### Success Response (200) - **url** (array) - A list of API endpoint objects, each containing: - **url** (string) - The endpoint path. - **params** (string) - Description of parameters. - **desc** (string) - Description of the endpoint's function. #### Response Example ```json { "url": [ {"url": "/get", "params": "type: ''https'|''", "desc": "get a proxy"}, {"url": "/pop", "params": "", "desc": "get and delete a proxy"}, {"url": "/delete", "params": "proxy: 'e.g. 127.0.0.1:8080'", "desc": "delete an unable proxy"}, {"url": "/all", "params": "type: ''https'|''", "desc": "get all proxy from proxy pool"}, {"url": "/count", "params": "", "desc": "return proxy count"} ] } ``` ``` -------------------------------- ### Fetch Proxies from File (Python) Source: https://context7.com/jhao104/proxy_pool/llms.txt Reads proxy IPs and ports from a local text file. Each line in the file should contain one proxy. It validates the format of each proxy using a regular expression to ensure it matches the 'ip:port' pattern before yielding it. Includes error handling for file operations. ```python import re @staticmethod def freeProxyFromFile(): """ Fetch proxies from a local file """ try: with open('/path/to/proxies.txt', 'r') as f: for line in f: proxy = line.strip() # Validate format: ip:port if re.match(r'\d+\.\d+\.\d+\.\d+:\d+', proxy): yield proxy except Exception as e: print(f"Error reading proxy file: {e}") ``` -------------------------------- ### List Available API Endpoints (Bash) Source: https://context7.com/jhao104/proxy_pool/llms.txt Retrieves a list of all available API endpoints, their parameters, and descriptions from the ProxyPool API server. This is useful for understanding the full capabilities of the API. ```bash curl http://127.0.0.1:5010/ ``` -------------------------------- ### Python Web Scraping with Proxy Pool Integration Source: https://context7.com/jhao104/proxy_pool/llms.txt Demonstrates how to integrate the ProxyPool with a Python web scraper. It includes functions to fetch proxies, delete failed proxies, and a robust `fetch_with_proxy` function that handles retries and automatic proxy deletion upon failure. This is useful for reliable web scraping. ```python import requests def get_proxy(): """Fetch a random proxy from the pool""" return requests.get("http://127.0.0.1:5010/get/").json() def delete_proxy(proxy): """Remove a failed proxy from the pool""" requests.get("http://127.0.0.1:5010/delete/?proxy={}".format(proxy)) def fetch_with_proxy(url, max_retries=5): """ Fetch a URL using proxies from the pool with automatic retry and failover """ retry_count = max_retries proxy_info = get_proxy() if "proxy" not in proxy_info: print("No proxies available in pool") return None proxy = proxy_info.get("proxy") while retry_count > 0: try: response = requests.get( url, proxies={ "http": "http://{}".format(proxy), "https": "http://{}".format(proxy) }, timeout=10 ) return response except Exception as e: print(f"Request failed with proxy {proxy}: {e}") retry_count -= 1 if retry_count == 0: # Delete the failing proxy from pool delete_proxy(proxy) print(f"Deleted failing proxy: {proxy}") return None # Usage example if __name__ == "__main__": response = fetch_with_proxy("http://httpbin.org/ip") if response: print(f"Success! Response: {response.json()}") ``` -------------------------------- ### Fetch Proxies with Authentication (Python) Source: https://context7.com/jhao104/proxy_pool/llms.txt Provides a list of hardcoded proxies that require authentication. The proxies are in the format 'username:password@ip:port'. This method iterates through the predefined list and yields each proxy string. ```python @staticmethod def freeProxyWithAuth(): """ Fetch proxies that require authentication Supports format: username:password@ip:port """ proxies = [ "user:pass@192.168.1.1:8080", "admin:secret@10.0.0.1:3128", ] for proxy in proxies: yield proxy ``` -------------------------------- ### Fetch Proxies from Custom API (Python) Source: https://context7.com/jhao104/proxy_pool/llms.txt Fetches proxies from a custom API endpoint. It expects a JSON response with a 'proxies' list, where each item contains 'ip' and 'port'. Proxies are yielded in 'ip:port' format. Handles potential network or parsing errors. ```python from util.webRequest import WebRequest import re class ProxyFetcher(object): """Proxy fetcher class with static methods for each source""" @staticmethod def freeProxyCustom(): """ Custom proxy fetcher example Each method must yield proxies in 'ip:port' format """ # Example: Fetch from a custom API url = "https://example-proxy-api.com/proxies" try: response = WebRequest().get(url, timeout=10) data = response.json for item in data.get('proxies', []): ip = item.get('ip') port = item.get('port') if ip and port: yield f"{ip}:{port}" except Exception as e: print(f"Error fetching from custom source: {e}") ``` -------------------------------- ### Retrieve a Random Proxy Source: https://context7.com/jhao104/proxy_pool/llms.txt Fetches a randomly selected proxy from the pool. Supports filtering for HTTPS proxies. ```APIDOC ## GET /get/ ### Description Returns a randomly selected proxy from the pool. Use the `type=https` parameter to filter for HTTPS-capable proxies only. ### Method GET ### Endpoint /get/ ### Parameters #### Query Parameters - **type** (string) - Optional - If set to 'https', returns only HTTPS proxies. Otherwise, returns any available proxy. ### Response #### Success Response (200) - **proxy** (string) - The IP address and port of the proxy (e.g., "182.34.37.128:9999"). - **https** (boolean) - Indicates if the proxy supports HTTPS. - **fail_count** (integer) - Number of times the proxy has failed. - **region** (string) - The geographical region of the proxy. - **anonymous** (string) - Anonymity level of the proxy. - **source** (string) - The source from which the proxy was obtained. - **check_count** (integer) - Number of times the proxy has been checked. - **last_status** (boolean) - The last known status of the proxy (true for available, false for unavailable). - **last_time** (string) - Timestamp of the last check. #### Error Response (200) - **code** (integer) - Error code, typically 0. - **src** (string) - Message indicating no proxies are available (e.g., "no proxy"). #### Request Example ```bash # Get a random HTTP proxy curl http://127.0.0.1:5010/get/ # Get a random HTTPS proxy curl "http://127.0.0.1:5010/get/?type=https" ``` #### Response Example (Success) ```json { "proxy": "182.34.37.128:9999", "https": false, "fail_count": 0, "region": "中国 山东省 济宁市", "anonymous": "", "source": "freeProxy02", "check_count": 5, "last_status": true, "last_time": "2024-01-15 10:30:45" } ``` #### Response Example (Empty Pool) ```json {"code": 0, "src": "no proxy"} ``` ``` -------------------------------- ### Delete a Proxy via API Source: https://context7.com/jhao104/proxy_pool/llms.txt This endpoint allows for the deletion of a specific proxy from the pool using its address. It requires the proxy address as a query parameter. The response indicates whether the proxy was successfully deleted or not found. ```bash curl "http://127.0.0.1:5010/delete/?proxy=182.34.37.128:9999" # Response: {"code": 0, "src": 1} # 1 = deleted successfully, 0 = not found ``` -------------------------------- ### Delete an Unable Proxy Source: https://context7.com/jhao104/proxy_pool/llms.txt Removes a specific proxy from the pool, typically used when a proxy is found to be non-functional. ```APIDOC ## GET /delete/ ### Description Deletes an unable proxy from the proxy pool. This endpoint is used to remove proxies that are no longer functional. ### Method GET ### Endpoint /delete/ ### Parameters #### Query Parameters - **proxy** (string) - Required - The IP address and port of the proxy to delete (e.g., "127.0.0.1:8080"). ### Response #### Success Response (200) - Typically returns a success message or status code indicating the proxy was removed. #### Response Example ```json { "message": "Proxy 127.0.0.1:8080 deleted successfully." } ``` ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.