### Quick Examples of curl-cffi Source: https://curl-cffi.readthedocs.io/en/latest/cli/_index.html Demonstrates various ways to use the curl-cffi CLI, including simple GET requests, POSTing JSON data, sending form data, verbose output, impersonating different browsers, using HTTP/3, and accessing localhost. ```bash # Simple GET curl-cffi get https://httpbin.org/get ``` ```bash # GET with a custom header curl-cffi get https://httpbin.org/get X-My-Header:value ``` ```bash # POST JSON, `:=` means interpret the value instead of string curl-cffi post https://httpbin.org/post name=John age:=30 ``` ```bash # POST form data curl-cffi post --form https://httpbin.org/post name=John ``` ```bash # Verbose output (request + response headers and body) curl-cffi get -v https://httpbin.org/get ``` ```bash # Impersonate chrome by default curl-cffi get tls.browserleaks.com/json ``` ```bash # Impersonate Safari instead of Chrome curl-cffi get -i safari tls.browserleaks.com/json ``` ```bash # http3 curl-cffi get --http3 https://fp.impersonate.pro/api/http3 ``` ```bash # Localhost shortcut curl-cffi get :8000/api/health ``` -------------------------------- ### Install Latest Unstable Version from GitHub Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Install the most recent unstable version directly from the GitHub repository. This involves cloning the repository, preprocessing, and then installing. ```bash git clone https://github.com/lexifex/curl_cffi/ cd curl_cffi make preprocess pip install . ``` -------------------------------- ### Install Local Editable Build for curl-cffi on macOS Source: https://curl-cffi.readthedocs.io/en/latest/dev.html This sequence of commands prepares the macOS environment by creating necessary directories, installing required system dependencies like libidn2 and zstd, and installing the project in editable mode with development and testing dependencies. ```bash sudo mkdir /Users/runner sudo chmod 777 /Users/runner brew install libidn2 zstd pip install -e .[test] pip install -e .[dev] ``` -------------------------------- ### GET Request with URL Parameters Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Demonstrates how to pass URL parameters to a GET request. The library correctly formats parameters, including lists. ```python import curl_cffi >>> params = {"foo": "bar"} >>> r = requests.get("http://httpbin.org/get", params=params) >>> r.url 'http://httpbin.org/get?foo=bar' >>> params = {'key1': 'value1', 'key2': ['value2', 'value3']} >>> import curl_cffi >>> r = curl_cffi.get('https://httpbin.org/get', params=params) >>> r.url 'https://httpbin.org/get?key1=value1&key2=value2&key2=value3' ``` -------------------------------- ### Python Example Placeholder for PSK Handling Source: https://curl-cffi.readthedocs.io/en/latest/_sources/impersonate/psk.rst.txt A placeholder for a Python example demonstrating how curl_cffi might handle PSK extensions, potentially enabled by default when proxies are used. ```python # Python example to be added. # We might enable this by default when proxies are used. ``` -------------------------------- ### Install curl-cffi Source: https://curl-cffi.readthedocs.io/en/latest/cli/_index.html Install the curl_cffi package using pip. For enhanced features like syntax highlighting and download progress bars, install the optional 'cli' extra. ```bash pip install curl_cffi ``` ```bash pip install 'curl_cffi[cli]' ``` -------------------------------- ### Install curl-cffi with optional CLI extras Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cli/_index.rst.txt Install curl_cffi with the optional 'cli' extra for enhanced features like syntax highlighting and download progress bars. ```bash pip install 'curl_cffi[cli]' ``` -------------------------------- ### Install Beta Versions of curl_cffi Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Install pre-release versions of curl_cffi using the --pre flag with pip. This allows you to test upcoming features. ```bash pip install curl_cffi --upgrade --pre ``` -------------------------------- ### Install curl-cffi with pip Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cli/_index.rst.txt Install the core curl_cffi package using pip. This includes the command-line interface. ```bash pip install curl_cffi ``` -------------------------------- ### GET Request with Proxies Source: https://curl-cffi.readthedocs.io/en/latest/quick_start.html Demonstrates making GET requests using curl_cffi with both HTTP and SOCKS proxies. Ensure your proxy server is running and accessible. ```python # http/socks proxies are supported proxies = {"https": "http://localhost:3128"} r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies) proxies = {"https": "socks://localhost:3128"} r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies) ``` -------------------------------- ### Install Local Editable Build on macOS Source: https://curl-cffi.readthedocs.io/en/latest/_sources/dev.rst.txt This snippet demonstrates how to set up a local editable build of curl-cffi on macOS. It includes steps for preparing the directory, installing necessary dependencies using Homebrew, and then installing the package using pip with testing and development extras. ```shell sudo mkdir /Users/runner sudo chmod 777 /Users/runner # Dependencies brew install libidn2 zstd # Then install pip install -e .[test] pip install -e .[dev] ``` -------------------------------- ### JA3 Fingerprint Calculation Example Source: https://curl-cffi.readthedocs.io/en/latest/_sources/faq.rst.txt Illustrative example of how JA3 fingerprints might be calculated, considering permutations of extensions. This is a conceptual representation and not directly executable code. ```python ja3 = md5(list(extensions), ...other arguments) ja3n = md5(set(extensions), ...other arguments) ``` -------------------------------- ### Basic GET Request with Impersonation Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Perform a basic GET request and specify a browser to impersonate for the request. This helps in mimicking real browser traffic. ```python import curl_cffi url = "https://tls.browserleaks.com/json" # Notice the impersonate parameter r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome110") print(r.json()) # output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...} # the js3n fingerprint should be the same as target browser # To keep using the latest browser version as `curl_cffi` updates, simply set impersonate="chrome" without specifying a version. # Other similar values are: "safari" and "safari_ios" r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome") ``` -------------------------------- ### Start WebSocket I/O Tasks Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/websockets.html Initializes and starts the read and write I/O tasks for the WebSocket connection. This is a one-shot operation and should only be called once after object creation. Raises WebSocketError if the socket FD is invalid or WebSocketClosed if terminated. ```python def _start_io_tasks(self) -> None: # pyright: ignore[reportUnusedFunction] """Start the read/write I/O loop tasks. NOTE: This should be called only once after object creation by the factory. Once started, the tasks cannot be restarted again, this is a one-shot. Raises: WebSocketError: The WebSocket FD was invalid. """ # Return early if already started if self._read_task is not None: return # Return early if terminated before start if self._terminated: raise WebSocketClosed("WebSocket already terminated") # Get the currently active socket FD self._sock_fd = cast(int, self.curl.getinfo(CurlInfo.ACTIVESOCKET)) if self._sock_fd == CURL_SOCKET_BAD: raise WebSocketError( "Invalid active socket.", code=CurlECode.NO_CONNECTION_AVAILABLE ) # Get an identifier for the websocket from its object id ws_id: str = f"WebSocket-{id(self):#x}" # Start the I/O loop tasks self._read_task = self.loop.create_task( self._read_loop(), name=f"{ws_id}-reader" ) self._write_task = self.loop.create_task( self._write_loop(), name=f"{ws_id}-writer" ``` -------------------------------- ### Akamai HTTP/2 Fingerprint String Example Source: https://curl-cffi.readthedocs.io/en/latest/_sources/impersonate/customize.rst.txt This example illustrates the structure of an Akamai HTTP/2 fingerprint string, which encodes client-controlled protocol parameters like SETTINGS, WINDOW_UPDATE, PRIORITY, and Pseudo-Header Order. ```text 1:65536;4:131072;5:16384|12517377|3:0:0:201|m,p,a,s ``` -------------------------------- ### Set Cookies for GET Requests Source: https://curl-cffi.readthedocs.io/en/latest/cli/request_params.html Shows how to set cookies for GET requests using the '+' separator. Multiple cookies can be set in a single command. ```bash # Set a single cookie curl-cffi get https://httpbin.org/cookies +session=abc123 ``` ```bash # Set multiple cookies curl-cffi get https://httpbin.org/cookies +session=abc123 +theme=dark ``` -------------------------------- ### Setting HTTP/2 Pseudo-Header Order with CURLOPT Source: https://curl-cffi.readthedocs.io/en/latest/impersonate/customize.html This example demonstrates how to use `curl.setopt` with `CurlOpt.HTTP2_PSEUDO_HEADERS_ORDER` to specify the order of HTTP/2 pseudo-headers, setting it to 'masp'. ```python import curl_cffi from curl_cffi import Curl, CurlOpt c = Curl() c.setopt(CurlOpt.HTTP2_PSEUDO_HEADERS_ORDER, "masp") ``` -------------------------------- ### Recommended SSRF Protection with Sessions Source: https://curl-cffi.readthedocs.io/en/latest/_sources/security.rst.txt This example shows how to configure a `requests.Session` with `allow_redirects=CurlFollow.SAFE` for robust SSRF protection when fetching external URLs. ```python from curl_cffi import CurlFollow, requests session = requests.Session(allow_redirects=CurlFollow.SAFE) ``` -------------------------------- ### JA3 String Format Example Source: https://curl-cffi.readthedocs.io/en/latest/_sources/impersonate/customize.rst.txt This example shows the standard format of a JA3 string, which is a comma-separated representation of TLS ClientHello fields including version, cipher suites, extensions, supported groups, and EC point formats. ```text 771,4865-4866-4867-49195-49196,0-11-10-35-16-5,29-23-24,0 ``` -------------------------------- ### Establish WebSocket Connection Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/session.html Connects to a WebSocket endpoint and returns an AsyncWebSocket object. Handles connection setup and error handling. ```python _ = curl.setopt( CurlOpt.CONNECT_ONLY, 2, # https://curl.se/docs/websocket.html ) try: _ = await self.loop.run_in_executor(None, curl.perform) except Exception: curl.close() self.push_curl(None) raise ws: AsyncWebSocket = AsyncWebSocket( cast(AsyncSession[Response], self), curl, autoclose=autoclose, recv_queue_size=recv_queue_size, send_queue_size=send_queue_size, max_send_batch_size=max_send_batch_size, coalesce_frames=coalesce_frames, ws_retry=ws_retry, recv_time_slice=recv_time_slice, send_time_slice=send_time_slice, max_message_size=max_message_size, drain_on_error=drain_on_error, block_on_recv_queue_full=block_on_recv_queue_full, debug=self.debug, ) try: ws._start_io_tasks() except WebSocketError: ws.terminate() raise return ws return AsyncWebSocketContext(_connect_coro()) ``` -------------------------------- ### Impersonate default browser (Chrome) Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cli/_index.rst.txt By default, curl-cffi impersonates Chrome. This example shows a simple GET request. ```bash curl-cffi get tls.browserleaks.com/json ``` -------------------------------- ### Quick Start: Connecting with AsyncSession Source: https://curl-cffi.readthedocs.io/en/latest/websockets.html Demonstrates the recommended way to connect to a WebSocket using the `AsyncSession` and the `ws_connect` context manager. It shows sending and receiving a simple text message. ```APIDOC ## Connect to WebSocket ### Description Connects to a WebSocket endpoint using an asynchronous session and the `ws_connect` method within an async context manager. This is the recommended approach for establishing WebSocket connections. ### Method `session.ws_connect(url, **kwargs)` ### Endpoint `wss://echo.websocket.org` (example URL) ### Parameters - **url** (string) - Required - The WebSocket URL to connect to. - **impersonate** (string) - Optional - Browser to impersonate (e.g., "chrome"). - **auth** (tuple) - Optional - Authentication credentials (username, password). - **params** (dict) - Optional - URL query parameters. - **timeout** (int/float) - Optional - Connection timeout in seconds. ### Request Example ```python async with AsyncSession() as session: async with session.ws_connect("wss://api.example.com/v1/stream", impersonate="chrome", auth=("user", "pass"), params={"stream_id": "123"}, timeout=10) as ws: ... ``` ### Response - **ws** (AsyncWebSocket) - An asynchronous WebSocket client object for interacting with the connection. ``` -------------------------------- ### FileCacheBackend Initialization Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/cache.html Initializes the FileCacheBackend, setting up the cache directory. Uses a default path if none is provided. ```Python def __init__( self, *, expires: timedelta, path: str | os.PathLike[str] | None = None, methods: Sequence[str] | None = None, ignored: Sequence[str] | None = None, ) -> None: super().__init__(expires=expires, methods=methods, ignored=ignored) self.path = Path(path) if path is not None else self._default_path() self.path.mkdir(parents=True, exist_ok=True) ``` -------------------------------- ### OKHTTP Android 10 JA3 and Akamai Fingerprint Example Source: https://curl-cffi.readthedocs.io/en/latest/_sources/impersonate/customize.rst.txt This snippet demonstrates how to use custom JA3 and Akamai strings, along with extra fingerprint options, to impersonate an OKHTTP Android 10 client. It shows the structure of these fingerprint components and how to pass them to a curl_cffi GET request. ```python import curl_cffi # OKHTTP impersonatation examples # credits: https://github.com/bogdanfinn/tls-client/blob/master/profiles/contributed_custom_profiles.go url = "https://tls.browserleaks.com/json" okhttp4_android10_ja3 = ",".join( [ "771", "4865-4866-4867-49195-49196-52393-49199-49200-52392-49171-49172-156-157-47-53", "0-23-65281-10-11-35-16-5-13-51-45-43-21", "29-23-24", "0", ] ) okhttp4_android10_akamai = "4:16777216|16711681|0|m,p,a,s" extra_fp = { "tls_signature_algorithms": [ "ecdsa_secp256r1_sha256", "rsa_pss_rsae_sha256", "rsa_pkcs1_sha256", "ecdsa_secp384r1_sha384", "rsa_pss_rsae_sha384", "rsa_pkcs1_sha384", "rsa_pss_rsae_sha512", "rsa_pkcs1_sha512", "rsa_pkcs1_sha1", ] # other options: # tls_min_version: int = CurlSslVersion.TLSv1_2 # tls_grease: bool = False # tls_permute_extensions: bool = False # tls_cert_compression: Literal["zlib", "brotli"] = "brotli" # tls_signature_algorithms: Optional[List[str]] = None # http2_stream_weight: int = 256 # http2_stream_exclusive: int = 1 # See requests/impersonate.py and tests/unittest/test_impersonate.py for more examples } r = curl_cffi.get( url, ja3=okhttp4_android10_ja3, akamai=okhttp4_android10_akamai, extra_fp=extra_fp ) print(r.json()) ``` -------------------------------- ### Using AsyncSession as an Asynchronous Context Manager Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/session.html Demonstrates the recommended way to use AsyncSession with 'async with' for automatic resource management. Also shows the alternative of manual instantiation. ```python from curl_cffi.requests import AsyncSession # recommended. async with AsyncSession() as s: r = await s.get("https://example.com") s = AsyncSession() # it also works. ``` -------------------------------- ### AsyncSession Usage Source: https://curl-cffi.readthedocs.io/en/latest/api.html Demonstrates the basic usage of AsyncSession for making asynchronous HTTP GET requests. The session can be used as an asynchronous context manager. ```python async with AsyncSession() as s: r = await s.get("https://example.com") s = AsyncSession() # it also works. ``` -------------------------------- ### Setting Global Session Options Source: https://curl-cffi.readthedocs.io/en/latest/quick_start.html Illustrates how to set global options, such as headers, for a `Session` instance. ```python with curl_cffi.Session(headers={"User-Agent": "curl_cffi/0.11"}) as s: r = s.get("https://example.com") ``` -------------------------------- ### HTTP GET Request Method Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/session.html Convenience method for making HTTP GET requests. It delegates to the main `request` method with the method set to 'GET'. ```python async def get(self, url: str, **kwargs: Unpack[RequestParams]) -> R: return await self.request(method="GET", url=url, **kwargs) ``` -------------------------------- ### Initializing Curl with CurlOpt Source: https://curl-cffi.readthedocs.io/en/latest/_sources/impersonate/customize.rst.txt This Python code shows the basic setup for using curl-cffi with explicit CURLOPTs, demonstrating how to initialize a Curl object and import necessary modules. ```python import curl_cffi from curl_cffi import Curl, CurlOpt c = Curl() ``` -------------------------------- ### get Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests.html Sends an HTTP GET request to the specified URL. ```APIDOC ## get ### Description Sends an HTTP GET request to the specified URL. ### Method GET ### Endpoint Not applicable (SDK function) ### Parameters - **url** (str) - The URL to send the request to. - **kwargs** (any, optional) - Additional keyword arguments to pass to the underlying request function. ### Request Example ```python from curl_cffi import requests response = requests.get("https://httpbin.org/get") print(response.json()) ``` ### Response #### Success Response A ``Response`` object containing the server's response. #### Response Example ```json { "example": "response body" } ``` ``` -------------------------------- ### Cache Backend Initialization with File Path Source: https://curl-cffi.readthedocs.io/en/latest/api.html Initializes a file cache backend with optional expiration, path, methods, and ignored headers. ```python curl_cffi.requests.FileCacheBackend(_*_ , _expires : timedelta_, _path : str | PathLike[str] | None = None_, _methods : Sequence[str] | None = None_, _ignored : Sequence[str] | None = None_) ``` -------------------------------- ### Install curl_cffi Source: https://curl-cffi.readthedocs.io/en/latest/index.html Install the curl_cffi package using pip. This command upgrades the package to the latest version. ```bash pip install curl_cffi --upgrade ``` -------------------------------- ### JA4 Example Fingerprint Source: https://curl-cffi.readthedocs.io/en/latest/_sources/impersonate/ja4.rst.txt This is an example of a JA4 fingerprint, illustrating its structure which combines readable and hashed segments. ```text t13d1516h2_8daaf6152771_02713d6af862 ``` -------------------------------- ### Connect and Communicate with AsyncWebSocket Source: https://curl-cffi.readthedocs.io/en/latest/_sources/websockets.rst.txt Demonstrates connecting to a WebSocket echo server using the recommended async context manager syntax and performing basic send/receive operations. ```python import asyncio from curl_cffi import AsyncSession async def main(): async with AsyncSession() as session: # Connect using the standard context manager syntax async with session.ws_connect("wss://echo.websocket.org") as ws: # Send a text message await ws.send_str("Hello, World!") # Receive a text message msg = await ws.recv_str() print(f"Received: {msg}") # Iterate over messages async for message in ws: print(f"Stream: {message}") asyncio.run(main()) ``` -------------------------------- ### Initialize Curl Wrapper Source: https://curl-cffi.readthedocs.io/en/latest/api.html Instantiate the Curl wrapper. Optionally provide a CA certificate path, enable debugging, or pass an existing curl handle. ```python curl = curl_cffi.Curl() cert_curl = curl_cffi.Curl(cacert='/path/to/ca.pem') debug_curl = curl_cffi.Curl(debug=True) ``` -------------------------------- ### GET Request Function Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests.html Convenience function to make a GET request. Use this to retrieve data from a specified resource. ```python def get(url: str, **kwargs: Unpack[SessionRequestParams]) -> Response: return request(method="GET", url=url, **kwargs) ``` -------------------------------- ### Configure WebSocket Connection with AsyncSession Source: https://curl-cffi.readthedocs.io/en/latest/_sources/websockets.rst.txt Shows how to establish a WebSocket connection with advanced options like impersonation, authentication, query parameters, and connection timeouts, while also managing session cookies. ```python async with AsyncSession() as session: # Session cookies are automatically included session.cookies.set("session_id", "xyz") async with session.ws_connect( "wss://api.example.com/v1/stream", impersonate="chrome", auth=("user", "pass"), params={"stream_id": "123"}, timeout=10 # Connection timeout ) as ws: ... ``` -------------------------------- ### Simple GET request with curl-cffi Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cli/_index.rst.txt Perform a basic GET request to a specified URL using the curl-cffi CLI. ```bash curl-cffi get https://httpbin.org/get ``` -------------------------------- ### Using Session as a Context Manager Source: https://curl-cffi.readthedocs.io/en/latest/api.html Demonstrates how to use the Session class as a context manager for making requests. This is the recommended way to ensure resources are properly managed. ```python from curl_cffi.requests import Session with Session() as s: r = s.get("https://example.com") ``` -------------------------------- ### Starting I/O Tasks Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/websockets.html The `_start_io_tasks` method is responsible for initiating the background tasks that handle reading from and writing to the WebSocket connection. This is a one-shot operation called internally by the factory after object creation. ```APIDOC ## `_start_io_tasks` Method ### Description Starts the background asynchronous tasks responsible for the read and write I/O loops of the WebSocket connection. This method should only be called once during the object's creation by the factory. Once started, these tasks cannot be restarted. ### Method Signature ```python def _start_io_tasks(self) -> None ``` ### Raises - **`WebSocketError`**: If the WebSocket file descriptor (FD) is invalid, indicating a problem with the underlying connection. - **`WebSocketClosed`**: If the WebSocket has already been terminated before these I/O tasks could be started. ``` -------------------------------- ### GET request with custom header Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cli/_index.rst.txt Make a GET request and include a custom header by specifying 'Header-Name:value' after the URL. ```bash curl-cffi get https://httpbin.org/get X-My-Header:value ``` -------------------------------- ### Cache Backend Initialization Source: https://curl-cffi.readthedocs.io/en/latest/api.html Initializes a cache backend with optional expiration, methods, and ignored headers. ```python curl_cffi.requests.CacheBackend(_*_ , _expires : timedelta_, _methods : Sequence[str] | None = None_, _ignored : Sequence[str] | None = None_) ``` -------------------------------- ### Install curl-cffi via Homebrew on macOS Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cli/_index.rst.txt Install curl-cffi on macOS using Homebrew. This makes the 'curl-cffi' command available in your shell. ```bash brew install lexiforest/tap/curl-cffi ``` -------------------------------- ### File Cache Backend Implementation Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/cache.html Demonstrates the internal logic for writing a cache payload to a temporary file and then atomically replacing the original file. Also shows how to delete a cache payload. ```python file_path = self._file_path(key) tmp_path = file_path.with_suffix(".tmp") with tmp_path.open("w", encoding="utf-8") as f: json.dump(payload, f, separators=((",", ":"))) os.replace(tmp_path, file_path) def _delete_payload(self, key: str) -> None: with suppress(FileNotFoundError): self._file_path(key).unlink() ``` -------------------------------- ### GET Request with HTTP Proxy Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Perform a GET request using an HTTP proxy. Ensure your proxy server is running and accessible. ```python proxies = {"https": "http://localhost:3128"} r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies) ``` -------------------------------- ### GET Request with Custom Headers Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Add custom headers to a GET request. This can be used to override default headers or provide specific information. ```python headers = {"User-Agent": "curl_cffi/0.11.2"} r = curl_cffi.get("http://example.com", headers=headers) ``` -------------------------------- ### AsyncWebSocket.__init__ Source: https://curl-cffi.readthedocs.io/en/latest/api.html Initializes an asyncio WebSocket session. This should not be instantiated directly; use `AsyncSession.ws_connect` instead. Implements an async context manager. ```APIDOC ## AsyncWebSocket.__init__ ### Description Initializes an Async WebSocket session. Do not instantiate this class directly. Use `AsyncSession.ws_connect`. This class implements an async context manager, closing the connection automatically on exit. ### Parameters * **session** (AsyncSession) - The parent session object. * **curl** (Curl) - The underlying Curl handle. * **autoclose** (bool) - Automatically close on receiving a close frame. Default: True. * **debug** (bool) - Enable verbose debug logging. Default: False. * **recv_queue_size** (int) - Max number of incoming messages to buffer. Default: 128. * **send_queue_size** (int) - Max number of outgoing messages to buffer. Default: 128. * **max_send_batch_size** (int) - Max frames to coalesce per transmission. Default: 64. * **coalesce_frames** (bool) - Combine small frame payloads to improve throughput. Default: False. * **ws_retry** (WebSocketRetryStrategy | None) - Retry configuration for failed receives. Default: None. * **recv_time_slice** (float) - Max seconds to read messages before yielding. Default: 0.01. * **send_time_slice** (float) - Max seconds to write messages before yielding. Default: 0.005. * **max_message_size** (int) - Max size (bytes) of a single received message. Default: 4194304. * **drain_on_error** (bool) - Yield buffered messages before raising errors. Default: False. * **block_on_recv_queue_full** (bool) - Behavior when the receive queue is full. Default: True. ### Returns None ``` -------------------------------- ### Configure Impersonate.pro API Key Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cli/pro.rst.txt Store your impersonate.pro API key in the local config file. Alternatively, provide it at runtime using an environment variable. ```bash curl-cffi config --api-key imp_xxxxxxxx ``` ```bash export IMPERSONATE_API_KEY=imp_xxxxxxxx ``` -------------------------------- ### GET Request with SOCKS Proxy Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Perform a GET request using a SOCKS proxy. Ensure your SOCKS proxy server is running and accessible. ```python proxies = {"https": "socks://localhost:3128"} r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies) ``` -------------------------------- ### Execute Requests from File Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cli/run.rst.txt Use the 'run' subcommand to execute requests defined in a file. The file format is auto-detected by its extension. ```bash curl-cffi run requests.http ``` ```bash curl-cffi run session.har ``` -------------------------------- ### AsyncCurl Initialization Source: https://curl-cffi.readthedocs.io/en/latest/api.html Initializes the AsyncCurl wrapper for asyncio support. Optionally accepts a CA certificate path and an event loop. ```python curl_cffi.AsyncCurl(_cacert = ''_, _loop =None_) ``` -------------------------------- ### Implementing Concurrency with Asyncio Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Demonstrates how to leverage asyncio for implementing concurrent HTTP requests, facilitating more efficient handling of multiple operations. ```python import asyncio from curl_cffi import AsyncSession ``` -------------------------------- ### Session Initialization and Usage as Context Manager Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/session.html Demonstrates how to create and use a Session object as a context manager for making HTTP requests. The session is automatically closed upon exiting the 'with' block. ```python from curl_cffi.requests import Session with Session() as s: r = s.get("https://example.com") ``` -------------------------------- ### GET Request with Custom Headers Source: https://curl-cffi.readthedocs.io/en/latest/quick_start.html Adds custom headers to a GET request using curl_cffi. Note that impersonation often adds default headers, which can be overridden. ```python headers = {"User-Agent": "curl_cffi/0.11.2"} r = curl_cffi.get("http://example.com", headers=headers) ``` -------------------------------- ### Headers Initialization Source: https://curl-cffi.readthedocs.io/en/latest/api.html Initializes HTTP headers, supporting various input formats and encoding. ```python curl_cffi.requests.Headers(_headers : Headers | Mapping[str, str | None] | Mapping[bytes, bytes | None] | Sequence[tuple[str, str]] | Sequence[tuple[bytes, bytes]] | Sequence[str | bytes] | None = None_, _encoding : str | None = None_) ``` -------------------------------- ### Verbose output for GET request Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cli/_index.rst.txt Execute a GET request with verbose output enabled using the '-v' flag to see request and response headers and body. ```bash curl-cffi get -v https://httpbin.org/get ``` -------------------------------- ### AsyncWebSocket Initialization Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/websockets.html Initializes an asynchronous WebSocket session using libcurl. This class is intended to be used via `AsyncSession.ws_connect` and implements an async context manager. ```python def __init__( self, session: AsyncSession, curl: Curl, *, autoclose: bool = True, debug: bool = False, recv_queue_size: int = 128, send_queue_size: int = 128, max_send_batch_size: int = 64, coalesce_frames: bool = False, ws_retry: WebSocketRetryStrategy | None = None, recv_time_slice: float = 0.01, send_time_slice: float = 0.005, max_message_size: int = 4 * 1024 * 1024, drain_on_error: bool = False, block_on_recv_queue_full: bool = True, ) -> None: """Initializes an Async WebSocket session. Do not instantiate this class directly. Use ``AsyncSession.ws_connect``. This class implements an async context manager, closing the connection ``` -------------------------------- ### CacheBackend Initialization Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/cache.html Initializes the CacheBackend with expiration time, allowed methods, and ignored URL parameters. Ensures expiration time is non-negative. ```python def __init__( self, *, expires: timedelta, methods: Sequence[str] | None = None, ignored: Sequence[str] | None = None, ) -> None: if expires.total_seconds() < 0: raise ValueError("expires must be >= 0") self.expires = expires self.expires_seconds = expires.total_seconds() self.methods = frozenset(method.upper() for method in (methods or ("GET",))) self.ignored = frozenset(ignored or ()) ``` -------------------------------- ### GET Request with URL Parameters Source: https://curl-cffi.readthedocs.io/en/latest/quick_start.html Shows how to pass URL parameters to a GET request using curl_cffi. Handles single values and lists of values for the same parameter key. ```python import curl_cffi >>> params = {"foo": "bar"} >>> r = requests.get("http://httpbin.org/get", params=params) >>> r.url 'http://httpbin.org/get?foo=bar' >>> params = {'key1': 'value1', 'key2': ['value2', 'value3']} >>> import curl_cffi >>> r = curl_cffi.get('https://httpbin.org/get', params=params) >>> r.url 'https://httpbin.org/get?key1=value1&key2=value2&key2=value3' ``` -------------------------------- ### Example curl-cffi Doctor Output Source: https://curl-cffi.readthedocs.io/en/latest/cli/doctor.html This is an example of the diagnostic information printed by the `curl-cffi doctor` command. It includes details about your Python environment, platform, curl-cffi version, and configuration. ```text curl-cffi doctor ---------------- python: 3.10.20 executable: /usr/bin/python3 platform: macOS-15.4-arm64-arm-64bit machine: arm64 curl_cffi: 0.15.0b4 libcurl: libcurl/8.15.0 api_root: https://api.impersonate.pro/v1 config_path: /Users/you/.config/impersonate/config.json config_present: True api_key_configured: True fingerprint_path: /Users/you/.config/impersonate/fingerprints.json fingerprint_present: True fingerprint_count: 42 ``` -------------------------------- ### WebSocket.connect Source: https://curl-cffi.readthedocs.io/en/latest/api.html Establishes a connection to the WebSocket server, handling various connection parameters and options. ```APIDOC ## WebSocket.connect ### Description Connect to the WebSocket. libcurl automatically handles pings and pongs. ref: https://curl.se/libcurl/c/libcurl-ws.html ### Method POST (implied by connection establishment) ### Endpoint [WebSocket URL] ### Parameters #### Path Parameters None #### Query Parameters - **params** (dict[str, object] | list[object] | tuple[str, int | list[str] | dict[str, str | int]] | None) - Optional - query string for the requests. #### Headers - **headers** (HeaderTypes | None) - Optional - headers to send. - **referer** (str | None) - Optional - shortcut for setting referer header. - **accept_encoding** (str | None) - Optional - shortcut for setting accept-encoding header. Defaults to 'gzip, deflate, br'. - **default_headers** (bool) - Optional - whether to set default browser headers. Defaults to True. #### Authentication - **auth** (tuple[str, str] | None) - Optional - HTTP basic auth, a tuple of (username, password), only basic auth is supported. #### Connection Options - **timeout** (float | tuple[float, float] | object | None) - Optional - how many seconds to wait before giving up. - **allow_redirects** (bool | CurlFollow | str) - Optional - whether to allow redirection. Can be a bool, a `CurlFollow` value, or the string "safe". Defaults to True. - **max_redirects** (int) - Optional - max redirect counts, default 30, use -1 for unlimited. - **proxies** (ProxySpec | None) - Optional - dict of proxies to use, prefer to use `proxy` if they are the same. format: `{"http": proxy_url, "https": proxy_url}`. - **proxy** (str | None) - Optional - proxy to use, format: “http://user@pass:proxy_url”. Can’t be used with proxies parameter. - **proxy_auth** (tuple[str, str] | None) - Optional - HTTP basic auth for proxy, a tuple of (username, password). - **verify** (bool | None) - Optional - whether to verify https certs. - **http_version** (CurlHttpVersion | None) - Optional - Limiting http version, defaults to http2. - **interface** (str | None) - Optional - interface name or local IP to bind to (bare IP = source address). - **cert** (str | tuple[str, str] | None) - Optional - a tuple of (cert, key) filenames for client cert. #### Fingerprinting - **impersonate** (BrowserTypeLiteral | str | Fingerprint | None) - Optional - which browser version or fingerprint to impersonate. - **ja3** (str | None) - Optional - ja3 string to impersonate. - **akamai** (str | None) - Optional - akamai string to impersonate. - **perk** (str | None) - Optional - perk string to impersonate. - **extra_fp** (ExtraFingerprints | ExtraFpDict | None) - Optional - extra fingerprints options, in complement to ja3 and akamai str. #### Other Options - **cookies** (CookieTypes | None) - Optional - cookies to use. - **auth** (tuple[str, str] | None) - Optional - HTTP basic auth, a tuple of (username, password), only basic auth is supported. - **max_recv_speed** (int) - Optional - maximum receive speed, bytes per second. Defaults to 0. - **curl_options** (dict[CurlOpt, str] | None) - Optional - extra curl options to use. - **quote** (str | Literal[False]) - Optional - Set characters to be quoted (percent-encoded). Default safe string is `!#$%&'()*+,/:;=?@[]~`. If set to a string, the characters will be removed from the safe string. If set to `False`, the URL is used as-is (you must encode it yourself). ### Request Example ```json { "url": "wss://example.com/socket", "headers": {"X-Custom-Header": "value"} } ``` ### Response #### Success Response (200 OK) - **Self** (Self) - The connected WebSocket instance. ``` -------------------------------- ### GET Request with Default Headers Disabled Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Perform a GET request impersonating Chrome but with default headers explicitly disabled using `default_headers=False`. This shows the minimal headers sent in this configuration. ```python >>> r = curl_cffi.get("https://httpbin.org/headers", impersonate="chrome", default_headers=False) >>> print(r.text) { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate, br", "Host": "httpbin.org", ``` -------------------------------- ### GET Request with Default Headers (Chrome Impersonation) Source: https://curl-cffi.readthedocs.io/en/latest/_sources/quick_start.rst.txt Perform a GET request impersonating Chrome, showing the default headers that are automatically added. This demonstrates the standard headers sent when impersonating a browser. ```python >>> r = curl_cffi.get("https://httpbin.org/headers", impersonate="chrome") >>> print(r.text) { "headers": { "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Accept-Encoding": "gzip, deflate, br, zstd", "Accept-Language": "en-US,en;q=0.9", "Host": "httpbin.org", "Priority": "u=0, i", "Sec-Ch-Ua": "\"Chromium\";v=\"136\", \"Google Chrome\";v=\"136\", \"Not.A/Brand\";v=\"99\"", "Sec-Ch-Ua-Mobile": "?0", "Sec-Ch-Ua-Platform": "\"macOS\"", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36", "X-Amzn-Trace-Id": "Root=1-68452cc9-7287427f222e720c57971297" } } ``` -------------------------------- ### Get Cookie Value by Name in Python Source: https://curl-cffi.readthedocs.io/en/latest/_modules/curl_cffi/requests/cookies.html Implements the __getitem__ method for dictionary-like access to cookie values. It retrieves the cookie value using the 'get' method and raises a KeyError if the cookie is not found. ```python def __getitem__(self, name: str) -> str: value = self.get(name) if value is None: raise KeyError(name) return value ``` -------------------------------- ### Persisting Cookies with Pickle Source: https://curl-cffi.readthedocs.io/en/latest/_sources/cookies.rst.txt Demonstrates how to save and load session cookies using the pickle module. This approach is recommended over simple dictionary dumps because cookies contain complex metadata. ```python import pickle import curl_cffi import os def save_cookies(client): with open("cookies.pk", "wb") as f: pickle.dump(client.cookies.jar._cookies, f) def load_cookies(): if not os.path.isfile("cookies.pk"): return None with open("cookies.pk", "rb") as f: return pickle.load(f) client = curl_cffi.Session() client.get("https://httpbin.org/cookies/set/foo/bar") save_cookies(client) client = curl_cffi.Session() client.cookies.jar._cookies.update(load_cookies()) print(client.cookies.get("foo")) ``` -------------------------------- ### Basic GET Request with Impersonation Source: https://curl-cffi.readthedocs.io/en/latest/quick_start.html Perform a basic GET request using curl_cffi, specifying a browser to impersonate for realistic client simulation. Supports latest browser versions by using 'chrome' or 'safari'. ```python import curl_cffi url = "https://tls.browserleaks.com/json" # Notice the impersonate parameter r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome110") print(r.json()) # output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...} # the js3n fingerprint should be the same as target browser # To keep using the latest browser version as `curl_cffi` updates, simply set impersonate="chrome" without specifying a version. # Other similar values are: "safari" and "safari_ios" r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome") ``` -------------------------------- ### WebSocket Class Initialization Source: https://curl-cffi.readthedocs.io/en/latest/api.html Initializes a WebSocket instance with various configuration options for callbacks and behavior. ```APIDOC ## WebSocket Class ### Description A WebSocket implementation using libcurl. ### Parameters #### Initialization Parameters - **autoclose** (bool) - Optional - whether to close the WebSocket after receiving a close frame. - **skip_utf8_validation** (bool) - Optional - whether to skip UTF-8 validation for text frames in run_forever(). - **debug** (bool) - Optional - print extra curl debug info. - **on_open** (ON_OPEN_T | None) - Optional - open callback, `def on_open(ws)` - **on_close** (ON_CLOSE_T | None) - Optional - close callback, `def on_close(ws, code, reason)` - **on_data** (ON_DATA_T | None) - Optional - raw data receive callback, `def on_data(ws, data, frame)` - **on_message** (ON_MESSAGE_T | None) - Optional - message receive callback, `def on_message(ws, message)` - **on_error** (ON_ERROR_T | None) - Optional - error callback, `def on_error(ws, exception)` ```