### Install from source Source: https://github.com/lexiforest/curl_cffi/blob/main/README.md Steps to build and install the library from the GitHub repository. ```bash git clone https://github.com/lexiforest/curl_cffi/ cd curl_cffi make preprocess pip install . ``` -------------------------------- ### Setup local development environment on macOS Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/dev.md Configures the necessary directory structure, installs system dependencies via Homebrew, and installs the project in editable mode with development and test dependencies. ```shell sudo mkdir /Users/runner sudo chmod 777 /Users/runner brew install libidn2 zstd pip install -e .[test] pip install -e .[dev] ``` -------------------------------- ### Install beta releases Source: https://github.com/lexiforest/curl_cffi/blob/main/README.md Command to install pre-release versions of the library. ```bash pip install curl_cffi --upgrade --pre ``` -------------------------------- ### Install curl-cffi with CLI extras Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Install the optional 'cli' extra for enhanced features like syntax highlighting and download progress bars. ```bash pip install 'curl_cffi[cli]' ``` -------------------------------- ### Basic GET Request Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Perform a simple GET request to a specified URL. ```bash curl-cffi get https://httpbin.org/get ``` -------------------------------- ### Perform GET requests with impersonation Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Demonstrates basic GET requests using the impersonate parameter to mimic specific browser TLS fingerprints and proxy configurations. ```python import curl_cffi url = "https://tls.browserleaks.com/json" # Notice the impersonate parameter r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome110") print(r.json()) # output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...} # the js3n fingerprint should be the same as target browser # To keep using the latest browser version as `curl_cffi` updates, # simply set impersonate="chrome" without specifying a version. # Other similar values are: "safari" and "safari_ios" r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome") # http/socks proxies are supported proxies = {"https": "http://localhost:3128"} r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies) proxies = {"https": "socks://localhost:3128"} r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies) ``` -------------------------------- ### Install curl-cffi with pip Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Install the curl_cffi package using pip. This includes the CLI functionality. ```bash pip install curl_cffi ``` -------------------------------- ### Perform Asynchronous HTTP Requests Source: https://github.com/lexiforest/curl_cffi/blob/main/README.md Demonstrates basic and concurrent GET requests using AsyncSession. ```python from curl_cffi import AsyncSession async with AsyncSession() as s: r = await s.get("https://example.com") ``` ```python import asyncio from curl_cffi import AsyncSession urls = [ "https://google.com/", "https://facebook.com/", "https://twitter.com/", ] async with AsyncSession() as s: tasks = [] for url in urls: task = s.get(url) tasks.append(task) results = await asyncio.gather(*tasks) ``` -------------------------------- ### GET Request with Custom Header Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Make a GET request and include a custom header in the request. ```bash curl-cffi get https://httpbin.org/get X-My-Header:value ``` -------------------------------- ### Implement Synchronous and Asynchronous WebSockets Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Provides examples for connecting to WebSockets using both blocking Session and non-blocking AsyncSession interfaces. ```python from curl_cffi import Session, WebSocket def on_message(ws: WebSocket, message): print(message) with Session() as session: ws = session.ws_connect( "wss://api.gemini.com/v1/marketdata/BTCUSD", on_message=on_message, ) ws.run_forever() # asyncio import asyncio from curl_cffi import AsyncSession async with AsyncSession() as session: async with session.ws_connect("wss://echo.websocket.org") as ws: await asyncio.gather(*[ws.send_str("Hello, World!") for _ in range(10)]) async for message in ws: print(message) ``` -------------------------------- ### Basic GET Request Source: https://github.com/lexiforest/curl_cffi/blob/main/skills/imp-fetch/SKILL.md Perform a simple GET request to a URL. Impersonates Chrome by default. ```bash curl-cffi get https://httpbin.org/get ``` -------------------------------- ### Python Example for Proxy Credential Reuse Control Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/impersonate/psk.md A placeholder for a Python example demonstrating how to enable or manage the `proxy_credential_no_reuse` option in curl_cffi, which binds TLS session caches to proxy credentials to prevent issues with rotating proxies. ```python # Python example to be added. # We might enable this by default when proxies are used. ``` -------------------------------- ### Use curl-cffi CLI Source: https://github.com/lexiforest/curl_cffi/blob/main/README.md Examples of using the command line interface to perform requests with impersonation. ```bash curl-cffi get tls.browserleaks.com/json # curl-cffi can be hard to type, use an alias if you want alias imp=curl-cffi imp get tls.browserleaks.com/json --impersonate chrome ``` -------------------------------- ### Doctor Command Example Output Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/doctor.md This is an example of the diagnostic information printed by the `doctor` command. It includes details about your Python environment, platform, curl_cffi version, and configuration. ```text curl-cffi doctor ---------------- python: 3.10.20 executable: /usr/bin/python3 platform: macOS-15.4-arm64-arm-64bit machine: arm64 curl_cffi: 0.15.0b4 libcurl: libcurl/8.15.0 api_root: https://api.impersonate.pro/v1 config_path: /Users/you/.config/impersonate/config.json config_present: True api_key_configured: True fingerprint_path: /Users/you/.config/impersonate/fingerprints.json fingerprint_present: True fingerprint_count: 42 ``` -------------------------------- ### Manage sessions Source: https://github.com/lexiforest/curl_cffi/blob/main/README.md Example of using the Session object to persist cookies across requests. ```python s = curl_cffi.Session() # httpbin is a http test website, this endpoint makes the server set cookies s.get("https://httpbin.org/cookies/set/foo/bar") print(s.cookies) # ]> # retrieve cookies again to verify r = s.get("https://httpbin.org/cookies") print(r.json()) # {'cookies': {'foo': 'bar'}} ``` -------------------------------- ### Perform requests with impersonation Source: https://github.com/lexiforest/curl_cffi/blob/main/README.md Examples of using the high-level API to perform requests with various browser impersonation and proxy configurations. ```python import curl_cffi # Notice the impersonate parameter r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome") print(r.json()) # output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...} # the js3n fingerprint should be the same as target browser # To keep using the latest browser version as `curl_cffi` updates, # simply set impersonate="chrome" without specifying a version. # Other similar values are: "safari" and "safari_ios" r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome") # Use http/3 with impersonation r = curl_cffi.get( "https://fp.impersonate.pro/api/http3", http_version="v3", impersonate="chrome" ) # To pin a specific version, use version numbers together. r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome124") # To impersonate other than browsers, bring your own ja3/akamai strings # See examples directory for details. r = curl_cffi.get("https://tls.browserleaks.com/json", ja3=..., akamai=...) # http/socks proxies are supported proxies = {"https": "http://localhost:3128"} r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome", proxies=proxies) proxies = {"https": "socks://localhost:3128"} r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome", proxies=proxies) ``` -------------------------------- ### Install curl-cffi via Homebrew Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Install curl-cffi on macOS using Homebrew. This makes the 'curl-cffi' command available in your shell. ```bash brew install lexiforest/tap/curl-cffi ``` -------------------------------- ### HTTP/3 Request Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Execute a GET request using the HTTP/3 protocol by specifying the --http3 flag. ```bash curl-cffi get --http3 https://fp.impersonate.pro/api/http3 ``` -------------------------------- ### Custom Header Source: https://github.com/lexiforest/curl_cffi/blob/main/skills/imp-fetch/SKILL.md Include a custom header in the GET request. ```bash curl-cffi get https://httpbin.org/get X-My-Header:value ``` -------------------------------- ### Impersonate Safari Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Make a GET request and explicitly impersonate Safari using the -i flag. ```bash curl-cffi get -i safari tls.browserleaks.com/json ``` -------------------------------- ### Verbose Output Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Execute a GET request with verbose output enabled (-v), showing request and response headers and body. ```bash curl-cffi get -v https://httpbin.org/get ``` -------------------------------- ### Akamai Fingerprint String Format Example Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/impersonate/customize.md The Akamai HTTP/2 fingerprint string encodes client-controlled protocol parameters: SETTINGS, WINDOW_UPDATE, PRIORITY, and Pseudo-Header Order. ```default 1:65536;4:131072;5:16384|12517377|3:0:0:201|m,p,a,s ``` -------------------------------- ### Initialize and use AsyncSession Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/api.md Demonstrates the recommended context manager usage for AsyncSession, as well as manual instantiation. ```python from curl_cffi.requests import AsyncSession # recommended. async with AsyncSession() as s: r = await s.get("https://example.com") s = AsyncSession() # it also works. ``` -------------------------------- ### Connect and Communicate with AsyncWebSocket Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/websockets.md Demonstrates establishing a connection, sending text, and iterating over incoming messages using the async context manager. ```python import asyncio from curl_cffi import AsyncSession async def main(): async with AsyncSession() as session: # Connect using the standard context manager syntax async with session.ws_connect("wss://echo.websocket.org") as ws: # Send a text message await ws.send_str("Hello, World!") # Receive a text message msg = await ws.recv_str() print(f"Received: {msg}") # Iterate over messages async for message in ws: print(f"Stream: {message}") asyncio.run(main()) ``` -------------------------------- ### Batch Execution with .http Files Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/vs_others.md Shows how to define multiple HTTP requests in a human-readable .http file format, separated by '###', and execute them using curl-cffi's run command. ```text ### Get users GET https://api.example.com/users ### Create user POST https://api.example.com/users Content-Type: application/json {"name": "Alice"} ``` ```bash curl-cffi run requests.http ``` -------------------------------- ### Handle URL parameters Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Shows how to pass query parameters as dictionaries, including support for lists. ```python import curl_cffi >>> params = {"foo": "bar"} >>> r = curl_cffi.get("http://httpbin.org/get", params=params) >>> r.url 'http://httpbin.org/get?foo=bar' >>> params = {'key1': 'value1', 'key2': ['value2', 'value3']} >>> import curl_cffi >>> r = curl_cffi.get('https://httpbin.org/get', params=params) >>> r.url 'https://httpbin.org/get?key1=value1&key2=value2&key2=value3' ``` -------------------------------- ### Create Custom Cache Backend Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/advanced.md Implement a custom storage backend by subclassing CacheBackend and defining the required storage methods. ```python from datetime import timedelta from curl_cffi import CacheBackend, Session class DictCache(CacheBackend): def __init__(self): super().__init__(expires=timedelta(minutes=10)) self.data = {} def _read_payload(self, key): return self.data.get(key) def _write_payload(self, key, payload): self.data[key] = payload def _delete_payload(self, key): self.data.pop(key, None) def clear(self): self.data.clear() with Session(cache=DictCache()) as s: response = s.get("https://example.com/api") ``` -------------------------------- ### curl_cffi.requests.WebSocket.__init__ Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/api.md Initializes a new WebSocket instance with optional event callbacks and configuration settings. ```APIDOC ## WebSocket.__init__ ### Description Initializes the WebSocket client. Callbacks can be provided to handle connection events and incoming data. ### Parameters - **autoclose** (bool) - Optional - Whether to close the WebSocket after receiving a close frame. - **skip_utf8_validation** (bool) - Optional - Whether to skip UTF-8 validation for text frames. - **debug** (bool) - Optional - Print extra curl debug info. - **on_open** (callable) - Optional - Callback function: `def on_open(ws)` - **on_close** (callable) - Optional - Callback function: `def on_close(ws, code, reason)` - **on_data** (callable) - Optional - Callback function: `def on_data(ws, data, frame)` - **on_message** (callable) - Optional - Callback function: `def on_message(ws, message)` - **on_error** (callable) - Optional - Callback function: `def on_error(ws, exception)` ``` -------------------------------- ### AsyncSession.__init__ Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/api.md Initializes a new asynchronous session. Parameters provided here serve as defaults for all requests made within the session. ```APIDOC ## __init__(loop=None, async_curl=None, max_clients=10, **kwargs) ### Description Initializes an async request session where cookies and connections are reused. Parameters set here can be overridden by individual request methods. ### Parameters - **loop** (asyncio.AbstractEventLoop) - Optional - The event loop to use. - **async_curl** (AsyncCurl) - Optional - The AsyncCurl object to use. - **max_clients** (int) - Optional - Maximum number of curl handles to use in the session. - **headers** (dict) - Optional - Default headers for the session. - **cookies** (dict) - Optional - Default cookies for the session. - **auth** (tuple) - Optional - HTTP basic auth (username, password). - **proxies** (dict) - Optional - Proxy configuration dictionary. - **proxy** (str) - Optional - Proxy URL. - **impersonate** (str) - Optional - Browser version or fingerprint to impersonate. ``` -------------------------------- ### Configure WebSocket Connections Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/websockets.md Shows how to pass network parameters, authentication, and browser impersonation to the ws_connect method. ```python async with AsyncSession() as session: # Session cookies are automatically included session.cookies.set("session_id", "xyz") async with session.ws_connect( "wss://api.example.com/v1/stream", impersonate="chrome", auth=("user", "pass"), params={"stream_id": "123"}, timeout=10 # Connection timeout ) as ws: ... ``` -------------------------------- ### Localhost Shortcut Source: https://github.com/lexiforest/curl_cffi/blob/main/skills/imp-fetch/SKILL.md Use a shortcut for localhost requests, defaulting to HTTP. ```bash curl-cffi get :8000/api/health ``` -------------------------------- ### ws_connect Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/api.md Establishes a WebSocket connection. ```APIDOC ## ws_connect(url, ...) ### Description Connects to a WebSocket server with various configuration options for authentication, proxies, and fingerprinting. ### Parameters - **url** (str) - Required - The WebSocket URL to connect to. - **autoclose** (bool) - Optional - Whether to automatically close the connection (default: True). - **params** (dict|list|tuple) - Optional - Query parameters. - **headers** (HeaderTypes) - Optional - Request headers. - **cookies** (CookieTypes) - Optional - Cookies to send. - **auth** (tuple) - Optional - Authentication credentials. - **timeout** (float|tuple|NotSetType) - Optional - Connection timeout. - **impersonate** (BrowserTypeLiteral|str|Fingerprint) - Optional - Browser impersonation settings. ### Returns - **AsyncWebSocketContext** - The context object for the WebSocket connection. ``` -------------------------------- ### List fingerprints via CLI Source: https://github.com/lexiforest/curl_cffi/blob/main/README.md Command to display the current list of available fingerprints. ```bash curl-cffi list ``` -------------------------------- ### Perform Asynchronous HTTP Requests Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Requires an AsyncSession context manager to execute asynchronous GET requests. ```python # You must use a session for asyncio async with curl_cffi.AsyncSession() as s: r = await s.get("https://example.com") ``` -------------------------------- ### Execute HTTP Requests from File Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/run.md Use the `run` subcommand to execute requests defined in a .http or .har file. The format is auto-detected by the file extension. ```bash curl-cffi run requests.http ``` ```bash curl-cffi run session.har ``` -------------------------------- ### Impersonate Default Browser (Chrome) Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Perform a GET request, impersonating Chrome by default, to a site that provides browser information. ```bash curl-cffi get tls.browserleaks.com/json ``` -------------------------------- ### Session.ws_connect Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/api.md Establishes a WebSocket connection to the specified URL with optional callback handlers. ```APIDOC ## ws_connect(url, on_message=None, on_error=None, on_open=None, on_close=None, **kwargs) ### Description Connects to a WebSocket URL. Note: This method is deprecated; use the WebSocket class directly. ### Parameters - **url** (str) - Required - The WebSocket URL. - **on_message** (Callable) - Optional - Callback for received messages. - **on_error** (Callable) - Optional - Callback for errors. - **on_open** (Callable) - Optional - Callback for connection open. - **on_close** (Callable) - Optional - Callback for connection close. ``` -------------------------------- ### HTTP Authentication with --auth Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/options.md Provide HTTP authentication credentials using the `--auth` flag in the format `user:password`. ```bash # Basic auth curl-cffi get -a user:password https://httpbin.org/basic-auth/user/password ``` -------------------------------- ### Run Requests from .http File Source: https://github.com/lexiforest/curl_cffi/blob/main/skills/imp-fetch/SKILL.md Execute multiple requests defined in an .http or .rest file. Shares session by default. ```bash curl-cffi run requests.http ``` -------------------------------- ### Basic POST Request: curl vs. curl-cffi Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/vs_others.md Demonstrates the difference in syntax for making a POST request. curl-cffi uses a more readable key-value format instead of flags like -H and -d. ```bash # curl curl -X POST https://api.example.com/users \ -H "Content-Type: application/json" \ -d '{"name": "Alice", "age": 30}' ``` ```bash # curl-cffi curl-cffi post https://api.example.com/users name=Alice age:=30 ``` -------------------------------- ### JA3 String Format Example Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/impersonate/customize.md A JA3 string is a comma-separated representation of TLS ClientHello fields: SSL/TLS Version, Cipher Suites, Extension IDs, Supported Groups, and EC Point Formats. ```default 771,4865-4866-4867-49195-49196,0-11-10-35-16-5,29-23-24,0 ``` -------------------------------- ### Randomize Browser Impersonation Version Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/impersonate/faq.md Demonstrates how to select a random browser version for impersonation from a predefined list. It is recommended to use realistic versions rather than generating custom fingerprints to avoid detection. ```python import random # Select a random browser version from a list of known valid versions version = random.choice(["chrome119", "chrome120", "safari17", "firefox120"]) ``` -------------------------------- ### Manage WebSocket Connections Source: https://github.com/lexiforest/curl_cffi/blob/main/README.md Shows how to handle WebSocket messages using a callback function. ```python from curl_cffi import WebSocket def on_message(ws: WebSocket, message: str | bytes): print(message) ws = WebSocket(on_message=on_message) ws.run_forever("wss://api.gemini.com/v1/marketdata/BTCUSD") ``` -------------------------------- ### Proxy Configuration Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/options.md Route requests through a proxy server by specifying the proxy URL with the `--proxy` option. ```bash # Use a proxy curl-cffi get --proxy http://proxy:8080 https://httpbin.org/get ``` -------------------------------- ### Persist cookies using pickle Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cookies.md Demonstrates how to save and load session cookies using the pickle module to maintain state across different session instances. This method is preferred over manual dictionary dumping due to the complex nature of cookie objects. ```python import pickle import curl_cffi import os def save_cookies(client): with open("cookies.pk", "wb") as f: pickle.dump(client.cookies.jar._cookies, f) def load_cookies(): if not os.path.isfile("cookies.pk"): return None with open("cookies.pk", "rb") as f: return pickle.load(f) client = curl_cffi.Session() client.get("https://httpbin.org/cookies/set/foo/bar") save_cookies(client) client = curl_cffi.Session() client.cookies.jar._cookies.update(load_cookies()) print(client.cookies.get("foo")) ``` -------------------------------- ### Configure API Key with curl-cffi Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/pro.md Store your impersonate.pro API key in the local config file. Alternatively, use the IMPERSONATE_API_KEY environment variable. ```bash curl-cffi config --api-key imp_xxxxxxxx ``` ```bash export IMPERSONATE_API_KEY=imp_xxxxxxxx ``` -------------------------------- ### Select HTTP version for requests Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/advanced.md Configure the HTTP version using the http_version parameter, supporting values like v1, v2, v3, and v3only. ```python import curl_cffi curl_cffi.get("https://cloudflare-quic.com", http_version="v3") ``` -------------------------------- ### Localhost Shortcut Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/_index.md Access a local service running on a specific port using the colon shortcut for localhost. ```bash curl-cffi get :8000/api/health ``` -------------------------------- ### Using Sessions for cookies and connections Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Sessions persist cookies and reuse connections across multiple requests. ```python s = curl_cffi.Session() # Cookies from server are stored in session s.get("https://httpbin.org/cookies/set/foo/bar") print(s.cookies) # ]> # Cookies are used in next request r = s.get("https://httpbin.org/cookies") print(r.json()) # {'cookies': {'foo': 'bar'}} # It's preferred to use a context manager with curl_cffi.Session() as s: r = s.get("https://example.com") ``` ```python with curl_cffi.Session(headers={"User-Agent": "curl_cffi/0.11"}) as s: r = s.get("https://example.com") ``` -------------------------------- ### Download Response Body to File Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/options.md Use the `--download` flag to save the response body to a file. The filename is derived from the `Content-Disposition` header or the URL path. Use `-o` to specify a custom output file path. ```bash # Download a file curl-cffi get --download https://example.com/file.zip ``` ```bash # Download to a specific path curl-cffi get --download -o myfile.zip https://example.com/file.zip ``` -------------------------------- ### Explicit vs. Implicit POST Method: HTTPie vs. curl-cffi Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/vs_others.md Highlights the difference in HTTP method specification. curl-cffi requires the method as an explicit subcommand, while HTTPie infers it from the presence of data. ```bash # HTTPie -- implicit POST because of data fields http example.com/api name=test ``` ```bash # curl-cffi -- method is always explicit curl-cffi post example.com/api name=test ``` -------------------------------- ### Replay HAR File Entries Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/run.md The `run` command can replay all entries from a HAR (HTTP Archive) file. Ensure sensitive data is included in the HAR export if needed. ```bash curl-cffi run captured.har ``` -------------------------------- ### Headers Only Output Source: https://github.com/lexiforest/curl_cffi/blob/main/skills/imp-fetch/SKILL.md Print only the response headers. ```bash curl-cffi get --headers https://httpbin.org/get ``` -------------------------------- ### Batch Execution with .har Files Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/vs_others.md Demonstrates replaying HTTP traffic from a .har file, exported from browser developer tools, using curl-cffi's run command. This is useful for reproducing real-world requests. ```bash curl-cffi run session.har ``` -------------------------------- ### Control Output with Headers and Body Flags Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/options.md Use `--headers` to print only response headers, `--body` to print only the response body, or `-v` for full verbose output including request and response headers and body. ```bash # Headers only curl-cffi get --headers https://httpbin.org/get ``` ```bash # Body only (no headers) curl-cffi get --body https://httpbin.org/get ``` ```bash # Full verbose output curl-cffi post -v https://httpbin.org/post name=test ``` -------------------------------- ### Package with PyInstaller specifying paths and data Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/faq.md When packaging with PyInstaller, specify Python paths and include necessary data files, such as DLLs and package data, using `--paths`, `--add-data`, and `--collect-data`. ```default pyinstaller --noconfirm --onefile --console \ --paths "C:/Users/Administrator/AppData/Local/Programs/Python/Python39" \ --add-data "C:/Users/Administrator/AppData/Local/Programs/Python/Python39/Lib/site-packages/curl_cffi.libs/libcurl-cbb416caa1dd01638554eab3f38d682d.dll;. " \ --collect-data "curl_cffi" \ "C:/Users/Administrator/Desktop/test_script.py" ``` -------------------------------- ### Use Session as a Context Manager Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/api.md The Session class supports the context manager protocol for automatic resource cleanup. ```python from curl_cffi.requests import Session with Session() as s: r = s.get("https://example.com") ``` -------------------------------- ### Batch Execution Error Summary Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/run.md When running multiple requests, execution continues on failure. A summary of failed requests is printed at the end. ```text --- --- [1] GET https://api.example.com/users --- [2] POST https://api.example.com/missing --- [3] GET https://api.example.com/health 1 request(s) failed. ``` -------------------------------- ### Configure curl_cffi for eventlet or gevent Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/community.md Demonstrates how to enable non-blocking I/O by setting the thread parameter to eventlet or gevent within a curl_cffi Session. This allows the library to work within greenlet-based concurrency environments. ```python from curl_cffi import requests s = requests.Session(thread="eventlet") s.get(url) ``` -------------------------------- ### Configure API Key via Python Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/fingerprints.md Set the API key programmatically in Python using FingerprintManager. This is useful for script-based configurations. ```python from curl_cffi.fingerprints import FingerprintManager FingerprintManager.set_api_key("imp_xxxxxxxx") ``` -------------------------------- ### Configure API Key via CLI Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/fingerprints.md Set your API key once using the curl-cffi config command. This configuration is persistent. ```sh curl-cffi config --api-key imp_xxxxxxxx ``` -------------------------------- ### curl_cffi.requests.Session.__init__ Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/api.md Initializes a new Session object. Parameters provided here are used as defaults for all requests made within the session and can be overridden by individual request methods. ```APIDOC ## class curl_cffi.requests.Session ### Description A request session where cookies and connections are reused. This object is thread-safe, though using a separate session per thread is recommended. ### Constructor `__init__(curl=None, thread=None, use_thread_local_curl=True, **kwargs)` ### Parameters - **curl** (Curl, optional) - Curl object to use. A fresh one is created if not provided or when accessed from another thread. - **thread** (ThreadType, optional) - Thread engine to use (e.g., 'eventlet', 'gevent'). - **headers** (dict, optional) - Headers to use in the session. - **cookies** (dict, optional) - Cookies to add to the session. - **auth** (tuple, optional) - HTTP basic auth as (username, password). - **proxies** (dict, optional) - Proxy configuration dictionary. - **proxy** (str, optional) - Proxy URL string. - **proxy_auth** (tuple, optional) - Proxy basic auth as (username, password). - **base_url** (str, optional) - Base URL for relative paths. - **params** (dict, optional) - Default query string parameters. - **verify** (bool, optional) - Whether to verify HTTPS certificates. - **timeout** (int, optional) - Seconds to wait before timing out. - **trust_env** (bool, optional) - Use environment variables like http_proxy. Default: True. - **allow_redirects** (bool/str/CurlFollow, optional) - Redirect handling policy. - **max_redirects** (int, optional) - Maximum number of redirects. Default: 30. - **retry** (int/RetryStrategy, optional) - Number of retries or strategy for failed requests. - **impersonate** (str, optional) - Browser version or fingerprint to impersonate. - **ja3** (str, optional) - JA3 string for impersonation. - **akamai** (str, optional) - Akamai string for impersonation. - **perk** (str, optional) - Perk string for impersonation. - **extra_fp** (dict, optional) - Additional fingerprint options. - **interface** (str, optional) - Interface name or local IP to bind to. - **doh_url** (str, optional) - DNS-over-HTTPS server URL. - **default_encoding** (str/callable, optional) - Encoding for response content. Default: 'utf-8'. - **cert** (tuple, optional) - (cert, key) filenames for client certificates. - **response_class** (type, optional) - Custom Response subtype. - **raise_for_status** (bool, optional) - Automatically raise HTTPError for 4xx/5xx status codes. ``` -------------------------------- ### Configure Queue Sizes for Backpressure Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/websockets.md Adjust internal buffer sizes to manage high-throughput streams or specific latency requirements. ```python # Increase queues for high-throughput streams (e.g., market data) ws = await session.ws_connect( url, recv_queue_size=512, send_queue_size=256 ) ``` ```python # High throughput, fast moving streams (e.g., video) ws = await session.ws_connect( url, recv_queue_size=2048, send_queue_size=2048 ) ``` -------------------------------- ### curl_cffi.requests.WebSocket.connect Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/api.md Establishes a connection to the specified WebSocket URL with various configuration options. ```APIDOC ## WebSocket.connect ### Description Connects to the WebSocket server. Supports headers, cookies, authentication, proxy settings, and browser impersonation. ### Parameters - **url** (str) - Required - The WebSocket URL to connect to. - **params** (dict/list/tuple) - Optional - Query string parameters. - **headers** (dict) - Optional - Headers to send. - **cookies** (dict) - Optional - Cookies to use. - **auth** (tuple) - Optional - HTTP basic auth (username, password). - **timeout** (float) - Optional - Connection timeout in seconds. - **impersonate** (str) - Optional - Browser version or fingerprint to impersonate. - **proxy** (str) - Optional - Proxy URL. - **verify** (bool) - Optional - Whether to verify HTTPS certificates. ``` -------------------------------- ### Execute Concurrent Requests with asyncio.gather Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Demonstrates managing multiple concurrent requests by gathering tasks created within an AsyncSession. ```python import asyncio from curl_cffi import AsyncSession urls = [ "https://google.com/", "https://facebook.com/", "https://apple.com/", ] async with AsyncSession() as s: tasks = [] for url in urls: task = s.get(url) tasks.append(task) results = await asyncio.gather(*tasks) ``` -------------------------------- ### List Fingerprints with curl-cffi Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/pro.md List native and cached fingerprints in a table format. Use the --json flag to output fingerprints as JSON. ```bash curl-cffi list ``` ```bash curl-cffi list --json ``` -------------------------------- ### Using CurlFollow.SAFE for Redirects Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/security.md Demonstrates how to use `CurlFollow.SAFE` to prevent SSRF by rejecting redirects to internal IP addresses. The string 'safe' is also accepted as a shorthand. ```python from curl_cffi import CurlFollow, requests # Recommended for server-side code that fetches user-supplied URLs r = requests.get(url, allow_redirects=CurlFollow.SAFE) # The string "safe" is also accepted r = requests.get(url, allow_redirects="safe") # Session-level setting s = requests.Session(allow_redirects=CurlFollow.SAFE) ``` -------------------------------- ### Verbose Output Source: https://github.com/lexiforest/curl_cffi/blob/main/skills/imp-fetch/SKILL.md Display full verbose output, including request and response headers and body. ```bash curl-cffi post -v https://httpbin.org/post name=test ``` -------------------------------- ### Update fingerprints via CLI Source: https://github.com/lexiforest/curl_cffi/blob/main/README.md Command to fetch the latest fingerprints from impersonate.pro. ```bash curl-cffi update ``` -------------------------------- ### Advanced fingerprint customization Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Demonstrates how to retrieve and modify a fingerprint object before passing it to the impersonate parameter. ```python fingerprint = curl_cffi.get_fingerprint("edge_146_macos_26") fingerprint.headers["User-Agent"] = "..." r = curl_cffi.get( "https://httpbin.org/headers", impersonate=fingerprint, ) ``` -------------------------------- ### Include Request Body from File Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/run.md Use the `< filepath` syntax to include the request body from an external file. This is useful for large or complex request bodies. ```text POST https://api.example.com/upload Content-Type: application/json < body.json ``` -------------------------------- ### Define Multiple HTTP Requests in a File Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/run.md The .http format allows defining multiple requests in a single file, separated by `###`. Each request can include a method, URL, headers, and body. ```text ### Get user list GET https://api.example.com/users ### Create a user POST https://api.example.com/users Content-Type: application/json {"name": "Alice", "email": "alice@example.com"} ### Get a specific user GET https://api.example.com/users/1 ``` -------------------------------- ### Integrate curl_cffi as a requests adapter Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/community.md Shows how to use curl-adapter to mount curl_cffi into a standard requests.Session. This enables the use of curl_cffi's impersonation features while maintaining the requests API. ```python import requests from curl_adapter import CurlCffiAdapter session = requests.Session() session.mount("http://", CurlCffiAdapter()) session.mount("https://", CurlCffiAdapter()) session.get("https://example.com") ``` -------------------------------- ### Set API Key via Environment Variable Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/fingerprints.md Override the configured API key at runtime by setting the IMPERSONATE_API_KEY environment variable. This takes precedence over configured keys. ```sh export IMPERSONATE_API_KEY=imp_xxxxxxxx ``` -------------------------------- ### TLS PSK Extension Flow with Rotating Proxies Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/impersonate/psk.md Illustrates a scenario where a TLS PSK identity, associated with one IP address, is reused with a different IP address, leading to a blocked connection. This highlights the problem solved by `proxy_credential_no_reuse`. ```text ┌───────────┐ ┌───────────┐ │ │ │ │ │ │ IP: 10.0.0.1 │ │ │ ┼─────────────TLS─Hello──────────────► │ │ │ │ │ │ ◄─────────────PSK:─xxx───────────────┼ │ │ │ │ │ │ │ │ │ │ │ │ Server │ │ Client │ IP: 10.0.0.2 │ │ │ ┼─────────────TLS─with─PSK───────────► │ │ │ │ │ │ ◄─────────────Blocked────────────────┼ │ │ │ │ │ │ │ PSK: xxx was │ │ │ │ associated with │ │ │ │ 10.0.0.1, not │ │ └───────────┘ 10.0.0.2 └───────────┘ ``` -------------------------------- ### Configure Reliability and Retries Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/websockets.md Set up automatic retry strategies and ensure data integrity during network errors. ```python from curl_cffi import WebSocketRetryStrategy # Retry transient read errors up to 5 times retry_policy = WebSocketRetryStrategy( retry=True, count=5 ) ws = await session.ws_connect( url, drain_on_error=True, ws_retry=retry_policy ) ``` -------------------------------- ### Upload files with multipart Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Use CurlMime to construct multipart requests as the files API is not supported. ```python mp = curl_cffi.CurlMime() mp.addpart( name="attachment", # field name in the form content_type="image/png", # mime type filename="image.png", # filename seen by remote server local_path="./image.png", # local file to upload data=file.read(), # if you already have the data in memory ) r = curl_cffi.post("https://httpbin.org/post", data={"foo": "bar"}, multipart=mp) print(r.json()) ``` -------------------------------- ### Set HTTP2 Pseudo-Headers Order with CurlOpt Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/impersonate/customize.md Use `curl.setopt` with `CurlOpt.HTTP2_PSEUDO_HEADERS_ORDER` to specify the exact order of HTTP/2 pseudo-headers. ```python import curl_cffi from curl_cffi import Curl, CurlOpt c = Curl() c.setopt(CurlOpt.HTTP2_PSEUDO_HEADERS_ORDER, "masp") ``` -------------------------------- ### Toggle default headers Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Compares the default header behavior with the option to disable them using default_headers=False. ```python >>> r = curl_cffi.get("https://httpbin.org/headers", impersonate="chrome") >>> print(r.text) { "headers": { "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Accept-Encoding": "gzip, deflate, br, zstd", "Accept-Language": "en-US,en;q=0.9", "Host": "httpbin.org", "Priority": "u=0, i", "Sec-Ch-Ua": "\"Chromium\";v=\"136\", \"Google Chrome\";v=\"136\", \"Not.A/Brand\";v=\"99\"", "Sec-Ch-Ua-Mobile": "?0", "Sec-Ch-Ua-Platform": "\"macOS\"", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36", "X-Amzn-Trace-Id": "Root=1-68452cc9-7287427f222e720c57971297" } } >>> r = curl_cffi.get("https://httpbin.org/headers", impersonate="chrome", default_headers=False) >>> print(r.text) { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate, br", "Host": "httpbin.org", "X-Amzn-Trace-Id": "Root=1-68452d20-2cf4cf00201987301c476c06" } } ``` -------------------------------- ### Customize HTTP Header Order with extra_fp Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/impersonate/customize.md Use the `extra_fp` dictionary to further refine TLS impersonation by specifying custom header ordering. ```python extra_fp = { "header_order": "User-Agent,Host,Connection" } ``` -------------------------------- ### Set Cookies with Curl-Cffi Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/request_params.md Shows how to set cookies for a request using the '+' separator. Multiple cookies can be set in a single command. ```bash # Set a single cookie curl-cffi get https://httpbin.org/cookies +session=abc123 ``` ```bash # Set multiple cookies curl-cffi get https://httpbin.org/cookies +session=abc123 +theme=dark ``` -------------------------------- ### Configure Proxies in curl_cffi Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/advanced.md Use the proxy parameter for a single proxy or a dictionary for protocol-specific proxies. ```python import curl_cffi curl_cffi.get(url, proxy="http://user:pass@example.com:3128") ``` ```python import curl_cffi proxies = { "http": "http://localhost:3128", "https": "http://localhost:3128" } curl_cffi.get(url, proxies=proxies) ``` -------------------------------- ### Handling cookies in redirects Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/quick_start.md Use sessions to ensure cookies are correctly tracked across redirects. ```python import curl_cffi r = curl_cffi.get("https://httpbin.org/redirect") # ❌ Cookie from previous redirect may be lost. do_something(r.cookies) s = curl_cffi.Session() r = s.get("https://httpbin.org/redirect") # ✅ Use a session instead, to retrive all cookies in the session do_something(s.cookies) ``` -------------------------------- ### Update Fingerprints with curl-cffi Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/pro.md Download the latest fingerprints for impersonate.pro. Successful updates will print the total number of fingerprints now stored in the local cache. ```bash curl-cffi update ``` -------------------------------- ### Manage Session State in Batch Execution Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/run.md By default, requests in a batch share a session (cookies and connections persist). Use `--no-session` to execute each request independently. ```bash # Default: shared session (cookies carry over between requests) curl-cffi run requests.http ``` ```bash # Independent requests (no shared state) curl-cffi run --no-session requests.http ``` -------------------------------- ### AsyncCurl Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/api.md A wrapper around the curl_multi handle to provide asyncio support using libcurl socket_action APIs. ```APIDOC ## class curl_cffi.AsyncCurl(cacert: str = '', loop=None) ### Description Provides asyncio support for curl operations by managing multiple handles. ### Methods - **add_handle(curl)**: Adds a curl handle to be managed. - **remove_handle(curl)**: Cancels a future for a given handle. - **setopt(option, value)**: Sets options for the multi handle. - **close()**: Closes and cleans up resources. ``` -------------------------------- ### Custom Output with --print Flag Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/cli/options.md Employ the `--print` flag with a character string to specify fine-grained output control. 'H' for request headers, 'B' for request body, 'h' for response headers, and 'b' for response body. ```bash # Custom: request headers + response headers curl-cffi get -p Hh https://httpbin.org/get ``` -------------------------------- ### Body Only Output Source: https://github.com/lexiforest/curl_cffi/blob/main/skills/imp-fetch/SKILL.md Print only the response body, useful for piping or parsing. ```bash curl-cffi get --body https://httpbin.org/get ``` -------------------------------- ### HTTP/3 Request Source: https://github.com/lexiforest/curl_cffi/blob/main/skills/imp-fetch/SKILL.md Force the request to use the HTTP/3 protocol. ```bash curl-cffi get --http3 https://fp.impersonate.pro/api/http3 ``` -------------------------------- ### Package with PyInstaller using --hidden-import Source: https://github.com/lexiforest/curl_cffi/blob/main/docs/faq.md When packaging with PyInstaller, use the `--hidden-import` option to include necessary modules like `_cffi_backend` and `--collect-all` for packages like `curl_cffi`. ```default pyinstaller -F .\example.py --hidden-import=_cffi_backend --collect-all curl_cffi ```