### Integration Examples Overview Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/INDEX.md Showcases real-world integration examples for multiple languages and platforms. ```markdown ### [integration-examples.md](integration-examples.md) Real-world integration examples for multiple languages and platforms. **Sections**: - Command-line usage patterns - Basic requests, verbose output - Custom headers, POST requests - Cookies, authentication - Proxy, timeouts, SSL handling - Python integration - subprocess approach - pycurl with LD_PRELOAD - requests fallback - C/C++ integration - Basic libcurl-impersonate - Custom headers - POST with JSON - Error handling - JavaScript/Node.js - child_process - Native addon template - PHP integration - Ruby integration - Go integration - Docker usage - Kubernetes deployment examples - AWS Lambda integration - Performance optimization **Code Examples**: - 20+ complete working examples - Error handling patterns - Multi-request patterns - Parallel request patterns - Container deployment examples ``` -------------------------------- ### Standard Installation Paths Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Default installation paths for binaries and shared libraries after building Curl-Impersonate. ```text /usr/local/bin/curl_chrome116 /usr/local/bin/curl_ff109 /usr/local/lib/libcurl-impersonate-chrome.so /usr/local/lib/libcurl-impersonate-ff.so ``` -------------------------------- ### Docker Installation Paths Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Installation paths within a Docker environment. ```text /usr/local/bin/ /usr/local/lib/ ``` -------------------------------- ### Complete Application Example Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/libcurl-api.md A full C application demonstrating how to use libcurl-impersonate to make an HTTP request, impersonating Chrome 116. This example includes setting up the handle, configuring the request, performing it, and cleaning up resources. ```c #include #include static size_t write_callback(void *contents, size_t size, size_t nmemb, void *userp) { fwrite(contents, size, nmemb, (FILE *)userp); return size * nmemb; } int main() { CURL *handle = curl_easy_init(); if (!handle) { fprintf(stderr, "curl_easy_init failed\n"); return 1; } // Impersonate Chrome 116 CURLcode ret = curl_easy_impersonate(handle, "chrome116", 1); if (ret != CURLE_OK) { fprintf(stderr, "curl_easy_impersonate failed: %s\n", curl_easy_strerror(ret)); curl_easy_cleanup(handle); return 1; } // Open output file FILE *outfile = fopen("output.html", "w"); if (!outfile) { fprintf(stderr, "Failed to open output file\n"); curl_easy_cleanup(handle); return 1; } // Configure request curl_easy_setopt(handle, CURLOPT_URL, "https://example.com"); curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, write_callback); curl_easy_setopt(handle, CURLOPT_WRITEDATA, (void *)outfile); curl_easy_setopt(handle, CURLOPT_TIMEOUT, 30L); curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1L); // Perform request ret = curl_easy_perform(handle); if (ret != CURLE_OK) { fprintf(stderr, "curl_easy_perform failed: %s\n", curl_easy_strerror(ret)); } // Cleanup fclose(outfile); curl_easy_cleanup(handle); return ret == CURLE_OK ? 0 : 1; } ``` -------------------------------- ### Docker Setup for curl-impersonate Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md This Dockerfile installs necessary libraries, copies the impersonate library, and sets environment variables for runtime injection and impersonation. ```docker FROM ubuntu:20.04 RUN apt-get update && apt-get install -y libnss3 ca-certificates COPY libcurl-impersonate-chrome.so /opt/ ENV LD_PRELOAD=/opt/libcurl-impersonate-chrome.so ENV CURL_IMPERSONATE=chrome116 ``` -------------------------------- ### Run curl-impersonate Wrapper Scripts Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Examples of using the installed wrapper scripts to make requests with specific browser impersonations. ```sh curl_ff98 https://www.wikipedia.org curl_chrome99 https://www.wikipedia.org ``` -------------------------------- ### Docker Setup for Curl-Impersonate Tests Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md This Dockerfile sets up an Ubuntu environment with necessary tools like tcpdump and curl, copies the project, installs it, and configures the default command to run pytest with specific capture interface settings. ```dockerfile FROM ubuntu:20.04 RUN apt-get update && apt-get install -y tcpdump curl WORKDIR /curl-impersonate COPY . . RUN make install CMD pytest tests/ --install_dir=/usr/local --capture_interface=eth0 -v ``` -------------------------------- ### Custom Prefix Installation Paths Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Installation paths when a custom prefix (e.g., --prefix=/opt/curl-impersonate) is used during the build. ```text /opt/curl-impersonate/bin/ /opt/curl-impersonate/lib/ ``` -------------------------------- ### Build Targets and Dependencies Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/INDEX.md Details on binary targets, build configuration, dependencies, and installation paths for curl-impersonate. ```markdown ### [build-targets.md](build-targets.md) Binary targets, build configuration, dependencies, and installation paths. **Sections**: - Binary targets - curl-impersonate-chrome (BoringSSL) - curl-impersonate-ff (NSS) - Library targets - libcurl-impersonate-chrome.so/dylib - libcurl-impersonate-ff.so/dylib - Wrapper scripts (18 total) - Configuration files - browsers.json - Makefile configuration - Patch files - Installation paths - Package configuration (pkg-config) - System dependencies by platform - Pre-built binaries - Docker images - Version information - Installation verification **Includes**: - Complete list of wrapper scripts - Patch file descriptions - Build system architecture - Docker image specifications ``` -------------------------------- ### Install Ubuntu Dependencies Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Installs necessary build tools and libraries for compiling curl-impersonate on Ubuntu. Includes specific packages for both Firefox and Chrome versions. ```sh sudo apt install build-essential pkg-config cmake ninja-build curl autoconf automake libtool # For the Firefox version only sudo apt install python3-pip libnss3 pip install gyp-next export PATH="$PATH:~/.local/bin" # Add gyp to PATH # For the Chrome version only sudo apt install golang-go unzip ``` -------------------------------- ### Usage with pkg-config Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Example of how to use pkg-config to compile an application linked against the Chrome variant of libcurl-impersonate. ```bash gcc myapp.c -o myapp `pkg-config --cflags --libs libcurl-impersonate-chrome` ``` -------------------------------- ### Autoconf/Automake Build Flow Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Illustrates the typical build process using autoconf and automake, starting from configure.ac and ending with generated binaries and libraries. ```text configure.ac ↓ (autoconf) configure (bash script) ↓ (run by user) config.h (generated) Makefile (from Makefile.in) ↓ (make) binaries and libraries ``` -------------------------------- ### Example curl Commands with Wrapper Scripts Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/command-line-interface.md Demonstrates how to use different wrapper scripts with common curl options for various tasks. ```bash curl_chrome116 -v -L https://example.com ``` ```bash curl_ff109 -o output.html https://example.com ``` ```bash curl_chrome104 -H "Custom-Header: value" https://api.example.com ``` -------------------------------- ### Link and Preload Firefox Library Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Example of how to link your application against the Firefox impersonation libcurl library and preload it for runtime usage. ```bash gcc myapp.c -o myapp -lcurl-impersonate-ff LD_PRELOAD=/path/to/libcurl-impersonate-ff.so myapp ``` -------------------------------- ### Install Curl-Impersonate via Homebrew Source: https://github.com/lwthiker/curl-impersonate/blob/main/README.md Installs curl-impersonate on macOS using Homebrew by tapping a custom repository and installing the formula. ```bash brew tap shakacode/brew brew install curl-impersonate ``` -------------------------------- ### Link and Preload Chrome Library Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Example of how to link your application against the Chrome impersonation libcurl library and preload it for runtime usage. ```bash gcc myapp.c -o myapp -lcurl-impersonate-chrome LD_PRELOAD=/path/to/libcurl-impersonate-chrome.so myapp ``` -------------------------------- ### Install Red Hat Dependencies Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Installs necessary build tools and libraries for compiling curl-impersonate on Red Hat-based systems. Includes specific packages for Firefox and Chrome versions. ```sh yum groupinstall "Development Tools" yum groupinstall "C Development Tools and Libraries" # Fedora only yum install cmake3 python3 python3-pip # Install Ninja. This may depend on your system. yum install ninja-build # OR pip3 install ninja # For the Firefox version, install NSS and gyp: yum install nss nss-pem pip3 install gyp-next # For the Chrome version, install Go. yum install golang ``` -------------------------------- ### Install macOS Dependencies Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Installs necessary build tools and libraries for compiling curl-impersonate on macOS using Homebrew. Includes specific packages for Firefox and Chrome versions. ```sh brew install pkg-config make cmake ninja autoconf automake libtool # For the Firefox version only brew install sqlite nss pip3 install gyp-next # For the Chrome version only brew install go ``` -------------------------------- ### Sec-CH-UA Client Hint Example Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/http-headers.md This header indicates browser capabilities, listing names and versions. The format is a comma-separated list of quoted names and versions. ```http Sec-CH-UA: "Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116" ``` -------------------------------- ### Browser Target Configuration Example Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Defines a supported browser target, including its name, version, operating system, and associated binary or wrapper script. Used for programmatic lookup of browser targets. ```json { "browsers": [ { "name": "chrome116", "browser": { "name": "chrome", "version": "116.0.5845.180", "os": "win10" }, "binary": "curl-impersonate-chrome", "wrapper_script": "curl_chrome116" }, // ... more entries ] } ``` -------------------------------- ### Sec-Fetch-Mode Header Example Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/http-headers.md Specifies how a fetch request was initiated. Possible values include navigate, cors, no-cors, and same-origin. ```text Sec-Fetch-Mode ``` -------------------------------- ### Sec-Fetch-Site Header Example Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/http-headers.md Indicates the relationship between the request initiator and the server. Values include same-origin, same-site, cross-site, and none. ```text Sec-Fetch-Site ``` -------------------------------- ### Run curl-impersonate Directly Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Examples of running the curl-impersonate executables directly with custom flags. ```sh curl-impersonate-ff https://www.wikipedia.org curl-impersonate-chrome https://www.wikipedia.org ``` -------------------------------- ### Sec-CH-UA-Platform Client Hint Examples Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/http-headers.md This header specifies the operating system platform. The platform name is provided in a quoted string format. ```http Sec-CH-UA-Platform: "Windows" ``` ```http Sec-CH-UA-Platform: "macOS" ``` ```http Sec-CH-UA-Platform: "Android" ``` -------------------------------- ### Sec-Fetch-User Header Example Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/http-headers.md Indicates whether a fetch request was initiated by user interaction. Values are ?0 (no) or ?1 (yes). ```text Sec-Fetch-User ``` -------------------------------- ### Sec-Fetch-Dest Header Example Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/http-headers.md Indicates the type of resource being fetched in a request. Common values include document, image, script, etc. ```text Sec-Fetch-Dest ``` -------------------------------- ### PHP: cURL extension example (requires custom build) Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Demonstrates using the standard PHP cURL extension. Note that direct integration with CURL_IMPERSONATE requires a custom PHP build with libcurl-impersonate patched in. ```php ``` -------------------------------- ### Basic Impersonation Configuration Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md Sets up basic impersonation using a specific Chrome version. This is the minimal configuration to start impersonating a browser. ```bash LD_PRELOAD=/opt/curl-impersonate/libcurl-impersonate-chrome.so \ CURL_IMPERSONATE=chrome116 \ ./myapp ``` -------------------------------- ### Preload libcurl-impersonate with LD_PRELOAD (Linux) Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md On Linux, LD_PRELOAD can be used to preload the libcurl-impersonate library before the application starts. ```bash export LD_PRELOAD=/path/to/libcurl-impersonate-chrome.so ./myapp ``` -------------------------------- ### Example YAML Signature Entry Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md Defines a specific client signature including TLS client hello details, HTTP/2 pseudo-headers, and other options. ```yaml name: chrome_116.0.5845.180_win10 signature: tls_client_hello: version: 771 cipher_suites: - 4865 - 4866 # ... more ciphers extensions: - type: supported_versions versions: - 772 - 771 # ... more extensions http2: pseudo_headers: - ":method" - ":scheme" - ":authority" - ":path" headers: - "user-agent" - "accept" # ... more headers options: tls_permute_extensions: false ``` -------------------------------- ### Verify Curl-Impersonate Binaries Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Verify that the installed curl-impersonate binaries are working correctly. This includes checking their versions and performing a test request. ```bash curl_chrome116 --version curl_ff109 --version # Test with real request curl_chrome116 -v https://www.example.com > /dev/null # Verify libraries installed pkg-config --list-all | grep curl-impersonate ldd /usr/local/lib/libcurl-impersonate-chrome.so ``` -------------------------------- ### Query Binary Version Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Check the version of the installed curl-impersonate binaries. These commands return the standard curl version with an impersonate suffix. ```bash curl-impersonate-chrome --version curl-impersonate-ff --version ``` -------------------------------- ### Pytest Command-Line Options Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md Define custom command-line arguments for pytest, such as network interface and installation directory. ```python def pytest_addoption(parser): parser.addoption("--capture_interface", default="eth0") parser.addoption("--install_dir", required=True) ``` -------------------------------- ### C Code Example with Built-in Headers Enabled Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md When CURL_IMPERSONATE_HEADERS is 'yes', built-in headers like User-Agent and Accept are automatically included alongside custom headers. ```c // Application code struct curl_slist *headers = NULL; headers = curl_slist_append(headers, "Custom-Header: value"); curl_easy_setopt(handle, CURLOPT_HTTPHEADER, headers); // Actual headers sent: // Custom-Header: value // User-Agent: Mozilla/5.0 ... (built-in) // Accept: text/html,... (built-in) // Accept-Encoding: gzip,... (built-in) // ... (other built-in headers) ``` -------------------------------- ### Example TLS and HTTP/2 Signature Database Entry Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/browser-signatures.md This YAML snippet shows a complete signature entry for Chrome 116 on Windows 10, including TLS Client Hello details and HTTP/2 configurations. ```yaml name: chrome_116.0.5845.180_win10 signature: tls_client_hello: version: 771 # TLS 1.2 cipher_suites: - 4865 # TLS_AES_128_GCM_SHA256 - 4866 # TLS_AES_256_GCM_SHA384 # ... more ciphers extensions: - type: supported_versions versions: - 772 # TLS 1.3 - 771 # TLS 1.2 - type: supported_groups groups: - 29 # x25519 - 23 # secp256r1 # ... more extensions http2: pseudo_headers: - ":method" - ":scheme" - ":authority" - ":path" headers: - "user-agent" - "accept" # ... more headers options: tls_permute_extensions: true # Allow randomized extension order ``` -------------------------------- ### C/C++: Set Custom HTTP Headers Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Example of setting custom HTTP headers like Authorization and X-Custom using libcurl-impersonate in C/C++. Useful for API interactions. ```c #include int main() { CURL *curl = curl_easy_init(); curl_easy_impersonate(curl, "chrome116", 0); // No default headers // Build custom header list struct curl_slist *headers = NULL; headers = curl_slist_append(headers, "Authorization: Bearer token"); headers = curl_slist_append(headers, "X-Custom: value"); headers = curl_slist_append(headers, "Accept: application/json"); curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers); curl_easy_setopt(curl, CURLOPT_URL, "https://api.example.com"); curl_easy_perform(curl); curl_slist_free_all(headers); curl_easy_cleanup(curl); return 0; } ``` -------------------------------- ### Example Chrome 116 TLS Extension Order Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/browser-signatures.md Illustrates the specific order of TLS extensions sent in a Client Hello by Chrome 116, which is crucial for signature matching. ```markdown 1. server_name (SNI) 2. status_request 3. supported_groups 4. ec_point_formats 5. application_layer_protocol_negotiation (ALPN) 6. signature_algorithms 7. signed_certificate_timestamp 8. key_share 9. psk_key_exchange_modes 10. supported_versions 11. (GREASE extensions interspersed) ``` -------------------------------- ### Configure and Compile curl-impersonate (Ubuntu) Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Configures, builds, and installs the Firefox and Chrome versions of curl-impersonate on Ubuntu. Updates the linker cache and optionally cleans up build files. ```sh mkdir build && cd build ../configure # Build and install the Firefox version make firefox-build sudo make firefox-install # Build and install the Chrome version make chrome-build sudo make chrome-install # You may need to update the linker's cache to find libcurl-impersonate sudo ldconfig # Optionally remove all the build files cd ../ && rm -Rf build ``` -------------------------------- ### Configure and Compile curl-impersonate (macOS) Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Configures, builds, and installs the Firefox and Chrome versions of curl-impersonate on macOS using gmake. Optionally cleans up build files. ```sh mkdir build && cd build ../configure # Build and install the Firefox version gmake firefox-build sudo gmake firefox-install # Build and install the Chrome version gmake chrome-build sudo gmake chrome-install # Optionally remove all the build files cd ../ && rm -Rf build ``` -------------------------------- ### Set SSL Certificate Compression Algorithm Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/libcurl-api.md Configure the certificate compression algorithm for SSL connections using CURLOPT_SSL_CERT_COMPRESSION. Examples include "brotli", "zstd", or "deflate". ```c curl_easy_setopt(handle, CURLOPT_SSL_CERT_COMPRESSION, "brotli"); ``` -------------------------------- ### Sec-CH-UA-Mobile Client Hint Examples Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/http-headers.md This header indicates whether the client is a mobile device. It uses a structured header format where '?0' signifies false (desktop) and '?1' signifies true (mobile). ```http Sec-CH-UA-Mobile: ?0 # Desktop (false) ``` ```http Sec-CH-UA-Mobile: ?1 # Mobile (true) ``` -------------------------------- ### Typical Build Update Process Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md This bash script outlines the typical process for updating Curl-Impersonate when upstream curl releases security updates. It involves checking out the upstream version, applying patches, configuring, building, and installing. ```bash cd curl-src git checkout upstream-version git apply patches/curl-impersonate.patch ./configure --prefix=/usr/local make make install ``` -------------------------------- ### Detailed Chrome 104 wrapper script example Source: https://github.com/lwthiker/curl-impersonate/blob/main/docs/02_USAGE.md This script demonstrates how to configure curl-impersonate-chrome to match Chrome 104's TLS and HTTP/2 signatures, including specific ciphers, headers, and TLS extensions. The '$@' passes any additional arguments to the underlying curl command. ```bash "$dir/curl-impersonate-chrome" \ --ciphers TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-CHACHA20-POLY1305,ECDHE-RSA-CHACHA20-POLY1305,ECDHE-RSA-AES128-SHA,ECDHE-RSA-AES256-SHA,AES128-GCM-SHA256,AES256-GCM-SHA384,AES128-SHA,AES256-SHA \ -H 'sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Google Chrome";v="104"' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'sec-ch-ua-platform: "Windows"' \ -H 'Upgrade-Insecure-Requests: 1' \ -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36' \ -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \ -H 'Sec-Fetch-Site: none' \ -H 'Sec-Fetch-Mode: navigate' \ -H 'Sec-Fetch-User: ?1' \ -H 'Sec-Fetch-Dest: document' \ -H 'Accept-Encoding: gzip, deflate, br' \ -H 'Accept-Language: en-US,en;q=0.9' \ --http2 --compressed \ --tlsv1.2 --no-npn --alps \ --cert-compression brotli \ "$@" ``` -------------------------------- ### Using curl_easy_impersonate with Standard libcurl Options Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/libcurl-api.md Demonstrates setting up impersonation using `curl_easy_impersonate` and then configuring standard libcurl options. Shows that impersonation settings can be overridden by subsequent standard options. ```c CURL *handle = curl_easy_init(); // Use curl_easy_impersonate to set up impersonation curl_easy_impersonate(handle, "chrome116", 1); // Standard options still work curl_easy_setopt(handle, CURLOPT_URL, "https://api.example.com"); curl_easy_setopt(handle, CURLOPT_CUSTOMREQUEST, "POST"); curl_easy_setopt(handle, CURLOPT_POSTFIELDS, "{\"key\": \"value\"}"); curl_easy_setopt(handle, CURLOPT_TIMEOUT, 30L); // Options set by curl_easy_impersonate can be overridden curl_easy_setopt(handle, CURLOPT_SSL_CIPHER_LIST, "custom-ciphers"); // Perform request curl_easy_perform(handle); ``` -------------------------------- ### Parse TLS String List Example Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md Example usage of the `parse_tls_int_list` function to parse a list of cipher suites from binary data. ```python ciphers, size = parse_tls_int_list(data, entry_size=2) ``` -------------------------------- ### Check Library Path and Dependencies Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md Verify the correct path to the library and check its dependencies using 'ls -la' and 'ldd'. ```bash ls -la /path/to/library ldd /path/to/library ``` -------------------------------- ### HTTP/2 Pseudo-Headers Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/tls-http2-configuration.md List of HTTP/2 pseudo-headers that start with a colon, followed by their meanings. ```text :method HTTP method (GET, POST, etc.) :scheme Protocol (https, http) :authority Host:port :path Request path and query ``` -------------------------------- ### Firefox pkg-config File Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Content of the libcurl-impersonate-ff.pc file, used for build system integration. ```text prefix=/usr/local exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include Name: libcurl-impersonate-ff Description: Browser-impersonating HTTP client library (Firefox variant) Version: 8.X.X Libs: -L${libdir} -lcurl-impersonate-ff Cflags: -I${includedir} Requires: nss ``` -------------------------------- ### Running curl-impersonate Tests Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md Execute the test suite using pytest, specifying the installation directory and capture interface. ```bash pytest tests/ --install_dir=/path/to/install --capture_interface=eth0 -v ``` -------------------------------- ### Basic CLI Usage Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/overview.md Use the wrapper script with standard curl options and a URL to make an impersonated request. ```bash curl_chrome116 [curl-options] URL ``` -------------------------------- ### Build Docker Image for Tests Source: https://github.com/lwthiker/curl-impersonate/blob/main/tests/README.md Build the Docker image containing the tests. Ensure you have built the curl-impersonate Docker images first. ```bash docker build -t curl-impersonate-tests tests/ ``` -------------------------------- ### Chrome pkg-config File Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Content of the libcurl-impersonate-chrome.pc file, used for build system integration. ```text prefix=/usr/local exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include Name: libcurl-impersonate-chrome Description: Browser-impersonating HTTP client library (Chrome variant) Version: 8.X.X Libs: -L${libdir} -lcurl-impersonate-chrome Cflags: -I${includedir} Requires: openssl ``` -------------------------------- ### CURLOPT_SSL_CERT_COMPRESSION Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/libcurl-api.md Sets the certificate compression algorithm. Examples include "brotli", "zstd", and "deflate". ```APIDOC ## CURLOPT_SSL_CERT_COMPRESSION ### Description Sets the certificate compression algorithm. Examples include "brotli", "zstd", and "deflate". ### Type CURLOPTTYPE_STRINGPOINT ### Example ```c curl_easy_setopt(handle, CURLOPT_SSL_CERT_COMPRESSION, "brotli"); ``` ``` -------------------------------- ### Launch HTTP/2 Server Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md Launches an HTTP/2 server using nghttpd for testing headers. The server is configured with a specific port and SSL certificates. The process is managed to ensure it's terminated after the test. ```python import asyncio @pytest.fixture async def nghttpd(): proc = await asyncio.create_subprocess_exec( "nghttpd", "-v", "8443", "ssl/server.key", "ssl/server.crt", stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE ) yield proc proc.terminate() await proc.wait() ``` -------------------------------- ### Python Test Method with Parametrization Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md Demonstrates how to use the parametrize decorator to run a custom test method against all configured curl binaries, environment variables, and preload settings. ```python @pytest.mark.parametrize( "curl_binary, env_vars, ld_preload, expected_signature", CURL_BINARIES_AND_SIGNATURES ) def test_custom_behavior(self, curl_binary, env_vars, ld_preload, expected_signature): # Test with each curl binary pass ``` -------------------------------- ### Configure curl Build Flags Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Demonstrates how to pass custom flags to curl's configure script when configuring curl-impersonate. ```sh ../configure CURL_CONFIG_FLAGS="--disable-rtsp" ``` -------------------------------- ### C Code Example with Built-in Headers Disabled Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md With CURL_IMPERSONATE_HEADERS set to 'no', only the headers explicitly defined in the application code are sent, excluding any built-in headers. ```c // Application code struct curl_slist *headers = NULL; headers = curl_slist_append(headers, "Custom-Header: value"); curl_easy_setopt(handle, CURLOPT_HTTPHEADER, headers); // Actual headers sent: // Custom-Header: value // (no built-in headers) ``` -------------------------------- ### Get libcurl Version Information Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/build-targets.md Retrieve version information for the libcurl library from C code. Use `curl_version()` for the version string and `LIBCURL_VERSION_NUM` for a numeric identifier. ```c curl_version() // Returns version string LIBCURL_VERSION_NUM // Numeric version identifier ``` -------------------------------- ### Verbose Output for Web Request Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Shows detailed information about the request and response, including TLS handshake and HTTP headers. ```bash curl_chrome116 -v https://example.com ``` -------------------------------- ### Basic Usage of Wrapper Script Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/command-line-interface.md Use the wrapper scripts like standard curl commands. The script automatically applies impersonation flags. ```bash curl_chrome116 [curl-options] ``` -------------------------------- ### TLS Client Hello Version Mapping Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/browser-signatures.md This table maps integer values to their corresponding TLS versions used in the Client Hello signature. ```markdown | Value | Version | |-------|---------| | 769 | SSL 3.0 | | 770 | TLS 1.0 | | 771 | TLS 1.2 | | 772 | TLS 1.3 | ``` -------------------------------- ### Node.js Addon: Call libcurl-impersonate directly Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Create a native Node.js addon to directly interface with the libcurl-impersonate library for advanced usage. Requires compiling the addon. ```cpp // addon.cc #include #include namespace addon { using v8::FunctionCallbackInfo; using v8::Isolate; using v8::String; void FetchUrl(const FunctionCallbackInfo& args) { Isolate* isolate = args.GetIsolate(); // Extract URL from arguments String::Utf8Value url(isolate, args[0]); // Use libcurl-impersonate CURL *curl = curl_easy_init(); if (!curl) { isolate->ThrowException(v8::Exception::Error( String::NewFromUtf8(isolate, "curl_easy_init failed").ToLocalChecked() )); return; } curl_easy_impersonate(curl, "chrome116", 1); curl_easy_setopt(curl, CURLOPT_URL, *url); curl_easy_perform(curl); curl_easy_cleanup(curl); } void Initialize(v8::Local exports) { NODE_SET_METHOD(exports, "fetchUrl", FetchUrl); } NODE_MODULE(addon, Initialize) } ``` -------------------------------- ### Replacing libcurl at Runtime on Linux Source: https://github.com/lwthiker/curl-impersonate/blob/main/docs/03_LIBCURL_IMPERSONATE_PHP.md Preload the patched libcurl-impersonate library using LD_PRELOAD and specify the desired impersonation profile to verify the curl version in PHP. ```bash LD_PRELOAD=/path/to/libcurl-impersonate-chrome.so CURL_IMPERSONATE=chrome101 php -r 'print_r(curl_version());' ``` -------------------------------- ### PHP: Use proc_open with LD_PRELOAD for libcurl-impersonate Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Utilize PHP's proc_open to execute a process with LD_PRELOAD set, enabling libcurl-impersonate for the executed command. This method injects the shared library at runtime. ```php array("pipe", "r"), 1 => array("pipe", "w"), 2 => array("pipe", "w") ); $env = array( 'LD_PRELOAD' => '/path/to/libcurl-impersonate-chrome.so', 'CURL_IMPERSONATE' => 'chrome116' ); $process = proc_open( 'php -r "curl_example.php"', $descriptorspec, $pipes, null, $env ); if (is_resource($process)) { $output = stream_get_contents($pipes[1]); fclose($pipes[1]); proc_close($process); echo $output; } ?> ``` -------------------------------- ### Download Multiple URLs Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Downloads multiple URLs sequentially in a single command. ```bash curl_chrome116 https://example.com https://example.org https://example.net ``` -------------------------------- ### Runtime Impersonation with LD_PRELOAD and CURL_IMPERSONATE Source: https://github.com/lwthiker/curl-impersonate/blob/main/README.md Preload `libcurl-impersonate.so` and set the `CURL_IMPERSONATE` environment variable to automatically apply impersonation settings on curl handle initialization and reset. ```bash LD_PRELOAD=/path/to/libcurl-impersonate.so CURL_IMPERSONATE=chrome116 my_app ``` -------------------------------- ### Link libcurl-impersonate with GCC Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/libcurl-api.md Link your C application with libcurl-impersonate using GCC. Ensure you include the necessary SSL and crypto libraries. ```bash gcc myapp.c -o myapp -lcurl-impersonate-chrome -lssl -lcrypto ``` -------------------------------- ### Set SSL Certificate File Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md Specifies the path to a CA certificate bundle file. Use this when all your CA certificates are in a single file. ```bash export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt curl_chrome116 https://example.com ``` -------------------------------- ### Basic Web Request with Chrome 116 Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Fetches a URL using the Chrome 116 user agent and outputs the response to standard output. ```bash curl_chrome116 https://example.com ``` -------------------------------- ### Test HTTP/2 Headers Order Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md Verifies HTTP/2 pseudo-header and header order. Asserts correct exit code and header order matching. ```python async def test_http2_headers(self, pytestconfig, nghttpd, curl_binary, env_vars, ld_preload, browser_signatures, expected_signature) ``` -------------------------------- ### Verify Environment Variable Export Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md Use the 'export' command to ensure an environment variable is set and the 'env | grep' command to verify the application can see it. ```bash export VAR=value env | grep CURL_IMPERSONATE ``` -------------------------------- ### C/C++: Basic libcurl-impersonate Integration Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Basic integration of libcurl-impersonate in C/C++ to make an HTTP request impersonating a specific browser. Writes output to a file. ```c #include #include size_t write_callback(void *contents, size_t size, size_t nmemb, void *userp) { ((size_t)size * nmemb) bytes written to userp fwrite(contents, size, nmemb, (FILE *)userp); return size * nmemb; } int main() { CURL *curl = curl_easy_init(); if (!curl) return 1; // Impersonate Chrome 116 curl_easy_impersonate(curl, "chrome116", 1); // Set URL curl_easy_setopt(curl, CURLOPT_URL, "https://example.com"); // Write to file FILE *f = fopen("output.html", "w"); curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_callback); curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)f); // Perform request CURLcode res = curl_easy_perform(curl); if (res != CURLE_OK) { fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res)); } fclose(f); curl_easy_cleanup(curl); return 0; } ``` -------------------------------- ### Configure Curl Handle for Browser Impersonation Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/libcurl-api.md Use `curl_easy_impersonate` to set up a curl handle to mimic a specific browser. Set `default_headers` to 1 to use built-in headers or 0 to provide custom headers. ```c #include int main() { CURL *handle = curl_easy_init(); if (!handle) return 1; // Configure for Chrome 116 with default headers CURLcode ret = curl_easy_impersonate(handle, "chrome116", 1); if (ret != CURLE_OK) { fprintf(stderr, "Impersonation failed\n"); curl_easy_cleanup(handle); return 1; } // Set URL and perform request curl_easy_setopt(handle, CURLOPT_URL, "https://example.com"); curl_easy_perform(handle); curl_easy_cleanup(handle); return 0; } ``` -------------------------------- ### Set Proxy for All Protocols with all_proxy Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md Use the all_proxy environment variable to specify a proxy server that applies to all protocols. ```bash export all_proxy=socks5://proxy.example.com:1080 curl_chrome116 https://example.com ``` -------------------------------- ### Common TLS Extension Types and Fields Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/browser-signatures.md This table outlines common TLS extension types and their associated fields, useful for understanding Client Hello configurations. ```markdown | Type | | Fields | | Description | | `supported_versions` | | `versions: [list]` | | TLS versions (1.3, 1.2) | | `supported_groups` | | `groups: [list]` | | Elliptic curves (x25519, secp256r1) | | `signature_algorithms` | | `algorithms: [list]` | | Signature algorithms | | `application_layer_protocol_negotiation` | | `protocols: [list]` | | ALPN (http/1.1, h2) | | `ec_point_formats` | | `formats: [list]` | | EC point formats | | `server_name` | | `names: [string]` | | Server names (SNI) | | `key_share` | | `groups: [list]` | | Key share groups | | `padding` | | `length: int` | | Padding length | | `keyshare` | | `shares: [object]` | | Shared keys | ``` -------------------------------- ### Using CURL_IMPERSONATE environment variable Source: https://github.com/lwthiker/curl-impersonate/blob/main/README.md This method allows runtime replacement of libcurl with libcurl-impersonate using `LD_PRELOAD`. It automatically configures impersonation for new or reset curl handles based on the `CURL_IMPERSONATE` environment variable. ```APIDOC ## Using CURL_IMPERSONATE env var ### Description This method allows runtime replacement of libcurl with libcurl-impersonate using `LD_PRELOAD` (Linux only). It automatically configures impersonation for new or reset curl handles based on the `CURL_IMPERSONATE` environment variable. ### Usage Example ```bash LD_PRELOAD=/path/to/libcurl-impersonate.so CURL_IMPERSONATE=chrome116 my_app ``` ### Environment Variables * **`CURL_IMPERSONATE`**: Specifies the target browser name (e.g., `chrome116`). When set, `curl_easy_impersonate()` is automatically called for new curl handles (`curl_easy_init()`) and after reset (`curl_easy_reset()`). * **`CURL_IMPERSONATE_HEADERS`**: If set to `no`, the built-in list of HTTP headers is disabled, allowing manual configuration via `curl_easy_setopt()` and `CURLOPT_HTTPHEADER`. ### Example with Disabled Built-in Headers ```bash LD_PRELOAD=/path/to/libcurl-impersonate.so CURL_IMPERSONATE=chrome116 CURL_IMPERSONATE_HEADERS=no my_app ``` ### Limitations * The `LD_PRELOAD` method does not work for the `curl` command-line tool itself because the tool overrides TLS settings. Use the wrapper scripts for the `curl` tool. ``` -------------------------------- ### Build Firefox version of curl-impersonate with Docker Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Create a Debian-based Docker image for the Firefox version of curl-impersonate. The resulting binaries are statically compiled. ```sh docker build -t curl-impersonate-ff firefox/ ``` -------------------------------- ### HTTP Basic Authentication Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Performs HTTP basic authentication using provided username and password. ```bash curl_chrome116 \ -u username:password \ https://example.com/protected ``` -------------------------------- ### Insert Libraries with DYLD_INSERT_LIBRARIES (macOS) Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md On macOS, DYLD_INSERT_LIBRARIES serves a similar purpose to LD_PRELOAD on Linux, allowing you to insert dynamic libraries. ```bash export DYLD_INSERT_LIBRARIES=/path/to/libcurl-impersonate-chrome.dylib ./myapp ``` -------------------------------- ### Docker: Run Firefox Impersonation Source: https://github.com/lwthiker/curl-impersonate/blob/main/README.md Pulls and runs the Firefox version of curl-impersonate from Docker Hub using an Alpine Linux base image. ```bash # Firefox version, Alpine Linux docker pull lwthiker/curl-impersonate:0.6-ff docker run --rm lwthiker/curl-impersonate:0.6-ff curl_ff109 https://www.wikipedia.org ``` -------------------------------- ### Test Suite Architecture Overview Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/INDEX.md Overview of the test suite architecture, focusing on signature parsing and methodologies. ```markdown ### [testing-signatures.md](testing-signatures.md) Test suite architecture, signature parsing, and test methodologies. **Sections**: - Signature module classes - TLSVersion, TLSExtensionType - TLSExtensionSignature, TLSClientHelloSignature - HTTP2Signature, BrowserSignature - Utility functions for parsing - Test suite classes and methods - TestSignatureModule - TestImpersonation (19+ test methods) - Test fixtures (tcpdump, nghttpd) - Binary packet parsing - YAML signature format - Platform-specific considerations - Test extension examples **Test Methods Coverage**: - TLS Client Hello signature verification - HTTP/2 header order verification - Content encoding verification - Built-in header override verification - User-Agent override verification ``` -------------------------------- ### Configure Shared Libraries with LD_LIBRARY_PATH (Linux) Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/environment-variables.md On Linux, use LD_LIBRARY_PATH to specify directories where shared libraries should be searched. ```bash export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH LD_PRELOAD=/path/to/libcurl-impersonate-chrome.so myapp ``` -------------------------------- ### Run Chrome v104 impersonated curl using wrapper script Source: https://github.com/lwthiker/curl-impersonate/blob/main/docs/02_USAGE.md Use a wrapper script to generate a TLS and HTTP/2 signature identical to Chrome version 104. Additional command-line flags can be appended. ```bash curl_chrome104 -v -L https://wikipedia.org ``` -------------------------------- ### Build and Upload curl-impersonate Layer for AWS Lambda Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Steps to build a Lambda layer containing curl-impersonate and upload it to AWS. This is necessary for using custom runtimes with curl-impersonate. ```bash # Build layer mkdir -p layer/bin cp /usr/local/bin/curl_chrome116 layer/bin/ cd layer && zip -r layer.zip . && cd .. # Upload layer to Lambda aws lambda publish-layer-version \ --layer-name curl-impersonate \ --zip-file fileb://layer.zip ``` -------------------------------- ### TLS Handshake Flow with HTTP/2 Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/tls-http2-configuration.md Illustrates a typical TLS 1.2 handshake followed by an HTTP/2 preface and request, showing the sequence of messages exchanged between client and server. ```text Client Server |-- ClientHello (+ extensions) -->| |<-- ServerHello (+ extensions) --| |<-- Certificate --------| |<-- ServerKeyExchange --| |<-- ServerHelloDone --| | |-- ClientKeyExchange ---| |-- ChangeCipherSpec ---| |-- Finished ----------->| |<-- ChangeCipherSpec ---| |<-- Finished -----------| | | --- HTTP/2 Preface ---| |--- Settings Frame ----| |--- HTTP Request ------>| ``` -------------------------------- ### Dockerfile Integration Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Integrate curl-impersonate into a Dockerfile by copying binaries and libraries from a base image. ```dockerfile FROM lwthiker/curl-impersonate:0.6-chrome as fetcher FROM ubuntu:20.04 COPY --from=fetcher /usr/local/bin/curl_* /usr/local/bin/ COPY --from=fetcher /usr/local/lib/libcurl* /usr/local/lib/ RUN apt-get update && apt-get install -y libnss3 ca-certificates ENTRYPOINT ["curl_chrome116"] ``` -------------------------------- ### Test TLS Client Hello Signature Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md Verifies TLS Client Hello signature via network capture. Asserts correct exit code, packet capture, and signature matching. ```python def test_tls_client_hello(self, pytestconfig, tcpdump, curl_binary, env_vars, ld_preload, browser_signatures, expected_signature, test_urls) ``` -------------------------------- ### Include Standard Curl Headers Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/libcurl-api.md Include the standard curl headers in your C/C++ project. No additional headers are required for libcurl-impersonate. ```c #include ``` -------------------------------- ### Clone curl-impersonate Repository Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Clones the curl-impersonate repository and navigates into the project directory. ```sh git clone https://github.com/lwthiker/curl-impersonate.git cd curl-impersonate ``` -------------------------------- ### Reuse Curl Handle for Multiple Requests Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/integration-examples.md Demonstrates how to reuse a curl handle for multiple requests to improve performance by avoiding repeated initialization and cleanup. Requires `curl_easy_impersonate` to set the impersonation profile. ```c // Reuse curl handle for multiple requests CURL *curl = curl_easy_init(); curl_easy_impersonate(curl, "chrome116", 1); for (int i = 0; i < 10; i++) { curl_easy_setopt(curl, CURLOPT_URL, urls[i]); curl_easy_perform(curl); } curl_easy_cleanup(curl); ``` -------------------------------- ### Build Chrome version of curl-impersonate with Docker Source: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md Create a Debian-based Docker image for the Chrome version of curl-impersonate. The resulting binaries are statically compiled. ```sh docker build -t curl-impersonate-chrome chrome/ ``` -------------------------------- ### Set SSL Signature Hash Algorithms Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/libcurl-api.md Specify a comma-separated list of signature hash algorithms for SSL connections using CURLOPT_SSL_SIG_HASH_ALGS. ```c curl_easy_setopt(handle, CURLOPT_SSL_SIG_HASH_ALGS, "ecdsa_secp256r1_sha256,rsa_pss_rsae_sha256"); ``` -------------------------------- ### Test Signature Parsing Utilities Source: https://github.com/lwthiker/curl-impersonate/blob/main/_autodocs/testing-signatures.md Tests for verifying YAML serialization roundtrip and parsing real Client Hello binary data. ```python class TestSignatureModule: def test_serialization(self, browser_signatures) def test_tls_client_hello_parsing(self, browser_signatures) ```