### Verify httpmorph Installation Source: https://httpmorph.readthedocs.io/en/latest/installation.html Import the library, print version information, and perform a basic GET request to verify the installation. ```python import httpmorph print(httpmorph.__version__) print(httpmorph.version()) # C library version # Test basic functionality response = httpmorph.get('https://httpbin.org/get') assert response.status_code == 200 ``` -------------------------------- ### httpmorph Quick Examples Source: https://httpmorph.readthedocs.io/en/latest/index.html Illustrates various httpmorph functionalities including GET, POST with JSON, session management with cookies, and HTTP/2 client usage. Import httpmorph before use. ```python import httpmorph # GET request response = httpmorph.get('https://httpbin.org/get') print(response.json()) # POST with JSON response = httpmorph.post( 'https://httpbin.org/post', json={'key': 'value'} ) # Session with cookies session = httpmorph.Session(browser='chrome') response = session.get('https://example.com') print(session.cookies) # HTTP/2 client = httpmorph.Client(http2=True) response = client.get('https://www.google.com') print(response.http_version) # '2.0' ``` -------------------------------- ### Install httpmorph Source: https://httpmorph.readthedocs.io/en/latest/index.html Installs the httpmorph library using pip. Ensure Python 3.8+ is installed. ```bash pip install httpmorph ``` -------------------------------- ### Install nghttp2 on Fedora/RHEL Source: https://httpmorph.readthedocs.io/en/latest/installation.html Troubleshoot import errors on Linux by installing the nghttp2 shared library. ```bash # Fedora/RHEL sudo dnf install libnghttp2 ``` -------------------------------- ### Build httpmorph from Source Source: https://httpmorph.readthedocs.io/en/latest/installation.html Steps to clone the repository, set up vendor dependencies, build Python extensions, and install in development mode. ```bash git clone https://github.com/arman-bd/httpmorph.git cd httpmorph # Build vendor dependencies (BoringSSL, nghttp2) ./scripts/setup_vendors.sh # Build Python extensions python setup.py build_ext --inplace # Install in development mode pip install -e ".[dev]" ``` -------------------------------- ### Install Optional Development Dependencies Source: https://httpmorph.readthedocs.io/en/latest/installation.html Install the 'dev' extra for httpmorph to include development tools like pytest, mypy, and ruff. ```bash pip install httpmorph[dev] ``` -------------------------------- ### Install nghttp2 on Ubuntu/Debian Source: https://httpmorph.readthedocs.io/en/latest/installation.html Troubleshoot import errors on Linux by installing the nghttp2 shared library. ```bash # Ubuntu/Debian sudo apt-get install libnghttp2-14 ``` -------------------------------- ### client.get() Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a GET request using the client instance. ```APIDOC ## Client GET Method ### Description Make a GET request using the client instance. ### Method GET ### Endpoint [URL] ### Parameters #### Path Parameters - **url** (str) - Required - URL to request #### Query Parameters - **kwargs** - See Request Parameters ### Response #### Success Response - **Response object** - The response from the GET request. ``` -------------------------------- ### Linux (Ubuntu/Debian) Build Requirements Source: https://httpmorph.readthedocs.io/en/latest/installation.html Install necessary build tools on Ubuntu/Debian systems using apt-get if building from source. ```bash sudo apt-get install cmake ninja-build libssl-dev pkg-config \ autoconf automake libtool libnghttp2-dev ``` -------------------------------- ### Install Xcode Command Line Tools on macOS Source: https://httpmorph.readthedocs.io/en/latest/installation.html Troubleshoot build errors on macOS by ensuring Xcode Command Line Tools are installed. ```bash xcode-select --install ``` -------------------------------- ### Windows Build Requirements Source: https://httpmorph.readthedocs.io/en/latest/installation.html Install necessary build tools on Windows using Chocolatey if building from source. ```bash choco install cmake golang nasm visualstudio2022buildtools -y ``` -------------------------------- ### Linux (Fedora/RHEL) Build Requirements Source: https://httpmorph.readthedocs.io/en/latest/installation.html Install necessary build tools on Fedora/RHEL systems using dnf if building from source. ```bash sudo dnf install cmake ninja-build openssl-devel pkg-config \ autoconf automake libtool libnghttp2-devel ``` -------------------------------- ### Simple GET Request Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Perform a basic GET request and print the status code and response text. ```python import httpmorph response = httpmorph.get('https://httpbin.org/get') print(response.status_code) print(response.text) ``` -------------------------------- ### Set and Get Cookies Manually Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Demonstrates how to manually set cookies for a session and retrieve them from a response. ```python session = httpmorph.Session() # Set cookies manually session.cookies = {'session_id': 'abc123'} # Make request with cookies response = session.get('https://example.com') # Get cookies from response print(session.cookies) ``` -------------------------------- ### Basic httpmorph Usage Source: https://httpmorph.readthedocs.io/en/latest/index.html Demonstrates simple GET requests and using sessions with browser profiles. Requires the httpmorph library to be imported. ```python import httpmorph # Simple GET request response = httpmorph.get('https://example.com') print(response.status_code, response.text) # Use a session with browser profile session = httpmorph.Session(browser='chrome') response = session.get('https://example.com') ``` -------------------------------- ### macOS Build Requirements Source: https://httpmorph.readthedocs.io/en/latest/installation.html Install necessary build tools on macOS using Homebrew if building from source. ```bash brew install cmake ninja libnghttp2 ``` -------------------------------- ### Perform GET Request with Session Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a GET request using an existing HTTP session. The response object contains the server's reply. ```python response = session.get(url, **kwargs) ``` -------------------------------- ### Perform Async GET Request Source: https://httpmorph.readthedocs.io/en/latest/api.html Make an asynchronous GET request using the AsyncClient. This method returns a coroutine. ```python response = await client.get(url, **kwargs) ``` -------------------------------- ### HTTP GET Request Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a GET request to a specified URL. Accepts additional keyword arguments for request parameters. ```python response = httpmorph.get(url, **kwargs) ``` -------------------------------- ### Async HTTP Request with HTTPMorph Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Demonstrates how to perform a single asynchronous HTTP GET request using HTTPMorph's AsyncClient. Ensure you have asyncio imported and configured. ```python import asyncio import httpmorph async def fetch(): async with httpmorph.AsyncClient() as client: response = await client.get('https://httpbin.org/get') print(response.status_code) return response asyncio.run(fetch()) ``` -------------------------------- ### Client HTTP Methods Source: https://httpmorph.readthedocs.io/en/latest/api.html Use client instance to make HTTP requests like GET and POST. Supports various data formats for POST requests. ```python response = client.get(url, **kwargs) ``` ```python response = client.post(url, data=None, json=None, **kwargs) ``` -------------------------------- ### Set Query Parameters Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Include query parameters in the URL for GET requests. The library automatically encodes them. ```python params = {'key': 'value', 'foo': 'bar'} response = httpmorph.get(url, params=params) # Requests: https://example.com?key=value&foo=bar ``` -------------------------------- ### Route Requests Through HTTP Proxy Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Configure a simple HTTP proxy for GET requests. Ensure the proxy server is accessible. ```python response = httpmorph.get( 'http://example.com', proxy='http://proxy.example.com:8080' ) ``` -------------------------------- ### httpmorph.get() Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a GET request to a specified URL. Accepts additional keyword arguments for request customization. ```APIDOC ## GET ### Description Make a GET request to a specified URL. ### Method GET ### Endpoint [URL] ### Parameters #### Path Parameters - **url** (str) - Required - URL to request #### Query Parameters - **kwargs** - See Request Parameters ### Response #### Success Response - **Response object** - The response from the GET request. ``` -------------------------------- ### Check HTTP Version of Response Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Provides an example of how to check the HTTP version used for a response. This can be helpful for debugging or verifying connection protocols. ```python response = client.get('https://www.google.com') if response.http_version == '2.0': print('Using HTTP/2') ``` -------------------------------- ### Get HTTPMorph Version Information Source: https://httpmorph.readthedocs.io/en/latest/api.html Retrieve the Python package version and the C library version using the httpmorph library. ```python import httpmorph # Python package version print(httpmorph.__version__) # C library version print(httpmorph.version()) ``` -------------------------------- ### Perform Non-Blocking HTTP Request with AsyncClient Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Demonstrates making a single non-blocking GET request using HTTPMorph's AsyncClient. Requires an asyncio event loop. ```python import asyncio import httpmorph async def main(): async with httpmorph.AsyncClient() as client: # Non-blocking request response = await client.get('https://httpbin.org/delay/2') print(response.status_code) asyncio.run(main()) ``` -------------------------------- ### Access Response Content Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Get the raw response body as bytes, decoded text, or parse JSON directly. ```python # Content print(response.body) # bytes print(response.text) # str (decoded) print(response.content) # bytes (alias) # JSON data = response.json() ``` -------------------------------- ### Access Final URL Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Get the URL of the response, which may differ from the original request URL if redirects occurred. ```python # URL print(response.url) # Final URL after redirects ``` -------------------------------- ### Access Response Status Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Retrieve the HTTP status code, check if the request was successful (status code 2xx-3xx), and get the reason phrase. ```python response = httpmorph.get('https://httpbin.org/get') # Status print(response.status_code) # 200 print(response.ok) # True for 200-399 print(response.reason) # 'OK' ``` -------------------------------- ### Initialize httpmorph Client with HTTP/2 Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Shows how to create an httpmorph Client that defaults to using HTTP/2, mimicking browser behavior. This can lead to performance improvements. ```python # Both Client and Session default to HTTP/2 (http2=True) client = httpmorph.Client() response = client.get('https://www.google.com') print(response.http_version) # '2.0' ``` -------------------------------- ### Initialize Async HTTP Client Source: https://httpmorph.readthedocs.io/en/latest/api.html Create an asynchronous HTTP client that uses non-blocking I/O. Note that DNS resolution remains blocking. ```python client = httpmorph.AsyncClient() ``` -------------------------------- ### All HTTP Methods Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Demonstrates the usage of all available HTTP methods provided by httpmorph. ```python httpmorph.get(url) httpmorph.post(url, data=...) httpmorph.put(url, data=...) httpmorph.delete(url) httpmorph.head(url) httpmorph.patch(url, data=...) httpmorph.options(url) ``` -------------------------------- ### Enable HTTP/2 with Client Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Create a client with HTTP/2 support enabled and check the HTTP version. ```python client = httpmorph.Client(http2=True) response = client.get('https://www.google.com') print(response.http_version) # '2.0' ``` -------------------------------- ### Initialize HTTP Client Source: https://httpmorph.readthedocs.io/en/latest/api.html Create an HTTP client instance. Defaults to HTTP/2 to match Chrome behavior. Can be configured to disable HTTP/2. ```python client = httpmorph.Client(http2=True) ``` -------------------------------- ### Initialize httpmorph Session with HTTP/2 Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Demonstrates creating an httpmorph Session configured to use HTTP/2, similar to Chrome's default. Sessions are useful for maintaining connection state across multiple requests. ```python session = httpmorph.Session(browser='chrome') response = session.get('https://www.google.com') print(response.http_version) # '2.0' ``` -------------------------------- ### Load Custom CA Bundle for SSL Verification Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Shows how to load a custom CA bundle using `certifi.where()` to enhance SSL certificate verification. This is a recommended practice for secure connections. ```python import certifi import httpmorph client = httpmorph.Client() client.load_ca_file(certifi.where()) response = client.get('https://example.com') ``` -------------------------------- ### Create and Use HTTPMorph Client Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Instantiate a client for more granular control over requests. ```python client = httpmorph.Client() response = client.get('https://example.com') ``` -------------------------------- ### Reuse HTTP Sessions for Efficiency Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Demonstrates the difference between creating new connections for each request versus reusing a session. Use sessions for multiple requests to the same host to improve performance. ```python # Bad - creates new connection each time for i in range(100): httpmorph.get('https://example.com') # Good - reuses connection session = httpmorph.Session() for i in range(100): session.get('https://example.com') ``` -------------------------------- ### httpmorph.Client() Source: https://httpmorph.readthedocs.io/en/latest/api.html Initialize an HTTP client for making requests. Defaults to HTTP/2. ```APIDOC ## Client Class ### Description HTTP client for making requests. Defaults to HTTP/2 to match Chrome behavior. ### Constructor `client = httpmorph.Client(http2=True)` ### Constructor Parameters - **http2** (bool) - Optional - Enable HTTP/2. Default: `True` ``` -------------------------------- ### Configure Per-Protocol Proxies Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Specify different proxy servers for HTTP and HTTPS protocols using a dictionary. This allows for granular control over routing. ```python proxies = { 'http': 'http://http-proxy.example.com:8080', 'https': 'http://https-proxy.example.com:8080' } response = httpmorph.get(url, proxies=proxies) ``` -------------------------------- ### client.post() Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a POST request using the client instance. Supports sending data or JSON. ```APIDOC ## Client POST Method ### Description Make a POST request using the client instance. Supports sending data or JSON in the request body. ### Method POST ### Endpoint [URL] ### Parameters #### Path Parameters - **url** (str) - Required - URL to request #### Request Body - **data** (bytes/str/dict) - Optional - Request body or form data - **json** (dict) - Optional - JSON data to send #### Query Parameters - **kwargs** - See Request Parameters ### Response #### Success Response - **Response object** - The response from the POST request. ``` -------------------------------- ### Initialize HTTP Session Source: https://httpmorph.readthedocs.io/en/latest/api.html Create an HTTP session mimicking a Chrome browser on macOS with HTTP/2 enabled. Sessions maintain persistent cookies and headers. ```python session = httpmorph.Session(browser='chrome', os='macos', http2=True) ``` -------------------------------- ### Proxy Authentication with Username and Password Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Set up basic authentication for proxy access by providing a tuple of username and password. Alternatively, embed credentials directly in the proxy URL. ```python response = httpmorph.get( url, proxy='http://proxy.example.com:8080', proxy_auth=('username', 'password') ) ``` ```python response = httpmorph.get( url, proxy='http://user:pass@proxy.example.com:8080' ) ``` -------------------------------- ### Verifying Connection Reuse with Timing Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Illustrates how to check connection reuse by comparing connect and TLS times for sequential requests. Lower or zero times indicate reuse. ```python client = httpmorph.Client() r1 = client.get('https://example.com') print(r1.connect_time_us, r1.tls_time_us) # Non-zero r2 = client.get('https://example.com') print(r2.connect_time_us, r2.tls_time_us) # Much lower or zero ``` -------------------------------- ### Upload File with Filename and Content Type Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Illustrates how to specify a custom filename and content type for an uploaded file. This is useful when the server needs specific metadata about the file. ```python # With filename and content type files = { 'file': ('report.pdf', open('report.pdf', 'rb'), 'application/pdf') } response = httpmorph.post(url, files=files) ``` -------------------------------- ### client.load_ca_file() Source: https://httpmorph.readthedocs.io/en/latest/api.html Load CA certificates from a file (PEM format) for the client. ```APIDOC ## Client load_ca_file Method ### Description Load CA certificates from file (PEM format). ### Method N/A (Client method) ### Parameters #### Path Parameters - **ca_file** (str) - Required - Path to CA certificate bundle ### Response #### Success Response - **bool** - True on success. ``` -------------------------------- ### Check HTTP Version Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Demonstrates how to check if a connection is using HTTP/2 or has fallen back to HTTP/1.1. ```python client = httpmorph.Client(http2=True) response = client.get('https://www.google.com') if response.http_version == '2.0': print('Using HTTP/2') else: print('Fell back to HTTP/1.1') ``` -------------------------------- ### Enabling HTTP/2 with Client Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Configures a httpmorph.Client to attempt HTTP/2 connections. The response will indicate the negotiated HTTP version. ```python client = httpmorph.Client(http2=True) response = client.get('https://www.google.com') print(response.http_version) # '2.0' if server supports it ``` -------------------------------- ### Initialize and Cleanup HTTPMorph Library Source: https://httpmorph.readthedocs.io/en/latest/api.html Initialize the library resources, which is called automatically on import. Cleanup library resources when no longer needed. ```python httpmorph.init() ``` ```python httpmorph.cleanup() ``` -------------------------------- ### Load CA Certificates Source: https://httpmorph.readthedocs.io/en/latest/api.html Load custom CA certificates from a PEM-formatted file into the client. Returns true on success. ```python client.load_ca_file(ca_file) ``` -------------------------------- ### httpmorph.options() Source: https://httpmorph.readthedocs.io/en/latest/api.html Make an OPTIONS request to a specified URL. ```APIDOC ## OPTIONS ### Description Make an OPTIONS request to a specified URL. ### Method OPTIONS ### Endpoint [URL] ### Parameters #### Path Parameters - **url** (str) - Required - URL to request #### Query Parameters - **kwargs** - See Request Parameters ### Response #### Success Response - **Response object** - The response from the OPTIONS request. ``` -------------------------------- ### Upload Multiple Files with httpmorph Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Demonstrates uploading multiple files simultaneously by providing a dictionary of file objects to the 'files' parameter. Each key in the dictionary corresponds to a form field name. ```python # Multiple files files = { 'file1': open('report.pdf', 'rb'), 'file2': open('data.csv', 'rb') } response = httpmorph.post(url, files=files) ``` -------------------------------- ### Basic Authentication Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Perform HTTP Basic Authentication by providing username and password. ```python # Basic authentication response = httpmorph.get( url, auth=('username', 'password') ) ``` -------------------------------- ### Use AsyncClient as Context Manager Source: https://httpmorph.readthedocs.io/en/latest/api.html Manage the lifecycle of an asynchronous HTTP client using an 'async with' statement. This ensures proper cleanup of resources. ```python async with httpmorph.AsyncClient() as client: response = await client.get(url) ``` -------------------------------- ### Automatic Connection Reuse with Client Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Demonstrates how a httpmorph.Client automatically pools and reuses connections for subsequent requests to the same host. ```python client = httpmorph.Client() # First request creates connection r1 = client.get('https://example.com/page1') # Second request reuses connection r2 = client.get('https://example.com/page2') ``` -------------------------------- ### Set Default Timeout with Client Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Configure a client with a default timeout value for requests. ```python client = httpmorph.Client(timeout=10) response = client.get('https://example.com') ``` -------------------------------- ### HTTP OPTIONS Request Source: https://httpmorph.readthedocs.io/en/latest/api.html Make an OPTIONS request to a specified URL. Accepts additional keyword arguments for request parameters. ```python response = httpmorph.options(url, **kwargs) ``` -------------------------------- ### Handle HTTP Exceptions with httpmorph Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Demonstrates how to catch specific httpmorph exceptions like Timeout, ConnectionError, and HTTPError, as well as the base RequestException. Use this to gracefully handle request failures. ```python import httpmorph try: response = httpmorph.get('https://example.com', timeout=5) response.raise_for_status() # Raise on 4xx/5xx except httpmorph.Timeout: print('Request timed out') except httpmorph.ConnectionError: print('Connection failed') except httpmorph.HTTPError as e: print(f'HTTP error: {e.response.status_code}') except httpmorph.RequestException as e: print(f'Request failed: {e}') ``` -------------------------------- ### HTTP/2 Fallback to HTTP/1.1 Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Demonstrates httpmorph's fallback behavior. If a server does not support HTTP/2, the client will automatically use HTTP/1.1 for the connection. ```python client = httpmorph.Client(http2=True) response = client.get('https://old-server.com') print(response.http_version) # May be '1.1' if server lacks HTTP/2 ``` -------------------------------- ### Analyze HTTP Request Performance Timings Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Measure connection, TLS, first byte, and total request times in microseconds. Useful for identifying bottlenecks in request performance. ```python response = httpmorph.get('https://example.com') print(f'Connection: {response.connect_time_us / 1000:.2f}ms') print(f'TLS: {response.tls_time_us / 1000:.2f}ms') print(f'First byte: {response.first_byte_time_us / 1000:.2f}ms') print(f'Total: {response.total_time_us / 1000:.2f}ms') ``` -------------------------------- ### Customize OS-Specific User Agents Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Allows customization of the User-Agent string to mimic different operating systems. This is useful for testing how a server responds to various OS environments. ```python # macOS user agent (default) session = httpmorph.Session(browser='chrome', os='macos') response = session.get('https://httpbin.org/user-agent') # User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ... ``` ```python # Windows user agent session = httpmorph.Session(browser='chrome', os='windows') response = session.get('https://httpbin.org/user-agent') # User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... ``` ```python # Linux user agent session = httpmorph.Session(browser='chrome', os='linux') response = session.get('https://httpbin.org/user-agent') # User-Agent: Mozilla/5.0 (X11; Linux x86_64) ... ``` -------------------------------- ### HTTPMorph Session as Context Manager Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Use an HTTPMorph session as a context manager for automatic resource management. ```python with httpmorph.Session() as session: response = session.get('https://example.com') ``` -------------------------------- ### httpmorph.head() Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a HEAD request to a specified URL. ```APIDOC ## HEAD ### Description Make a HEAD request to a specified URL. ### Method HEAD ### Endpoint [URL] ### Parameters #### Path Parameters - **url** (str) - Required - URL to request #### Query Parameters - **kwargs** - See Request Parameters ### Response #### Success Response - **Response object** - The response from the HEAD request. ``` -------------------------------- ### Use HTTPS Proxy with CONNECT Tunneling Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Configure an HTTP proxy for HTTPS requests using CONNECT tunneling. The proxy must support this method. ```python response = httpmorph.get( 'https://example.com', proxy='http://proxy.example.com:8080' ) ``` -------------------------------- ### AsyncClient Class Source: https://httpmorph.readthedocs.io/en/latest/api.html An asynchronous HTTP client utilizing non-blocking I/O. ```APIDOC ## AsyncClient Class ### Description An asynchronous HTTP client that uses non-blocking I/O (epoll/kqueue). Note that DNS resolution remains blocking. ### Constructor `httpmorph.AsyncClient()` ### Methods All HTTP methods return coroutines. #### GET `response = await client.get(url, **kwargs)` #### POST `response = await client.post(url, data=None, json=None, **kwargs)` ### Context Manager ```python async with httpmorph.AsyncClient() as client: response = await client.get(url) ``` ``` -------------------------------- ### Use AsyncClient for Concurrent I/O-Bound Requests Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Compares synchronous and asynchronous approaches for making multiple concurrent requests. The asynchronous method using AsyncClient is significantly faster for I/O-bound operations. ```python # Synchronous - slow urls = ['https://example.com'] * 100 for url in urls: httpmorph.get(url) # Asynchronous - fast async def fetch_all(): async with httpmorph.AsyncClient() as client: tasks = [client.get(url) for url in urls] await asyncio.gather(*tasks) asyncio.run(fetch_all()) ``` -------------------------------- ### Concurrent HTTP Requests with HTTPMorph Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Shows how to make multiple asynchronous HTTP requests concurrently using asyncio.gather and HTTPMorph's AsyncClient. This is useful for fetching data from multiple URLs simultaneously. ```python import asyncio import httpmorph async def fetch_all(urls): async with httpmorph.AsyncClient() as client: tasks = [client.get(url) for url in urls] responses = await asyncio.gather(*tasks) return responses urls = [ 'https://httpbin.org/get', 'https://httpbin.org/headers', 'https://httpbin.org/user-agent' ] responses = asyncio.run(fetch_all(urls)) for response in responses: print(response.status_code, response.url) ``` -------------------------------- ### Use Session as Context Manager Source: https://httpmorph.readthedocs.io/en/latest/api.html Manage an HTTP session's lifecycle using a 'with' statement. Ensures the session is properly closed after use. ```python with httpmorph.Session() as session: response = session.get(url) ``` -------------------------------- ### Automatic Connection Reuse with Session Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Shows how httpmorph.Session also benefits from connection pooling, reusing a single connection for multiple requests within a session context. ```python with httpmorph.Session() as session: for i in range(100): response = session.get(f'https://example.com/page{i}') # All requests reuse the same connection ``` -------------------------------- ### Upload Single File with httpmorph Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Shows how to upload a single file using the 'files' parameter in an httpmorph POST request. Ensure the file is opened in binary read mode ('rb'). ```python # Single file files = {'file': open('report.pdf', 'rb')} response = httpmorph.post('https://httpbin.org/post', files=files) ``` -------------------------------- ### Use Chrome Browser Profiles in Session Source: https://httpmorph.readthedocs.io/en/latest/api.html Instantiate an httpmorph Session using different Chrome browser profiles. The default profile mimics Chrome 143. ```python # Use Chrome 143 profile (default) session = httpmorph.Session(browser='chrome') # Explicitly use Chrome 143 session = httpmorph.Session(browser='chrome143') # With specific OS session = httpmorph.Session(browser='chrome', os='windows') ``` ```python session = httpmorph.Session(browser='chrome127') session = httpmorph.Session(browser='chrome135') # etc. ``` ```python # Randomly selects a browser profile for each session from available Chrome profiles. session = httpmorph.Session(browser='random') ``` -------------------------------- ### Simulate OS-Specific User Agents Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Use the `os` parameter with a 'chrome' browser profile to simulate requests from different operating systems. This primarily affects the User-Agent string. ```python import httpmorph # macOS (default) session = httpmorph.Session(browser='chrome', os='macos') # User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ... # Windows session = httpmorph.Session(browser='chrome', os='windows') # User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... # Linux session = httpmorph.Session(browser='chrome', os='linux') # User-Agent: Mozilla/5.0 (X11; Linux x86_64) ... ``` -------------------------------- ### Access Timing Information Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Inspect various timing metrics of the request, such as total time, connection time, and time to first byte, in microseconds or as a timedelta object. ```python print(response.total_time_us) # Total time in microseconds print(response.connect_time_us) # Connection time print(response.tls_time_us) # TLS handshake time print(response.first_byte_time_us) # Time to first byte print(response.elapsed) # As timedelta ``` -------------------------------- ### Handle GREASE Values for TLS Extensibility Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Demonstrates how Chrome 143 uses GREASE values, which are randomized per request to maintain TLS ecosystem extensibility. While JA3 fingerprints may differ slightly due to this randomization, the normalized JA3N remains consistent. ```python session = httpmorph.Session(browser='chrome') # Each request gets different GREASE values r1 = session.get('https://example.com') r2 = session.get('https://example.com') # JA3 fingerprints will differ slightly due to GREASE randomization # However, JA3N (normalized) remains consistent ``` -------------------------------- ### Per-Request HTTP/2 Override Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Shows how to override the client's default HTTP/2 setting for individual requests. This allows forcing HTTP/2 or using the default (HTTP/1.1) on a per-request basis. ```python # Client defaults to HTTP/1.1 client = httpmorph.Client(http2=False) # Force HTTP/2 for this request r1 = client.get('https://www.google.com', http2=True) print(r1.http_version) # '2.0' # Use default (HTTP/1.1) for this request r2 = client.get('https://example.com') print(r2.http_version) # '1.1' ``` -------------------------------- ### Implement Retry with Exponential Backoff Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Use this pattern to automatically retry failed requests with increasing delays between attempts. Handles timeouts and connection errors. ```python import time import httpmorph def fetch_with_retry(url, max_retries=3): for attempt in range(max_retries): try: response = httpmorph.get(url, timeout=10) response.raise_for_status() return response except (httpmorph.Timeout, httpmorph.ConnectionError): if attempt < max_retries - 1: time.sleep(2 ** attempt) # Exponential backoff else: raise response = fetch_with_retry('https://example.com') ``` -------------------------------- ### HTTP HEAD Request Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a HEAD request to a specified URL. Accepts additional keyword arguments for request parameters. ```python response = httpmorph.head(url, **kwargs) ``` -------------------------------- ### HTTP Request Parameters Source: https://httpmorph.readthedocs.io/en/latest/api.html This section outlines the common parameters that can be passed to any HTTP request method within HTTPMorph. These parameters control various aspects of the request, including headers, query parameters, cookies, authentication, request body, connection settings, SSL/TLS configuration, HTTP/2 support, redirect behavior, and streaming responses. ```APIDOC ## Request Parameters Common parameters accepted by all request methods: ### HTTP Parameters * `headers` (dict) - HTTP headers * `params` (dict) - URL query parameters * `cookies` (dict) - Cookies to send * `auth` (tuple) - Basic authentication: `(username, password)` ### Body Parameters * `data` (bytes/str/dict) - Request body or form data * `json` (dict) - JSON data (auto-serialized with Content-Type header) * `files` (dict) - Files to upload (multipart/form-data) ### Connection Parameters * `timeout` (int/float) - Request timeout in seconds * `proxy` (str) - Proxy URL * `proxy_auth` (tuple) - Proxy authentication: `(username, password)` * `proxies` (dict) - Proxy dict: `{'http': '...', 'https': '...'}` ### SSL/TLS Parameters * `verify` or `verify_ssl` (bool) - Verify SSL certificates. Default: `True` * `tls_version` (str/tuple) - TLS version constraint (e.g., `"1.2"` or `(min, max)`) ### HTTP/2 Parameters * `http2` (bool) - Enable HTTP/2 for this request (overrides client/session default) ### Redirect Parameters * `allow_redirects` (bool) - Follow redirects. Default: `True` * `max_redirects` (int) - Maximum number of redirects. Default: `10` ### Other Parameters * `stream` (bool) - Stream response (use with `iter_content()`) ## Exceptions All exceptions inherit from `RequestException`. ### RequestException Base exception for all httpmorph errors. ### HTTPError Raised when `raise_for_status()` is called on a 4xx/5xx response. **Attributes:** * `response` - The Response object ### ConnectionError Raised when connection fails. ### Timeout Raised when request times out. ### TooManyRedirects Raised when max redirects is exceeded. ``` -------------------------------- ### Perform POST Request with Session Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a POST request with session data. Supports sending raw data or JSON payloads. ```python response = session.post(url, data=None, json=None, **kwargs) ``` -------------------------------- ### httpmorph.post() Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a POST request to a specified URL. Supports sending data or JSON in the request body. ```APIDOC ## POST ### Description Make a POST request to a specified URL. Supports sending data or JSON in the request body. ### Method POST ### Endpoint [URL] ### Parameters #### Path Parameters - **url** (str) - Required - URL to request #### Request Body - **data** (bytes/str/dict) - Optional - Request body or form data - **json** (dict) - Optional - JSON data to send #### Query Parameters - **kwargs** - See Request Parameters ### Response #### Success Response - **Response object** - The response from the POST request. ``` -------------------------------- ### Print Request Details Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Retrieves and prints details of an HTTP request, including URL, status code, headers, and timing breakdown. ```python response = httpmorph.get('https://httpbin.org/get') # Request details (stored in response) print('URL:', response.url) print('Status:', response.status_code) print('Headers:', response.headers) # Timing breakdown print('Connect:', response.connect_time_us / 1000, 'ms') print('TLS:', response.tls_time_us / 1000, 'ms') print('Total:', response.total_time_us / 1000, 'ms') ``` -------------------------------- ### Check Cookie Count Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Shows how to check the number of cookies stored in the session's cookie jar after a request. ```python session = httpmorph.Session() session.get('https://example.com') print(len(session.cookie_jar)) # Number of cookies ``` -------------------------------- ### Perform Async POST Request Source: https://httpmorph.readthedocs.io/en/latest/api.html Make an asynchronous POST request with the AsyncClient, supporting data or JSON payloads. Returns a coroutine. ```python response = await client.post(url, data=None, json=None, **kwargs) ``` -------------------------------- ### HTTPMorph Session for Persistent Cookies Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Utilize a session to maintain cookies and headers across multiple requests. ```python session = httpmorph.Session() # First request sets cookies session.get('https://example.com/login') # Subsequent requests include cookies session.get('https://example.com/protected') # Access cookies print(session.cookies) ``` -------------------------------- ### POST Request with Form Data Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Send a POST request with form data. ```python response = httpmorph.post( 'https://httpbin.org/post', data={'field': 'value'} ) ``` -------------------------------- ### Control Redirects Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Manage how the client handles HTTP redirects. You can disable them or set a maximum limit. ```python # Disable redirect following response = httpmorph.get(url, allow_redirects=False) ``` ```python # Limit max redirects response = httpmorph.get(url, max_redirects=5) ``` -------------------------------- ### Override Session Headers with Per-Request Headers Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Demonstrates how headers provided directly in a request method call take precedence over headers defined in the httpmorph.Session. Ensures flexibility for specific requests. ```python session = httpmorph.Session() session.headers = {'User-Agent': 'MyApp/1.0'} # Uses session User-Agent session.get('https://example.com') # Overrides session User-Agent for this request session.get('https://example.com', headers={'User-Agent': 'CustomBot/2.0'}) ``` -------------------------------- ### Mimic Chrome Browser Fingerprints Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Use the `browser` parameter to mimic Chrome. Defaults to Chrome 143. You can specify a version or use 'random' for selection. ```python import httpmorph # Chrome browser profile (defaults to Chrome 143) session = httpmorph.Session(browser='chrome') response = session.get('https://example.com') # Use specific Chrome version (127-143 supported) session = httpmorph.Session(browser='chrome143') response = session.get('https://example.com') # Random browser selection session = httpmorph.Session(browser='random') ``` -------------------------------- ### HTTP POST Request Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a POST request to a specified URL. Supports sending data as bytes, string, dictionary, or JSON. ```python response = httpmorph.post(url, data=None, json=None, **kwargs) ``` -------------------------------- ### Set Request Headers Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Add custom headers to your HTTP request. This is useful for setting User-Agent, API keys, or other metadata. ```python headers = {'User-Agent': 'MyApp/1.0'} response = httpmorph.get(url, headers=headers) ``` -------------------------------- ### Process Response Line by Line Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Handle responses, such as large log files, by processing them line by line using `iter_lines`. This is memory-efficient for text-based responses. ```python response = httpmorph.get('https://example.com/large-log.txt', stream=True) for line in response.iter_lines(): process_line(line) ``` -------------------------------- ### Retrieve TLS Debugging Information Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Fetches TLS-related information from a response, such as the TLS version, cipher suite, and JA3 fingerprint. ```python response = httpmorph.get('https://example.com') print('TLS Version:', response.tls_version) print('Cipher Suite:', response.tls_cipher) print('JA3 Fingerprint:', response.ja3_fingerprint) ``` -------------------------------- ### Access HTTP Version Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Determine the HTTP protocol version used for the response (e.g., '1.1' or '2.0'). ```python print(response.http_version) # '1.1' or '2.0' ``` -------------------------------- ### Iterate Over Response Lines Source: https://httpmorph.readthedocs.io/en/latest/api.html Use `iter_lines()` to iterate over the response body line by line. The delimiter can be specified, and lines can be decoded as text by default. ```python response.iter_lines(delimiter=None, decode_unicode=True) ``` -------------------------------- ### Session Class Source: https://httpmorph.readthedocs.io/en/latest/api.html Represents an HTTP session with persistent cookies and headers, defaulting to HTTP/2. ```APIDOC ## Session Class ### Description Manages HTTP sessions with persistent cookies and headers. Defaults to HTTP/2 to mimic Chrome browser behavior. ### Constructor `httpmorph.Session(browser='chrome', os='macos', http2=True)` #### Parameters * `browser` (str) - Browser profile to mimic. Options: `'chrome'`, `'chrome127'`-`'chrome143'`, `'random'`. Default: `'chrome'` (Chrome 143) * `os` (str) - Operating system for User-Agent. Options: `'macos'`, `'windows'`, `'linux'`. Default: `'macos'` * `http2` (bool) - Enable HTTP/2. Default: `True` ### Attributes * `cookies` (dict) - Session cookies. * `cookie_jar` - Cookie jar object with length. * `headers` (dict) - Persistent headers (can be set). ### Methods All standard HTTP methods are available as functions on the session object. #### GET `response = session.get(url, **kwargs)` #### POST `response = session.post(url, data=None, json=None, **kwargs)` ### Context Manager ```python with httpmorph.Session() as session: response = session.get(url) ``` ### Other Methods #### close `session.close()` Explicitly close the session and release resources. ``` -------------------------------- ### Stream Large File Downloads Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Download large files efficiently by streaming the response content in chunks, avoiding loading the entire file into memory. Use `stream=True` and `iter_content`. ```python response = httpmorph.get('https://example.com/large-file.zip', stream=True) with open('large-file.zip', 'wb') as f: for chunk in response.iter_content(chunk_size=8192): f.write(chunk) ``` -------------------------------- ### Iterate Over Response Content in Chunks Source: https://httpmorph.readthedocs.io/en/latest/api.html Use `iter_content()` to iterate over the response body in chunks of a specified size. This is memory-efficient for large responses. You can optionally decode the chunks as text. ```python response.iter_content(chunk_size=1024, decode_unicode=False) ``` -------------------------------- ### Configure HTTP Proxy Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Route requests through an HTTP proxy. Supports basic proxy URLs and dictionary formats. ```python # HTTP proxy response = httpmorph.get( url, proxy='http://proxy.example.com:8080' ) ``` ```python # With authentication response = httpmorph.get( url, proxy='http://proxy.example.com:8080', proxy_auth=('user', 'pass') ) ``` ```python # Proxy dict (requests-compatible) proxies = { 'http': 'http://proxy.example.com:8080', 'https': 'https://proxy.example.com:8080' } response = httpmorph.get(url, proxies=proxies) ``` -------------------------------- ### Response Class Attributes and Methods Source: https://httpmorph.readthedocs.io/en/latest/api.html Provides details on accessing response data, status, and utility methods. ```APIDOC ## Response Class Returned by all request methods. ### Status Attributes * `status_code` (int) - HTTP status code (e.g., 200, 404) * `ok` (bool) - True if status is 200-399 * `reason` (str) - Status reason phrase (e.g., “OK”, “Not Found”) * `url` (str) - Final URL after redirects ### Content Attributes * `body` (bytes) - Raw response body * `content` (bytes) - Alias for body * `text` (str) - Decoded response body (lazy, UTF-8 with fallback) * `headers` (dict) - Response headers ### Methods ``` response.json() ``` Parse response body as JSON. **Returns:** Parsed JSON object (dict/list) **Raises:** `ValueError` if body is not valid JSON ``` response.raise_for_status() ``` Raise `HTTPError` if status is 4xx or 5xx. **Raises:** `HTTPError` ``` response.iter_content(chunk_size=1024, decode_unicode=False) ``` Iterate over response body in chunks. **Parameters:** * `chunk_size` (int) - Size of chunks in bytes * `decode_unicode` (bool) - Decode chunks as text **Yields:** bytes or str ``` response.iter_lines(delimiter=None, decode_unicode=True) ``` Iterate over response body line by line. **Parameters:** * `delimiter` (str) - Line delimiter (default: `\n`) * `decode_unicode` (bool) - Decode lines as text **Yields:** str or bytes ### Timing Attributes All timing values are in microseconds: * `total_time_us` (int) - Total request time * `connect_time_us` (int) - Connection establishment time * `tls_time_us` (int) - TLS handshake time * `first_byte_time_us` (int) - Time to first byte * `elapsed` (timedelta) - Total time as timedelta object ### TLS Attributes For HTTPS requests: * `tls_version` (str) - TLS version used (e.g., “TLSv1.3”) * `tls_cipher` (str) - Cipher suite name * `ja3_fingerprint` (str) - JA3 TLS fingerprint ### HTTP Version * `http_version` (str) - HTTP protocol version: “1.0”, “1.1”, or “2.0” ### Redirect Handling * `history` (list) - List of `Response` objects from redirects * `is_redirect` (bool) - True if status is 3xx ### Streaming * `raw` (BytesIO) - Raw response stream (lazy) ### Error Attributes * `error` (int) - Error code (0 if no error) * `error_message` (str) - Human-readable error message ``` -------------------------------- ### Disable HTTP/2 for a Specific Request Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Explains how to override the default HTTP/2 setting for a single request using the http2=False parameter. This is useful for compatibility with servers that do not support HTTP/2. ```python # Per-request override (disable HTTP/2 for specific request) client = httpmorph.Client() # Defaults to HTTP/2 response = client.get('https://example.com', http2=False) ``` -------------------------------- ### Access Response Headers Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Retrieve all response headers or access a specific header by its key. ```python # Headers print(response.headers) print(response.headers['Content-Type']) ``` -------------------------------- ### Set Request Timeout Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Configure a timeout for the request to prevent indefinite waiting. Specified in seconds. ```python # 5 second timeout response = httpmorph.get(url, timeout=5) ``` -------------------------------- ### HTTP PUT Request Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a PUT request to a specified URL. Supports sending data as bytes, string, or dictionary. ```python response = httpmorph.put(url, data=None, **kwargs) ``` -------------------------------- ### POST Request with JSON Data Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Send a POST request with JSON data and print the JSON response. ```python response = httpmorph.post( 'https://httpbin.org/post', json={'name': 'value'} ) print(response.json()) ``` -------------------------------- ### Disable SSL Verification Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Optionally disable SSL certificate verification. Use with caution as it compromises security. ```python # Disable SSL verification (not recommended) response = httpmorph.get(url, verify=False) ``` -------------------------------- ### Access TLS Information Source: https://httpmorph.readthedocs.io/en/latest/quickstart.html Retrieve details about the TLS/SSL connection, including the protocol version, cipher suite, and JA3 fingerprint. ```python print(response.tls_version) # 'TLSv1.3' print(response.tls_cipher) # Cipher suite name print(response.ja3_fingerprint) # JA3 fingerprint ``` -------------------------------- ### Graceful Error Handling for Network Requests Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Safely fetch data from a URL, returning None and printing an error message if timeouts, HTTP errors, or connection issues occur. Catches specific httpmorph exceptions. ```python def safe_fetch(url): try: response = httpmorph.get(url, timeout=5) response.raise_for_status() return response.json() except httpmorph.Timeout: print(f'Timeout fetching {url}') return None except httpmorph.HTTPError as e: print(f'HTTP {e.response.status_code}: {url}') return None except httpmorph.ConnectionError: print(f'Connection failed: {url}') return None ``` -------------------------------- ### httpmorph.put() Source: https://httpmorph.readthedocs.io/en/latest/api.html Make a PUT request to a specified URL. Supports sending data in the request body. ```APIDOC ## PUT ### Description Make a PUT request to a specified URL. Supports sending data in the request body. ### Method PUT ### Endpoint [URL] ### Parameters #### Path Parameters - **url** (str) - Required - URL to request #### Request Body - **data** (bytes/str/dict) - Optional - Request body #### Query Parameters - **kwargs** - See Request Parameters ### Response #### Success Response - **Response object** - The response from the PUT request. ``` -------------------------------- ### Generate Chrome 143 TLS Fingerprints Source: https://httpmorph.readthedocs.io/en/latest/advanced.html Generates accurate Chrome 143 TLS fingerprints including JA3N, JA4, JA4_R, and Akamai matching. The default session uses the Chrome 143 profile. ```python session = httpmorph.Session(browser='chrome') response = session.get('https://example.com') print('JA3:', response.ja3_fingerprint) print('TLS:', response.tls_version) print('Cipher:', response.tls_cipher) print('HTTP:', response.http_version) ```