### GET Request Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Demonstrates how to perform a simple GET request to fetch a webpage. ```APIDOC ## GET Request ### Description Fetches a webpage using a GET request. The response object contains details about the request and response, including cookies. ### Method GET ### Endpoint `https://httpbin.org/get` ### Parameters N/A ### Request Example ```python r = tls_requests.get('https://httpbin.org/get') print(r) ``` ### Response #### Success Response (200) - **r** (`Response` object) - The response object from the GET request. #### Response Example ``` ``` ``` -------------------------------- ### Authentication Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Demonstrates how to perform Basic Authentication with requests. ```APIDOC ## Authentication ### Description Perform Basic Authentication with requests by providing a username and password tuple to the `auth` parameter. ### Method GET ### Endpoint https://httpbin.org/get ### Parameters #### Query Parameters - **auth** (tuple) - A tuple containing the username and password for Basic Authentication. ### Request Example ```python r = tls_requests.get("https://httpbin.org/get", auth=("admin", "admin")) ``` ``` -------------------------------- ### Redirection Handling Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Explains how to control redirect behavior using the `follow_redirects` parameter and how to inspect redirect history. ```APIDOC ## Redirection Handling ### Description Control redirect behavior using the `follow_redirects` parameter and inspect the redirect history. ### Method GET ### Endpoint https://httpbin.org/absolute-redirect/{n} ### Parameters #### Query Parameters - **follow_redirects** (bool) - If `True` (default), redirects are automatically followed. If `False`, the redirect response is returned. ### Request Example (Disabling Redirects) ```python redirect_url = 'https://httpbin.org/absolute-redirect/3' r = tls_requests.get(redirect_url, follow_redirects=False) print(r) print(r.history) print(r.next) ``` ### Request Example (Following Redirects) ```python redirect_url = 'https://httpbin.org/absolute-redirect/3' r = tls_requests.get(redirect_url, follow_redirects=True) print(r.status_code) print(r.history) ``` ### Response #### Success Response (200) - **history** (list) - A list of `Response` objects for any followed redirects. - **next** (Request) - The next `Request` object in the redirect chain if `follow_redirects` is `False`. ``` -------------------------------- ### Other HTTP Methods Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Illustrates the usage of PUT, DELETE, HEAD, and OPTIONS HTTP methods. ```APIDOC ## Other HTTP Methods ### Description Demonstrates how to perform PUT, DELETE, HEAD, and OPTIONS requests using the `tls_requests` library. ### Method PUT, DELETE, HEAD, OPTIONS ### Endpoint - `https://httpbin.org/put` - `https://httpbin.org/delete` - `https://httpbin.org/get` (for HEAD and OPTIONS examples) ### Parameters #### Request Body (for PUT) - **data** (dict) - A dictionary containing the data to be sent in the PUT request body. ### Request Example ```python # PUT Request r = tls_requests.put('https://httpbin.org/put', data={'key': 'value'}) print(r) # DELETE Request r = tls_requests.delete('https://httpbin.org/delete') print(r) # HEAD Request r = tls_requests.head('https://httpbin.org/get') print(r) # OPTIONS Request r = tls_requests.options('https://httpbin.org/get') print(r) ``` ### Response #### Success Response (200) - **r** (`Response` object) - The response object for each respective request. #### Response Example ``` ``` ``` -------------------------------- ### HTTP/2 Support Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Details on how to enable HTTP/2 support using the `http2` parameter. ```APIDOC ## HTTP/2 Support ### Description Enable or control HTTP/2 support for requests using the `http2` parameter. This allows for potentially faster communication with servers that support HTTP/2. ### Method GET ### Endpoint `https://httpbin.org/get` ### Parameters #### Query Parameters - **http2** (string or boolean) - Controls HTTP/2 behavior: - `auto` or `None`: Automatically switch between HTTP/2 and HTTP/1, preferring HTTP/2. Handles redirects. - `http1` or `False`: Force the request to use HTTP/1.1. - `http2` or `True`: Force the request to use HTTP/2. - **client_identifier** (string) - Optional. The name of the client identifier profile. ### Request Example ```python # Force HTTP/2 r = tls_requests.get('https://httpbin.org/get', http2=True, client_identifier="chrome_120") # Auto mode (default behavior if not specified) r = tls_requests.get('https://httpbin.org/get', http2='auto', client_identifier="chrome_120") # Force HTTP/1.1 r = tls_requests.get('https://httpbin.org/get', http2=False, client_identifier="chrome_120") ``` ### Response #### Success Response (200) - **r** (`Response` object) - The response object from the GET request. #### Response Example ``` ``` ``` -------------------------------- ### POST Request Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Shows how to make a POST request with data payload. ```APIDOC ## POST Request ### Description Makes a POST request to a specified URL with a data payload. ### Method POST ### Endpoint `https://httpbin.org/post` ### Parameters #### Request Body - **data** (dict) - A dictionary containing the data to be sent in the request body. ### Request Example ```python r = tls_requests.post('https://httpbin.org/post', data={'key': 'value'}) ``` ### Response #### Success Response (200) - **r** (`Response` object) - The response object from the POST request. #### Response Example ``` ``` ``` -------------------------------- ### Exceptions Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Explains how to handle exceptions for network errors or invalid responses, specifically `HTTPError`. ```APIDOC ## Exceptions ### Description Handle exceptions for network errors or invalid responses. The `HTTPError` is raised for non-2xx status codes when `raise_for_status()` is called. ### Method GET ### Endpoint https://httpbin.org/status/404 ### Parameters #### Query Parameters - **None** ### Request Example (Handling HTTPError) ```python try: r = tls_requests.get('https://httpbin.org/status/404') r.raise_for_status() except tls_requests.exceptions.HTTPError as e: print(e) ``` ### Response #### Error Response (4xx/5xx) - **HTTPError** - Raised when a response status code is 4xx or 5xx and `raise_for_status()` is called. ``` -------------------------------- ### Make GET Request with tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Perform an HTTP GET request to a specified URL. This example demonstrates fetching data from a web resource. The response object contains status and content. ```python r = tls_requests.get('https://httpbin.org/get') r # Cookies now have proper domain backfilled from request URL ``` -------------------------------- ### Timeouts Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Shows how to set custom timeouts for requests to prevent them from hanging indefinitely. ```APIDOC ## Timeouts ### Description Set custom timeouts for requests to control how long the client will wait for a response. ### Method GET ### Endpoint https://github.com/ ### Parameters #### Query Parameters - **timeout** (int or float) - The number of seconds to wait for the server to send data before giving up. ### Request Example ```python tls_requests.get('https://github.com/', timeout=10) ``` ``` -------------------------------- ### Custom Headers Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Adds custom headers to HTTP requests. ```APIDOC ## Custom Headers ### Description Add custom headers to your HTTP requests, such as `User-Agent`, to modify request behavior or provide specific information to the server. ### Method GET ### Endpoint `https://httpbin.org/headers` ### Parameters #### Request Headers - **headers** (dict) - A dictionary where keys are header names and values are the corresponding header values. ### Request Example ```python url = 'https://httpbin.org/headers' headers = {'user-agent': 'my-app/1.0.0'} r = tls_requests.get(url, headers=headers) print(r.json()) ``` ### Response #### Success Response (200) - **r.json()** (dict) - A dictionary containing the headers sent with the request. #### Response Example ```json { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate, br", "Host": "httpbin.org", "User-Agent": "my-app/1.0.0" } } ``` ``` -------------------------------- ### Set Request Timeouts in Python Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Provides an example of setting a custom timeout for a request using the 'timeout' parameter in tls_requests. This helps prevent requests from hanging indefinitely. ```python tls_requests.get('https://github.com/', timeout=10) ``` -------------------------------- ### Importing tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md This snippet shows how to import the tls_requests library and configure basic logging. ```APIDOC ## Importing tls_requests ### Description Import the `tls_requests` library and set up basic logging. ### Method Import Statement ### Endpoint N/A ### Parameters N/A ### Request Example ```python import tls_requests import logging logging.basicConfig(level=logging.INFO) ``` ### Response N/A ``` -------------------------------- ### Handling Response Content Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Demonstrates how to access and process response content in text, binary, and JSON formats. ```APIDOC ## Handling Response Content ### Description Provides methods to access and process the content of an HTTP response, including decoding text, accessing raw binary data, and parsing JSON. ### Method GET ### Endpoint `https://httpbin.org/get` ### Parameters N/A ### Request Example ```python r = tls_requests.get('https://httpbin.org/get') ``` ### Response #### Success Response (200) - **r.text** (string) - The response body decoded as text. - **r.content** (bytes) - The response body as raw bytes. - **r.json()** (dict) - The response body parsed as a JSON object. - **r.encoding** (string) - The encoding used to decode the response text. #### Response Example ```python # Accessing text content print(r.text) # Output might look like: # { # "args": {}, # "headers": { # "Accept": "*/*", # "Accept-Encoding": "gzip, deflate, br", # "Host": "httpbin.org", # ... # }, # ... # } print(r.encoding) # Output: UTF-8 # Accessing binary content print(r.content) # Output: b'{\n "args": {}, \n "headers": {\n "Accept": "*/*", ...}' # Parsing JSON content print(r.json()) # Output: # { # "args": {}, # "headers": { # "Accept": "*/*", # "Accept-Encoding": "gzip, deflate, br", # "Host": "httpbin.org", # ... # }, # ... # } ``` ``` -------------------------------- ### Control Redirect Handling in Python Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Shows how to manage HTTP redirect behavior using the 'follow_redirects' parameter. Demonstrates disabling redirects and inspecting the redirect history. ```python redirect_url = 'https://httpbin.org/absolute-redirect/3' r = tls_requests.get(redirect_url, follow_redirects=False) print(r) print(r.history) print(r.next) ``` ```python redirect_url = 'https://httpbin.org/absolute-redirect/3' r = tls_requests.get(redirect_url, follow_redirects=True) print(r.status_code) print(r.history) ``` -------------------------------- ### Perform Basic Authentication in Python Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Demonstrates how to perform Basic Authentication for HTTP requests using the 'auth' parameter with a tuple of username and password. ```python r = tls_requests.get("https://httpbin.org/get", auth=("admin", "admin")) ``` -------------------------------- ### Pass URL Parameters with tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Include query parameters in a GET request using the `params` keyword. This allows sending key-value pairs that will be appended to the URL. Supports lists for multiple values. ```python import tls_requests params = {'key1': 'value1', 'key2': 'value2'} r = tls_requests.get('https://httpbin.org/get', params=params) r.url '' r.url.url 'https://httpbin.org/get' r.url.params ``` ```python params = {'key1': 'value1', 'key2': ['value2', 'value3']} r = tls_requests.get('https://httpbin.org/get?order_by=asc', params=params) r.url '' ``` -------------------------------- ### Using Client Identifiers Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Explains how to specify a TLS client profile using the `client_identifier` parameter. ```APIDOC ## Using Client Identifiers ### Description Specify a TLS client profile using the `client_identifier` parameter to mimic specific browser TLS fingerprints. ### Method GET ### Endpoint `https://httpbin.org/get` ### Parameters #### Query Parameters - **client_identifier** (string) - The name of the client identifier profile (e.g., "chrome_120"). ### Request Example ```python r = tls_requests.get('https://httpbin.org/get', client_identifier="chrome_120") ``` ### Response #### Success Response (200) - **r** (`Response` object) - The response object from the GET request. #### Response Example ``` ``` ``` -------------------------------- ### Form-Encoded Data Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Shows how to send form-encoded data in POST requests, including handling multiple values for a single key. ```APIDOC ## Form-Encoded Data ### Description Send form-encoded data in POST requests. The `data` parameter accepts a dictionary, and it can handle cases where a single key maps to multiple values. ### Method POST ### Endpoint `https://httpbin.org/post` ### Parameters #### Request Body - **data** (dict) - A dictionary containing the form data. Keys are form field names, and values are the corresponding data. Values can be strings or lists of strings. ### Request Example ```python # Sending simple form data data = {'key1': 'value1', 'key2': 'value2'} r = tls_requests.post("https://httpbin.org/post", data=data) print(r.text) # Sending form data with multiple values for a key data = {'key1': ['value1', 'value2']} r = tls_requests.post("https://httpbin.org/post", data=data) print(r.text) ``` ### Response #### Success Response (200) - **r.text** (string) - The response body, which will contain details about the received form data. #### Response Example ``` # For the first example: { "args": {}, "data": "key1=value1&key1=value2", "files": {}, "form": {}, ... } # For the second example: { ... "form": { "key1": [ "value1", "value2" ] }, ... } ``` ``` -------------------------------- ### URL Parameters Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md How to pass query parameters in the URL, including lists and merging with existing query strings. ```APIDOC ## URL Parameters ### Description Pass query parameters to a request using the `params` keyword argument. This method supports single values, lists for multiple values, and merging with existing URL query strings. ### Method GET ### Endpoint `https://httpbin.org/get` ### Parameters #### Query Parameters - **params** (dict) - A dictionary where keys are parameter names and values are the parameter values. Values can be strings, lists of strings, or other types that can be stringified. ### Request Example ```python # Simple parameters params = {'key1': 'value1', 'key2': 'value2'} r = tls_requests.get('https://httpbin.org/get', params=params) print(r.url) # Parameters with list values and merging with existing query string params = {'key1': 'value1', 'key2': ['value2', 'value3']} r = tls_requests.get('https://httpbin.org/get?order_by=asc', params=params) print(r.url) ``` ### Response #### Success Response (200) - **r.url** (string) - The constructed URL with query parameters. - **r.url.url** (string) - The base URL without query parameters. - **r.url.params** (URLParams object) - The parsed query parameters. #### Response Example ``` # For the first example: '' 'https://httpbin.org/get' # For the second example: '' ``` ``` -------------------------------- ### Import tls_requests Library Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Import the necessary tls_requests library and configure basic logging. This is the initial step for using any functionality within the library. ```python import tls_requests import logging logging.basicConfig(level=logging.INFO) ``` -------------------------------- ### Multipart File Uploads Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Demonstrates how to upload files using the `files` parameter in POST requests. It also shows how to specify custom filenames and MIME types, and how to include non-file data in multipart form requests. ```APIDOC ## Multipart File Uploads ### Description Upload files using the `files` parameter. You can also add custom filenames or MIME types, and include non-file data fields using the `data` parameter. ### Method POST ### Endpoint https://httpbin.org/get ### Parameters #### Request Body - **files** (dict) - A dictionary where keys are form field names and values are file objects or tuples of (filename, file object, content_type). - **data** (dict) - Optional. A dictionary of non-file data to include in the multipart form. ### Request Example (Basic File Upload) ```python files = {'image': open('static/coingecko.png', 'rb')} r = tls_requests.post("https://httpbin.org/get", files=files) print(r.text) ``` ### Request Example (Custom Filename/MIME Type) ```python files = {'image': ('image.png', open('static/coingecko.png', 'rb'), 'image/*')} r = tls_requests.post("https://httpbin.bin/get", files=files) print(r.text) ``` ### Request Example (With Additional Data) ```python data = {'key1': ['value1', 'value2']} files = {'image': open('static/coingecko.png', 'rb')} r = tls_requests.post("https://httpbin.org/get", data=data, files=files) print(r.text) ``` ### Response Example (Success) ```json { "args": {}, "data": "", "files": { "image": "data:image/png;base64, ..." }, "form": { "key1": [ "value1", "value2" ] }, ... } ``` ``` -------------------------------- ### Sending JSON Data Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Demonstrates how to send complex JSON data structures in the request body using the `json` parameter. ```APIDOC ## Sending JSON Data ### Description Send complex JSON data structures in the request body using the `json` parameter. ### Method POST ### Endpoint https://httpbin.org/post ### Parameters #### Request Body - **json** (dict) - A dictionary representing the JSON payload to send. ### Request Example ```python data = { 'integer': 1, 'boolean': True, 'list': ['1', '2', '3'], 'data': {'key': 'value'} } r = tls_requests.post("https://httpbin.org/post", json=data) print(r.text) ``` ### Response Example (Success) ```json { ... "json": { "boolean": true, "data": { "key": "value" }, "integer": 1, "list": [ "1", "2", "3" ] }, ... } ``` ``` -------------------------------- ### Inspecting Responses Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Covers how to inspect various aspects of an HTTP response, including status codes, headers, and cookies. ```APIDOC ## Inspecting Responses ### Description This section details how to inspect different parts of an HTTP response, such as status codes, headers, and cookies. ### Status Codes Check the HTTP status code of a response using the `status_code` attribute. You can also use `raise_for_status()` to raise an exception for non-2xx responses. #### Request Example (Checking Status Code) ```python r = tls_requests.get('https://httpbin.org/get') print(r.status_code) ``` #### Request Example (Raising for Status) ```python not_found = tls_requests.get('https://httpbin.org/status/404') not_found.raise_for_status() # This will raise an HTTPError ``` ### Headers Access response headers as a case-insensitive dictionary using the `headers` attribute. #### Request Example (Accessing Headers) ```python r = tls_requests.get('https://httpbin.org/get') print(r.headers) print(r.headers['Content-Type']) ``` ### Cookies Access cookies received in the response using the `cookies` attribute. Cookies can also be included in requests. #### Request Example (Accessing Cookies) ```python url = 'https://httpbin.org/cookies/set?foo=bar' r = tls_requests.get(url, follow_redirects=True) print(r.cookies['foo']) ``` ``` -------------------------------- ### Handle Cookies in Python with tls-requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Explains how to access cookies from a response using the 'cookies' attribute and how to include cookies in requests. Supports automatic cookie handling for redirects. ```python url = 'https://httpbin.org/cookies/set?foo=bar' r = tls_requests.get(url, follow_redirects=True) print(r.cookies['foo']) ``` -------------------------------- ### Handle HTTP Errors in Python Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Illustrates how to use a try-except block to catch and handle 'HTTPError' exceptions raised by tls_requests for non-2xx status codes, providing robust error management. ```python try: r = tls_requests.get('https://httpbin.org/status/404') r.raise_for_status() except tls_requests.exceptions.HTTPError as e: print(e) ``` -------------------------------- ### Upload Files with Multipart Encoding in Python Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Demonstrates how to upload files using the 'files' parameter in tls_requests. Supports basic file uploads, custom filenames and MIME types, and inclusion of non-file data fields. ```python files = {'image': open('static/coingecko.png', 'rb')} r = tls_requests.post("https://httpbin.org/get", files=files) print(r.text) ``` ```python files = {'image': ('image.png', open('static/coingecko.png', 'rb'), 'image/*')} r = tls_requests.post("https://httpbin.org/get", files=files) print(r.text) ``` ```python data = {'key1': ['value1', 'value2']} files = {'image': open('static/coingecko.png', 'rb')} r = tls_requests.post("https://httpbin.org/get", data=data, files=files) print(r.text) ``` -------------------------------- ### Check HTTP Status Codes in Python Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Illustrates how to access the HTTP status code of a response using the 'status_code' attribute and how to raise exceptions for non-2xx responses with 'raise_for_status()'. ```python r = tls_requests.get('https://httpbin.org/get') print(r.status_code) ``` ```python not_found = tls_requests.get('https://httpbin.org/status/404') print(not_found.status_code) not_found.raise_for_status() ``` ```python r = tls_requests.get('https://httpbin.org/get') raw = r.raise_for_status().text print(raw) ``` -------------------------------- ### Access Response Headers in Python Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Demonstrates how to access response headers as a case-insensitive dictionary using the 'headers' attribute. This allows easy retrieval of information like 'Content-Type'. ```python r = tls_requests.get('https://httpbin.org/get') print(r.headers) ``` ```python print(r.headers['Content-Type']) ``` -------------------------------- ### Enable HTTP/2 Support in tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Enable HTTP/2 protocol for requests by setting the `http2` parameter to `True`. This can improve performance for certain connections. The `client_identifier` is also specified. ```python r = tls_requests.get('https://httpbin.org/get', http2=True, client_identifier="chrome_120") ``` -------------------------------- ### Handle Binary Content from Response with tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Retrieve the raw binary content of an HTTP response. This is useful for non-textual data like images or files. ```python r.content b'{\n "args": {}, \n "headers": {\n "Accept": "*/*", ...' ``` -------------------------------- ### Make Other HTTP Requests (PUT, DELETE, HEAD, OPTIONS) with tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Demonstrates making various other HTTP requests including PUT, DELETE, HEAD, and OPTIONS using the tls_requests library. These methods are used for different server interactions. ```python r = tls_requests.put('https://httpbin.org/put', data={'key': 'value'}) r r = tls_requests.delete('https://httpbin.org/delete') r r = tls_requests.head('https://httpbin.org/get') r r = tls_requests.options('https://httpbin.org/get') r ``` -------------------------------- ### Make POST Request with tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Execute an HTTP POST request with data payload. This is used for sending information to a server, such as form submissions. The data is provided as a dictionary. ```python r = tls_requests.post('https://httpbin.org/post', data={'key': 'value'}) ``` -------------------------------- ### Bypass Cloudflare Bot Fight Mode Example Source: https://github.com/thewebscraping/tls-requests/blob/main/README.md A simple example demonstrating how to make a GET request to a website like coingecko.com using tls_requests. This illustrates the library's capability to bypass anti-bot measures. ```python import tls_requests r = tls_requests.get('https://www.coingecko.com/') r ``` -------------------------------- ### Setting Custom TLS Binary Path (Bash) Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/tls/install.md This example shows how to configure the tls-requests library to use a custom binary by setting the `TLS_LIBRARY_PATH` environment variable. This overrides the default binary discovery mechanism. ```bash export TLS_LIBRARY_PATH=/path/to/your/custom/library.so ``` -------------------------------- ### Full Anti-Detection Setup with TLS Requests Client Source: https://context7.com/thewebscraping/tls-requests/llms.txt This example illustrates a comprehensive anti-detection setup using the `tls_requests.Client`. It configures header rotation, TLS identifier rotation, follows redirects, and enables HTTP/2. The client is then used to make multiple requests to a protected site, with a delay between them to simulate human behavior. ```python import tls_requests import time with tls_requests.Client( headers=tls_requests.HeaderRotator(), client_identifier=tls_requests.TLSIdentifierRotator(), follow_redirects=True, http2=True ) as client: # Rotate fingerprints on each request response = client.get("https://protected-site.com") # Add delays between requests to appear more human time.sleep(2) response = client.get("https://protected-site.com/api/data") ``` -------------------------------- ### Add Custom Headers with tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Attach custom HTTP headers to a request using the `headers` parameter. This is useful for setting user agents, authentication tokens, or other required headers. ```python url = 'https://httpbin.org/headers' headers = {'user-agent': 'my-app/1.0.0'} r = tls_requests.get(url, headers=headers) r.json() { "headers": { ... "Host": "httpbin.org", "User-Agent": "my-app/1.0.0", ... } } ``` -------------------------------- ### Handle Text Content from Response with tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Access the decoded text content of an HTTP response. The library automatically handles encoding, providing the response body as a string. ```python r = tls_requests.get('https://httpbin.org/get') print(r.text) { "args": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate, br", "Host": "httpbin.org", ... }, ... } r.encoding 'UTF-8' ``` -------------------------------- ### Specify Client Identifier in tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Customize the TLS client profile for requests using the `client_identifier` parameter. This allows emulating specific browser clients for more realistic interactions. ```python r = tls_requests.get('https://httpbin.org/get', client_identifier="chrome_120") ``` -------------------------------- ### Send JSON Data with tls-requests in Python Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Shows how to send complex JSON data structures in the request body using the 'json' parameter with tls_requests. The library automatically handles serialization and sets the appropriate Content-Type header. ```python data = { 'integer': 1, 'boolean': True, 'list': ['1', '2', '3'], 'data': {'key': 'value'} } r = tls_requests.post("https://httpbin.org/post", json=data) print(r.text) ``` -------------------------------- ### Install TLS Requests Library Source: https://context7.com/thewebscraping/tls-requests/llms.txt Instructions for installing the TLS Requests library using pip, including options for PyPI, uv, and direct GitHub installation. It also shows how to manually update the underlying TLS library. ```bash # Install via PyPI pip install wrapper-tls-requests # Or using uv uv add wrapper-tls-requests # Install from GitHub pip install git+https://github.com/thewebscraping/tls-requests.git # Update TLS library manually (optional) python -m tls_requests.models.libraries ``` -------------------------------- ### Install TLS Requests from GitHub Source: https://github.com/thewebscraping/tls-requests/blob/main/README.md Installs the tls-requests library directly from its GitHub repository. This is useful for installing the latest development version or a specific commit. ```shell pip install git+https://github.com/thewebscraping/tls-requests.git ``` -------------------------------- ### Enable Proxy Rotation with a List of Proxies Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/advanced/rotators.md Shows how to enable proxy rotation by providing a list of proxy strings to the 'proxy' parameter. The library uses a weighted strategy by default, prioritizing well-performing proxies. This example includes various proxy formats. ```python import tls_requests proxy_list = [ "http://user1:pass1@proxy.example.com:8080", "http://user2:pass2@proxy.example.com:8081", "socks5://proxy.example.com:8082", "proxy.example.com:8083", # (defaults to http) "http://user:pass@proxy.example.com:8084|1.0|US", # http://user:pass@host:port|weight|region ] # Provide a list to enable proxy rotation. with tls_requests.Client(proxy=proxy_list) as client: response = client.get("https://httpbin.org/get") ``` -------------------------------- ### Send Form-Encoded Data with tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Submit form-encoded data in POST requests using the `data` parameter. This method is commonly used for submitting HTML form values. It supports sending multiple values for the same key. ```python data = {'key1': 'value1', 'key2': 'value2'} r = tls_requests.post("https://httpbin.org/post", data=data) print(r.text) { "args": {}, "data": "key1=value1&key1=value2", "files": {}, "form": {}, ... } ``` ```python data = {'key1': ['value1', 'value2']} r = tls_requests.post("https://httpbin.org/post", data=data) print(r.text) { ... "form": { "key1": [ "value1", "value2" ] }, ... } ``` -------------------------------- ### Install TLS Requests via PyPI Source: https://github.com/thewebscraping/tls-requests/blob/main/README.md Installs the tls-requests library from the Python Package Index (PyPI). This is the standard method for adding the library to your Python environment. ```shell pip install wrapper-tls-requests ``` -------------------------------- ### Manual TLS Binary Download (Python) Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/tls/install.md This code allows for manual downloading of a specific version of the TLS binary. This is useful in environments with restricted internet access or when a particular version is required. ```python from tls_requests import TLSLibrary # Download a specific version TLSLibrary.download(version='1.13.1') ``` -------------------------------- ### Automatic TLS Binary Download and Usage (Python) Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/tls/install.md This snippet demonstrates the automatic management of TLS binaries. The first time `tls_requests.get` is called, it checks for and downloads the appropriate binary if it's missing. Subsequent calls reuse the cached binary. ```python import tls_requests # The first call will trigger the binary download if it doesn't exist response = tls_requests.get('https://httpbin.org/get') print(response.status_code) ``` -------------------------------- ### Basic Async Request with tls-requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/advanced/async_client.md Demonstrates how to make a single asynchronous GET request using tls_requests.AsyncClient. It requires the asyncio library and the tls_requests library. The function fetches data from a URL and prints the status code and JSON response. ```python import asyncio import tls_requests async def main(): async with tls_requests.AsyncClient() as client: response = await client.get("https://httpbin.org/get") print(f"Status: {response.status_code}") print(f"Data: {response.json()}") if __name__ == "__main__": asyncio.run(main()) ``` -------------------------------- ### Python: Logging Requests and Responses with Hooks Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/advanced/hooks.md This example demonstrates how to define and use request and response hooks in tls-requests to log HTTP request details and response status codes. It requires the tls-requests library. ```python import tls_requests def log_request(request): print(f"Request event: {request.method} {request.url}") def log_response(response): print(f"Response event: {response.status_code} for {response.url}") # Create a client with hooks client = tls_requests.Client(hooks={ 'request': [log_request], 'response': [log_response] }) ``` -------------------------------- ### Executing All HTTP Methods with TLS Requests Source: https://context7.com/thewebscraping/tls-requests/llms.txt Shows how to use the tls_requests library to perform all standard HTTP methods (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS). It also demonstrates the generic `request` method for flexibility. ```python import tls_requests # GET request response = tls_requests.get("https://httpbin.org/get") # POST request response = tls_requests.post("https://httpbin.org/post", json={"key": "value"}) # PUT request response = tls_requests.put("https://httpbin.org/put", data={"update": "data"}) # PATCH request response = tls_requests.patch("https://httpbin.org/patch", json={"partial": "update"}) # DELETE request response = tls_requests.delete("https://httpbin.org/delete") # HEAD request (no body returned) response = tls_requests.head("https://httpbin.org/get") print(response.headers) # OPTIONS request response = tls_requests.options("https://httpbin.org/get") # Generic request method response = tls_requests.request("GET", "https://httpbin.org/get") ``` -------------------------------- ### Initialize and Use Client with Context Manager (Python) Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/advanced/client.md Demonstrates the recommended way to use the tls_requests.Client by leveraging a context manager (`with` statement). This ensures the client's resources are automatically managed and closed upon exiting the block, preventing resource leaks. It makes a GET request and prints the status code. ```python import tls_requests with tls_requests.Client() as client: response = client.get("https://httpbin.org/get") print(response.status_code) ``` -------------------------------- ### Basic Authentication with Tuple Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/advanced/authentication.md Demonstrates how to use basic HTTP authentication by providing a username and password tuple when initializing the Client. ```APIDOC ## Basic Authentication with Tuple ### Description Use basic HTTP authentication by passing a tuple `(username, password)` to the `tls_requests.Client` constructor. This automatically adds the `Authorization` header to all requests made by this client. ### Method `Client(auth=(username, password))` ### Endpoint N/A (Client initialization) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python import tls_requests client = tls_requests.Client(auth=("username", "secret")) response = client.get("https://httpbin.org/basic-auth/username/secret") ``` ### Response #### Success Response (200) Returns the response from the server if authentication is successful. #### Response Example ```json { "authenticated": true, "user": "username" } ``` ``` -------------------------------- ### Parse JSON Content from Response with tls_requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/quickstart.md Easily parse JSON responses directly into Python dictionaries or lists using the `.json()` method. This simplifies working with APIs that return JSON. ```python r.json() { "args": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate, br", "Host": "httpbin.org", ... }, ... } ``` -------------------------------- ### Configure Client with Global Headers and Proxy (Python) Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/advanced/client.md Illustrates how to initialize a tls_requests.Client with persistent configurations, such as custom headers and a proxy server. These settings will be applied to all subsequent requests made using this client instance, simplifying the configuration for multiple requests. ```python import tls_requests # Set global headers and a proxy client = tls_requests.Client( headers={"User-Agent": "MyCustomBrowser/1.0"}, proxy="http://127.0.0.1:8080" ) # This request will use the custom User-Agent and Proxy response = client.get("https://httpbin.org/headers") ``` -------------------------------- ### TLSClient Initialization Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/tls/index.md Demonstrates how to manually initialize the TLSClient, although it usually initializes automatically. ```APIDOC ## TLSClient Initialization ### Description Manual initialization of the TLSClient. This is typically handled automatically upon first use. ### Method `initialize()` ### Endpoint N/A (Class method) ### Parameters None ### Request Example ```python from tls_requests import TLSClient TLSClient.initialize() ``` ### Response None ``` -------------------------------- ### Python: Managing Hooks During Client Initialization Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/advanced/hooks.md Demonstrates how to configure request and response hooks for a tls-requests client directly during its initialization. This allows for setting up multiple hooks for different events. Requires the tls-requests library. ```python import tls_requests def log_request(request): print(f"Request event: {request.method} {request.url}") def log_response(response): print(f"Response event: {response.status_code} for {response.url}") def raise_on_4xx_5xx(response): response.raise_for_status() client = tls_requests.Client(hooks={ 'request': [log_request], 'response': [log_response, raise_on_4xx_5xx], }) ``` -------------------------------- ### Utilize Client for Reusable TLS Configurations Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/tls/configuration.md Demonstrates using the `Client` class for creating reusable and shared TLS configurations. This approach is efficient for making multiple requests with consistent settings, including proxy, HTTP/2, timeouts, and custom TLS client configurations. ```python import tls_requests client = tls_requests.Client( proxy = "http://127.0.0.1:8080", http2 = True, timeout = 10.0, follow_redirects = True, verify = True, client_identifier = "chrome_120", **config_obj.to_dict(), ) r = client.get(url = "https://httpbin.org/get",) r ``` -------------------------------- ### Initialize and Use Client Manually (Python) Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/advanced/client.md Shows how to manually manage the lifecycle of a tls_requests.Client instance. This involves explicitly creating the client, making requests, and then calling the `.close()` method to release resources. This approach is useful when a context manager cannot be used. ```python import tls_requests client = tls_requests.Client() response = client.get("https://httpbin.org/get") # ... do more work ... client.close() ``` -------------------------------- ### Basic GET Request with TLS Requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/index.md Demonstrates a simple GET request using the tls_requests library to fetch data from a specified URL. It shows how to initiate a request and access the response object. ```python import tls_requests r = tls_requests.get("https://httpbin.org/get") r r.status_code 200 ``` -------------------------------- ### Basic GET Request with Client Identifier Source: https://github.com/thewebscraping/tls-requests/blob/main/README.md Demonstrates a basic GET request using tls_requests. It automatically synchronizes headers like User-Agent and Sec-CH-UA based on the provided client_identifier, mimicking a specific browser. ```python import tls_requests # The library automatically injects matching User-Agent and Sec-CH-UA headers r = tls_requests.get("https://httpbin.org/headers", client_identifier="chrome_133") r.json()["headers"]["User-Agent"] ``` -------------------------------- ### GET Request with Proxy Rotator and TLS Identifier Rotator Source: https://github.com/thewebscraping/tls-requests/blob/main/README.md Shows how to perform a GET request using a ProxyRotator for rotating proxies and a TLSIdentifierRotator for rotating TLS fingerprints. This enhances anonymity and bypasses bot detection. ```python import tls_requests proxy_rotator = tls_requests.ProxyRotator([ "http://user1:pass1@proxy.example.com:8080", "http://user2:pass2@proxy.example.com:8081", "socks5://proxy.example.com:8082", "proxy.example.com:8083", # defaults to http "http://user:pass@proxy.example.com:8084|1.0|US", # weight and region support ]) r = tls_requests.get( "https://httpbin.org/get", proxy=proxy_rotator, client_identifier=tls_requests.TLSIdentifierRotator() ) r r.status_code ``` -------------------------------- ### Make HTTP GET Request with TLS Fingerprint Spoofing (Python) Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/tls/profiles.md Demonstrates how to make an HTTP GET request using the tls-requests library, either with the default client configuration or a specific client identifier to spoof a TLS fingerprint. This is useful for emulating browser behavior. ```python import tls_requests # Using the default profile response = tls_requests.get("https://httpbin.org/get") # Using a specific profile response = tls_requests.get("https://httpbin.org/get", client_identifier="firefox_132") ``` -------------------------------- ### Basic GET Request with TLS Fingerprinting Source: https://context7.com/thewebscraping/tls-requests/llms.txt Demonstrates how to perform a basic GET request using the tls_requests library. It shows how to access the response status code, JSON content, raw text, binary content, headers, cookies, and encoding. ```python import tls_requests # Simple GET request with default Chrome TLS profile response = tls_requests.get("https://httpbin.org/get") print(response.status_code) # 200 print(response.json()) # {'args': {}, 'headers': {...}, ...} # Access response properties print(response.text) # Raw text content print(response.content) # Binary content print(response.headers) # Response headers print(response.cookies) # Response cookies print(response.encoding) # 'UTF-8' ``` -------------------------------- ### Configure HTTP Proxy for tls-requests Client Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/advanced/proxies.md Demonstrates how to set an HTTP proxy when initializing a tls_requests.Client. This routes all subsequent requests made by this client through the specified proxy server. The response from the target URL is then printed. ```python import tls_requests with tls_requests.Client(proxy="http://127.0.0.1:8080") as client: response = client.get("https://httpbin.org/ip") print(response.json()) ``` -------------------------------- ### Get Cookies Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/tls/index.md Retrieves cookies associated with a specific TLS session and URL. ```APIDOC ## Get Cookies ### Description Retrieves cookies associated with a specific TLS session and URL. ### Method `get_cookies(session_id: str, url: str) -> dict` ### Endpoint N/A (Class method) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters * `session_id` (str): The unique identifier for the TLS session. * `url` (str): The URL for which cookies are requested. ### Request Example ```python from tls_requests import TLSClient cookies = TLSClient.get_cookies(session_id="my-session-123", url="https://httpbin.org") ``` ### Response #### Success Response (200) * `dict`: A dictionary containing the cookies. ``` -------------------------------- ### Initialize TLSConfig for HTTP Requests Source: https://github.com/thewebscraping/tls-requests/blob/main/docs/tls/configuration.md Demonstrates initializing a TLSConfig object with various settings for HTTP requests. This configuration can be used to fine-tune aspects like headers, redirects, and timeouts. It's a foundational step for customizing TLS behavior. ```python import tls_requests config_data = { "catchPanics": False, "followRedirects": False, "forceHttp1": False, "headers": { "accept": "text/html,application/xhtml+xml", "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/105.0.0.0 Safari/537.36", }, "insecureSkipVerify": False, "proxyUrl": "", "requestMethod": "GET", "requestUrl": "https://httpbin.org/get", "sessionId": "my-custom-session", "timeoutSeconds": 30, "tlsClientIdentifier": "chrome_120", } config = tls_requests.TLSConfig.from_kwargs(**config_data) # Use the config in a request response = tls_requests.get("https://httpbin.org/get", **config.to_dict()) ```