### UserAgentBuilder Example - Python Source: https://github.com/requests/toolbelt/blob/master/docs/user-agent.md Illustrates the usage of the `UserAgentBuilder` class for constructing a User-Agent string with explicit control over included components. This example shows how to include implementation details, system information, and extra package versions. ```python from requests_toolbelt.utils.user_agent import UserAgentBuilder user_agent_str = UserAgentBuilder( name='requests-toolbelt', version='17.4.0', ).include_implementation( ).include_system( ).include_extras([ ('requests', '2.14.2'), ('urllib3', '1.21.2'), ]).build() ``` -------------------------------- ### Build and Serve Documentation Locally Source: https://github.com/requests/toolbelt/blob/master/docs/contributing.md Build the project's documentation using tox and serve it locally using Python's built-in HTTP server for preview. Ensure tox is installed before running. ```shell $ tox -e docs $ python2.7 -m SimpleHTTPServer # or $ python3.4 -m http.server ``` -------------------------------- ### Create Request Pool from URLs Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md This example shows a shortcut to create a `Pool` directly from a list of URLs. After joining all threads, it iterates through the responses and prints the URL and status code for each. This is a convenient way to handle a predefined set of URLs. ```python from requests_toolbelt.threaded import pool urls = [ # My list of URLs to get ] p = pool.Pool.from_urls(urls) p.join_all() for response in p.responses(): print('GET {}. Returned {}.'.format(response.request_kwargs['url'], response.status_code)) ``` -------------------------------- ### Stream Data from Standard Input with Requests Source: https://github.com/requests/toolbelt/blob/master/docs/uploading-data.md Demonstrates how to upload data from standard input using requests. The first example shows the default behavior with chunked transfer encoding. The second example utilizes StreamingIterator to stream the data without chunking, provided the data size is known. ```python import requests import sys r = requests.post(url, data=sys.stdin) ``` ```python import requests from requests_toolbelt.streaming_iterator import StreamingIterator import sys stream = StreamingIterator(size, sys.stdin) r = requests.post(url, data=stream, headers={'Content-Type': content_type}) ``` -------------------------------- ### Build User-Agent with Specific Components - Python Source: https://github.com/requests/toolbelt/blob/master/docs/user-agent.md Utilizes the `UserAgentBuilder` to gain fine-grained control over the User-Agent string's content. This example demonstrates including only the package name, version, and specific extra information, excluding default system and implementation details. ```python import requests from requests_toolbelt.utils import user_agent as ua s = requests.Session() s.headers['User-Agent'] = ua.UserAgentBuilder( 'mypackage', '0.0.1', ).include_extras([ ('requests', requests.__version__), ]).build() ``` -------------------------------- ### AuthHandler Example: GitHub and GitLab Authentication Source: https://github.com/requests/toolbelt/blob/master/docs/authentication.md Demonstrates how to use AuthHandler to manage different authentication methods for multiple domains (GitHub and GitLab) within a single requests session. It shows setting up basic authentication for GitHub and a custom token-based authentication for GitLab. ```python import requests from requests_toolbelt.auth.handler import AuthHandler def gitlab_auth(request): request.headers['PRIVATE-TOKEN'] = 'asecrettoken' return request handler = AuthHandler({ 'https://api.github.com': ('sigmavirus24', 'apassword'), 'https://gitlab.com': gitlab_auth, }) session = requests.Session() session.auth = handler r = session.get('https://api.github.com/user') # assert r.ok r2 = session.get('https://gitlab.com/api/v3/projects') # assert r2.ok ``` -------------------------------- ### BaseUrlSession: GET request with partial path (Python) Source: https://github.com/requests/toolbelt/blob/master/docs/sessions.md Demonstrates making a GET request using BaseUrlSession with a base URL and a partial resource path. The resulting request URL is constructed by joining the base URL and the partial path. This functionality is useful for setting a common domain or API endpoint. ```python >>> from requests_toolbelt import sessions >>> s = sessions.BaseUrlSession( ... base_url='https://example.com/resource/') >>> r = s.get('sub-resource/', params={'foo': 'bar'}) >>> print(r.request.url) https://example.com/resource/sub-resource/?foo=bar ``` -------------------------------- ### Mount SocketOptionsAdapter to re-enable Nagle's Algorithm Source: https://github.com/requests/toolbelt/blob/master/docs/adapters.md Mounts the SocketOptionsAdapter to a Requests Session to set custom socket options. This example demonstrates re-enabling Nagle's Algorithm for all HTTP and HTTPS connections made with the session. ```python import socket import requests from requests_toolbelt.adapters.socket_options import SocketOptionsAdapter nagles = [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 0)] session = requests.Session() for scheme in session.adapters.keys(): session.mount(scheme, SocketOptionsAdapter(socket_options=nagles)) ``` -------------------------------- ### Retry Failed Requests Using Pool Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md This example demonstrates how to retry requests that previously raised exceptions. It first creates a pool, joins all threads, and then creates a new pool using `Pool.from_exceptions()` with the exceptions from the first pool. The new pool is then joined to re-attempt the failed requests. ```python from requests_toolbelt.threaded import pool urls = [ # My list of URLs to get ] p = pool.Pool.from_urls(urls) p.join_all() new_pool = pool.Pool.from_exceptions(p.exceptions()) new_pool.join_all() ``` -------------------------------- ### BaseUrlSession: GET request with leading slash (Python) Source: https://github.com/requests/toolbelt/blob/master/docs/sessions.md Illustrates the behavior of BaseUrlSession when the partial resource path starts with a '/'. This causes `urllib.parse.urljoin` to treat the partial path as absolute relative to the base URL's scheme and netloc, potentially overriding parts of the base URL. This is a key difference in URL construction compared to paths without a leading slash. ```python >>> from requests_toolbelt import sessions >>> s = sessions.BaseUrlSession( ... base_url='https://example.com/resource/') >>> r = s.get('/sub-resource/', params={'foo': 'bar'}) >>> print(r.request.url) https://example.com/sub-resource/?foo=bar ``` -------------------------------- ### Instantiate and Mount X509Adapter in Requests Session (Python) Source: https://github.com/requests/toolbelt/blob/master/docs/adapters.md Demonstrates how to instantiate the X509Adapter with specified parameters and then mount it to a Requests session for HTTPS connections. This adapter is used for authenticating with X.509 certificates. ```python import requests from requests_toolbelt.adapters.x509 import X509Adapter s = requests.Session() a = X509Adapter(max_retries=3, cert_bytes=b'...', pk_bytes=b'...', encoding='...') ``` ```python s.mount('https://', a) ``` -------------------------------- ### Get Exception from Pool (Python) Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md Retrieves an exception that occurred during a request within the pool. This method is essential for error handling in asynchronous operations. It returns a ThreadException object. ```python from requests_toolbelt.threaded import pool # Assuming 'my_pool' is an initialized Pool object try: thread_exc = my_pool.get_exception() print(f"Exception message: {thread_exc.message}") except StopIteration: print("No exceptions found.") ``` -------------------------------- ### X509Adapter Initialization Source: https://github.com/requests/toolbelt/blob/master/docs/adapters.md This section details the parameters required to instantiate the X509Adapter for use with Requests sessions. ```APIDOC ## X509Adapter ### Description Adapter for use with X.509 certificates. Provides an interface for Requests sessions to contact HTTPS urls and authenticate with an X.509 cert by implementing the Transport Adapter interface. This class will need to be manually instantiated and mounted to the session. ### Method `__init__` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters: - **pool_connections** (int) - Optional - The number of urllib3 connection pools to cache. - **pool_maxsize** (int) - Optional - The maximum number of connections to save in the pool. - **max_retries** (int or urllib3.Retry) - Optional - The maximum number of retries each connection should attempt. Note, this applies only to failed DNS lookups, socket connections and connection timeouts, never to requests where data has made it to the server. By default, Requests does not retry failed connections. If you need granular control over the conditions under which we retry a request, import urllib3’s `Retry` class and pass that instead. - **pool_block** (bool) - Optional - Whether the connection pool should block for connections. - **cert_bytes** (bytes) - Required - bytes object containing contents of a cryptography.x509.Certificate object using the encoding specified by the `encoding` parameter. - **pk_bytes** (bytes) - Required - bytes object containing contents of a object that implements `cryptography.hazmat.primitives.serialization.PrivateFormat` using the encoding specified by the `encoding` parameter. - **password** (str or bytes) - Optional - string or utf8 encoded bytes containing the passphrase used for the private key. None if unencrypted. Defaults to None. - **encoding** (str) - Optional - Enumeration detailing the encoding method used on the `cert_bytes` parameter. Can be either PEM or DER. Defaults to PEM. ### Request Example ```python import requests from requests_toolbelt.adapters.x509 import X509Adapter s = requests.Session() a = X509Adapter(max_retries=3, cert_bytes=b'...', pk_bytes=b'...', encoding='PEM') s.mount('https://', a) ``` ### Response N/A (This is a constructor) ### Response Example N/A ``` -------------------------------- ### Get Response from Pool (Python) Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md Fetches a response object from the pool. This method allows you to access the results of completed requests. It returns a ThreadResponse object, which wraps the original requests.Response. ```python from requests_toolbelt.threaded import pool # Assuming 'my_pool' is an initialized Pool object try: thread_response = my_pool.get_response() print(f"Response status code: {thread_response.status_code}") print(f"Response JSON: {thread_response.json()}") except StopIteration: print("No responses available yet.") ``` -------------------------------- ### BaseUrlSession: Prepare request with partial path (Python) Source: https://github.com/requests/toolbelt/blob/master/docs/sessions.md Shows how to prepare a request using BaseUrlSession before sending it. The `prepare_request` method constructs the full URL from the base URL and the provided request object's URL. This is useful for inspecting or modifying the request before it's sent. ```python >>> from requests import Request >>> from requests_toolbelt import sessions >>> s = sessions.BaseUrlSession( ... base_url='https://example.com/resource/') >>> request = Request(method='GET', url='sub-resource/') >>> prepared_request = s.prepare_request(request) >>> r = s.send(prepared_request) >>> print(r.request.url) https://example.com/resource/sub-resource ``` -------------------------------- ### Tagging and Releasing a New Version Source: https://github.com/requests/toolbelt/blob/master/docs/contributing.md Steps to tag a release commit and upload a new version to PyPI using git and tox. This involves bumping the version, creating a signed tag, and pushing it to the repository. ```shell git tag -s -a $VERSION -m "Release v$VERSION" git push --tags tox -e release ``` -------------------------------- ### Build User-Agent String - Python Source: https://context7.com/requests/toolbelt/llms.txt The `user_agent` function constructs standard User-Agent strings. It includes the package name, version, Python implementation, and system information. It allows for the inclusion of extra dependencies to provide more detailed information. ```python import requests from requests_toolbelt import user_agent # Simple user agent ua = user_agent('my-application', '1.2.3') print(f"User-Agent: {ua}") # Output: my-application/1.2.3 CPython/3.9.0 Linux/5.10.0 # User agent with extra dependencies ua_detailed = user_agent('my-application', '1.2.3', extras=[ ('requests', '2.28.0'), ('urllib3', '1.26.9') ]) print(f"Detailed User-Agent: {ua_detailed}") # Use in requests headers = {'User-Agent': user_agent('api-client', '2.0.0')} response = requests.get('https://api.github.com/users/octocat', headers=headers) print(f"Response: {response.status_code}") ``` -------------------------------- ### Construct Custom User-Agent - Python Source: https://context7.com/requests/toolbelt/llms.txt The `UserAgentBuilder` class provides fine-grained control over User-Agent string construction. It offers a fluent interface for including implementation details, system information, and extra dependencies. This is useful for creating highly specific User-Agent strings for different use cases. ```python from requests_toolbelt.utils.user_agent import UserAgentBuilder import requests # Build custom user agent with full control user_agent_str = UserAgentBuilder( name='custom-client', version='3.1.4' ).include_implementation( ).include_system( ).include_extras([ ('requests', '2.28.0'), ('requests-toolbelt', '0.10.1') ]).build() print(f"Custom UA: {user_agent_str}") # Use in session session = requests.Session() session.headers.update({'User-Agent': user_agent_str}) response = session.get('https://httpbin.org/user-agent') print(f"Server received: {response.json()['user-agent']}") ``` -------------------------------- ### AuthHandler Initialization with Different Auth Types Source: https://github.com/requests/toolbelt/blob/master/docs/authentication.md Illustrates initializing AuthHandler with various authentication strategies, including basic HTTP authentication for one domain and HTTP Digest Authentication for another. It shows how to apply these strategies to requests made directly or through a session. ```python from requests import HTTPDigestAuth from requests_toolbelt.auth.handler import AuthHandler import requests auth = AuthHandler({ 'https://api.github.com': ('sigmavirus24', 'fakepassword'), 'https://example.com': HTTPDigestAuth('username', 'password') }) r = requests.get('https://api.github.com/user', auth=auth) # => r = requests.get('https://example.com/some/path', auth=auth) # => s = requests.Session() s.auth = auth r = s.get('https://api.github.com/user') # => ``` -------------------------------- ### Get Unicode from HTTP Response Source: https://github.com/requests/toolbelt/blob/master/docs/deprecated.md Decodes the content of an HTTP response into a unicode string. It prioritizes encoding information from response headers and falls back to analyzing the content if headers are insufficient. This function helps ensure proper text rendering across different character sets. ```python import requests from requests_toolbelt.utils import deprecated r = requests.get(url) text = deprecated.get_unicode_from_response(r) ``` -------------------------------- ### Constructing a User-Agent String using Python Source: https://github.com/requests/toolbelt/blob/master/README.rst Shows how to create a custom User-Agent string for requests using the `user_agent` function from 'requests_toolbelt'. This is useful for identifying your application to servers. It takes the package name and version as input. ```python from requests_toolbelt import user_agent headers = { 'User-Agent': user_agent('my_package', '0.0.1') } r = requests.get('https://api.github.com/users', headers=headers) ``` -------------------------------- ### Generate Basic User-Agent String - Python Source: https://github.com/requests/toolbelt/blob/master/docs/user-agent.md Generates a basic User-Agent string using the package name and version. This string typically includes the package information, Python implementation and version, and system information. It is useful for identifying your application to servers. ```python >>> import requests_toolbelt >>> requests_toolbelt.user_agent('mypackage', '0.0.1') 'mypackage/0.0.1 CPython/2.7.5 Darwin/13.0.0' ``` -------------------------------- ### Add Extra Information to User-Agent - Python Source: https://github.com/requests/toolbelt/blob/master/docs/user-agent.md Shows how to include additional version information for other libraries or packages used by your application within the User-Agent string. This is achieved by passing a list of tuples containing the name and version of the extra components. ```python import requests import requests_toolbelt from requests_toolbelt.utils import user_agent as ua user_agent = ua.user_agent('mypackage', '0.0.1', extras=[('requests', requests.__version__), ('requests-toolbelt', requests_toolbelt.__version__)]) s = requests.Session() s.headers['User-Agent'] = user_agent ``` -------------------------------- ### MultipartEncoder for File Uploads using Python Source: https://github.com/requests/toolbelt/blob/master/README.rst Demonstrates how to use MultipartEncoder to stream multipart/form-data with file uploads. It requires the 'requests_toolbelt' and 'requests' libraries. The input is a dictionary of fields, where one field can be a tuple representing a file. ```python from requests_toolbelt import MultipartEncoder import requests m = MultipartEncoder( fields={'field0': 'value', 'field1': 'value', 'field2': ('filename', open('file.py', 'rb'), 'text/plain')} ) r = requests.post('http://httpbin.org/post', data=m, headers={'Content-Type': m.content_type}) ``` -------------------------------- ### SSLAdapter for Customizing SSL Protocols in Python Source: https://github.com/requests/toolbelt/blob/master/README.rst Demonstrates how to use SSLAdapter to specify a particular SSL protocol for HTTPS connections within a `requests.Session`. This requires 'requests_toolbelt' and the 'ssl' module. It allows explicit control over the SSL version used. ```python from requests_toolbelt import SSLAdapter import requests import ssl s = requests.Session() s.mount('https://', SSLAdapter(ssl.PROTOCOL_TLSv1)) ``` -------------------------------- ### AuthHandler: Retrieving Strategy for a URL Source: https://github.com/requests/toolbelt/blob/master/docs/authentication.md Demonstrates how to retrieve the appropriate authentication strategy for a given URL using the `get_strategy_for` method of the AuthHandler. This method helps in determining which credentials to apply based on the URL's domain. ```python import requests a = AuthHandler({'example.com', ('foo', 'bar')}) strategy = a.get_strategy_for('http://example.com/example') assert isinstance(strategy, requests.auth.HTTPBasicAuth) ``` -------------------------------- ### Customizing Session Initialization for Threaded Requests Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md Shows how to provide a custom `initializer` function to `threaded.map`. This function is executed for each new `requests.Session` created, allowing for pre-configuration of headers, authentication, or other session settings. ```python from requests_toolbelt import threaded, user_agent def initialize_session(session): session.headers['User-Agent'] = user_agent('my-scraper', '0.1') session.headers['Accept'] = 'application/json' urls_to_get = [{ 'url': 'https://api.github.com/users/sigmavirus24', 'method': 'GET', }, { 'url': 'https://api.github.com/repos/requests/toolbelt', 'method': 'GET', }, { 'url': 'https://google.com', 'method': 'GET', }] responses, errors = threaded.map(urls_to_get, initializer=initialize_session) ``` -------------------------------- ### Run Tests with Tox Source: https://github.com/requests/toolbelt/blob/master/docs/contributing.md Execute tests for the project using tox across specified Python versions and flake8 style enforcers. Tox manages virtual environments and dependencies. ```shell $ tox -e py27,py34 $ tox -e py27-flake8 # or $ tox -e py34-flake8 ``` -------------------------------- ### MultipartEncoder for Simple Form Data using Python Source: https://github.com/requests/toolbelt/blob/master/README.rst Illustrates using MultipartEncoder for simple multipart/form-data requests without file attachments. This requires 'requests_toolbelt' and 'requests'. The input is a dictionary of key-value pairs. ```python from requests_toolbelt import MultipartEncoder import requests m = MultipartEncoder(fields={'field0': 'value', 'field1': 'value'}) r = requests.post('http://httpbin.org/post', data=m, headers={'Content-Type': m.content_type}) ``` -------------------------------- ### Configure SSL/TLS Certificates with X509Adapter Source: https://github.com/requests/toolbelt/blob/master/docs/adapters.md The X509Adapter allows passing a full certificate chain as part of a request without decrypting it into a .pem file. It supports specifying certificate bytes, private key bytes, and encoding. Useful for scenarios where .pem files are not practical or desired. ```python import requests from requests_toolbelt.adapters.x509 import X509Adapter s = requests.Session() a = X509Adapter(max_retries=3, cert_bytes=b'...', pk_bytes=b'...', encoding='...') s.mount('https://', a) ``` -------------------------------- ### Monitor Multipart Upload Progress with MultipartEncoderMonitor (Python) Source: https://github.com/requests/toolbelt/blob/master/docs/uploading-data.md Demonstrates how to wrap a MultipartEncoder with MultipartEncoderMonitor to track upload progress and trigger a callback function. This is useful for providing user feedback like progress bars during large uploads. It requires the 'requests' and 'requests-toolbelt' libraries. ```Python import requests from requests_toolbelt.multipart import encoder def my_callback(monitor): # Your callback function to process progress updates pass e = encoder.MultipartEncoder( fields={'field0': 'value', 'field1': 'value', 'field2': ('filename', open('file.py', 'rb'), 'text/plain')} ) m = encoder.MultipartEncoderMonitor(e, my_callback) r = requests.post('http://httpbin.org/post', data=m, headers={'Content-Type': m.content_type}) ``` ```Python import requests from requests_toolbelt.multipart.encoder import MultipartEncoderMonitor def my_callback(monitor): # Your callback function pass m = MultipartEncoderMonitor.from_fields( fields={'field0': 'value', 'field1': 'value', 'field2': ('filename', open('file.py', 'rb'), 'text/plain')}, callback=my_callback ) r = requests.post('http://httpbin.org/post', data=m, headers={'Content-Type': m.content_type}) ``` ```Python from requests_toolbelt import ( MultipartEncoder, MultipartEncoderMonitor ) import requests def callback(monitor): # Do something with this information, e.g., update a progress bar pass m = MultipartEncoder(fields={'field0': 'value0'}) monitor = MultipartEncoderMonitor(m, callback) headers = {'Content-Type': monitor.content_type} r = requests.post('https://httpbin.org/post', data=monitor, headers=headers) ``` ```Python from requests_toolbelt import MultipartEncoderMonitor import requests def callback(monitor): # Do something with this information pass monitor = MultipartEncoderMonitor.from_fields( fields={'field0': 'value0'}, callback ) headers = {'Content-Type': montior.content_type} r = requests.post('https://httpbin.org/post', data=monitor, headers=headers) ``` -------------------------------- ### Upload data with MultipartEncoder in Python Source: https://github.com/requests/toolbelt/blob/master/docs/uploading-data.md Demonstrates how to use the MultipartEncoder to construct and send a multipart/form-data upload. This method is useful for uploading files without loading them entirely into memory. It requires the 'requests' and 'requests_toolbelt' libraries. ```python import requests from requests_toolbelt.multipart.encoder import MultipartEncoder m = MultipartEncoder( fields={'field0': 'value', 'field1': 'value', 'field2': ('filename', open('file.py', 'rb'), 'text/plain')} ) r = requests.post('http://httpbin.org/post', data=m, headers={'Content-Type': m.content_type}) ``` ```python import requests from requests_toolbelt import MultipartEncoder encoder = MultipartEncoder({'field': 'value', 'other_field': 'other_value'}) r = requests.post('https://httpbin.org/post', data=encoder, headers={'Content-Type': encoder.content_type}) ``` ```python r = requests.post('https://httpbin.org/post', data=encoder.to_string(), headers={'Content-Type': encoder.content_type}) ``` ```python encoder = MultipartEncoder([('field', 'value'), ('other_field', 'other_value')]) ``` ```python encoder = MultipartEncoder({ 'field': ('file_name', b'{"a": "b"}', 'application/json', {'X-My-Header': 'my-value'}) ]) ``` -------------------------------- ### BaseUrlSession - Session with Base URL Source: https://context7.com/requests/toolbelt/llms.txt Extends `requests.Session` to automatically prepend a base URL to all requests. This simplifies API client implementations by reducing repetitive URL construction. ```APIDOC ## BaseUrlSession - Session with Base URL ### Description Extends requests.Session to automatically prepend a base URL to all requests, simplifying API client implementations. ### Method GET, POST (examples) ### Endpoint `https://api.example.com/v1/users` `https://api.example.com/v1/users/123` `https://api.example.com/v1/posts` `https://api.example.com/status` (absolute path) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body (for POST /posts) - **title** (string) - Required - The title of the post. - **content** (string) - Required - The content of the post. ### Request Example ```python import requests from requests_toolbelt.sessions import BaseUrlSession # Create session with base URL session = BaseUrlSession(base_url='https://api.example.com/v1/') # All requests are relative to base URL users = session.get('users').json() print(f"Users: {len(users)}") user_detail = session.get('users/123').json() print(f"User: {user_detail['name']}") # Create new resource new_post = session.post('posts', json={ 'title': 'New Post', 'content': 'Post content here' }) print(f"Created post: {new_post.status_code}") # Leading slash goes to absolute path root = session.get('/status') print(f"Absolute path request URL: {root.request.url}") ``` ### Response #### Success Response (200) - **users** (array) - List of users. - **name** (string) - Name of the user. - **status_code** (integer) - HTTP status code. #### Response Example ```json [ {"id": "1", "name": "Alice"}, {"id": "2", "name": "Bob"} ] ``` ```json { "name": "Alice" } ``` ```json { "status_code": 200 } ``` ``` -------------------------------- ### Basic Threaded Request Mapping Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md Demonstrates the fundamental usage of `threaded.map` to send multiple HTTP requests concurrently. It takes a list of dictionaries, where each dictionary specifies the URL and method for a request, and returns responses and any errors encountered. ```python from requests_toolbelt import threaded urls_to_get = [{ 'url': 'https://api.github.com/users/sigmavirus24', 'method': 'GET', }, { 'url': 'https://api.github.com/repos/requests/toolbelt', 'method': 'GET', }, { 'url': 'https://google.com', 'method': 'GET', }] responses, errors = threaded.map(urls_to_get) ``` -------------------------------- ### Configuring Number of Processes for Threaded Requests Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md Illustrates how to control the number of worker threads used by `threaded.map` by specifying the `num_processes` keyword argument. This allows tuning performance based on available system resources. ```python from requests_toolbelt import threaded urls_to_get = [{ 'url': 'https://api.github.com/users/sigmavirus24', 'method': 'GET', }, { 'url': 'https://api.github.com/repos/requests/toolbelt', 'method': 'GET', }, { 'url': 'https://google.com', 'method': 'GET', }] responses, errors = threaded.map(urls_to_get, num_processes=10) ``` -------------------------------- ### Create Request Pool from Queue Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md This snippet demonstrates how to create a `Pool` object from a `queue.Queue` containing job dictionaries. It then processes all jobs and iterates through the responses, printing the URL and status code. This is useful for managing a dynamic list of requests. ```python import queue from requests_toolbelt.threaded import pool jobs = queue.Queue() urls = [ # My list of URLs to get ] for url in urls: jobs.put({'method': 'GET', 'url': url}) p = pool.Pool(job_queue=jobs) p.join_all() for response in p.responses(): print('GET {}. Returned {}.'.format(response.request_kwargs['url'], response.status_code)) ``` -------------------------------- ### Mount SourceAddressAdapter with IP tuple Source: https://github.com/requests/toolbelt/blob/master/docs/adapters.md Mounts the SourceAddressAdapter to a Requests Session to specify a source IP address and port for outgoing HTTPS connections. This allows requests to originate from a specific local address and port. ```python import requests from requests_toolbelt.adapters.source import SourceAddressAdapter s = requests.Session() s.mount('https://', SourceAddressAdapter(('10.10.10.10', 8999))) ``` -------------------------------- ### requests_toolbelt.downloadutils.tee.tee_to_file Source: https://github.com/requests/toolbelt/blob/master/docs/downloadutils.md Streams response content and writes it to a specified file. ```APIDOC ## requests_toolbelt.downloadutils.tee.tee_to_file ### Description Streams the response content both to a generator and a file. This function opens a file and writes the response body bytes to it. ### Method N/A (Generator function) ### Endpoint N/A ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python resp = requests.get(url, stream=True) for chunk in tee_to_file(resp, 'save_file'): # do stuff with chunk ``` ### Response #### Success Response (Generator yielding chunks) - **chunk** (bytes) - A chunk of the streamed response content. #### Response Example ```python # The generator yields chunks of bytes ``` ``` -------------------------------- ### SocketOptionsAdapter Source: https://github.com/requests/toolbelt/blob/master/docs/adapters.md Allows users to pass specific options to be set on created sockets. This adapter leverages urllib3's support for setting arbitrary socket options, useful for fine-tuning network behavior like disabling Nagle's Algorithm. ```APIDOC ## POST /requests/toolbelt/socket_options ### Description The `SocketOptionsAdapter` allows a user to pass specific options to be set on created sockets when constructing the Adapter without subclassing. ### Method (Adapter Class - Used within a requests.Session) ### Endpoint (Adapter Class - Used within a requests.Session) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **socket_options** (list) - Required - A list of three-item tuples representing socket options. ### Request Example ```python import socket import requests from requests_toolbelt.adapters.socket_options import SocketOptionsAdapter nagles = [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 0)] session = requests.Session() for scheme in session.adapters.keys(): session.mount(scheme, SocketOptionsAdapter(socket_options=nagles)) ``` ### Response #### Success Response (200) (Standard HTTP Response) #### Response Example (Standard HTTP Response) ``` -------------------------------- ### requests_toolbelt.threaded.pool.Pool Class Initialization Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md Details on initializing the `Pool` class, which manages threads and `requests.Session` instances. It outlines the parameters for controlling job queues, session initialization, authentication generation, and the number of worker processes. ```python # Class definition and constructor would typically go here. class Pool: def __init__(self, job_queue, initializer=None, auth_generator=None, num_processes=None, session=requests.Session): """Pool that manages the threads containing sessions. * **Parameters:** * **job_queue** ([*queue.Queue*](https://docs.python.org/3/library/queue.html#queue.Queue)) – The queue you’re expected to use to which you should add items. * **initializer** (*collections.Callable*) – Function used to initialize an instance of `session`. * **auth_generator** (*collections.Callable*) – Function used to generate new auth credentials for the session. * **num_processes** ([*int*](https://docs.python.org/3/library/functions.html#int)) – Number of threads to create. * **session** (*requests.Session*) """ pass ``` -------------------------------- ### Create Pool from URLs (Python) Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md Initializes a Pool object from an iterable of URLs. It accepts optional keyword arguments for request methods and Pool initialization. This class method is useful for setting up concurrent requests. ```python from requests_toolbelt.threaded import pool # Example usage: urls = ['http://example.com/page1', 'http://example.com/page2'] request_kwargs = {'timeout': 5} my_pool = pool.Pool.from_urls(urls, request_kwargs=request_kwargs, size=10) ``` -------------------------------- ### AuthHandler: Adding a New Authentication Strategy Source: https://github.com/requests/toolbelt/blob/master/docs/authentication.md Shows how to dynamically add a new authentication strategy for a specific domain to an existing AuthHandler instance using the `add_strategy` method. This is useful for configuring authentication on the fly. ```python a = AuthHandler({}) a.add_strategy('https://api.github.com', ('username', 'password')) ``` -------------------------------- ### Dump All Request-Response Cycles with Python Source: https://github.com/requests/toolbelt/blob/master/docs/dumputils.md The `dump_all` function from `requests_toolbelt.utils.dump` captures all request-response pairs, including redirects, from a given response object. It returns a bytearray containing the formatted data. Optional `request_prefix` and `response_prefix` can be provided to customize the output. ```python import requests from requests_toolbelt.utils import dump resp = requests.get('https://httpbin.org/redirect/5') data = dump.dump_all(resp) print(data.decode('utf-8')) ``` -------------------------------- ### Simplified Request Mapping with `threaded.map` Source: https://github.com/requests/toolbelt/blob/master/docs/threading.md This snippet shows the simplest way to perform multiple requests concurrently using the `requests_toolbelt.threaded.map` function. It takes a list of request dictionaries and returns generators for responses and exceptions, abstracting away the `Pool` management. ```python from requests_toolbelt import threaded requests = [{ 'method': 'GET', 'url': 'https://httpbin.org/get', # ... }, { # ... }, { # ... }] responses_generator, exceptions_generator = threaded.map(requests) for response in responses_generator: # Do something ``` -------------------------------- ### Specify Socket Options with SocketOptionsAdapter Source: https://github.com/requests/toolbelt/blob/master/docs/adapters.md The SocketOptionsAdapter allows users to specify custom socket options before establishing a connection. It can be used with default options or extended with custom ones. Requires requests version 2.4.0 or newer. ```python import socket import requests from requests_toolbelt.adapters import socket_options s = requests.Session() opts = [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 0)] adapter = socket_options.SocketOptionsAdapter(socket_options=opts) s.mount('http://', adapter) # To use default options in addition to custom ones: # opts = socket_options.SocketOptionsAdapter.default_options + opts ``` -------------------------------- ### Mount SourceAddressAdapter with IP string Source: https://github.com/requests/toolbelt/blob/master/docs/adapters.md Mounts the SourceAddressAdapter to a Requests Session to specify a source IP address for outgoing HTTP connections. This allows requests to originate from a specific network interface. ```python import requests from requests_toolbelt.adapters.source import SourceAddressAdapter s = requests.Session() s.mount('http://', SourceAddressAdapter('10.10.10.10')) ``` -------------------------------- ### GuessAuth - Automatic Authentication Detection (Python) Source: https://context7.com/requests/toolbelt/llms.txt Automatically detects and handles Basic or Digest authentication based on the WWW-Authenticate header returned by the server. Simplifies authentication by removing the need to manually specify the authentication type. ```python import requests from requests_toolbelt import GuessAuth session = requests.Session() session.auth = GuessAuth('username', 'password') ``` -------------------------------- ### SSLAdapter - Custom SSL/TLS Protocol Selection with Python Requests Source: https://context7.com/requests/toolbelt/llms.txt Allows specifying SSL/TLS protocol versions for HTTPS connections. This is particularly useful when interacting with legacy servers that require specific protocol versions due to security constraints or compatibility issues. It takes an SSL protocol constant as input. ```python import requests import ssl from requests_toolbelt import SSLAdapter session = requests.Session() # Force TLS 1.2 for legacy server compatibility session.mount('https://', SSLAdapter(ssl.PROTOCOL_TLSv1_2)) try: response = session.get('https://legacy-api.example.com/data') print(f"Connected with TLS 1.2: {response.status_code}") print(f"Data: {response.json()}") except requests.exceptions.SSLError as e: print(f"SSL Error: {e}") ``` -------------------------------- ### SourceAddressAdapter - Bind to Specific Network Interface Source: https://context7.com/requests/toolbelt/llms.txt Enables binding HTTP requests to a specific local IP address and optionally a specific port. This is beneficial for multi-homed systems where you need to control the source network interface for outgoing connections. ```APIDOC ## SourceAddressAdapter - Bind to Specific Network Interface ### Description Enables binding HTTP requests to a specific local IP address and optionally a specific port, useful for multi-homed systems. ### Method GET (example) ### Endpoint `https://httpbin.org/ip` `http://httpbin.org/get` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python import requests from requests_toolbelt.adapters.source import SourceAddressAdapter session = requests.Session() # Bind to specific network interface IP session.mount('http://', SourceAddressAdapter('192.168.1.100')) session.mount('https://', SourceAddressAdapter('192.168.1.100')) response = session.get('https://httpbin.org/ip') print(f"Request sent from: {response.json()['origin']}") # With explicit port (use carefully with connection pooling) session2 = requests.Session() session2.mount('http://', SourceAddressAdapter(('192.168.1.100', 8999))) response2 = session2.get('http://httpbin.org/get') print(f"Status: {response2.status_code}") ``` ### Response #### Success Response (200) - **origin** (string) - The originating IP address of the request. - **status_code** (integer) - The HTTP status code of the response. #### Response Example ```json { "origin": "192.168.1.100" } ``` ```json { "status_code": 200 } ``` ``` -------------------------------- ### SourceAddressAdapter Source: https://github.com/requests/toolbelt/blob/master/docs/adapters.md Enables specifying a local source address for network connections. This allows you to control the IP address and port from which your HTTP requests originate, useful for testing or network configurations. ```APIDOC ## POST /requests/toolbelt/source_address ### Description A Source Address Adapter for Python Requests that enables you to choose the local address to bind to. This allows you to send your HTTP requests from a specific interface and IP address. ### Method (Adapter Class - Used within a requests.Session) ### Endpoint (Adapter Class - Used within a requests.Session) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python import requests from requests_toolbelt.adapters.source import SourceAddressAdapter s = requests.Session() s.mount('http://', SourceAddressAdapter('10.10.10.10')) s.mount('https://', SourceAddressAdapter(('10.10.10.10', 8999))) ``` ### Response #### Success Response (200) (Standard HTTP Response) #### Response Example (Standard HTTP Response) ``` -------------------------------- ### SSLAdapter - Custom SSL/TLS Protocol Selection Source: https://context7.com/requests/toolbelt/llms.txt Allows selection of specific SSL/TLS protocol versions for HTTPS connections. This is useful for ensuring compatibility with legacy systems or enforcing specific security standards. ```APIDOC ## SSLAdapter - Custom SSL/TLS Protocol Selection ### Description Allows selecting specific SSL/TLS protocol versions for HTTPS connections, useful for legacy systems or security requirements. ### Method GET (example) ### Endpoint `https://legacy-api.example.com/data` ### Parameters #### Query Parameters None #### Request Body None ### Request Example ```python import requests import ssl from requests_toolbelt import SSLAdapter session = requests.Session() # Force TLS 1.2 for legacy server compatibility session.mount('https://', SSLAdapter(ssl.PROTOCOL_TLSv1_2)) try: response = session.get('https://legacy-api.example.com/data') print(f"Connected with TLS 1.2: {response.status_code}") print(f"Data: {response.json()}") except requests.exceptions.SSLError as e: print(f"SSL Error: {e}") ``` ### Response #### Success Response (200) - **data** (object) - The data retrieved from the server. #### Response Example ```json { "key": "value" } ``` ``` -------------------------------- ### BaseUrlSession - Session with Predefined Base URL in Python Requests Source: https://context7.com/requests/toolbelt/llms.txt Extends requests.Session to automatically prepend a base URL to all generated request URLs. This simplifies building API clients by reducing redundancy when making multiple calls to the same API endpoint. It takes a base_url string during initialization. ```python import requests from requests_toolbelt.sessions import BaseUrlSession # Create session with base URL session = BaseUrlSession(base_url='https://api.example.com/v1/') # All requests are relative to base URL users = session.get('users').json() print(f"Users: {len(users)}") user_detail = session.get('users/123').json() print(f"User: {user_detail['name']}") # Create new resource new_post = session.post('posts', json={ 'title': 'New Post', 'content': 'Post content here' }) print(f"Created post: {new_post.status_code}") # Leading slash goes to absolute path root = session.get('/status') print(f"Absolute path request URL: {root.request.url}") ``` -------------------------------- ### requests_toolbelt.downloadutils.tee.tee_to_bytearray Source: https://github.com/requests/toolbelt/blob/master/docs/downloadutils.md Streams response content and appends it to a provided bytearray. ```APIDOC ## requests_toolbelt.downloadutils.tee.tee_to_bytearray ### Description Streams response content and appends it to a provided bytearray. This function uses `bytearray.extend()` by default. ### Method N/A (Generator function) ### Endpoint N/A ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python b = bytearray() resp = requests.get(url, stream=True) for chunk in tee_to_bytearray(resp, b): # do stuff with chunk ``` ### Response #### Success Response (Generator yielding chunks) - **chunk** (bytes) - A chunk of the streamed response content. #### Response Example ```python # The generator yields chunks of bytes ``` ```