urllib3

https://urllib3.readthedocs.io/en/stable/
Admin
urllib3 is a powerful, user-friendly HTTP client for Python that provides features like thread...
Tokens:133,576
Snippets:938
Trust Score:9.6
Update:1 day ago
Show doc for...
Tokens:
Raw
### Basic urllib3 request example

Source: https://urllib3.readthedocs.io/en/stable/index.html

Shows how to make a simple GET request using urllib3.request and access the response status and data.

```python
>>> import urllib3
>>> resp = urllib3.request("GET", "https://httpbin.org/robots.txt")
>>> resp.status
200
>>> resp.data
b"User-agent: *\nDisallow: /deny\n"
```

--------------------------------

### Install urllib3 from source

Source: https://urllib3.readthedocs.io/en/stable/index.html

Commands to clone the urllib3 repository from GitHub and install it locally.

```bash
$ git clone https://github.com/urllib3/urllib3.git
$ cd urllib3
$ pip install .
```

--------------------------------

### Basic urllib3 request in Pyodide

Source: https://urllib3.readthedocs.io/en/stable/reference/contrib/emscripten.html

Shows how to make an HTTP request using urllib3 in a Pyodide environment. The response status, headers, and JSON body are printed. This example is from the 'Getting started' section.

```python
import urllib3
resp = urllib3.request("GET", "https://httpbin.org/anything")

print(resp.status)  # 200
print(resp.headers) # HTTPHeaderDict(...)
print(resp.json())  # {"headers": {"Accept": "*/*", ...}, ...}
```

--------------------------------

### Using requests with urllib3 in Pyodide

Source: https://urllib3.readthedocs.io/en/stable/reference/contrib/emscripten.html

Shows how to use the requests library, which is built on urllib3, in a Pyodide environment. The response status code and headers are printed. This example is from the 'Getting started' section.

```python
import requests
resp = requests.request("GET", "https://httpbin.org/anything")

print(resp.status_code)  # 200
print(resp.headers)
```

--------------------------------

### Install nox for development environment

Source: https://urllib3.readthedocs.io/en/stable/contributing.html

Install nox with pip to set up the development environment. This is the only required tool for contributing.

```bash
$ python -m pip install --user --upgrade nox
```

--------------------------------

### Install urllib3 with SOCKS support

Source: https://urllib3.readthedocs.io/en/stable/advanced-usage.html

To use SOCKS proxies, install PySocks or install urllib3 with the socks extra.

```bash
python -m pip install urllib3[socks]
```

--------------------------------

### make_headers example

Source: https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html

Shows how to use urllib3.util.make_headers to generate HTTP headers. The example prints the resulting headers for keep_alive and user_agent, and for accept_encoding.

```python
import urllib3

print(urllib3.util.make_headers(keep_alive=True, user_agent="Batman/1.0"))
# {'connection': 'keep-alive', 'user-agent': 'Batman/1.0'}
print(urllib3.util.make_headers(accept_encoding=True))
# {'accept-encoding': 'gzip,deflate'}
```

--------------------------------

### Install urllib3 with pip

Source: https://urllib3.readthedocs.io/en/stable/index.html

Command to install urllib3 using pip.

```bash
$ python -m pip install urllib3
```

--------------------------------

### Install pyOpenSSL dependencies

Source: https://urllib3.readthedocs.io/en/stable/reference/contrib/pyopenssl.html

Install the required packages for the pyOpenSSL backend: pyOpenSSL, cryptography, and idna. Use this command before using the module.

```bash
$ python -m pip install pyopenssl cryptography idna
```

--------------------------------

### Install urllib3 with Brotli extra

Source: https://urllib3.readthedocs.io/en/stable/advanced-usage.html

Install the Brotli package via the urllib3[brotli] extra to enable Brotli compression support.

```bash
$ python -m pip install urllib3[brotli]
```

--------------------------------

### Using urllib3.request() to send a GET request

Source: https://urllib3.readthedocs.io/en/stable/v2-migration-guide.html

Shows the new top-level urllib3.request() function, which uses a global poolmanager instance. The example sends a GET request to https://example.com and prints the response status (200).

```python
>>> import urllib3
>>> resp = urllib3.request("GET", "https://example.com")
>>> resp.status
200
```

--------------------------------

### ProxyManager example: sharing pools for HTTP and HTTPS requests

Source: https://urllib3.readthedocs.io/en/stable/reference/urllib3.poolmanager.html

This example creates a ProxyManager pointing to a local HTTPS proxy and makes requests to different hosts. It shows that plain HTTP requests share a single connection pool, while each HTTPS target gets its own pool, as indicated by the printed pool counts.

```python
import urllib3

proxy = urllib3.ProxyManager("https://localhost:3128/")

resp1 = proxy.request("GET", "http://google.com/")
resp2 = proxy.request("GET", "http://httpbin.org/")

# One pool was shared by both plain HTTP requests.
print(len(proxy.pools))
# 1

resp3 = proxy.request("GET", "https://httpbin.org/")
resp4 = proxy.request("GET", "https://twitter.com/")

# A separate pool was added for each HTTPS target.
print(len(proxy.pools))
# 3
```

--------------------------------

### Create a connection pool from a URL and make a GET request

Source: https://urllib3.readthedocs.io/en/stable/reference/urllib3.connectionpool.html

Shows how to use connection_from_url() to create a ConnectionPool instance from a URL and then call request('GET', '/') on it. The URL must include the scheme; port is optional. Additional keyword arguments can be passed to the pool constructor.

```python
>>> conn = connection_from_url('http://google.com/')
>>> r = conn.request('GET', '/')
```

--------------------------------

### Timeout.start_connect()

Source: https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html

Starts the timeout clock, used during a connect() attempt. Raises TimeoutStateError if the timer has already been started.

```APIDOC
## Timeout.start_connect()

### Description
Starts the timeout clock, used during a connect() attempt. Raises TimeoutStateError if the timer has already been started.

### Method
GET

### Endpoint
urllib3.util.Timeout.start_connect

### Parameters
None

### Response
#### Success Response (200)
- **return** (float) - The start time.

#### Response Example
```
0.0
```

### Errors
- **urllib3.exceptions.TimeoutStateError** - If you attempt to start a timer that has been started already.
```

--------------------------------

### Install certifi

Source: https://urllib3.readthedocs.io/en/stable/user-guide.html

Installs the certifi package, which provides Mozilla's root certificate bundle for certificate verification.

```bash
$ python -m pip install certifi
```

--------------------------------

### Install urllib3 v2.x locally

Source: https://urllib3.readthedocs.io/en/stable/v2-migration-guide.html

Upgrade urllib3 to version 2.x in your local environment to test your package against it.

```bash
$ python -m pip install -U 'urllib3>=2'
```

--------------------------------

### Make a GET request with PoolManager

Source: https://urllib3.readthedocs.io/en/stable/user-guide.html

Sends a GET request to httpbin.org/robots.txt and prints the response data.

```python
import urllib3

# Creating a PoolManager instance for sending requests.
http = urllib3.PoolManager()

# Sending a GET request and getting back response as HTTPResponse object.
resp = http.request("GET", "https://httpbin.org/robots.txt")

# Print the returned data.
print(resp.data)
# b"User-agent: *\nDisallow: /deny\n"
```

--------------------------------

### Install urllib3 with Zstandard extra

Source: https://urllib3.readthedocs.io/en/stable/advanced-usage.html

Install the urllib3[zstd] extra to enable Zstandard support on Python 3.13 and earlier. On Python 3.14+, Zstandard support is included in the standard library. Requires zstandard package v0.18.0 or later.

```bash
# This is only necessary on Python 3.13 and earlier.
# Otherwise zstandard support is included in the Python standard library.
$ python -m pip install urllib3[zstd]
```

--------------------------------

### Example output of getproxies() with problematic HTTPS proxy

Source: https://urllib3.readthedocs.io/en/stable/advanced-usage.html

This output shows a system-level proxy configuration where the HTTPS proxy uses https://, which is the problem.

```json
{
  "http": "http://127.0.0.1:8888",
  "https": "https://127.0.0.1:8888"  # <--- This setting is the problem!
}
```

--------------------------------

### Example FutureWarning message

Source: https://urllib3.readthedocs.io/en/stable/v2-migration-guide.html

Shows the format of a deprecation warning for the 'ssl_version' option, which is deprecated in favor of 'ssl_minimum_version'.

```text
FutureWarning: 'ssl_version' option is deprecated and will be removed
in urllib3 v3.0. Instead use 'ssl_minimum_version'
```

--------------------------------

### Run specific test sessions with nox

Source: https://urllib3.readthedocs.io/en/stable/contributing.html

Runs the test-3.12 and test-pypy3.11 sessions. Nox creates virtualenvs if needed, installs dependencies, and runs commands in order.

```bash
$ nox --reuse-existing-virtualenvs --sessions test-3.12 test-pypy3.11
[ Nox will create virtualenv if needed, install the specified dependencies, and run the commands in order.]
```

--------------------------------

### Timeout.get_connect_duration()

Source: https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html

Gets the time elapsed since the call to start_connect(). Raises TimeoutStateError if the timer hasn't been started.

```APIDOC
## Timeout.get_connect_duration()

### Description
Gets the time elapsed since the call to start_connect().

### Method
get_connect_duration()

### Returns
Elapsed time in seconds.

### Return Type
float

### Raises
- **urllib3.exceptions.TimeoutStateError** - if you attempt to get duration for a timer that hasn't been started.
```

--------------------------------

### Build documentation locally with nox -rs docs

Source: https://urllib3.readthedocs.io/en/stable/contributing.html

Builds the documentation locally using the nox tool. Run this command from the project root.

```bash
$ nox -rs docs
```

--------------------------------

### Pass query parameters via fields for GET/HEAD/DELETE

Source: https://urllib3.readthedocs.io/en/stable/user-guide.html

For `GET`, `HEAD`, and `DELETE` requests, pass query parameters as a dictionary in the `fields` argument to `urllib3.request()`. The example sends a GET request and prints the `args` from the response.

```python
import urllib3

resp = urllib3.request(
    "GET",
    "https://httpbin.org/get",
    fields={"arg": "value"}
)

print(resp.json()["args"])
# {"arg": "value"}
```

--------------------------------

### Run an HTTP proxy locally with python -m dummyserver.proxy

Source: https://urllib3.readthedocs.io/en/stable/contributing.html

Starts a local HTTP proxy server using the dummyserver module. Useful when developing proxy-related features.

```bash
$ python -m dummyserver.proxy
```

--------------------------------

### Read Set-Cookie header from response

Source: https://urllib3.readthedocs.io/en/stable/user-guide.html

Cookies provided by the server are stored in the `Set-Cookie` header. The example sends a GET request with `redirect=False` and prints the `Set-Cookie` header.

```python
import urllib3

resp = urllib3.request(
    "GET",
    "https://httpbin.org/cookies/set/session/f3efe9db",
    redirect=False
)

print(resp.headers["Set-Cookie"])
# session=f3efe9db; Path=/
```

--------------------------------

### Send request with custom headers using headers argument

Source: https://urllib3.readthedocs.io/en/stable/user-guide.html

Specify headers as a dictionary in the `headers` argument to `urllib3.request()`. The example sends a GET request to httpbin.org/headers and prints the response headers.

```python
import urllib3

resp = urllib3.request(
    "GET",
    "https://httpbin.org/headers",
    headers={
        "X-Something": "value"
    }
)

print(resp.json()["headers"])
# {"X-Something": "value", ...}
```

--------------------------------

### Create multi-valued headers with HTTPHeaderDict

Source: https://urllib3.readthedocs.io/en/stable/user-guide.html

Use `urllib3.HTTPHeaderDict` to add multiple values for the same header. The example adds two `Accept` headers and sends a GET request, resulting in a comma-separated header value.

```python
import urllib3

# Create an HTTPHeaderDict and add headers
headers = urllib3.HTTPHeaderDict()
headers.add("Accept", "application/json")
headers.add("Accept", "text/plain")

# Make the request using the headers
resp = urllib3.request(
    "GET",
    "https://httpbin.org/headers",
    headers=headers
)

print(resp.json()["headers"])
# {"Accept": "application/json, text/plain", ...}
```

--------------------------------

### Check and fix proxy environment variables in bash

Source: https://urllib3.readthedocs.io/en/stable/advanced-usage.html

Inspect HTTP_PROXY, HTTPS_PROXY, and ALL_PROXY environment variables. Ensure they start with http://, not https://. The example shows a problematic HTTPS_PROXY and how to fix it in the current session and persist it in the shell profile.

```bash
# Check your existing environment variables in bash
$ env | grep "_PROXY"
HTTP_PROXY=http://127.0.0.1:8888
HTTPS_PROXY=https://127.0.0.1:8888  # <--- This setting is the problem!

# Make the fix in your current session and test your script
$ export HTTPS_PROXY="http://127.0.0.1:8888"
$ python test-proxy.py  # This should now pass.

# Persist your change in your shell 'profile' (~/.bashrc, ~/.profile, ~/.bash_profile, etc)
# You may need to logout and log back in to ensure this works across all programs.
$ vim ~/.bashrc
```

--------------------------------

### Run an HTTPS proxy locally with python -m dummyserver.https_proxy

Source: https://urllib3.readthedocs.io/en/stable/contributing.html

Starts a local HTTPS proxy server using the dummyserver module. Useful when developing proxy-related features that require HTTPS.

```bash
$ python -m dummyserver.https_proxy
```

--------------------------------

### Sending JSON in a request and deserializing a JSON response

Source: https://urllib3.readthedocs.io/en/stable/v2-migration-guide.html

Demonstrates the new json= parameter for requests, which automatically adds 'Content-Type: application/json' by default, and the HTTPResponse.json() method for deserializing response bodies. The example shows a POST request with a JSON body and a GET request whose response is parsed with resp.json().

```python
import urllib3

# Send a request with a JSON body.
# This adds 'Content-Type: application/json' by default.
resp = urllib3.request(
    "POST", "https://example.api.com",
    json={"key": "value"}
)

# Receive a JSON body in the response.
resp = urllib3.request("GET", "https://xkcd.com/2347/info.0.json")

# There's always an XKCD...
resp.json()
{
  "num": 2347,
  "img": "https://imgs.xkcd.com/comics/dependency.png",
  "title": "Dependency",
  ...
}
```

--------------------------------

### Example: parse_url and Url.url round-trip

Source: https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html

Shows how to parse a URL with urllib3.util.parse_url and convert it back to a string with the Url.url property. The output demonstrates that the round-trip preserves the URL, and that constructing a Url with explicit components yields the expected string.

```python
import urllib3

U = urllib3.util.parse_url("https://google.com/mail/")

print(U.url)
# "https://google.com/mail/"

print( urllib3.util.Url("https", "username:password",
                        "host.com", 80, "/path", "query", "fragment"
                        ).url
    )
# "https://username:password@host.com:80/path?query#fragment"
```

--------------------------------

### HTTPConnection.__init__

Source: https://urllib3.readthedocs.io/en/stable/reference/urllib3.connection.html

Constructor for HTTPConnection. Accepts host, port, and various keyword arguments to configure the connection, such as timeout, source_address, blocksize, socket_options, proxy, and proxy_config.

```APIDOC
## HTTPConnection.__init__

### Description
Initializes a new HTTPConnection instance. Provides a backwards-compatibility layer between older and newer Pythons.

### Method
Constructor

### Signature
`HTTPConnection(host, port=None, *, timeout=_TYPE_DEFAULT.token, source_address=None, blocksize=16384, socket_options=[(6, 1, 1)], proxy=None, proxy_config=None)`

### Parameters
- **host** (str) - Required - The host to connect to.
- **port** (int | None) - Optional - The port to connect to. Defaults to None.
- **timeout** (_TYPE_TIMEOUT) - Optional - Timeout for the connection.
- **source_address** (tuple[str, int] | None) - Optional - Source address for the connection.
- **blocksize** (int) - Optional - Block size for the connection. Defaults to 16384.
- **socket_options** (list[tuple[int, int, int | bytes]] | None) - Optional - Socket options to set on the underlying socket. If not specified, defaults are loaded from `HTTPConnection.default_socket_options`.
- **proxy** (Url | None) - Optional - Proxy URL.
- **proxy_config** (ProxyConfig | None) - Optional - Proxy configuration.

### Notes
- `socket_options` can be used to enable TCP Keep Alive by adding to the defaults, or disable defaults by passing an empty list.

### Example
```python
HTTPConnection.default_socket_options + [
    (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
]
```
```

--------------------------------

### connection_from_url(url, **kw)

Source: https://urllib3.readthedocs.io/en/stable/reference/urllib3.connectionpool.html

Given a url, return a ConnectionPool instance of its host. This is a shortcut for not having to parse out the scheme, host, and port of the url before creating a ConnectionPool instance.

```APIDOC
## connection_from_url(url, **kw)

### Description
Given a url, return a ConnectionPool instance of its host. This is a shortcut for not having to parse out the scheme, host, and port of the url before creating a ConnectionPool instance.

### Method
connection_from_url(url, **kw)

### Parameters
- **url** (str) - Required - Absolute URL string that must include the scheme. Port is optional.
- **\*\*kw** (Any) - Optional - Passes additional parameters to the constructor of the appropriate ConnectionPool. Useful for specifying things like timeout, maxsize, headers, etc.

### Return Type
HTTPConnectionPool

### Example
```
>>> conn = connection_from_url('http://google.com/')
>>> r = conn.request('GET', '/')
```
```

--------------------------------

### Create PoolManager and make requests

Source: https://urllib3.readthedocs.io/en/stable/reference/urllib3.poolmanager.html

Shows how to instantiate urllib3.PoolManager with num_pools=2, make three GET requests, and print the number of pools. The output is 2 because the pool manager caches up to num_pools pools before discarding the least recently used.

```python
import urllib3

http = urllib3.PoolManager(num_pools=2)

resp1 = http.request("GET", "https://google.com/")
resp2 = http.request("GET", "https://google.com/mail")
resp3 = http.request("GET", "https://yahoo.com/")

print(len(http.pools))
# 2
```

--------------------------------

### Check system-level proxy settings with urllib.request.getproxies()

Source: https://urllib3.readthedocs.io/en/stable/advanced-usage.html

On Windows or macOS, proxy settings may be set at the system level. Run this command to see the current proxy configuration.

```python
$ python -c 'import urllib.request; print(urllib.request.getproxies())'
```

--------------------------------

### PoolManager.connection_from_url()

Source: https://urllib3.readthedocs.io/en/stable/reference/index.html

Gets a connection from a URL.

```APIDOC
## PoolManager.connection_from_url()

### Description
Gets a connection from a URL.

### Method
Method call

### Endpoint
PoolManager.connection_from_url()

### Parameters
Refer to the detailed documentation for parameter definitions.

### Request Example
```python
conn = manager.connection_from_url(url)
```

### Response
Returns a connection object.
```

--------------------------------

### Update install_requires to support urllib3 1.26 and 2.x

Source: https://urllib3.readthedocs.io/en/stable/v2-migration-guide.html

Set your package's dependency specifier to allow both urllib3 1.26.x and 2.x. Shown for setup.py (setuptools) and pyproject.toml (hatch).

```python
# setup.py (setuptools)
setup(
  ...
  install_requires=["urllib3>=1.26,<3"]
)
```

```toml
# pyproject.toml (hatch)
[project]
dependencies = [
  "urllib3>=1.26,<3"
]
```

--------------------------------

### PoolManager.connection_from_pool_key()

Source: https://urllib3.readthedocs.io/en/stable/reference/index.html

Gets a connection from a pool key.

```APIDOC
## PoolManager.connection_from_pool_key()

### Description
Gets a connection from a pool key.

### Method
Method call

### Endpoint
PoolManager.connection_from_pool_key()

### Parameters
Refer to the detailed documentation for parameter definitions.

### Request Example
```python
conn = manager.connection_from_pool_key(pool_key)
```

### Response
Returns a connection object.
```

--------------------------------

### PoolManager.connection_from_host()

Source: https://urllib3.readthedocs.io/en/stable/reference/index.html

Gets a connection from a host and port.

```APIDOC
## PoolManager.connection_from_host()

### Description
Gets a connection from a host and port.

### Method
Method call

### Endpoint
PoolManager.connection_from_host()

### Parameters
Refer to the detailed documentation for parameter definitions.

### Request Example
```python
conn = manager.connection_from_host(host, port, scheme)
```

### Response
Returns a connection object.
```

--------------------------------

### urllib3.util.create_urllib3_context()

Source: https://urllib3.readthedocs.io/en/stable/reference/index.html

Creates an SSLContext with urllib3's default settings.

```APIDOC
## create_urllib3_context()

### Description
Creates an SSLContext with urllib3's default settings.

### Parameters
None

### Returns
SSLContext - The created SSL context.
```

--------------------------------

### PoolManager.connection_from_context()

Source: https://urllib3.readthedocs.io/en/stable/reference/index.html

Gets a connection from a context object.

```APIDOC
## PoolManager.connection_from_context()

### Description
Gets a connection from a context object.

### Method
Method call

### Endpoint
PoolManager.connection_from_context()

### Parameters
Refer to the detailed documentation for parameter definitions.

### Request Example
```python
conn = manager.connection_from_context(context)
```

### Response
Returns a connection object.
```

--------------------------------

### Access response status, data, and headers

Source: https://urllib3.readthedocs.io/en/stable/user-guide.html

Shows how to make a request and print the status, data, and headers of the HTTPResponse object. The output demonstrates the expected format for each attribute.

```python
import urllib3

# Making the request (The request function returns HTTPResponse object)
resp = urllib3.request("GET", "https://httpbin.org/ip")

print(resp.status)
# 200
print(resp.data)
# b"{\n  "origin": "104.232.115.37"\n}\n"
print(resp.headers)
# HTTPHeaderDict({"Content-Length": "32", ...})
```
Tokens:
Raw