### Install w3lib
Source: https://context7.com/scrapy/w3lib/llms.txt
Command to install the w3lib library via pip.
```bash
pip install w3lib
```
--------------------------------
### Get Meta Refresh Info from HTML - Python
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Parses HTML content to find meta refresh tags. Returns a tuple containing the refresh interval in seconds and the target URL. If no meta refresh tag is found, it returns (None, None). It can ignore specified HTML tags during parsing.
```python
import w3lib.html
html_content = b''
refresh_info = w3lib.html.get_meta_refresh(html_content)
# refresh_info will be (5.0, 'http://example.com/page2')
html_no_refresh = b'No refresh here'
refresh_info_none = w3lib.html.get_meta_refresh(html_no_refresh)
# refresh_info_none will be (None, None)
```
--------------------------------
### Get Base URL from HTML - Python
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Extracts the base URL declared in HTML text. It resolves relative URLs against a provided base URL. If no base URL is found in the HTML, the original baseurl is returned. Handles both string and byte inputs for HTML content.
```python
import w3lib.html
html_content = b''
base_url = w3lib.html.get_base_url(html_content, baseurl='http://default.com')
# base_url will be 'http://example.com/'
```
--------------------------------
### Generate Safe Download URL with w3lib
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Creates a safe URL suitable for downloading by applying safe_url_string and stripping any fragment. It also normalizes the path and ensures it stays within the document root if necessary. Supports both string and bytes inputs for the URL.
```python
import w3lib.url
# Example usage (assuming a URL exists)
# download_url = w3lib.url.safe_download_url('http://example.com/some/path?query=1#fragment')
# print(download_url)
```
--------------------------------
### Convert Filesystem Paths and URIs
Source: https://context7.com/scrapy/w3lib/llms.txt
Utilities to convert between local filesystem paths and file URIs, or normalize any input into a valid URI format.
```python
import w3lib.url
# Convert path to file URI
uri = w3lib.url.path_to_file_uri('/home/user/documents/file.txt')
# Convert file URI back to path
path = w3lib.url.file_uri_to_path('file:///home/user/documents/file.txt')
# Handle any input (path or URI)
uri = w3lib.url.any_to_uri('/home/user/file.txt')
```
--------------------------------
### HTTP Module Utilities
Source: https://context7.com/scrapy/w3lib/llms.txt
Functions for handling HTTP headers, including creating basic authentication headers and parsing/converting raw headers to/from dictionaries.
```APIDOC
## w3lib.http.basic_auth_header
### Description
Generates the `Authorization` header value for HTTP Basic Authentication.
### Method
`w3lib.http.basic_auth_header(username, password, encoding='utf-8')`
### Parameters
- **username** (str) - The username for authentication.
- **password** (str) - The password for authentication.
- **encoding** (str, optional) - The encoding to use for the username and password before Base64 encoding. Defaults to 'utf-8'.
### Request Example
```python
import w3lib.http
# Create basic auth header
auth = w3lib.http.basic_auth_header('username', 'password')
print(auth)
# Output: b'Basic dXNlcm5hbWU6cGFzc3dvcmQ='
# Use in HTTP requests
import urllib.request
req = urllib.request.Request('http://example.com/api')
req.add_header('Authorization', w3lib.http.basic_auth_header('user', 'pass'))
# Custom encoding for special characters
auth = w3lib.http.basic_auth_header('user', 'pässwörd', encoding='utf-8')
print(auth)
# Output: b'Basic dXNlcjpww6Rzc3fDtnJk'
```
## w3lib.http.headers_raw_to_dict
### Description
Parses raw HTTP headers (provided as bytes) into a dictionary. Headers with the same name will have their values stored in a list.
### Method
`w3lib.http.headers_raw_to_dict(raw_headers)`
### Parameters
- **raw_headers** (bytes) - The raw HTTP headers string.
### Request Example
```python
import w3lib.http
# Parse raw headers
raw_headers = b"Content-Type: text/html\r\nAccept: gzip\r\nAccept: deflate\r\n"
headers = w3lib.http.headers_raw_to_dict(raw_headers)
print(headers)
# Output: {b'Content-Type': [b'text/html'], b'Accept': [b'gzip', b'deflate']}
# Multiple values for same header are collected in a list
raw = b"Set-Cookie: session=abc\r\nSet-Cookie: token=xyz\r\n"
headers = w3lib.http.headers_raw_to_dict(raw)
print(headers[b'Set-Cookie'])
# Output: [b'session=abc', b'token=xyz']
# Invalid headers are skipped
headers = w3lib.http.headers_raw_to_dict(b"Invalid header without colon\r\n")
print(headers)
# Output: {}
```
## w3lib.http.headers_dict_to_raw
### Description
Converts a dictionary of HTTP headers back into the raw byte format, suitable for sending in an HTTP request.
### Method
`w3lib.http.headers_dict_to_raw(headers_dict)`
### Parameters
- **headers_dict** (dict) - A dictionary where keys are header names (bytes) and values are header values (bytes or list of bytes).
### Request Example
```python
import w3lib.http
# Convert dict to raw headers
headers_dict = {b'Content-Type': b'text/html', b'Accept': b'gzip'}
raw = w3lib.http.headers_dict_to_raw(headers_dict)
print(raw)
# Output: b'Content-Type: text/html\r\nAccept: gzip'
# Handle multiple values for a header
headers_dict = {b'Set-Cookie': [b'session=abc', b'token=xyz']}
raw = w3lib.http.headers_dict_to_raw(headers_dict)
print(raw)
# Output: b'Set-Cookie: session=abc\r\nSet-Cookie: token=xyz'
```
```
--------------------------------
### Convert Path to File URI with w3lib
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Converts a local filesystem path into a legal File URI. This function ensures that the resulting URI is compliant with the File URI scheme specification.
```python
import w3lib.url
# Example usage (assuming a local path exists)
# local_path = "/path/to/your/file.txt"
# file_uri = w3lib.url.path_to_file_uri(local_path)
# print(file_uri)
```
--------------------------------
### Safe Download URL
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Creates a safe URL suitable for downloading, normalizing the path and stripping fragments.
```APIDOC
## Safe Download URL
### Description
Generates a URL string that is safe for downloading. This function first calls `safe_url_string` for basic safety and normalization, then strips any fragment identifier. The path component of the URL is also normalized. If the path is outside the document root, it will be adjusted to be within it.
### Method
N/A (This is a Python function, not an HTTP endpoint)
### Endpoint
N/A
### Parameters
- **url** (str | bytes) - Required - The URL to make safe for download.
- **encoding** (str) - Optional - Defaults to 'utf8'. The encoding to use for the URL.
- **path_encoding** (str) - Optional - Defaults to 'utf8'. The encoding to use for the path component of the URL.
### Request Example
```python
import w3lib.url
# Example of creating a safe download URL
print(w3lib.url.safe_download_url('http://example.com/path/to/resource#fragment'))
# Output: http://example.com/path/to/resource
```
### Response
#### Success Response (str)
- **safe_url** (str) - The normalized and safe URL string suitable for downloading.
```
--------------------------------
### Create HTTP Basic Auth Header
Source: https://context7.com/scrapy/w3lib/llms.txt
Generates a base64-encoded Authorization header for HTTP Basic Authentication. Supports custom character encoding.
```python
import w3lib.http
auth = w3lib.http.basic_auth_header('username', 'password')
print(auth)
```
--------------------------------
### File URI to Path Conversion
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Converts a File URI scheme to a local filesystem path.
```APIDOC
## File URI to Path Conversion
### Description
Converts a given File URI string into its corresponding local filesystem path, adhering to the specifications of the File URI scheme.
### Method
N/A (This is a Python function, not an HTTP endpoint)
### Endpoint
N/A
### Parameters
- **uri** (str) - Required - The File URI string to convert.
### Request Example
```python
import w3lib.url
# Example conversion (assuming a file exists at this path)
print(w3lib.url.file_uri_to_path('file:///home/user/document.txt'))
# Output: /home/user/document.txt
```
### Response
#### Success Response (str)
- **file_path** (str) - The local filesystem path derived from the File URI.
```
--------------------------------
### Path to File URI Conversion
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Converts a local filesystem path to a legal File URI.
```APIDOC
## Path to File URI Conversion
### Description
Converts a local filesystem path into a legal File URI, ensuring compliance with the File URI scheme specification.
### Method
N/A (This is a Python function, not an HTTP endpoint)
### Endpoint
N/A
### Parameters
- **path** (str | PathLike) - Required - The local filesystem path to convert.
### Request Example
```python
import w3lib.url
# Example conversion
print(w3lib.url.path_to_file_uri('/home/user/document.txt'))
# Output: file:///home/user/document.txt
```
### Response
#### Success Response (str)
- **file_uri** (str) - The legal File URI string generated from the input path.
```
--------------------------------
### Convert File URI to Path with w3lib
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Converts a File URI string into a local filesystem path. This function adheres to the specifications outlined in the File URI scheme.
```python
import w3lib.url
# Example usage (assuming a file URI exists)
# file_uri = "file:///path/to/your/file.txt"
# local_path = w3lib.url.file_uri_to_path(file_uri)
# print(local_path)
```
--------------------------------
### w3lib.url Module Functions
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Provides utility functions for URL manipulation, including parsing data URIs, adding/replacing URL parameters, and converting paths to file URIs.
```APIDOC
## w3lib.url.add_or_replace_parameter
### Description
Adds a new parameter to a URL or replaces the value of an existing parameter. If the URL already contains parameters, the new parameter is appended, or the existing one is updated.
### Method
N/A (Function)
### Endpoint
N/A (Function)
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Request Example
```python
import w3lib.url
url1 = 'http://www.example.com/index.php'
print(w3lib.url.add_or_replace_parameter(url1, 'arg', 'v'))
# Expected output: 'http://www.example.com/index.php?arg=v'
url2 = 'http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3'
print(w3lib.url.add_or_replace_parameter(url2, 'arg4', 'v4'))
# Expected output: 'http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3&arg4=v4'
print(w3lib.url.add_or_replace_parameter(url2, 'arg3', 'v3new'))
# Expected output: 'http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3new'
```
### Response
#### Success Response (200)
- **str** (str) - The modified URL with the parameter added or replaced.
```
```APIDOC
## w3lib.url.add_or_replace_parameters
### Description
Adds or replaces multiple parameters in a URL based on a dictionary of key-value pairs. If a parameter already exists, its value is updated; otherwise, it's added.
### Method
N/A (Function)
### Endpoint
N/A (Function)
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Request Example
```python
import w3lib.url
url1 = 'http://www.example.com/index.php'
params1 = {'arg': 'v'}
print(w3lib.url.add_or_replace_parameters(url1, params1))
# Expected output: 'http://www.example.com/index.php?arg=v'
url2 = 'http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3'
params2 = {'arg4': 'v4', 'arg3': 'v3new'}
print(w3lib.url.add_or_replace_parameters(url2, params2))
# Expected output: 'http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3new&arg4=v4'
```
### Response
#### Success Response (200)
- **str** (str) - The modified URL with the parameters added or replaced.
```
```APIDOC
## w3lib.url.any_to_uri
### Description
Converts a given path name into its corresponding File URI. If the input is already a URI, it is returned unmodified.
### Method
N/A (Function)
### Endpoint
N/A (Function)
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Request Example
```python
import w3lib.url
path = '/home/user/document.txt'
print(w3lib.url.any_to_uri(path))
# Expected output: 'file:///home/user/document.txt'
uri = 'http://example.com'
print(w3lib.url.any_to_uri(uri))
# Expected output: 'http://example.com'
```
### Response
#### Success Response (200)
- **str** (str) - The File URI representation of the path, or the original input if it was already a URI.
```
--------------------------------
### w3lib Library Overview
Source: https://github.com/scrapy/w3lib/blob/master/docs/index.md
The w3lib library provides various modules for web-related operations. Note that this library is a Python package and not a REST API; the following documentation outlines the functional interface.
```APIDOC
## Library Interface: w3lib
### Description
w3lib is a Python utility library. It does not expose HTTP endpoints but provides functional modules for web data processing.
### Modules
- **encoding**: Functions for character encoding detection and conversion (e.g., `html_to_unicode`, `resolve_encoding`).
- **html**: Tools for cleaning and parsing HTML (e.g., `remove_tags`, `replace_entities`, `get_base_url`).
- **http**: Utilities for HTTP headers and authentication (e.g., `headers_dict_to_raw`, `basic_auth_header`).
- **url**: Functions for URL sanitization and manipulation (e.g., `canonicalize_url`, `url_query_parameter`).
### Installation
```bash
pip install w3lib
```
### Usage Example
```python
from w3lib.html import remove_tags
html = "Hello World
"
clean_text = remove_tags(html) # Returns: "Hello World"
```
```
--------------------------------
### Generate Basic Auth Header - Python
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Generates an Authorization header value for HTTP Basic Access Authentication according to RFC 2617. It takes a username and password, and returns the header value as bytes.
```python
import w3lib.http
w3lib.http.basic_auth_header('someuser', 'somepass')
```
--------------------------------
### Parse Data URIs with w3lib
Source: https://context7.com/scrapy/w3lib/llms.txt
Parses a data URI string into its constituent parts: media type, parameters, and the decoded binary data.
```python
import w3lib.url
# Parse a text data URI
result = w3lib.url.parse_data_uri('data:text/plain;charset=UTF-8,Hello%20World')
# Parse a base64-encoded image data URI
result = w3lib.url.parse_data_uri('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA')
```
--------------------------------
### Normalize URLs for Safe Downloading
Source: https://context7.com/scrapy/w3lib/llms.txt
Cleans URLs by normalizing paths, resolving directory traversals, and removing URL fragments to ensure safe resource retrieval.
```python
import w3lib.url
# Normalize and clean URL for download
url = w3lib.url.safe_download_url('http://example.com/../path/./to/file.zip#section')
```
--------------------------------
### Parse and Convert HTTP Headers
Source: https://context7.com/scrapy/w3lib/llms.txt
Utilities to convert between raw byte-encoded HTTP headers and Python dictionaries. Handles multiple values for the same header key.
```python
import w3lib.http
raw_headers = b"Content-Type: text/html\r\nAccept: gzip\r\n"
headers = w3lib.http.headers_raw_to_dict(raw_headers)
raw = w3lib.http.headers_dict_to_raw(headers)
```
--------------------------------
### Convert Raw HTTP Headers to Dictionary
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Parses a raw multi-line bytestring of HTTP headers into a dictionary where keys map to lists of values. It handles malformed input by returning an empty dictionary and returns None if the input is None.
```python
import w3lib.http
# Convert raw headers to dict
headers = w3lib.http.headers_raw_to_dict(b"Content-type: text/html\n\rAccept: gzip\n\n")
# Result: {'Content-type': ['text/html'], 'Accept': ['gzip']}
```
--------------------------------
### Detect Byte Order Mark (BOM)
Source: https://context7.com/scrapy/w3lib/llms.txt
Analyzes a byte stream to detect the presence of a Byte Order Mark (BOM) and returns the corresponding encoding and the BOM bytes themselves. Returns None if no BOM is present.
```python
import w3lib.encoding
# Detect UTF-16 Big Endian BOM
encoding, bom = w3lib.encoding.read_bom(b'\xfe\xff\x00H\x00i')
print(f"Encoding: {encoding}, BOM: {bom!r}")
# Detect UTF-8 BOM
encoding, bom = w3lib.encoding.read_bom(b'\xef\xbb\xbfHello')
print(f"Encoding: {encoding}, BOM: {bom!r}")
```
--------------------------------
### Convert Headers Dictionary to Raw - Python
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Converts a dictionary of HTTP headers into a raw bytes format suitable for sending in an HTTP request or response. Keys and values must be bytes. If the input is None, it returns None.
```python
import w3lib.http
w3lib.http.headers_dict_to_raw({b'Content-type': b'text/html', b'Accept': b'gzip'})
w3lib.http.headers_dict_to_raw(None)
```
--------------------------------
### Remove HTML Tags with w3lib
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Demonstrates how to remove HTML tags from a string. Users can either remove all tags, keep specific tags, or remove only a defined subset of tags.
```python
import w3lib.html
doc = ''
# Remove all tags
print(w3lib.html.remove_tags(doc))
# Keep specific tags
print(w3lib.html.remove_tags(doc, keep=('div',)))
# Remove specific tags
print(w3lib.html.remove_tags(doc, which_ones=('a','b')))
```
--------------------------------
### Read Byte Order Mark (BOM) - Python
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Reads the Byte Order Mark (BOM) from a byte string to determine the encoding. Returns the encoding and the BOM if found, otherwise returns (None, None). Handles various UTF encodings like UTF-16 and UTF-32.
```python
import w3lib.encoding
w3lib.encoding.read_bom(b'\xfe\xff\x6c\x34')
# ('utf-16-be', b'\xfe\xff')
w3lib.encoding.read_bom(b'\xff\xfe\x34\x6c')
# ('utf-16-le', b'\xff\xfe')
w3lib.encoding.read_bom(b'\x00\x00\xfe\xff\x00\x00\x6c\x34')
# ('utf-32-be', b'\x00\x00\xfe\xff')
w3lib.encoding.read_bom(b'\xff\xfe\x00\x00\x34\x6c\x00\x00')
# ('utf-32-le', b'\xff\xfe\x00\x00')
w3lib.encoding.read_bom(b'\x01\x02\x03\x04')
# (None, None)
```
--------------------------------
### Manipulate URL Parameters with w3lib
Source: https://context7.com/scrapy/w3lib/llms.txt
Functions to add, replace, or update query parameters in a URL string. These utilities ensure query strings are correctly formatted.
```python
import w3lib.url
# Add to existing parameters
url = w3lib.url.add_or_replace_parameter('http://www.example.com/index.php?arg1=v1&arg2=v2', 'arg3', 'v3')
# Replace existing parameter
url = w3lib.url.add_or_replace_parameter('http://www.example.com/index.php?arg1=v1&arg2=v2', 'arg2', 'new_value')
# Add or replace multiple parameters at once
url = w3lib.url.add_or_replace_parameters('http://www.example.com/?a=1&b=2', {'b': 'new', 'c': '3'})
```
--------------------------------
### Extract Meta Refresh Redirects
Source: https://context7.com/scrapy/w3lib/llms.txt
Parses HTML meta refresh tags to retrieve the redirect URL and delay interval. Returns None if no refresh tag is present.
```python
import w3lib.html
html = ''
interval, url = w3lib.html.get_meta_refresh(html, baseurl='http://example.com/')
print(f"Redirect to {url} after {interval} seconds")
```
--------------------------------
### w3lib.http.headers_raw_to_dict
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Converts a raw, multi-line bytestring of HTTP headers into a Python dictionary. Handles various header formats and returns an empty dictionary for invalid inputs or None.
```APIDOC
## w3lib.http.headers_raw_to_dict
### Description
Converts a single multi-line bytestring of raw HTTP headers into a dictionary. Each header line is parsed into a key-value pair, where the value is a list of strings.
### Method
N/A (Function)
### Endpoint
N/A (Function)
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Request Example
```python
import w3lib.http
raw_headers = b"Content-type: text/html\n\rAccept: gzip\n\n"
headers_dict = w3lib.http.headers_raw_to_dict(raw_headers)
print(headers_dict)
# Expected output: {'Content-type': ['text/html'], 'Accept': ['gzip']}
# Example with invalid input
invalid_headers = b"Content-typt gzip\n\n"
print(w3lib.http.headers_raw_to_dict(invalid_headers))
# Expected output: {}
# Example with None input
print(w3lib.http.headers_raw_to_dict(None))
# Expected output: None
```
### Response
#### Success Response (200)
- **dict** (dict) - A dictionary where keys are header names and values are lists of header values.
```
--------------------------------
### FUNCTION read_bom
Source: https://context7.com/scrapy/w3lib/llms.txt
Detects and returns the byte order mark (BOM) encoding if present in the byte stream.
```APIDOC
## FUNCTION read_bom
### Description
Checks the beginning of a byte stream for a Byte Order Mark (BOM) to determine the encoding.
### Parameters
#### Arguments
- **data** (bytes) - Required - The byte stream to check.
### Response
- **encoding** (string|None) - The detected encoding, or None.
- **bom** (bytes|None) - The detected BOM bytes, or None.
```
--------------------------------
### Convert Raw Headers to Dictionary - Python
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Parses raw HTTP headers (bytes) into a dictionary format. The dictionary keys and values are bytes, and values can be lists of bytes if a header appears multiple times.
```python
import w3lib.http
# Example usage would go here, but no direct code example was provided in the input.
```
--------------------------------
### FUNCTION http_content_type_encoding
Source: https://context7.com/scrapy/w3lib/llms.txt
Extracts the charset parameter from an HTTP Content-Type header string.
```APIDOC
## FUNCTION http_content_type_encoding
### Description
Parses an HTTP Content-Type header string to extract the charset value.
### Parameters
#### Arguments
- **content_type** (string) - Required - The full Content-Type header string.
### Response
- **encoding** (string|None) - The extracted charset, or None if not specified.
```
--------------------------------
### Extract Base URL from HTML
Source: https://context7.com/scrapy/w3lib/llms.txt
Extracts the base URL from an HTML document's tag. If no tag is found, it falls back to a provided default URL.
```python
import w3lib.html
html = 'Link'
base = w3lib.html.get_base_url(html, baseurl='http://example.com/')
print(base)
```
--------------------------------
### HTML Utilities
Source: https://context7.com/scrapy/w3lib/llms.txt
Functions for parsing and manipulating HTML content, including tag replacement, base URL extraction, meta refresh handling, whitespace stripping, and escape character removal.
```APIDOC
## w3lib.html.replace_tags
### Description
Replaces HTML tags in a string with spaces or removes them entirely.
### Method
`w3lib.html.replace_tags(text, replace_with='')`
### Parameters
- **text** (str) - The input HTML string.
- **replace_with** (str, optional) - The string to replace tags with. Defaults to an empty string (removing tags).
### Request Example
```python
import w3lib.html
# Replace tags with spaces
text = w3lib.html.replace_tags('Hello
World
', ' ')
print(text)
# Output: ' Hello World '
# Remove tags (default behavior)
text = w3lib.html.replace_tags('This is formatted text')
print(text)
# Output: 'This is formatted text'
```
## w3lib.html.get_base_url
### Description
Extracts the base URL declared in an HTML document's `` tag, which is useful for resolving relative links.
### Method
`w3lib.html.get_base_url(html, baseurl=None)`
### Parameters
- **html** (str) - The HTML content.
- **baseurl** (str, optional) - A fallback base URL to use if no `` tag is found. Defaults to None.
### Request Example
```python
import w3lib.html
html = '''
Link
'''
base = w3lib.html.get_base_url(html, baseurl='http://example.com/')
print(base)
# Output: 'http://example.com/subdir/'
# Returns provided baseurl if no base tag found
html = 'No base tag'
base = w3lib.html.get_base_url(html, baseurl='http://fallback.com/')
print(base)
# Output: 'http://fallback.com/'
```
## w3lib.html.get_meta_refresh
### Description
Extracts the URL and delay from HTML `` tags.
### Method
`w3lib.html.get_meta_refresh(html, baseurl=None)`
### Parameters
- **html** (str) - The HTML content.
- **baseurl** (str, optional) - The base URL to resolve relative redirect URLs. Defaults to None.
### Request Example
```python
import w3lib.html
html = '''
'''
interval, url = w3lib.html.get_meta_refresh(html, baseurl='http://example.com/')
print(f"Redirect to {url} after {interval} seconds")
# Output: 'Redirect to http://example.com/newpage after 5.0 seconds'
# Handle relative URLs
html = ''
interval, url = w3lib.html.get_meta_refresh(html, baseurl='http://example.com/page/')
print(url)
# Output: 'http://example.com/login'
# Returns (None, None) if no meta refresh found
html = 'No redirect'
interval, url = w3lib.html.get_meta_refresh(html)
print(interval, url)
# Output: None None
```
## w3lib.html.strip_html5_whitespace
### Description
Strips leading and trailing whitespace from a string according to the W3C HTML5 specifications, which includes standard whitespace characters and a few others.
### Method
`w3lib.html.strip_html5_whitespace(value)`
### Parameters
- **value** (str) - The string to strip whitespace from.
### Request Example
```python
from w3lib.html import strip_html5_whitespace
# Clean attribute values
href = strip_html5_whitespace(' http://example.com \n')
print(href)
# Output: 'http://example.com'
# Handles HTML5 space characters (space, tab, newline, carriage return, form feed)
text = strip_html5_whitespace('\t\n\r\x0c content \x0c\r\n\t')
print(text)
# Output: 'content'
```
## w3lib.html.replace_escape_chars
### Description
Removes or replaces common escape characters such as newline (`\n`), tab (`\t`), and carriage return (`\r`) from a string.
### Method
`w3lib.html.replace_escape_chars(text, replace_by='', which_ones=('\n', '\t', '\r'))`
### Parameters
- **text** (str) - The input string.
- **replace_by** (str, optional) - The string to replace the escape characters with. Defaults to an empty string (removal).
- **which_ones** (tuple, optional) - A tuple of escape characters to replace. Defaults to ('\n', '\t', '\r').
### Request Example
```python
import w3lib.html
# Remove default escape characters (\n, \t, \r)
text = w3lib.html.replace_escape_chars("Hello\n\tWorld\r\n")
print(repr(text))
# Output: 'HelloWorld'
# Replace with custom string
text = w3lib.html.replace_escape_chars("Line1\nLine2\nLine3", replace_by=' ')
print(text)
# Output: 'Line1 Line2 Line3'
# Specify which characters to replace
text = w3lib.html.replace_escape_chars("Hello\n\tWorld", which_ones=('\n',))
print(repr(text))
# Output: 'Hello\tWorld'
```
```
--------------------------------
### Add or Replace URL Parameters
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Modifies a URL string by adding a new parameter or updating an existing one. Supports both single parameter updates and bulk updates using a dictionary.
```python
import w3lib.url
# Add single parameter
url1 = w3lib.url.add_or_replace_parameter('http://www.example.com/index.php', 'arg', 'v')
# Add multiple parameters
args = {'arg4': 'v4', 'arg3': 'v3new'}
url2 = w3lib.url.add_or_replace_parameters('http://www.example.com/index.php?arg1=v1&arg2=v2&arg3=v3', args)
```
--------------------------------
### Convert HTML Bytes to Unicode with w3lib
Source: https://context7.com/scrapy/w3lib/llms.txt
Converts raw HTML byte strings into Unicode by automatically detecting the encoding from meta tags or provided headers. This function is critical for processing web pages with unknown or mixed character sets.
```python
import w3lib.encoding
# Auto-detect encoding from meta tags
html_bytes = b'\n\nCaf\xc3\xa9 and \xc3\xa9clair
'
encoding, text = w3lib.encoding.html_to_unicode(None, html_bytes)
print(f"Detected encoding: {encoding}")
print(text)
# Use Content-Type header for encoding
html_bytes = b'Caf\xe9'
encoding, text = w3lib.encoding.html_to_unicode('text/html; charset=ISO-8859-1', html_bytes)
print(f"Encoding: {encoding}, Text: {text}")
```
--------------------------------
### w3lib.html.get_meta_refresh
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Extracts the meta refresh interval and URL from HTML content.
```APIDOC
## FUNCTION w3lib.html.get_meta_refresh
### Description
Parses HTML text to find meta refresh tags and returns the redirect interval and target URL.
### Parameters
- **text** (str|bytes) - Required - The HTML content to parse.
- **baseurl** (str) - Optional - The base URL for resolving relative redirects.
- **encoding** (str) - Optional - The character encoding of the text (default: 'utf-8').
### Response
Returns a tuple (interval, url) where interval is a float and url is a string. Returns (None, None) if no redirect is found.
```
--------------------------------
### Canonicalize URL with w3lib
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Canonicalizes a URL by applying several procedures: making it safe, sorting query arguments, normalizing spaces and percent encodings, and optionally removing blank query values or fragments. The function accepts string, bytes, or ParseResult as input and always returns a string.
```python
import w3lib.url
# sorting query arguments
print(w3lib.url.canonicalize_url('http://www.example.com/do?c=3&b=5&b=2&a=50'))
# UTF-8 conversion + percent-encoding of non-ASCII characters
print(w3lib.url.canonicalize_url('http://www.example.com/r\u00e9sum\u00e9'))
```
--------------------------------
### FUNCTION html_to_unicode
Source: https://context7.com/scrapy/w3lib/llms.txt
Converts raw HTML bytes to a Unicode string by automatically detecting the encoding from meta tags or headers.
```APIDOC
## FUNCTION html_to_unicode
### Description
Converts raw HTML bytes to a Unicode string, automatically detecting the encoding from meta tags or provided Content-Type headers.
### Parameters
#### Arguments
- **content_type** (string) - Optional - The HTTP Content-Type header string.
- **html_bytes** (bytes) - Required - The raw HTML content in bytes.
- **auto_detect_fun** (callable) - Optional - A custom function for encoding detection.
### Response
- **encoding** (string) - The detected encoding name.
- **text** (string) - The decoded Unicode string.
```
--------------------------------
### Parse Data URI with w3lib
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Parses a data: URI into a structured result object, which includes details about the data's media type, encoding, and the data itself. The function accepts either a string or bytes representation of the URI.
```python
import w3lib.url
# Example usage (assuming a data URI exists)
# data_uri = "data:text/plain;charset=US-ASCII;base64,SGVsbG8sIFdvcmxkIQ=="
# parsed_data = w3lib.url.parse_data_uri(data_uri)
# print(parsed_data)
```
--------------------------------
### Replace HTML Tags with Token - Python
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Replaces all markup tags in a given text with a specified token. By default, it removes all tags by using an empty string as the token. It handles both unicode and byte strings, always returning a unicode string.
```python
import w3lib.html
w3lib.html.replace_tags('This text contains some tag')
w3lib.html.replace_tags('Je ne parle pas fran\xe7ais
', ' -- ', 'latin-1')
```
--------------------------------
### Convert HTML Bytes to Unicode
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Converts raw HTML bytes into a Unicode string using html_to_unicode. It follows a browser-like hierarchy of detection including BOM, headers, meta tags, and optional auto-detection functions.
```python
import w3lib.encoding
content_type = None
html_bytes = b""
encoding, unicode_str = w3lib.encoding.html_to_unicode(content_type, html_bytes)
```
--------------------------------
### safe_url_string
Source: https://context7.com/scrapy/w3lib/llms.txt
Converts a URL string into a safe, browser-compatible format by applying proper encoding.
```APIDOC
## safe_url_string
### Description
Converts a URL to a safe format, ensuring it is valid for web browsers and servers, conforming to RFC 3986.
### Parameters
#### Path Parameters
- **url** (string) - Required - The raw URL string to sanitize.
#### Query Parameters
- **quote_path** (boolean) - Optional - Whether to quote the path component. Defaults to True.
### Response
- **url** (string) - The sanitized, percent-encoded URL string.
```
--------------------------------
### Clean URL Query Parameters with url_query_cleaner
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Filters URL query parameters based on a provided list. Supports keeping or removing specific keys, handling duplicates, and managing URL fragments.
```python
from w3lib.url import url_query_cleaner
# Keep only specific parameters
cleaned = url_query_cleaner("product.html?id=200&foo=bar&name=wired", ('id',))
# Remove specific parameters
removed = url_query_cleaner("product.html?id=200&foo=bar&name=wired", ['id'], remove=True)
# Preserve fragments
with_frag = url_query_cleaner('http://domain.tld/?bla=123#123123', ['bla'], remove=True, keep_fragments=True)
```
--------------------------------
### Parse Data URI
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Parses a data URI into its components.
```APIDOC
## Parse Data URI
### Description
Parses a data URI string into its constituent parts, returning a structured result object.
### Method
N/A (This is a Python function, not an HTTP endpoint)
### Endpoint
N/A
### Parameters
- **uri** (str | bytes) - Required - The data URI string to parse.
### Request Example
```python
import w3lib.url
# Example parsing a base64 encoded data URI
data_uri = 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ=='
parsed_data = w3lib.url.parse_data_uri(data_uri)
print(parsed_data)
# Output: ParseDataURIResult(mediatype='text/plain', encoding='base64', data=b'Hello, World!')
```
### Response
#### Success Response (ParseDataURIResult)
- **ParseDataURIResult** - An object containing the parsed data URI components, typically including mediatype, encoding, and data.
```
--------------------------------
### Sanitize URLs with safe_url_string
Source: https://context7.com/scrapy/w3lib/llms.txt
Converts a URL string into a safe format compliant with web standards. It handles spaces, non-ASCII characters, and international domain names.
```python
import w3lib.url
# Basic URL sanitization
url = w3lib.url.safe_url_string("http://example.com/path with spaces")
print(url)
# Handle non-ASCII characters in URLs
url = w3lib.url.safe_url_string("http://example.com/résumé")
print(url)
# Handle international domain names (IDNA)
url = w3lib.url.safe_url_string("http://例え.jp/path")
print(url)
# Disable path quoting when needed
url = w3lib.url.safe_url_string("http://example.com/already%20encoded", quote_path=False)
print(url)
```
--------------------------------
### Filter Query Parameters with url_query_cleaner
Source: https://context7.com/scrapy/w3lib/llms.txt
Filters URL query parameters by keeping or removing specific keys. It also supports handling duplicate parameters and preserving fragments.
```python
import w3lib.url
# Keep only specific parameters
url = w3lib.url.url_query_cleaner("product.html?id=200&foo=bar&name=wired", ('id',))
print(url)
# Keep multiple parameters
url = w3lib.url.url_query_cleaner("product.html?id=200&foo=bar&name=wired", ['id', 'name'])
print(url)
# Remove specific parameters instead
url = w3lib.url.url_query_cleaner("product.html?id=200&foo=bar&name=wired", ['id'], remove=True)
print(url)
# Keep duplicate parameters
url = w3lib.url.url_query_cleaner("product.html?d=1&e=b&d=2&d=3&other=other", ['d'], unique=False)
print(url)
# Preserve URL fragments
url = w3lib.url.url_query_cleaner('http://domain.tld/?bla=123#section', ['bla'], remove=True, keep_fragments=True)
print(url)
```
--------------------------------
### w3lib.encoding.read_bom
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Detects and returns the byte order mark (BOM) and its corresponding encoding from a byte string.
```APIDOC
## FUNCTION w3lib.encoding.read_bom
### Description
Reads the byte order mark in the provided data, if present, and returns the encoding represented by the BOM and the BOM itself.
### Parameters
#### Arguments
- **data** (bytes) - Required - The byte string to inspect for a BOM.
### Response
Returns a tuple (encoding, bom). If no BOM is detected, returns (None, None).
### Example
```python
import w3lib.encoding
w3lib.encoding.read_bom(b'\xfe\xff\x6c\x34')
# Returns: ('utf-16-be', b'\xfe\xff')
```
```
--------------------------------
### Normalize URLs with canonicalize_url
Source: https://context7.com/scrapy/w3lib/llms.txt
Canonicalizes URLs by sorting query arguments, normalizing encoding, and managing fragments. This is useful for deduplication and comparison tasks.
```python
import w3lib.url
# Sort query arguments alphabetically
url = w3lib.url.canonicalize_url('http://www.example.com/do?c=3&b=5&b=2&a=50')
print(url)
# Normalize percent-encoding and non-ASCII characters
url = w3lib.url.canonicalize_url('http://www.example.com/résumé')
print(url)
# Keep fragments when needed
url = w3lib.url.canonicalize_url('http://example.com/page?b=2&a=1#section', keep_fragments=True)
print(url)
# Remove blank query values
url = w3lib.url.canonicalize_url('http://example.com/?a=1&b=&c=3', keep_blank_values=False)
print(url)
```
--------------------------------
### FUNCTION resolve_encoding
Source: https://context7.com/scrapy/w3lib/llms.txt
Resolves various encoding aliases to their canonical, browser-compatible names.
```APIDOC
## FUNCTION resolve_encoding
### Description
Normalizes encoding names and aliases to their canonical forms, applying common browser-compatible translations.
### Parameters
#### Arguments
- **encoding** (string) - Required - The encoding name to resolve.
### Response
- **canonical_name** (string|None) - The resolved canonical encoding name, or None if invalid.
```
--------------------------------
### url_query_cleaner
Source: https://context7.com/scrapy/w3lib/llms.txt
Filters URL query parameters based on a whitelist or blacklist.
```APIDOC
## url_query_cleaner
### Description
Filters URL query parameters, keeping only specified ones or removing specified ones.
### Parameters
#### Query Parameters
- **url** (string) - Required - The URL to clean.
- **parameters** (list) - Required - List of parameters to filter.
- **remove** (boolean) - Optional - If True, removes the specified parameters instead of keeping them.
- **unique** (boolean) - Optional - Whether to keep only unique parameters. Defaults to True.
- **keep_fragments** (boolean) - Optional - Whether to preserve fragments. Defaults to False.
### Response
- **url** (string) - The cleaned URL.
```
--------------------------------
### Convert to Unicode - Python
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Converts a byte string to a Unicode string using the specified encoding. Characters that cannot be decoded are replaced with the Unicode replacement character ('\ufffd').
```python
import w3lib.encoding
# Example usage (assuming you have bytes data and an encoding)
data_bytes = b'\xe9' # Example byte for 'é' in latin1
encoding = 'latin1'
unicode_string = w3lib.encoding.to_unicode(data_bytes, encoding)
# unicode_string will be 'é'
```
--------------------------------
### w3lib.url.safe_url_string
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Ensures a URL is safe for web browsers and servers by percent-encoding characters according to various URL standards.
```APIDOC
## safe_url_string
### Description
Returns a URL that is considered valid by a wide range of web browsers and web servers. It parses the URL according to the URL living standard and percent-encodes additional characters to comply with various URL standards including RFC 3986, RFC 2396, RFC 2732, and Java 8's java.net.URI class interpretation. If `quote_path` is True, the path component is encoded and quoted using the specified `path_encoding`. Otherwise, the path component is not encoded or quoted. The `encoding` parameter is used for the query string or form data.
### Method
N/A (Python function)
### Endpoint
N/A (Python function)
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Request Example
```python
import w3lib.url
# Example usage
safe_url = w3lib.url.safe_url_string("http://example.com/path with spaces?query=value")
print(safe_url)
```
### Response
#### Success Response
- **str**: A URL string that is safe for web transmission.
#### Response Example
```json
{
"example": "http://example.com/path%20with%20spaces?query=value"
}
```
```
--------------------------------
### Sanitize URLs with safe_url_string
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Ensures a URL string is compliant with multiple standards including RFC 3986 and the URL living standard. It handles character encoding and percent-encoding for paths.
```python
from w3lib.url import safe_url_string
# Basic usage
safe_url = safe_url_string("http://example.com/path with spaces")
# Output: 'http://example.com/path%20with%20spaces'
```
--------------------------------
### Remove HTML Tags and Content
Source: https://github.com/scrapy/w3lib/blob/master/docs/w3lib.md
Removes specified HTML tags along with the content contained within them. If no tags are specified, the original string is returned.
```python
import w3lib.html
doc = ''
print(w3lib.html.remove_tags_with_content(doc, which_ones=('b',)))
```
--------------------------------
### Convert HTML Entities
Source: https://context7.com/scrapy/w3lib/llms.txt
Converts named and numeric HTML entities into their corresponding Unicode characters.
```python
import w3lib.html
# Convert named entities
text = w3lib.html.replace_entities('Price: £100 & €50')
# Convert numeric entities
text = w3lib.html.replace_entities('© 2024 — All rights reserved')
```