Webencodings

https://pythonhosted.org/webencodings/
Admin
Webencodings is a Python implementation of the WHATWG Encoding standard, providing tools for...
Tokens:4,549
Snippets:33
Trust Score:9.5
Update:3 months ago
Show doc for...
Tokens:
Raw
### Detect BOM in byte strings

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Identifies the encoding based on the Byte Order Mark (BOM) at the start of the input and returns the encoding along with the remaining data.

```python
def _detect_bom(input):
    """Return (bom_encoding, input), with any BOM removed from the input."""
    if input.startswith(b'\xFF\xFE'):
        return _UTF16LE, input[2:]
    if input.startswith(b'\xFE\xFF'):
        return _UTF16BE, input[2:]
    if input.startswith(b'\xEF\xBB\xBF'):
        return UTF8, input[3:]
    return None, input
```

--------------------------------

### IncrementalEncoder.__init__(encoding='utf-8', errors='strict')

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Initializes a new IncrementalEncoder instance with a specified encoding and error handling policy.

```APIDOC
## IncrementalEncoder.__init__

### Description
Initializes the encoder with a specific encoding and error handling strategy.

### Parameters
- **encoding** (Encoding object or string) - Optional - The encoding to use (defaults to UTF8).
- **errors** (string) - Optional - Error handling scheme (see codecs.register).

### Raises
- **LookupError** - Raised if the provided encoding label is unknown.
```

--------------------------------

### Lookup Encoding by Label

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Retrieves an Encoding object based on a provided label string, following the WHATWG get-an-encoding algorithm.

```python
def lookup(label):
    """
    Look for an encoding by its label.
    This is the spec’s `get an encoding
    <http://encoding.spec.whatwg.org/#concept-encoding-get>`_ algorithm.
    Supported labels are listed there.

    :param label: A string.
    :returns:
        An :class:`Encoding` object, or :obj:`None` for an unknown label.

    """
    # Only strip ASCII whitespace: U+0009, U+000A, U+000C, U+000D, and U+0020.
    label = ascii_lower(label.strip('\t\n\f\r '))
    name = LABELS.get(label)
    if name is None:
        return None
    encoding = CACHE.get(name)
    if encoding is None:
        if name == 'x-user-defined':
            from .x_user_defined import codec_info
        else:
            python_name = PYTHON_NAMES.get(name, name)
            # Any python_name value that gets to here should be valid.
            codec_info = codecs.lookup(python_name)
        encoding = Encoding(name, codec_info)
        CACHE[name] = encoding
    return encoding
```

--------------------------------

### decode(input, fallback_encoding=None)

Source: https://pythonhosted.org/webencodings/_sources/index.rst.txt

Decode a byte string using the specified encoding or a fallback.

```APIDOC
## decode(input, fallback_encoding=None)

### Description
Decode a byte string. The default error handling is 'replace'.
```

--------------------------------

### webencodings.lookup(label)

Source: https://pythonhosted.org/webencodings/index.html

Look for an encoding by its label. Returns an Encoding object or None if the label is unknown.

```APIDOC
## webencodings.lookup(label)

### Description
Look for an encoding by its label. This is the spec’s get an encoding algorithm.

### Parameters
- **label** (string) - Required - A string representing the encoding label.

### Returns
- **Encoding** (object) - An Encoding object, or None for an unknown label.
```

--------------------------------

### Incremental Decoding Logic

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Handles incremental decoding by checking for a BOM and managing the input buffer.

```python
        input = self._buffer + input
        encoding, input = _detect_bom(input)
        if encoding is None:
            if len(input) < 3 and not final:  # Not enough data yet.
                self._buffer = input
                return ''
            else:  # No BOM
                encoding = self._fallback_encoding
        decoder = encoding.codec_info.incrementaldecoder(self._errors).decode
        self._decoder = decoder
        self.encoding = encoding
        return decoder(input, final)
```

--------------------------------

### IncrementalDecoder(fallback_encoding, errors='replace')

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

A class for push-based decoding of byte strings into Unicode strings.

```APIDOC
## IncrementalDecoder.decode(input, final=False)

### Description
Decodes a chunk of input bytes into a Unicode string.

### Parameters
- **input** (bytes) - Required - A byte string chunk.
- **final** (boolean) - Optional - Indicates if no more input is available.

### Returns
- **string** - The decoded Unicode string.
```

--------------------------------

### webencodings.iter_decode(input, fallback_encoding, errors='replace')

Source: https://pythonhosted.org/webencodings/index.html

Pull-based decoder for an iterable of byte strings.

```APIDOC
## webencodings.iter_decode(input, fallback_encoding, errors='replace')

### Description
Pull-based decoder that processes an iterable of byte strings.

### Parameters
- **input** (iterable) - Required - An iterable of byte strings.
- **fallback_encoding** (Encoding object or string) - Required - The encoding to use if input does not have a BOM.
- **errors** (string) - Optional - Type of error handling (default: 'replace').

### Returns
- **tuple** (iterable, Encoding) - An iterable of Unicode strings and the Encoding used.
```

--------------------------------

### decode(input, fallback_encoding, errors='replace')

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Decodes a byte string using a fallback encoding if no Byte Order Mark (BOM) is detected.

```APIDOC
## decode(input, fallback_encoding, errors='replace')

### Description
Decode a single byte string. If the input does not have a BOM, the provided fallback_encoding is used.

### Parameters
- **input** (byte string) - Required - The byte string to decode.
- **fallback_encoding** (Encoding object or string) - Required - The encoding to use if no BOM is found.
- **errors** (string) - Optional - Type of error handling (default: 'replace').

### Returns
- **tuple** (output, encoding) - A tuple containing the decoded Unicode string and the Encoding object used.
```

--------------------------------

### Compare ascii_lower with standard string lower

Source: https://pythonhosted.org/webencodings/index.html

Demonstrates how ascii_lower preserves non-ASCII characters compared to the standard string lower() method.

```python
>>> keyword = u'Bac\N{KELVIN SIGN}ground'
>>> assert keyword.lower() == u'background'
>>> assert ascii_lower(keyword) != keyword.lower()
>>> assert ascii_lower(keyword) == u'bac\N{KELVIN SIGN}ground'
```

--------------------------------

### iter_decode(input, fallback_encoding=None)

Source: https://pythonhosted.org/webencodings/_sources/index.rst.txt

Incrementally decode a byte string.

```APIDOC
## iter_decode(input, fallback_encoding=None)

### Description
Incrementally decode a byte string, yielding decoded chunks.
```

--------------------------------

### encode(input, encoding=UTF8)

Source: https://pythonhosted.org/webencodings/_sources/index.rst.txt

Encode a string into a byte string.

```APIDOC
## encode(input, encoding=UTF8)

### Description
Encode a string into a byte string using the specified encoding.
```

--------------------------------

### iter_encode(input, encoding=UTF8)

Source: https://pythonhosted.org/webencodings/_sources/index.rst.txt

Incrementally encode a string.

```APIDOC
## iter_encode(input, encoding=UTF8)

### Description
Incrementally encode a string, yielding encoded chunks.
```

--------------------------------

### Incremental decoding class

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Provides a push-based interface for decoding byte streams incrementally.

```python
class IncrementalDecoder(object):
    """
    “Push”-based decoder.

    :param fallback_encoding:
        An :class:`Encoding` object or a label string.
        The encoding to use if :obj:`input` does note have a BOM.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.

    """
    def __init__(self, fallback_encoding, errors='replace'):
        # Fail early if `encoding` is an invalid label.
        self._fallback_encoding = _get_encoding(fallback_encoding)
        self._errors = errors
        self._buffer = b''
        self._decoder = None
        #: The actual :class:`Encoding` that is being used,
        #: or :obj:`None` if that is not determined yet.
        #: (Ie. if there is not enough input yet to determine
        #: if there is a BOM.)
        self.encoding = None  # Not known yet.

    def decode(self, input, final=False):
        """Decode one chunk of the input.

        :param input: A byte string.
        :param final:
            Indicate that no more input is available.
            Must be :obj:`True` if this is the last call.
        :returns: An Unicode string.

        """
        decoder = self._decoder
        if decoder is not None:
            return decoder(input, final)
```

--------------------------------

### lookup(name)

Source: https://pythonhosted.org/webencodings/_sources/index.rst.txt

Look up an encoding by its name.

```APIDOC
## lookup(name)

### Description
Look up an encoding by its name. Returns an Encoding object or None if not found.
```

--------------------------------

### encode(input, encoding=UTF8, errors='strict')

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Encodes a single Unicode string into a byte string using the specified encoding.

```APIDOC
## encode(input, encoding=UTF8, errors='strict')

### Description
Encodes a single Unicode string into a byte string.

### Parameters
- **input** (string) - Required - An Unicode string.
- **encoding** (Encoding/string) - Optional - An Encoding object or a label string.
- **errors** (string) - Optional - Type of error handling (see codecs.register).

### Returns
- **byte string** - The encoded byte string.

### Raises
- **LookupError** - If the encoding label is unknown.
```

--------------------------------

### webencodings.iter_encode(input, encoding=UTF8, errors='strict')

Source: https://pythonhosted.org/webencodings/index.html

Pull-based encoder for an iterable of Unicode strings.

```APIDOC
## webencodings.iter_encode(input, encoding=UTF8, errors='strict')

### Description
Pull-based encoder that processes an iterable of Unicode strings.

### Parameters
- **input** (iterable) - Required - An iterable of Unicode strings.
- **encoding** (Encoding object or string) - Optional - The encoding to use (default: UTF8).
- **errors** (string) - Optional - Type of error handling (default: 'strict').

### Returns
- **iterable** - An iterable of byte strings.
```

--------------------------------

### webencodings.decode(input, fallback_encoding, errors='replace')

Source: https://pythonhosted.org/webencodings/index.html

Decode a single byte string into a Unicode string.

```APIDOC
## webencodings.decode(input, fallback_encoding, errors='replace')

### Description
Decode a single byte string. Returns a tuple of the decoded Unicode string and the Encoding object used.

### Parameters
- **input** (byte string) - Required - The byte string to decode.
- **fallback_encoding** (Encoding object or string) - Required - The encoding to use if input does not have a BOM.
- **errors** (string) - Optional - Type of error handling (default: 'replace').

### Returns
- **tuple** (Unicode string, Encoding) - A tuple containing the output string and the encoding used.
```

--------------------------------

### IncrementalEncoder.encode(input, final=False)

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Encodes a Unicode string into a byte string, supporting incremental processing.

```APIDOC
## IncrementalEncoder.encode

### Description
Encodes the provided Unicode string input into a byte string.

### Parameters
- **input** (string) - Required - The Unicode string to encode.
- **final** (boolean) - Optional - Indicates if this is the final call and no more input is available (defaults to False).

### Returns
- **byte string** - The encoded result.
```

--------------------------------

### webencodings.ascii_lower(string)

Source: https://pythonhosted.org/webencodings/index.html

Transform ASCII letters in a string to lower case.

```APIDOC
## webencodings.ascii_lower(string)

### Description
Transform only ASCII letters to lower case (A-Z to a-z). Used for ASCII case-insensitive matching of encoding labels.

### Parameters
- **string** (Unicode string) - Required - The string to transform.

### Returns
- **string** - A new Unicode string with ASCII characters lowercased.
```

--------------------------------

### webencodings.encode(input, encoding=UTF8, errors='strict')

Source: https://pythonhosted.org/webencodings/index.html

Encode a single Unicode string into a byte string.

```APIDOC
## webencodings.encode(input, encoding=UTF8, errors='strict')

### Description
Encode a single Unicode string into a byte string.

### Parameters
- **input** (Unicode string) - Required - The string to encode.
- **encoding** (Encoding object or string) - Optional - The encoding to use (default: UTF8).
- **errors** (string) - Optional - Type of error handling (default: 'strict').

### Returns
- **byte string** - The encoded byte string.
```

--------------------------------

### Decode Byte String

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Decodes a byte string using a fallback encoding, automatically detecting BOM if present.

```python
def decode(input, fallback_encoding, errors='replace'):
    """
    Decode a single string.

    :param input: A byte string
    :param fallback_encoding:
        An :class:`Encoding` object or a label string.
        The encoding to use if :obj:`input` does note have a BOM.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
    :return:
        A ``(output, encoding)`` tuple of an Unicode string
        and an :obj:`Encoding`.

    """
    # Fail early if `encoding` is an invalid label.
    fallback_encoding = _get_encoding(fallback_encoding)
    bom_encoding, input = _detect_bom(input)
    encoding = bom_encoding or fallback_encoding
    return encoding.codec_info.decode(input, errors)[0], encoding
```

--------------------------------

### Pull-based stream encoding

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Encodes an iterable of Unicode strings into an iterable of byte strings.

```python
def iter_encode(input, encoding=UTF8, errors='strict'):
    """
    “Pull”-based encoder.

    :param input: An iterable of Unicode strings.
    :param encoding: An :class:`Encoding` object or a label string.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
    :returns: An iterable of byte strings.

    """
    # Fail early if `encoding` is an invalid label.
    encode = IncrementalEncoder(encoding, errors).encode
    return _iter_encode_generator(input, encode)
```

```python
def _iter_encode_generator(input, encode):
    for chunck in input:
        output = encode(chunck)
        if output:
            yield output
    output = encode('', final=True)
    if output:
        yield output
```

--------------------------------

### Define Encoding Class

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Represents a character encoding with its canonical name and codec information.

```python
class Encoding(object):
    """Reresents a character encoding such as UTF-8,
    that can be used for decoding or encoding.

    .. attribute:: name

        Canonical name of the encoding

    .. attribute:: codec_info

        The actual implementation of the encoding,
        a stdlib :class:`~codecs.CodecInfo` object.
        See :func:`codecs.register`.

    """
    def __init__(self, name, codec_info):
        self.name = name
        self.codec_info = codec_info

    def __repr__(self):
        return '<Encoding %s>' % self.name
```

--------------------------------

### Pull-based stream decoding

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Decodes an iterable of byte strings, determining the encoding from a BOM or falling back to a provided default.

```python
def iter_decode(input, fallback_encoding, errors='replace'):
    """
    "Pull"-based decoder.

    :param input:
        An iterable of byte strings.

        The input is first consumed just enough to determine the encoding
        based on the precense of a BOM,
        then consumed on demand when the return value is.
    :param fallback_encoding:
        An :class:`Encoding` object or a label string.
        The encoding to use if :obj:`input` does note have a BOM.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
    :returns:
        An ``(output, encoding)`` tuple.
        :obj:`output` is an iterable of Unicode strings,
        :obj:`encoding` is the :obj:`Encoding` that is being used.

    """

    decoder = IncrementalDecoder(fallback_encoding, errors)
    generator = _iter_decode_generator(input, decoder)
    encoding = next(generator)
    return generator, encoding
```

```python
def _iter_decode_generator(input, decoder):
    """Return a generator that first yields the :obj:`Encoding`,
    then yields output chukns as Unicode strings.

    """
    decode = decoder.decode
    input = iter(input)
    for chunck in input:
        output = decode(chunck)
        if output:
            assert decoder.encoding is not None
            yield decoder.encoding
            yield output
            break
    else:
        # Input exhausted without determining the encoding
        output = decode(b'', final=True)
        assert decoder.encoding is not None
        yield decoder.encoding
        if output:
            yield output
        return

    for chunck in input:
        output = decode(chunck)
        if output:
            yield output
    output = decode(b'', final=True)
    if output:
        yield output
```

--------------------------------

### IncrementalEncoder Class Definition

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Defines the IncrementalEncoder class for push-based encoding with configurable error handling.

```python
[docs]class IncrementalEncoder(object):
    """
    “Push”-based encoder.

    :param encoding: An :class:`Encoding` object or a label string.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.

    .. method:: encode(input, final=False)

        :param input: An Unicode string.
        :param final:
            Indicate that no more input is available.
            Must be :obj:`True` if this is the last call.
        :returns: A byte string.

    """
    def __init__(self, encoding=UTF8, errors='strict'):
        encoding = _get_encoding(encoding)
        self.encode = encoding.codec_info.incrementalencoder(errors).encode
```

--------------------------------

### Transform ASCII to Lowercase

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Converts ASCII letters to lowercase while preserving non-ASCII characters, suitable for encoding label matching.

```python
def ascii_lower(string):
    r"""Transform (only) ASCII letters to lower case: A-Z is mapped to a-z.

    :param string: An Unicode string.
    :returns: A new Unicode string.

    This is used for `ASCII case-insensitive
    <http://encoding.spec.whatwg.org/#ascii-case-insensitive>`_
    matching of encoding labels.
    The same matching is also used, among other things,
    for `CSS keywords <http://dev.w3.org/csswg/css-values/#keywords>`_.

    This is different from the :meth:`~py:str.lower` method of Unicode strings
    which also affect non-ASCII characters,
    sometimes mapping them into the ASCII range:

        >>> keyword = u'Bac\N{KELVIN SIGN}ground'
        >>> assert keyword.lower() == u'background'
        >>> assert ascii_lower(keyword) != keyword.lower()
        >>> assert ascii_lower(keyword) == u'bac\N{KELVIN SIGN}ground'

    """
    # This turns out to be faster than unicode.translate()
    return string.encode('utf8').lower().decode('utf8')
```

--------------------------------

### Encode Unicode strings

Source: https://pythonhosted.org/webencodings/_modules/webencodings.html

Encodes a single Unicode string using the specified encoding and error handling strategy.

```python
def encode(input, encoding=UTF8, errors='strict'):
    """
    Encode a single string.

    :param input: An Unicode string.
    :param encoding: An :class:`Encoding` object or a label string.
    :param errors: Type of error handling. See :func:`codecs.register`.
    :raises: :exc:`~exceptions.LookupError` for an unknown encoding label.
    :return: A byte string.

    """
    return _get_encoding(encoding).codec_info.encode(input, errors)[0]
```
Tokens:
Raw