htmldate

https://htmldate.readthedocs.io/en/latest/
Admin
htmldate is a Python package that extracts and identifies original and updated publication dates...
Tokens:28,905
Snippets:201
Trust Score:9.6
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install optional speed dependencies

Source: https://htmldate.readthedocs.io/en/latest/index.html

Command to install htmldate with additional speed-enhancing libraries.

```bash
$ pip install htmldate[speed] # install with additional functionality
```

--------------------------------

### Install htmldate with additional components

Source: https://htmldate.readthedocs.io/en/latest/evaluation.html

Use this command to install the package with all optional dependencies for improved performance and coverage.

```bash
pip/pip3/pipenv htmldate[all]
```

--------------------------------

### Install htmldate package

Source: https://htmldate.readthedocs.io/en/latest/index.html

Commands for installing the htmldate package via pip, including options for upgrading and installing from source.

```bash
$ pip install htmldate # pip3 install on systems where both Python 2 and 3 are installed
$ pip install --upgrade htmldate # to make sure you have the latest version
$ pip install git+https://github.com/adbar/htmldate.git # latest available code (see build status above)
```

--------------------------------

### Install Htmldate with Optional Speed Enhancements

Source: https://htmldate.readthedocs.io/en/latest/_sources/index.rst.txt

Install Htmldate along with optional libraries like `cchardet` for improved execution speed. This command installs the `[speed]` extra, which includes packages that enhance performance but may not be universally compatible.

```bash
pip install htmldate[speed]
```

--------------------------------

### Run library tests

Source: https://htmldate.readthedocs.io/en/latest/options.html

Install pytest and execute the test suite for the library.

```bash
$ python3 -m pip install pytest
$ pytest
```

--------------------------------

### Install Htmldate with Pip

Source: https://htmldate.readthedocs.io/en/latest/_sources/index.rst.txt

Install the main Htmldate package using pip. This command ensures you have the latest stable version compatible with Python 3.8 and upwards. For systems with both Python 2 and 3, use `pip3`.

```bash
pip install htmldate
```

```bash
pip install --upgrade htmldate
```

```bash
pip install git+https://github.com/adbar/htmldate.git
```

--------------------------------

### Handling Missing Dates

Source: https://htmldate.readthedocs.io/en/latest/index.html

Example behavior when no date can be extracted from a web page.

```python
>>> r = requests.get('https://example.com')
>>> htmldate.find_date(r.text)
>>>
```

--------------------------------

### Extract publication date via Python and CLI

Source: https://htmldate.readthedocs.io/en/latest/index.html

Basic usage examples for finding a publication date from a URL using the Python library and the command-line interface.

```python
>>> from htmldate import find_date
>>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')
'2016-12-23'
```

```bash
$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html
'2016-12-23'
```

--------------------------------

### Compile Regex for YYYY Start Pattern

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/extractors.html

Compiles a regular expression to match a year at the beginning of a string, followed by non-digit characters.

```python
YMD_YEAR = re.compile(rf"^({YEAR_RE})")
```

--------------------------------

### Find Publication Date on Command-Line

Source: https://htmldate.readthedocs.io/en/latest/_sources/index.rst.txt

Execute the `htmldate` command with the `-u` flag followed by the URL to retrieve the publication date directly from the terminal. This is a convenient way to quickly get the date for a single web page.

```bash
$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html
'2016-12-23'
```

--------------------------------

### Get Minimum Date

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/validators.html

Validates the provided minimum date input, defaulting to the earliest plausible date if the input is invalid or not provided. Uses `check_date_input` for validation.

```python
def get_min_date(min_date: Optional[Union[datetime, str]]) -> datetime:
    """Validates the minimum date and/or defaults to earliest plausible date"""
    return check_date_input(min_date, MIN_DATE)
```

--------------------------------

### CLI Help Documentation

Source: https://htmldate.readthedocs.io/en/latest/index.html

Display available command-line arguments and options.

```bash
$ htmldate --help
htmldate [-h] [-f] [-i INPUTFILE] [--original] [-min MINDATE] [-max MAXDATE] [-u URL] [-v] [--version]
optional arguments:
    -h, --help            show this help message and exit
    -f, --fast            fast mode: disable extensive search
    -i INPUTFILE, --inputfile INPUTFILE
                          name of input file for batch processing (similar to wget -i)
    --original            original date prioritized
    -min MINDATE, --mindate MINDATE
                          earliest acceptable date (ISO 8601 YMD)
    -max MAXDATE, --maxdate MAXDATE
                          latest acceptable date (ISO 8601 YMD)
    -u URL, --URL URL     custom URL download
    -v, --verbose         increase output verbosity
    --version             show version information and exit
```

--------------------------------

### Basic Usage and Input Formats

Source: https://htmldate.readthedocs.io/en/latest/_sources/options.rst.txt

Demonstrates how to use the `find_date` function with different input types, including HTML strings, LXML trees, and URLs.

```APIDOC
## Basic Usage and Input Formats

### Description
This section covers the fundamental usage of the `find_date` function, illustrating how to process HTML content provided as strings, parsed LXML tree objects, or directly via URLs.

### Method
`find_date(html_content, **kwargs)`

### Endpoint
N/A (Python library function)

### Parameters
- **html_content** (str | lxml.html.HtmlElement) - Required - The HTML content or an LXML tree object to parse.
- **outputformat** (str) - Optional - The desired output format for the date (e.g., `"%Y-%m-%d"`, `"%d %B %Y"`). Defaults to `"%Y-%m-%d"`.
- **original_date** (bool) - Optional - If True, prioritizes the original publication date over the last modified date. Defaults to False.

### Request Example
```python
from htmldate import find_date
from lxml import html

# Using an HTML string
print(find_date('<html><body><span class="entry-date">July 12th, 2016</span></body></html>'))

# Using an LXML tree object
mytree = html.fromstring('<html><body><span class="entry-date">July 12th, 2016</span></body></html>')
print(find_date(mytree))

# Using a URL
print(find_date('https://creativecommons.org/about/'))
```

### Response
#### Success Response (200)
- **date_string** (str) - The extracted date in the specified or default format.

#### Response Example
```json
{
  "date_string": "2016-07-12"
}
```
```

--------------------------------

### Compile htmldate with mypyc

Source: https://htmldate.readthedocs.io/en/latest/_sources/index.rst.txt

Experimental compilation steps to improve processing speed using mypyc.

```bash
pip3 install mypy
python setup.py --use-mypyc bdist_wheel
pip3 install dist/...
```

--------------------------------

### Get Maximum Date

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/validators.html

Validates the provided maximum date input, defaulting to the current date and time if the input is invalid or not provided. Uses `check_date_input` for validation.

```python
def get_max_date(max_date: Optional[Union[datetime, str]]) -> datetime:
    """Validates the maximum date and/or defaults to latest plausible date"""
    return check_date_input(max_date, datetime.now())
```

--------------------------------

### Fetch and extract dates from URLs

Source: https://htmldate.readthedocs.io/en/latest/options.html

Demonstrates using external libraries like requests, internal fetch_url utilities, or direct URL strings to extract dates.

```python
>>> from htmldate.core import find_date
# using requests
>>> import requests
>>> r = requests.get('https://creativecommons.org/about/')
>>> find_date(r.text)
'2017-11-28' # may have changed since
# using htmldate's own fetch_url function
>>> from htmldate.utils import fetch_url
>>> htmldoc = fetch_url('https://blog.wikimedia.org/2018/06/28/interactive-maps-now-in-your-language/')
>>> find_date(htmldoc)
'2018-06-28'
# or simply
>>> find_date('https://blog.wikimedia.org/2018/06/28/interactive-maps-now-in-your-language/') # URL detected
'2018-06-28'
```

--------------------------------

### Initialize External Date Parser

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/extractors.html

Configures the DateDataParser with specific settings for normalization and preferred date formats.

```python
EXTERNAL_PARSER = DateDataParser(
    languages=None,
    locales=None,
    region=None,
    settings={
        "NORMALIZE": True,  # False may be faster
        "PARSERS": [
            "custom-formats",
            "absolute-time",
        ],
        "PREFER_DATES_FROM": "past",
        "PREFER_LOCALE_DATE_ORDER": True,
        "RETURN_AS_TIMEZONE_AWARE": False,
        "STRICT_PARSING": True,
    },
)
```

--------------------------------

### Utility Functions

Source: https://htmldate.readthedocs.io/en/latest/genindex.html

Utility functions for fetching URLs and loading HTML.

```APIDOC
## fetch_url()

### Description
Fetches content from a given URL.

### Method
(Assumed to be a function call, not an HTTP endpoint)

### Endpoint
N/A

### Parameters
(No specific parameters listed in the provided text for this function)

### Request Example
N/A

### Response
(No specific return value described in the provided text)

## load_html()

### Description
Loads HTML content from a source.

### Method
(Assumed to be a function call, not an HTTP endpoint)

### Endpoint
N/A

### Parameters
(No specific parameters listed in the provided text for this function)

### Request Example
N/A

### Response
(No specific return value described in the provided text)
```

--------------------------------

### Batch Processing via CLI

Source: https://htmldate.readthedocs.io/en/latest/index.html

Process a list of URLs from a file using the -i flag.

```bash
$ htmldate --fast -i list-of-urls.txt
```

--------------------------------

### Customizing Output Format

Source: https://htmldate.readthedocs.io/en/latest/_sources/options.rst.txt

Explains how to change the date output format using the `outputformat` parameter.

```APIDOC
## Customizing Output Format

### Description
This section details how to modify the output date format using the `outputformat` parameter in the `find_date` function. The format string should be compatible with Python's `datetime` module.

### Method
`find_date(html_content, outputformat=...)`

### Endpoint
N/A (Python library function)

### Parameters
- **outputformat** (str) - Required - The desired output format for the date string. Examples include `"%Y-%m-%d"`, `"%d %B %Y"`, `"%Y-%m-%dT%H:%M:%S%z"`.

### Request Example
```python
from htmldate import find_date

# Custom format: Day Month Year
print(find_date('https://www.gnu.org/licenses/gpl-3.0.en.html', outputformat='%d %B %Y'))

# Custom format: ISO 8601 with timezone
print(find_date('http://blog.python.org/2016/12/python-360-is-now-available.html', outputformat='%Y-%m-%dT%H:%M:%S%z'))
```

### Response
#### Success Response (200)
- **date_string** (str) - The extracted date formatted according to the `outputformat` parameter.

#### Response Example
```json
{
  "date_string": "18 November 2016"
}
```
```

--------------------------------

### is_valid_format(outputformat)

Source: https://htmldate.readthedocs.io/en/latest/options.html

Validates if the provided output format string is compatible with the library settings.

```APIDOC
## is_valid_format

### Description
Validates the output format string against the library's internal settings.

### Parameters
#### Request Body
- **outputformat** (str) - Required - The format string to validate.

### Response
#### Success Response (200)
- **result** (bool) - Returns True if the format is valid, False otherwise.
```

--------------------------------

### Fetch URL Content

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/utils.html

Downloads a page using urllib3 and returns the decoded HTML content.

```python
[docs]
def fetch_url(url: str) -> Optional[str]:
    """Fetches page using urllib3 and decodes the response.

    Args:
        url: URL of the page to fetch.

    Returns:
        HTML code as string, or Urllib3 response object (headers + body), or empty string in case
        the result is invalid, or None if there was a problem with the network.

    """
    # send
    try:
        # read by streaming chunks (stream=True, iter_content=xx)
        # so we can stop downloading as soon as MAX_FILE_SIZE is reached
        response = HTTP_POOL.request("GET", url, timeout=30)  # type: ignore
    except Exception as err:
        LOGGER.error("download error: %s %s", url, err)  # sys.exc_info()[0]
    else:
        # safety checks
        if response.status != 200:
            LOGGER.error("not a 200 response: %s for URL %s", response.status, url)
        elif is_wrong_document(response.data):
            LOGGER.error("incorrect input data for URL %s", url)
        else:
            return decode_response(response.data)
    return None
```

--------------------------------

### Examine HTML Header for Date Cues

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Parses header elements to find date cues using an LXML parsed tree and extraction options.

```python
def examine_header(
    tree: HtmlElement,
    options: Extractor,
) -> Optional[str]:
    """
    Parse header elements to find date cues

    :param tree:
        LXML parsed tree object
    :type tree: LXML tree
    :param options:
        Options for extraction
    :type options: Extractor
    :return: Returns a valid date expression as a string, or None

    """
    headerdate, reserve = None, None
    tryfunc = partial(
        try_date_expr,
        outputformat=options.format,
        extensive_search=options.extensive,
        min_date=options.min,
        max_date=options.max,
    )
    # loop through all meta elements
    for elem in tree.iterfind(".//meta"):
        # safeguard
        if (
            not elem.attrib
            or "content" not in elem.attrib
            and "datetime" not in elem.attrib
        ):
            continue
        # name attribute, most frequent
        if "name" in elem.attrib:
            attribute = elem.get("name", "").lower()
            # url
            if attribute == "og:url":
                reserve = extract_url_date(elem.get("content"), options)
            # date
            elif attribute in DATE_ATTRIBUTES:
                LOGGER.debug("examining meta name: %s", logstring(elem))
                headerdate = tryfunc(elem.get("content"))
            # modified
            elif attribute in NAME_MODIFIED:
                LOGGER.debug("examining meta name: %s", logstring(elem))
                if not options.original:
                    headerdate = tryfunc(elem.get("content"))
                else:
                    reserve = tryfunc(elem.get("content"))
        # property attribute
        elif "property" in elem.attrib:
            attribute = elem.get("property", "").lower()
            if attribute in DATE_ATTRIBUTES or attribute in PROPERTY_MODIFIED:
                LOGGER.debug("examining meta property: %s", logstring(elem))
                attempt = tryfunc(elem.get("content"))
                if attempt is not None:
                    if (attribute in DATE_ATTRIBUTES and options.original) or (
                        attribute in PROPERTY_MODIFIED and not options.original
                    ):
                        headerdate = attempt
                    # hurts precision
                    else:
                        reserve = attempt
        # itemprop
        elif "itemprop" in elem.attrib:
            attribute = elem.get("itemprop", "").lower()
            # original: store / updated: override date
            if attribute in ITEMPROP_ATTRS:
                LOGGER.debug("examining meta itemprop: %s", logstring(elem))
                attempt = tryfunc(elem.get("datetime") or elem.get("content"))
                # store value
                if attempt is not None:
                    if (attribute in ITEMPROP_ATTRS_ORIGINAL and options.original) or (
                        attribute in ITEMPROP_ATTRS_MODIFIED and not options.original
                    ):
                        headerdate = attempt
                    # put on hold: hurts precision
                    # else:
                    #    reserve = attempt
            # reserve with copyrightyear
            elif attribute == "copyrightyear":
                LOGGER.debug("examining meta itemprop: %s", logstring(elem))
                if "content" in elem.attrib:
                    attempt = "-".join([elem.get("content", ""), "01", "01"])
                    if is_valid_date(
                        attempt, "%Y-%m-%d", earliest=options.min, latest=options.max
                    ):
                        reserve = attempt
        # pubdate, relatively rare
        elif "pubdate" in elem.attrib:
            if elem.get("pubdate", "").lower() == "pubdate":
                LOGGER.debug("examining meta pubdate: %s", logstring(elem))
                headerdate = tryfunc(elem.get("content"))
        # http-equiv, rare
        elif "http-equiv" in elem.attrib:
            attribute = elem.get("http-equiv", "").lower()
            if attribute == "date":
                LOGGER.debug("examining meta http-equiv: %s", logstring(elem))
                if options.original:
                    headerdate = tryfunc(elem.get("content"))
                else:
                    reserve = tryfunc(elem.get("content"))
            elif attribute == "last-modified":
                LOGGER.debug("examining meta http-equiv: %s", logstring(elem))
                if not options.original:
                    headerdate = tryfunc(elem.get("content"))
                else:
                    reserve = tryfunc(elem.get("content"))
        # exit loop
        if headerdate is not None:
            break
    # if nothing was found, look for lower granularity (so far: "copyright year")
    if headerdate is None and reserve is not None:
        LOGGER.debug("opting for reserve date with less granularity")
        headerdate = reserve
```

--------------------------------

### Utility and Helper Functions

Source: https://htmldate.readthedocs.io/en/latest/_sources/corefunctions.rst.txt

Helper functions for date validation, conversion, and network operations.

```APIDOC
## htmldate.validators & htmldate.utils
### Functions
- **is_valid_date**: Validates if a date string is in a correct format.
- **convert_date**: Converts date strings to standardized formats.
- **load_html**: Loads HTML content for processing.
- **fetch_url**: Fetches content from a given URL.
```

--------------------------------

### Examine Text for Date Extraction

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Prepares text by trimming and cleaning before attempting to extract a date using configured options.

```python
def examine_text(
    text: str,
    options: Extractor,
) -> Optional[str]:
    "Prepare text and try to extract a date."
    text = trim_text(text)

    if len(text) <= MIN_SEGMENT_LEN:
        return None

    text = NON_DIGITS_REGEX.sub("", text[:MAX_SEGMENT_LEN])
    return try_date_expr(
        text, options.format, options.extensive, options.min, options.max
    )
```

--------------------------------

### Compile Core Date Regex Components

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/extractors.html

Compiles core regular expression components for matching dates with day, month, and year, supporting different separators.

```python
THREE_COMP_REGEX_A = re.compile(rf"({DAY_RE})[/.-]({MONTH_RE})[/.-]({YEAR_RE})")
```

```python
THREE_COMP_REGEX_B = re.compile(
    rf"({DAY_RE})/({MONTH_RE})/([0-9]{{2}})|({DAY_RE})[.-]({MONTH_RE})[.-]([0-9]{{2}})"
)
```

```python
TWO_COMP_REGEX = re.compile(rf"({MONTH_RE})[/.-]({YEAR_RE})")
```

--------------------------------

### Command-Line Interface Usage

Source: https://htmldate.readthedocs.io/en/latest/index.html

Execute date extraction directly from the terminal using the htmldate command.

```bash
$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html
'2016-12-23'
$ wget -qO- "http://blog.python.org/2016/12/python-360-is-now-available.html" | htmldate
'2016-12-23'
```

--------------------------------

### Define Extractor Configuration

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/utils.html

Stores extraction parameters including date ranges and output formatting preferences.

```python
class Extractor:
    "Defines a class to store all extraction options."

    __slots__ = ["extensive", "format", "max", "min", "original"]

    # consider dataclasses for Python 3.7+
    def __init__(
        self,
        extensive_search: bool,
        max_date: datetime,
        min_date: datetime,
        original_date: bool,
        outputformat: str,
    ) -> None:
        self.extensive: bool = extensive_search
        self.format: str = outputformat
        self.max: datetime = max_date
        self.min: datetime = min_date
        self.original: bool = original_date
```

--------------------------------

### Import Statements for htmldate.utils

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/utils.html

Imports necessary modules for HTML processing, date handling, and network requests. Ensure these are available in your environment.

```python
# pylint:disable-msg=E0611,I1101
"""
Module bundling functions related to HTML processing.
"""

import logging
import re

from datetime import datetime
from typing import Any, List, Optional, Set, Union

import urllib3

```

--------------------------------

### Customize output date format

Source: https://htmldate.readthedocs.io/en/latest/options.html

Use the outputformat parameter to specify a format compatible with Python's datetime module.

```python
>>> find_date('https://www.gnu.org/licenses/gpl-3.0.en.html', outputformat='%d %B %Y')
'18 November 2016' # may have changed since
>>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html', outputformat='%Y-%m-%dT%H:%M:%S%z')
'2016-12-23T05:11:00-0500'
```

--------------------------------

### Configure extensive search

Source: https://htmldate.readthedocs.io/en/latest/_sources/index.rst.txt

Toggle the extensive_search parameter to control whether the library performs a full document screening.

```python
>>> find_date('https://creativecommons.org/about/')
'2017-08-11' # has been updated since
>>> find_date('https://creativecommons.org/about/', extensive_search=False)
>>>
```

--------------------------------

### Date Comparison Utilities

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/validators.html

Helper functions for comparing date expressions against a reference value.

```APIDOC
## Function: compare_values

### Description
Compares a date expression string against a reference timestamp. It parses the `attempt` string using the format specified in `options.format` and updates the `reference` timestamp based on whether `options.original` is set (to find the minimum) or not (to find the maximum).

### Parameters
- **reference** (int) - The initial reference timestamp.
- **attempt** (str) - The date expression string to compare.
- **options** (Extractor) - An object containing formatting and comparison options (e.g., `format`, `original`).

### Returns
- **int** - The updated reference timestamp after comparison.
```

--------------------------------

### Perform advanced date extraction with find_date

Source: https://htmldate.readthedocs.io/en/latest/index.html

Demonstrates using find_date with debug output and controlling the extensive_search parameter.

```python
>>> from htmldate import find_date
>>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')
# DEBUG analyzing: <h2 class="date-header"><span>Friday, December 23, 2016</span></h2>
# DEBUG result: 2016-12-23
'2016-12-23'
```

```python
>>> find_date('https://creativecommons.org/about/')
'2017-08-11' # has been updated since
>>> find_date('https://creativecommons.org/about/', extensive_search=False)
>>>
```

--------------------------------

### Examine abbr elements

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Scans HTML abbr elements for eligible date content, handling data-utime attributes and class-based metadata.

```python
def examine_abbr_elements(
    tree: HtmlElement,
    options: Extractor,
) -> Optional[str]:
    """Scan the page for abbr elements and check if their content contains an eligible date"""
    elements = tree.findall(".//abbr")
    if 0 < len(elements) < MAX_POSSIBLE_CANDIDATES:
        reference = 0
        for elem in elements:
            # data-utime (mostly Facebook)
            if "data-utime" in elem.attrib:
                try:
                    candidate = int(elem.get("data-utime", ""))
                except ValueError:
                    continue
                LOGGER.debug("data-utime found: %s", candidate)
                # look for original date
                if options.original and (reference == 0 or candidate < reference):
                    reference = candidate
                # look for newest (i.e. largest time delta)
                elif not options.original and candidate > reference:
                    reference = candidate
            # class
            elif elem.get("class") in CLASS_ATTRS:
                # other attributes
                if "title" in elem.attrib:
                    trytext = elem.get("title")
                    LOGGER.debug("abbr published-title found: %s", trytext)
                    # shortcut
                    if options.original:
                        attempt = try_date_expr(
                            trytext,
                            options.format,
                            options.extensive,
                            options.min,
                            options.max,
                        )
                        if attempt is not None:
                            return attempt
                    else:
                        reference = compare_reference(reference, trytext, options)
                        # faster execution
                        if reference > 0:
                            break
                # dates, not times of the day
                elif elem.text and len(elem.text) > 10:
                    LOGGER.debug("abbr published found: %s", elem.text)
                    reference = compare_reference(reference, elem.text, options)
        # return or try rescue in abbr content
```

--------------------------------

### htmldate.utils.fetch_url

Source: https://htmldate.readthedocs.io/en/latest/corefunctions.html

Fetches a web page using urllib3 and decodes the response.

```APIDOC
## htmldate.utils.fetch_url

### Description
Fetches page using urllib3 and decodes the response.

### Parameters
- **url** (str) - Required - URL of the page to fetch.

### Response
- **Return type** (str | None) - HTML code as string, or Urllib3 response object, or empty string, or None if there was a network problem.
```

--------------------------------

### Cite htmldate in BibTeX format

Source: https://htmldate.readthedocs.io/en/latest/_sources/used-by.rst.txt

Use this BibTeX entry to reference the htmldate library in academic publications.

```shell
@article{barbaresi-2020-htmldate,
  title = {{htmldate: A Python package to extract publication dates from web pages}},
  author = "Barbaresi, Adrien",
  journal = "Journal of Open Source Software",
  volume = 5,
  number = 51,
  pages = 2439,
  url = {https://doi.org/10.21105/joss.02439},
  publisher = {The Open Journal},
  year = 2020,
}
```

--------------------------------

### Load and validate HTML objects

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/utils.html

Normalizes input (bytes, string, URL, or existing tree) into an lxml HtmlElement, handling encoding and parsing fallbacks.

```python
def load_html(htmlobject: Union[bytes, str, HtmlElement]) -> Optional[HtmlElement]:
    """Load object given as input and validate its type
    (accepted: lxml.html tree, bytestring and string)
    """
    # use tree directly
    if isinstance(htmlobject, HtmlElement):
        return htmlobject
    # do not accept any other type after this point
    if not isinstance(htmlobject, (bytes, str)):
        raise TypeError("incompatible input type: %s", type(htmlobject))
    # the string is a URL, download it
    if (
        isinstance(htmlobject, str)
        and htmlobject.startswith("http")
        and " " not in htmlobject
    ):
        LOGGER.debug("URL detected, downloading: %s", htmlobject)
        htmlobject = fetch_url(htmlobject)  # type: ignore[assignment]
        # log the error and quit
        if htmlobject is None:
            raise ValueError("URL couldn't be processed: %s", htmlobject)
    # start processing
    tree = None
    # try to guess encoding and decode file: if None then keep original
    htmlobject = decode_file(htmlobject)
    # sanity checks
    beginning = htmlobject[:50].lower()
    # repair first
    htmlobject = repair_faulty_html(htmlobject, beginning)
    # first pass: use Unicode string
    fallback_parse = False
    try:
        tree = fromstring(htmlobject, parser=HTML_PARSER)
    except ValueError:
        # "Unicode strings with encoding declaration are not supported."
        fallback_parse = True
        tree = fromstring_bytes(htmlobject)
    except Exception as err:  # pragma: no cover
        LOGGER.error("lxml parsing failed: %s", err)
    # second pass: try passing bytes to LXML
    if (tree is None or len(tree) < 1) and not fallback_parse:
        tree = fromstring_bytes(htmlobject)
    # rejection test: is it (well-formed) HTML at all?
    # log parsing errors
    if tree is not None and is_dubious_html(beginning) and len(tree) < 2:
        LOGGER.error(
            "parsed tree length: %s, wrong data type or not valid HTML", len(tree)
        )
        tree = None
    return tree
```

--------------------------------

### Define Month Mappings for Date Parsing

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/extractors.html

Maps month abbreviations and full names across multiple languages to their corresponding numerical month. Used for text-based date parsing.

```python
MONTHS = [
    ("jan", "januar", "jänner", "january", "januari", "janvier", "ocak", "oca"),
    ("feb", "februar", "feber", "february", "februari", "février", "şubat", "şub"),
    ("mar", "mär", "märz", "march", "maret", "mart", "mars"),
    ("apr", "april", "avril", "nisan", "nis"),
    ("may", "mai", "mei", "mayıs"),
    ("jun", "juni", "june", "juin", "haziran", "haz"),
    ("jul", "juli", "july", "juillet", "temmuz", "tem"),
    ("aug", "august", "agustus", "ağustos", "ağu", "aout"),
    ("sep", "september", "septembre", "eylül", "eyl"),
    ("oct", "oktober", "october", "octobre", "okt", "ekim", "eki"),
    ("nov", "november", "kasım", "kas", "novembre"),
    ("dec", "dez", "dezember", "december", "desember", "décembre", "aralık", "ara"),
]

TEXT_MONTHS = { month: mnum for mnum, mlist in enumerate(MONTHS, start=1) for month in mlist }
```

--------------------------------

### Validate output format

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/validators.html

Verifies that the provided output format string is compatible with datetime formatting.

```python
@lru_cache(maxsize=16)
def is_valid_format(outputformat: str) -> bool:
    """Validate the output format in the settings"""
    # test with date object
    dateobject = datetime(2017, 9, 1, 0, 0)
    try:
        dateobject.strftime(outputformat)
    except (TypeError, ValueError) as err:
        LOGGER.error("wrong output format or type: %s %s", outputformat, err)
        return False
    # test in abstracto (could be the only test)
    if not isinstance(outputformat, str) or "%" not in outputformat:
        LOGGER.error("malformed output format: %s", outputformat)
        return False
    return True
```

--------------------------------

### Search for JSON time patterns

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/extractors.html

Parses JSON-LD or application settings scripts within an HTML tree to locate date information.

```python
def json_search(
    tree: HtmlElement,
    options: Extractor,
) -> Optional[str]:
    """Look for JSON time patterns in JSON sections of the tree"""
    # determine pattern
    json_pattern = JSON_PUBLISHED if options.original else JSON_MODIFIED
    # look throughout the HTML tree
    for elem in tree.xpath(
        './/script[@type="application/ld+json" or @type="application/settings+json"]'
    ):
        if not elem.text or '"date' not in elem.text:
            continue
        return pattern_search(elem.text, json_pattern, options)
    return None
```

--------------------------------

### Swap Day and Month if Feasible

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/extractors.html

Swaps the day and month values if the month value is greater than 12 and the day value is less than or equal to 12, suggesting a potential DD/MM vs MM/DD ambiguity.

```python
def try_swap_values(day: int, month: int) -> Tuple[int, int]:
    """Swap day and month values if it seems feasible."""
    return (month, day) if month > 12 and day <= 12 else (day, month)
```

--------------------------------

### Search page for date patterns

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Opportunistically searches HTML text for copyright symbols and common date patterns.

```python
def search_page(htmlstring: str, options: Extractor) -> Optional[str]:
    """
    Opportunistically search the HTML text for common text patterns

    :param htmlstring:
        The HTML document in string format, potentially cleaned and stripped to
        the core (much faster)
    :type htmlstring: string
    :param options:
        Define extraction options
    :type options: Extractor
    :return: Returns a valid date expression as a string, or None

    """

    # copyright symbol
    LOGGER.debug("looking for copyright/footer information")
    copyear = 0
    bestmatch = search_pattern(
        htmlstring,
        COPYRIGHT_PATTERN,
        YEAR_PATTERN,
        YEAR_PATTERN,
        options,
    )
    if bestmatch is not None:
        year = int(bestmatch[0])
        if is_valid_date(
            datetime(year, 1, 1), "%Y", earliest=options.min, latest=options.max
        ):
            LOGGER.debug("copyright year/footer pattern found: %s", year)
            copyear = year

    # 3 components
    LOGGER.debug("3 components")
    # target URL characteristics
    # then more loosely structured data
    for patterns in THREE_COMP_PATTERNS:
        bestmatch = search_pattern(
            htmlstring,
            patterns[0],
            patterns[1],
            YEAR_PATTERN,
            options,
        )
        result = filter_ymd_candidate(
            bestmatch,
            patterns[0],
            options.original,
            copyear,
            options.format,
            options.min,
            options.max,
        )
        if result is not None:
            return result
```

--------------------------------

### Extract date using core search logic

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Executes a multi-stage search for date information, falling back to extensive text analysis if initial patterns fail.

```python
htmlstring = tostring(search_tree, pretty_print=False).decode("utf-8", "ignore")

    # date regex timestamp rescue
    # try image elements
    # precise patterns and idiosyncrasies
    result = (
        pattern_search(htmlstring, TIMESTAMP_PATTERN, options)
        or img_search(search_tree, options)
        or idiosyncrasies_search(htmlstring, options)
    )
    if result is not None:
        return result

    # last resort
    if extensive_search:
        LOGGER.debug("extensive search started")
        # TODO: further tests & decide according to original_date
        reference = 0
        for segment in FREE_TEXT_EXPRESSIONS(search_tree):
            segment = segment.strip()
            if not MIN_SEGMENT_LEN < len(segment) < MAX_SEGMENT_LEN:
                continue
            reference = compare_reference(reference, segment, options)
        converted = check_extracted_reference(reference, options)
        # return or search page HTML
        return converted or search_page(htmlstring, options)

    return None
```

--------------------------------

### Compile Regex for Copyright Year

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/extractors.html

Compiles a regular expression to extract the year from copyright notices, supporting various symbols and formats.

```python
COPYRIGHT_PATTERN = re.compile(
    rf"(?:©|&copy;|Copyright|\(c\))\D*(?:{YEAR_RE})?-?({YEAR_RE})\D"
)
```

--------------------------------

### Prioritize original publication date

Source: https://htmldate.readthedocs.io/en/latest/options.html

Set original_date=True to prioritize the original publication date over the last modified date.

```python
>>> find_date('https://netzpolitik.org/2016/die-cider-connection-abmahnungen-gegen-nutzer-von-creative-commons-bildern/') # default setting
'2019-06-24'
>>> find_date('https://netzpolitik.org/2016/die-cider-connection-abmahnungen-gegen-nutzer-von-creative-commons-bildern/', original_date=True) # modified behavior
'2016-06-23'
```

--------------------------------

### Extract dates from HTML strings and LXML trees

Source: https://htmldate.readthedocs.io/en/latest/index.html

Shows how to pass raw HTML strings or pre-parsed LXML tree objects to the find_date function.

```python
# simple HTML document as string
>>> htmldoc = '<html><body><span class="entry-date">July 12th, 2016</span></body></html>'
>>> find_date(htmldoc)
'2016-07-12'
# parsed LXML tree
>>> from lxml import html
>>> mytree = html.fromstring('<html><body><span class="entry-date">July 12th, 2016</span></body></html>')
>>> find_date(mytree)
'2016-07-12'
```

--------------------------------

### Select candidate date

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Selects the most plausible date candidate from a collection of occurrences using frequency and year-based heuristics.

```python
def select_candidate(
    occurrences: Counter_Type[str],
    catch: Pattern[str],
    yearpat: Pattern[str],
    options: Extractor,
) -> Optional[Match[str]]:
    """Select a candidate among the most frequent matches"""
    if not occurrences or len(occurrences) > MAX_POSSIBLE_CANDIDATES:
        return None

    if len(occurrences) == 1:
        return catch.search(next(iter(occurrences)))

    # select among most frequent: more than 10? more than 2 candidates?
    firstselect = occurrences.most_common(10)
    LOGGER.debug("firstselect: %s", firstselect)
    # sort and find probable candidates
    bestones = sorted(firstselect, reverse=not options.original)[:2]
    LOGGER.debug("bestones: %s", bestones)

    # plausibility heuristics
    patterns, counts = zip(*bestones)

    years = []
    for pattern in patterns:
        year_match = yearpat.search(pattern)
        if year_match:
            years.append(year_match[1])

    validation = [
        is_valid_date(
            datetime(int(year), 1, 1), "%Y", earliest=options.min, latest=options.max
        )
        for year in years
    ]

    # safety net: plausibility
    if all(validation):
        # same number of occurrences: always take top of the pile?
        if counts[0] == counts[1]:
            match = catch.search(patterns[0])
        # safety net: newer date but up to 50% less frequent
        elif years[1] != years[0] and counts[1] / counts[0] > 0.5:
            match = catch.search(patterns[1])
        # not newer or hopefully not significant
        else:
            match = catch.search(patterns[0])
    elif any(validation):
        match = catch.search(patterns[validation.index(True)])
    else:
        LOGGER.debug("no suitable candidate: %s %s", years[0], years[1])
        match = None
    return match
```

--------------------------------

### Compare reference date

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Compares a candidate date against a reference value, incorporating date validation and older/newer testing.

```python
@lru_cache(maxsize=CACHE_SIZE)
def compare_reference(
    reference: int,
    expression: str,
    options: Extractor,
) -> int:
    """Compare candidate to current date reference (includes date validation and older/newer test)"""
    attempt = try_date_expr(
        expression, options.format, options.extensive, options.min, options.max
    )
    if attempt is not None:
        return compare_values(reference, attempt, options)
    return reference
```

--------------------------------

### Return header date

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Returns the extracted header date value.

```python
# return value
return headerdate
```

--------------------------------

### Compile Regex for Date Strings Capture (YYYYMMDD)

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/extractors.html

Compiles a regular expression to capture year, month, and day from a YYYYMMDD date string.

```python
DATESTRINGS_CATCH = re.compile(rf"({YEAR_RE})([01][0-9])([0-3][0-9])")
```

--------------------------------

### Examine time elements

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Scans the HTML tree for <time> elements and evaluates their datetime attributes or text content for eligible dates.

```python
def examine_time_elements(
    tree: HtmlElement,
    options: Extractor,
) -> Optional[str]:
    """Scan the page for time elements and check if their content contains an eligible date"""
    elements = tree.findall(".//time")
    if 0 < len(elements) < MAX_POSSIBLE_CANDIDATES:
        # scan all the tags and look for the newest one
        reference = 0
        for elem in elements:
            shortcut_flag = False
            datetime_attr = elem.get("datetime", "")
            # go for datetime
            if len(datetime_attr) > 6:
                # shortcut: time pubdate
                if (
                    "pubdate" in elem.attrib
                    and elem.get("pubdate") == "pubdate"
                    and options.original
                ):
                    shortcut_flag = True
                    LOGGER.debug("shortcut for time pubdate found: %s", datetime_attr)
                # shortcuts: class attribute
                elif "class" in elem.attrib:
                    if options.original and (
                        elem.get("class", "").startswith("entry-date")
                        or elem.get("class", "").startswith("entry-time")
                    ):
                        shortcut_flag = True
                        LOGGER.debug(
                            "shortcut for time/datetime found: %s", datetime_attr
                        )
                    # updated time
                    elif not options.original and elem.get("class") == "updated":
                        shortcut_flag = True
                        LOGGER.debug(
                            "shortcut for updated time/datetime found: %s",
                            datetime_attr,
                        )
                # datetime attribute
                else:
                    LOGGER.debug("time/datetime found: %s", datetime_attr)
                # analyze attribute
                if shortcut_flag:
                    attempt = try_date_expr(
                        datetime_attr,
                        options.format,
                        options.extensive,
                        options.min,
                        options.max,
                    )
                    if attempt is not None:
                        return attempt
                else:
                    reference = compare_reference(reference, datetime_attr, options)
            # bare text in element
            elif elem.text is not None and len(elem.text) > 6:
                LOGGER.debug("time/datetime found in text: %s", elem.text)
                reference = compare_reference(reference, elem.text, options)
            # else...?
        # return
        return check_extracted_reference(reference, options)
    return None
```

--------------------------------

### Search pattern

Source: https://htmldate.readthedocs.io/en/latest/_modules/htmldate/core.html

Chains candidate filtering and selection processes to identify a date match.

```python
def search_pattern(
    htmlstring: str,
    pattern: Pattern[str],
    catch: Pattern[str],
    yearpat: Pattern[str],
    options: Extractor,
) -> Optional[Match[str]]:
    """Chained candidate filtering and selection"""
    candidates = plausible_year_filter(
        htmlstring,
        pattern=pattern,
        yearpat=yearpat,
        earliest=options.min,
        latest=options.max,
    )
    return select_candidate(candidates, catch, yearpat, options)
```
Tokens:
Raw