Goose3

https://goose3.readthedocs.io/en/latest/
Admin
Goose3 is a Python library designed for robust article extraction from web pages, providing tools to...
Tokens:9,870
Snippets:67
Trust Score:9.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Goose3 from source

Source: https://goose3.readthedocs.io/en/latest/quickstart.html

Install the library by running the setup script from the cloned repository.

```bash
$ python setup.py install
```

--------------------------------

### Install Goose3 via pip

Source: https://goose3.readthedocs.io/en/latest/quickstart.html

Install the library using the pip package manager.

```bash
$ pip install goose3
```

--------------------------------

### Use configuration helper classes

Source: https://goose3.readthedocs.io/en/latest/quickstart.html

Utilize pattern classes to define specific extraction rules for articles, dates, and authors.

```python
from goose3 import Goose
from goose3.configuration import Configuration, ArticleContextPattern, PublishDatePattern, AuthorPattern

config = Configuration()

# we know of a particular article location in the site we are pulling from
config.known_context_patterns = ArticleContextPattern(attr="id", value="my-site-article")

# publish date
config.known_publish_date_tags = PublishDatePattern(attr="id", value="pubdate", content="content")

# author
config.known_author_patterns = AuthorPattern(attr="id", value="writer", content="content")
```

--------------------------------

### Initialize Configuration class

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

The Configuration class holds all settings for the Goose3 extraction process.

```python
class Configuration:
    def __init__(self):
        # parser information
        self._available_parsers = list(AVAILABLE_PARSERS.keys())
        self._parser_class = "lxml"

        # URL extraction parameters
        self._browser_user_agent = f"Goose/{__version__}"
        self._http_timeout = 30.0
        self._http_auth = None
        self._http_proxies = None
        self._http_headers = None

        # extraction information
        self._local_storage_path = os.path.join(tempfile.gettempdir(), "goose")
        self._known_context_patterns = KNOWN_ARTICLE_CONTENT_PATTERNS[:]
        self._known_publish_date_tags = KNOWN_PUBLISH_DATE_TAGS[:]
        self._known_author_patterns = KNOWN_AUTHOR_PATTERNS[:]
        self._target_language = "en"
        self._use_meta_language = True

        # general configuration
        self._strict = True
        self._log_level = "ERROR"
        self._stopwords_class = StopWords

        # imagemagick executable paths
        self._imagemagick_convert_path = "/opt/local/bin/convert"  # Not used
        self._imagemagick_identify_path = "/opt/local/bin/identify"  # not used

        # image extraction
        self._enable_image_fetching = False
        self._images_min_bytes = 4000
        # Do we need to allow setting one's own ImageExtractor class?

        self._parse_lists = True
        self._pretty_lists = True
        self._parse_headers = True
        self._keep_footnotes = True
```

--------------------------------

### Configure Goose3 using a dictionary

Source: https://goose3.readthedocs.io/en/latest/quickstart.html

Pass configuration options as a dictionary to the Goose constructor.

```python
from goose3 import Goose

config = {}
config['strict'] = False  # turn of strict exception handling
config['browser_user_agent'] = 'Mozilla 5.0'  # set the browser agent string
config['http_timeout'] = 5.05  # set http timeout in seconds

with Goose(config) as g:
    pass
```

```python
from goose3 import Goose

with Goose({'http_timeout': 5.0}) as g:
    pass
```

--------------------------------

### Package with PyInstaller

Source: https://goose3.readthedocs.io/en/latest/quickstart.html

Include required resources when bundling the application with PyInstaller.

```bash
pyinstaller \
    --add-data="goose3/resources/images/*;goose3/resources/images/" \
    --add-data="goose3/resources/text/*;goose3/resources/text/" \
    my_prog.py
```

--------------------------------

### Initialize and Use Goose Class

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3.html

The Goose class serves as the main entry point for content extraction. It supports initialization with a configuration object or dictionary and can be used as a context manager.

```python
"""
This is a python port of "Goose" orignialy licensed to Gravity.com
under one or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.

Python port was written by Xavier Grangier for Recrutae

Gravity.com licenses this file
to you under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""

import logging
import os
import weakref
from tempfile import mkstemp
from typing import List, Union

from goose3.article import Article  # noqa: F401
from goose3.configuration import ArticleContextPattern, AuthorPattern, Configuration, PublishDatePattern  # noqa: F401
from goose3.crawler import CrawlCandidate, Crawler
from goose3.image import Image  # noqa: F401
from goose3.network import NetworkFetcher
from goose3.video import Video  # noqa: F401

logger = logging.getLogger(__name__)




[docs]
class Goose:
    """Extract most likely article content and aditional metadata from a URL or previously fetched HTML document

    Args:
        config (Configuration, dict): A configuration file or dictionary representation of the configuration file
    Returns:
        Goose: An instance of the goose extraction object"""

    def __init__(self, config: Union[Configuration, dict, None] = None):
        # Use the passed in configuration if it is of the right type, otherwise
        # use the default as a base
        if isinstance(config, Configuration):
            self.config = config
        else:
            self.config = Configuration()

        # if config was a passed in dict, parse it into the stored configuration
        if isinstance(config, dict):
            for k, v in config.items():
                if hasattr(self.config, k):
                    setattr(self.config, k, v)

        # setup a single network connection
        self.fetcher = NetworkFetcher(self.config)
        self.finalizer = weakref.finalize(self, self.close)

        # we don't need to go further if image extractor or local_storage is not set
        if not self.config.local_storage_path or not self.config.enable_image_fetching:
            return

        # test if config.local_storage_path is a directory
        if not os.path.isdir(self.config.local_storage_path):
            os.makedirs(self.config.local_storage_path)

        if not os.path.isdir(self.config.local_storage_path):
            msg = (
                f"{self.config.local_storage_path} directory does not seem to exist, "
                "you need to set this for image processing downloads"
            )
            raise Exception(msg)

        # test to write a dummy file to the directory to check is directory is writable
        level, path = mkstemp(dir=self.config.local_storage_path)
        try:
            with os.fdopen(level, "w"):
                pass
            os.remove(path)
        except OSError as exc:
            msg = (
                f"{self.config.local_storage_path} directory is not writeble, "
                "you need to set this for image processing downloads"
            )
            raise Exception(msg) from exc

    def __enter__(self):
        """Setup the context manager"""
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        """Define what to do when the context manager exits"""
        self.close()



[docs]
    def close(self):
        """Close the network connection and perform any other required cleanup

        Note:
            Auto closed when using goose as a context manager or when garbage collected"""
        if self.fetcher is not None:
            self.shutdown_network()
        self.finalizer.atexit = False  # turn off the garbage collection close



[docs]
    def extract(self, url: Union[str, None] = None, raw_html: Union[str, None] = None) -> Article:
        """Extract the most likely article content from the html page

        Args:
            url (str): URL to pull and parse
            raw_html (str): String representation of the HTML page
        Returns:
            Article: Representation of the article contents including other parsed and extracted metadata"""
        if not url and not raw_html:
            raise ValueError("Either url or raw_html should be provided")
        crawl_candidate = CrawlCandidate(self.config, url, raw_html)
        return self.__crawl(crawl_candidate)



[docs]
    def shutdown_network(self):
        """Close the network connection
```

--------------------------------

### Configure Goose3 using Configuration object

Source: https://goose3.readthedocs.io/en/latest/quickstart.html

Set extraction parameters using the dedicated Configuration class.

```python
from goose3 import Goose
from goose3.configuration import Configuration

config = Configuration()
config.strict = False  # turn of strict exception handling
config.browser_user_agent = 'Mozilla 5.0'  # set the browser agent string
config.http_timeout = 5.05  # set http timeout in seconds

with Goose(config) as g:
    ...
```

--------------------------------

### Initialize known author patterns

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

A list of predefined AuthorPattern instances used for author extraction.

```python
KNOWN_AUTHOR_PATTERNS = [
    AuthorPattern(attr="itemprop", value="author", subpattern=AuthorPattern(attr="itemprop", value="name")),
    AuthorPattern(attr="name", value="author", content="content"),
]
```

--------------------------------

### Goose.__init__

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3.html

Initializes the Goose extraction object with an optional configuration.

```APIDOC
## Goose(config)

### Description
Initializes a new instance of the Goose extraction object. Users can provide a Configuration object or a dictionary to customize the extraction behavior.

### Parameters
- **config** (Configuration, dict) - Optional - A configuration object or dictionary containing settings for the extraction process.
```

--------------------------------

### Extract multiple URLs using a context manager

Source: https://goose3.readthedocs.io/en/latest/quickstart.html

Use the context manager to handle multiple extractions efficiently.

```python
from goose3 import Goose

urls = [...]
with Goose() as g:
    for tmp in urls:
        article = g.extract(url=tmp)
        print(article.cleaned_text)
```

--------------------------------

### Manage known context patterns

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Getter and setter methods for managing article context patterns.

```python
@property
def known_context_patterns(self) -> list:
    """list: The context patterns to search to find the likely article content

    Note:
        Each entry must be a dictionary with the following keys: `attr` and `value` or just `tag`
    """
    return self._known_context_patterns

@known_context_patterns.setter
def known_context_patterns(self, val: Union[dict, List[dict]]):
    """val must be an ArticleContextPattern, a dictionary, or list of dictionaries
    e.g., {'attr': 'class', 'value': 'my-article-class'}
        or [{'attr': 'class', 'value': 'my-article-class'},
            {'attr': 'id', 'value': 'my-article-id'}]
    """

    def create_pat_from_dict(val):
        """Helper function used to create an ArticleContextPattern from a dictionary"""
        if "tag" in val:
            pat = ArticleContextPattern(tag=val["tag"])
            if "attr" in val:
                pat.attr = val["attr"]
                pat.value = val["value"]
        elif "attr" in val:
            pat = ArticleContextPattern(attr=val["attr"], value=val["value"])

        if "domain" in val:
            pat.domain = val["domain"]

        return pat

    if isinstance(val, list):
        self._known_context_patterns = [
            x if isinstance(x, ArticleContextPattern) else create_pat_from_dict(x) for x in val
        ] + self.known_context_patterns
    elif isinstance(val, ArticleContextPattern):
        self._known_context_patterns.insert(0, val)
    elif isinstance(val, dict):
        self._known_context_patterns.insert(0, create_pat_from_dict(val))
    else:
        raise Exception(f"Unknown type: {type(val)}. Use a ArticleContextPattern.")
```

--------------------------------

### Configure known_author_patterns

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Sets the patterns used to identify authors. Each entry requires attribute, value, and content keys.

```python
@known_author_patterns.setter
    def known_author_patterns(self, val: Union[dict, List[dict], AuthorPattern]):
        """val must be a dictionary or list of dictionaries
        e.g., {'attrribute': 'name', 'value': 'my-pubdate', 'content': 'datetime'}
            or [{'attrribute': 'name', 'value': 'my-pubdate', 'content': 'datetime'},
                {'attrribute': 'property', 'value': 'pub_time', 'content': 'content'}]
        """

        def create_pat_from_dict(val):
            """Helper function used to create an AuthorPatterns from a dictionary"""
            if "tag" in val:
                pat = AuthorPattern(tag=val["tag"])
                if "attribute" in val:
                    pat.attr = val["attribute"]
                    pat.value = val["value"]
            elif "attribute" in val:
                pat = AuthorPattern(attr=val["attribute"], value=val["value"], content=val["content"])
            if "subpattern" in val:
                pat.subpattern = create_pat_from_dict(val["subpattern"])

            return pat

        if isinstance(val, list):
            self._known_author_patterns = [
                x if isinstance(x, AuthorPattern) else create_pat_from_dict(x) for x in val
            ] + self.known_author_patterns
        elif isinstance(val, AuthorPattern):
            self._known_author_patterns.insert(0, val)
        elif isinstance(val, dict):
            self._known_author_patterns.insert(0, create_pat_from_dict(val))
        else:
            raise Exception(f"Unknown type: {type(val)}. Use an AuthorPattern.")
```

--------------------------------

### Configure known_publish_date_tags

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Sets the tags used to identify publish dates. Accepts a dictionary or a list of dictionaries containing attribute, value, and content keys.

```python
@known_publish_date_tags.setter
    def known_publish_date_tags(self, val: Union[dict, List[dict]]):
        """val must be a dictionary or list of dictionaries
        e.g., {'attrribute': 'name', 'value': 'my-pubdate', 'content': 'datetime'}
            or [{'attrribute': 'name', 'value': 'my-pubdate', 'content': 'datetime'},
                {'attrribute': 'property', 'value': 'pub_time', 'content': 'content'}]
        """

        def create_pat_from_dict(val):
            """Helper function used to create an PublishDatePattern from a dictionary"""
            if "tag" in val:
                pat = PublishDatePattern(tag=val["tag"])
                if "attribute" in val:
                    pat.attr = val["attribute"]
                    pat.value = val["value"]
            elif "attribute" in val:
                pat = PublishDatePattern(attr=val["attribute"], value=val["value"], content=val["content"])
                if "subcontent" in val:
                    pat.subcontent = val["subcontent"]

            if "domain" in val:
                pat.domain = val["domain"]

            return pat

        if isinstance(val, list):
            self._known_publish_date_tags = [
                x if isinstance(x, PublishDatePattern) else create_pat_from_dict(x) for x in val
            ] + self.known_publish_date_tags
        elif isinstance(val, PublishDatePattern):
            self._known_publish_date_tags.insert(0, val)
        elif isinstance(val, dict):
            self._known_publish_date_tags.insert(0, create_pat_from_dict(val))
        else:
            raise Exception(f"Unknown type: {type(val)}. Use a PublishDatePattern.")
```

--------------------------------

### Configure Goose3 extraction properties

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

These property decorators manage settings for image fetching, text formatting, and metadata handling.

```python
        Note:
            Defaults to `True`"""
        return self._use_meta_language

    @use_meta_language.setter
    def use_meta_language(self, val: bool):
        """set the use_meta_language property"""
        self._use_meta_language = bool(val)

    @property
    def enable_image_fetching(self) -> bool:
        """bool: Turn on or off image extraction

        Note:
            Defaults to `False`"""
        return self._enable_image_fetching

    @enable_image_fetching.setter
    def enable_image_fetching(self, val: bool):
        """set the enable_image_fetching property"""
        self._enable_image_fetching = bool(val)

    @property
    def images_min_bytes(self) -> int:
        """int: Minimum number of bytes for an image to be evaluated to be the main image of the site

        Note:
            Defaults to 4500 bytes"""
        return self._images_min_bytes

    @images_min_bytes.setter
    def images_min_bytes(self, val: int):
        """set the images_min_bytes property"""
        self._images_min_bytes = int(val)

    @property
    def pretty_lists(self) -> bool:
        """bool: Specify if lists should be pretty printed in the cleaned_text output

        Note:
            Defaults to `True`"""
        return self._pretty_lists

    @pretty_lists.setter
    def pretty_lists(self, val: bool):
        """set if lists should be pretty printed"""
        self._pretty_lists = bool(val)

    @property
    def parse_lists(self) -> bool:
        return self._parse_lists

    @parse_lists.setter
    def parse_lists(self, val: bool):
        """set if headers should be parsed"""
        self._parse_lists = bool(val)

    @property
    def parse_headers(self) -> bool:
        """bool: Specify if headers should be pulled or not in the cleaned_text output

        Note:
            Defaults to `True`"""
        return self._parse_headers

    @parse_headers.setter
    def parse_headers(self, val: bool):
        """set if headers should be parsed"""
        self._parse_headers = bool(val)

    @property
    def keep_footnotes(self) -> bool:
        """bool: Specify if footnotes should be kept or not in the cleaned_text output

        Note:
            Defaults to `True`"""
        return self._keep_footnotes

    @keep_footnotes.setter
    def keep_footnotes(self, val: bool):
        """set if headers should be parsed"""
        self._keep_footnotes = bool(val)
```

--------------------------------

### goose3.Configuration

Source: https://goose3.readthedocs.io/en/latest/_sources/code.rst.txt

Configuration class used to customize the behavior of the Goose3 extraction process.

```APIDOC
## class goose3.Configuration

### Description
Allows users to modify how Goose3 parses and extracts content by setting various configuration attributes.
```

--------------------------------

### Define Publish Date Patterns

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Use PublishDatePattern to define how to extract publication dates from HTML elements. Requires either (attr and value) or (tag) to be set.

```python
class PublishDatePattern:
    """Ensure correctly formed publish date patterns; to be used in conjuntion with the configuration
    `known_publish_date_tags` property

    Args:
        attr (str): The attribute type: class, id, etc
        value (str): The value of the attribute
        content (str): The name of another attribute (of the element) that contains the value
        subcontent (str): The name of a json object key (optional)
        tag (str): The type of tag, such as `time` that contains the publish date
        domain (str): The domain to which this pattern pertains (optional)
    Note:
        Must provide, at a minimum, (attr and value) or (tag)
    """

    __slots__ = ["attr", "value", "content", "subcontent", "tag", "domain"]

    def __init__(self, *, attr=None, value=None, content=None, subcontent=None, tag=None, domain=None):
        if (not attr and not value) and not tag:
            raise Exception("`attr` and `value` must be set or `tag` must be set")
        self.attr = attr
        self.value = value
        self.content = content
        self.subcontent = subcontent
        self.tag = tag
        self.domain = domain

    def __repr__(self):
        if self.tag:
            return f"PublishDatePattern(tag={self.tag}, attr={self.attr}, value={self.value} domain={self.domain})"
        res = (
            f"PublishDatePattern(attr={self.attr}, value={self.value} content={self.content} "
            f"subcontent={self.subcontent} domain={self.domain})"
        )
        return res
```

--------------------------------

### Internal Crawl Wrapper Implementation

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3.html

Handles the crawling process by attempting multiple parsers if the initial attempt fails due to decoding or value errors.

```python
    def __crawl(self, crawl_candidate: CrawlCandidate):
        """wrap the crawling functionality"""

        def crawler_wrapper(parser: str, parsers: List[str], crawl_candidate: CrawlCandidate):
            try:
                crawler = Crawler(self.config, self.fetcher)
                article = crawler.crawl(crawl_candidate)
            except (UnicodeDecodeError, ValueError) as ex:
                logger.error("Parser %s failed to parse the content", parser)
                if parsers:
                    parser = parsers.pop(0)  # remove it also!
                    return crawler_wrapper(parser, parsers, crawl_candidate)
                raise ex
            return article

        # use the wrapper
        parsers = list(self.config.available_parsers)
        parsers.remove(self.config.parser_class)
        return crawler_wrapper(self.config.parser_class, parsers, crawl_candidate)
```

--------------------------------

### Configure http_timeout

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Sets the timeout duration in seconds for HTTP requests.

```python
@http_timeout.setter
    def http_timeout(self, val):
        """set the http_timeout property"""
        self._http_timeout = float(val)
```

--------------------------------

### Configure local_storage_path

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Sets the directory path used for storing temporary files.

```python
@local_storage_path.setter
    def local_storage_path(self, val):
        """set the local_storage_path property"""
        self._local_storage_path = val
```

--------------------------------

### Set Browser User Agent

Source: https://goose3.readthedocs.io/en/latest/code.html

Configures the user agent string used for URL requests. Using a standard browser agent is recommended for frequent requests.

```python
>>> config.browser_user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2)'
>>> config.browser_user_agent = 'AppleWebKit/534.52.7 (KHTML, like Gecko)'
>>> config.browser_user_agent = 'Version/5.1.2 Safari/534.52.7'
```

--------------------------------

### close

Source: https://goose3.readthedocs.io/en/latest/code.html

Closes network connections and performs cleanup.

```APIDOC
## close()

### Description
Closes the network connection and performs required cleanup. This is automatically handled when using the object as a context manager or during garbage collection.
```

--------------------------------

### Access Article Properties

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/article.html

Standard read-only properties for accessing article metadata and document structures.

```python
@property
def links(self):
    """list(str): A listing of URL links within the article

    Note:
        Read only"""
    return self._links

@property
def authors(self):
    """list(str): A listing of authors as parsed from the meta tags

    Note:
        Read only"""
    return self._authors

@property
def final_url(self):
    """str: The URL that was used to pull and parsed; `None` if raw_html was used and no url element was found.

    Note:
        Read only"""
    return self._final_url

@property
def link_hash(self):
    """str: The hash of the final url to be used for various identification tasks

    Note:
        Read only"""
    return self._link_hash

@property
def raw_html(self):
    """str: The HTML represented as a string

    Note:
        Read only"""
    return self._raw_html

@property
def doc(self):
    """etree: lxml document that is being processed

    Note:
        Read only"""
    return self._doc

@property
def raw_doc(self):
    """etree: Original, uncleaned, and untouched lxml document to be processed

    Note:
        Read only"""
    return self._raw_doc

@property
def schema(self):
    """dict: All schema tag data

    Note:
        Read only"""
    return self._schema

@property
def publish_date(self):
    """str: The date the article was published based on meta tag extraction

    Note:
        Read only"""
    return self._publish_date

@property
def publish_datetime_utc(self):
    """datetime.datetime: The date time version of the published date based on meta tag extraction
    in the UTC timezone, if timezone information is known

    Note:
        Read only"""
    return self._publish_datetime_utc

@property
def additional_data(self):
    """dict: A property bucket for consumers of goose3 to store custom data extractions

    Note:
        Read only"""
    return self._additional_data
```

--------------------------------

### Define the Video class

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/video.html

The Video class serves as a container for video metadata, providing read-only properties for accessing embed details.

```python
"""
This is a python port of "Goose" orignialy licensed to Gravity.com
under one or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.

Python port was written by Xavier Grangier for Recrutae

Gravity.com licenses this file
to you under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""




[docs]
class Video:
    """Video object"""

    def __init__(self):

        # type of embed
        # embed, object, iframe
        self._embed_type = None

        # video provider name
        self._provider = None

        # width
        self._width = None

        # height
        self._height = None

        # embed code
        self._embed_code = None

        # src
        self._src = None

    @property
    def embed_type(self):
        """str: The type of embeding such as embed, object, or iframe

        Note:
            Read only"""
        return self._embed_type

    @property
    def provider(self):
        """str: The video provider

        Note:
            Read only"""
        return self._provider

    @property
    def width(self):
        """int: The video width in pixels

        Note:
            Read only"""
        return self._width

    @property
    def height(self):
        """int: The video height in pixels

        Note:
            Read only"""
        return self._height

    @property
    def embed_code(self):
        """str: The embed code of the video

        Note:
            Read only"""
        return self._embed_code

    @property
    def src(self):
        """str: The URL source of the video

        Note:
            Read only"""
        return self._src
```

--------------------------------

### Retrieve Article Information Summary

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/article.html

The infos property aggregates all extracted article data into a single dictionary.

```python
@property
def infos(self):
    """dict: The summation of all data available about the extracted article

    Note:
        Read only"""
    data = {
        "meta": {
            "description": self.meta_description,
            "lang": self.meta_lang,
            "keywords": self.meta_keywords,
            "favicon": self.meta_favicon,
            "canonical": self.canonical_link,
            "encoding": self.meta_encoding,
        },
        "image": None,
        "domain": self.domain,
        "title": self.title,
        "cleaned_text": self.cleaned_text,
        "opengraph": self.opengraph,
        "tags": self.tags,
        "tweets": self.tweets,
        "movies": [],
        "links": self.links,
        "authors": self.authors,
        "publish_date": self.publish_date,
    }

    # image
    if self.top_image is not None:
        data["image"] = {
            "url": self.top_image.src,
            "width": self.top_image.width,
            "height": self.top_image.height,
            "type": "image",
        }

    # movies
    for movie in self.movies:
        data["movies"].append(
            {
                "embed_type": movie.embed_type,
                "provider": movie.provider,
                "width": movie.width,
                "height": movie.height,
                "embed_code": movie.embed_code,
                "src": movie.src,
            }
        )

    return data
```

--------------------------------

### Goose.close

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3.html

Closes the network connection and performs cleanup.

```APIDOC
## close()

### Description
Closes the active network connection and performs necessary cleanup. This is automatically called when using the object as a context manager or during garbage collection.
```

--------------------------------

### Define Article Context Patterns

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Use ArticleContextPattern to specify HTML attributes or tags that identify the main article body. At least one of (attr and value) or (tag) must be provided.

```python
class ArticleContextPattern:
    """Help ensure correctly generated article context patterns

    Args:
        attr (str): The attribute type: class, id, etc
        value (str): The value of the attribute
        tag (str): The type of tag, such as `article` that contains the main article body
        domain (str): The domain to which this pattern pertains (optional)
    Note:
        Must provide, at a minimum, (attr and value) or (tag)
    """

    __slots__ = ["attr", "value", "tag", "domain"]

    def __init__(self, *, attr=None, value=None, tag=None, domain=None):
        if (not attr and not value) and not tag:
            raise Exception("`attr` and `value` must be set or `tag` must be set")
        self.attr = attr
        self.value = value
        self.tag = tag
        self.domain = domain

    def __repr__(self):
        return f"ArticleContextPattern(attr={self.attr} value={self.value} tag={self.tag} domain={self.domain})"
```

--------------------------------

### Modify configuration after object creation

Source: https://goose3.readthedocs.io/en/latest/quickstart.html

Update configuration properties directly on the Goose object instance.

```python
from goose3 import Goose

g = Goose()
g.config.browser_user_agent = 'Mozilla 5.0'
```

--------------------------------

### Define ImageDetails Class

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/image.html

The ImageDetails class provides getters and setters for basic image properties like dimensions and MIME type.

```python
class ImageDetails:
    def __init__(self):
        # the width of the image
        self.width = 0

        # height of the image
        self.height = 0

        # the mime_type of the image JPEG / PNG
        self.mime_type = None

    def get_width(self):
        return self.width

    def set_width(self, width):
        self.width = width

    def get_height(self):
        return self.height

    def set_height(self, height):
        self.height = height

    def get_mime_type(self):
        return self.mime_type

    def set_mime_type(self, mime_type):
        self.mime_type = mime_type
```

--------------------------------

### Configure strict mode

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Enables or disables strict mode, which determines whether exceptions are raised or swallowed.

```python
@strict.setter
    def strict(self, val: bool):
        """set the strict property"""
        self._strict = bool(val)
```

--------------------------------

### Extract article content

Source: https://goose3.readthedocs.io/en/latest/quickstart.html

Basic usage to extract text from a URL. Remember to close the Goose object after use.

```python
from goose3 import Goose

g = Goose()
article = g.extract(url='http://this-url.html')
print(article.cleaned_text)
g.close()
```

--------------------------------

### Access known publish date tags

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Property to retrieve the list of tags used to identify publication dates.

```python
@property
def known_publish_date_tags(self):
    """list: The tags to search to find the likely published date

    Note:
        Each entry must be a dictionary with the following keys: `attribute`, `value`, and `content`.
    """
    return self._known_publish_date_tags
```

--------------------------------

### extract

Source: https://goose3.readthedocs.io/en/latest/code.html

Extracts article content and metadata from a provided URL or raw HTML string.

```APIDOC
## extract(url=None, raw_html=None)

### Description
Parses the provided URL or raw HTML string to extract the main article content and associated metadata.

### Parameters
- **url** (str) - Optional - The URL to fetch and parse.
- **raw_html** (str) - Optional - The raw HTML string to parse.

### Returns
- **Article** - An object containing the extracted article content and metadata.
```

--------------------------------

### goose3.Article

Source: https://goose3.readthedocs.io/en/latest/_sources/code.rst.txt

The object returned after a successful extraction process containing the parsed article data.

```APIDOC
## class goose3.Article

### Description
Represents the result of the extraction process, containing fields such as title, cleaned text, meta information, and media objects.
```

--------------------------------

### Retrieve the active parser

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

Returns the parser class currently configured for extraction.

```python
[docs]
    def get_parser(self) -> Union[Parser, ParserSoup, Any]:
        """Retrieve the current parser class to use for extraction

        Returns:
            Parser: The parser to use"""
        return AVAILABLE_PARSERS[self.parser_class]
```

--------------------------------

### Goose.close()

Source: https://goose3.readthedocs.io/en/latest/code.html

Closes the Goose instance and releases resources.

```APIDOC
## Goose.close()

### Description
Closes the current Goose instance and cleans up associated resources.
```

--------------------------------

### Article Class Definition

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/article.html

The Article class initializes with various attributes to store extracted web page data and provides read-only properties for accessing these fields.

```python
[docs]
class Article:
    def __init__(self):
        self._title = ""
        self._cleaned_text = ""
        self._meta_description = ""
        self._meta_lang = ""
        self._meta_favicon = ""
        self._meta_keywords = ""
        self._meta_encoding = []
        self._canonical_link = ""
        self._domain = ""
        self._top_node = None
        self._top_node_raw_html = None
        self._top_image = None
        self._tags = []
        self._opengraph = {}
        self._tweets = []
        self._movies = []
        self._links = []
        self._authors = []
        self._final_url = ""
        self._link_hash = ""
        self._raw_html = ""
        self._schema = None
        self._doc = None
        self._raw_doc = None
        self._publish_date = None
        self._publish_datetime_utc = None
        self._additional_data = {}

    @property
    def title(self):
        """str: Title extracted from the HTML source

        Note:
            Read only"""
        return self._title

    @property
    def cleaned_text(self):
        """str: Cleaned text of the article without HTML tags; most commonly desired property

        Note:
            Read only"""
        return self._cleaned_text

    @property
    def meta_description(self):
        """str: Contents of the meta-description field from the HTML source

        Note:
            Read only"""
        return self._meta_description

    @property
    def meta_lang(self):
        """str: Contents of the meta-lang field from the HTML source

        Note:
            Read only"""
        return self._meta_lang

    @property
    def meta_favicon(self):
        """str: Contents of the meta-favicon field from the HTML source

        Note:
            Read only"""
        return self._meta_favicon

    @property
    def meta_keywords(self):
        """str: Contents of the meta-keywords field from the HTML source

        Note:
            Read only"""
        return self._meta_keywords

    @property
    def meta_encoding(self):
        """str: Contents of the encoding/charset field from the HTML source

        Note:
            Read only"""
        return self._meta_encoding

    @property
    def canonical_link(self):
        """str: The canonical link of the article if found in the meta data

        Note:
            Read only"""
        return self._canonical_link

    @property
    def domain(self):
        """str: Domain of the article parsed

        Note:
            Read only"""
        return self._domain

    @property
    def top_node(self):
        """etree: The top Element that is a candidate for the main body of the article

        Note:
            Read only"""
        return self._top_node

    @property
    def top_node_raw_html(self):
        """etree: The top html of Element that is a candidate for the main body of the article without cleaning

        Note:
            Read only"""
        return self._top_node_raw_html

    @property
    def top_image(self):
        """Image: The top image object that likely represents the article

        Returns:
            Image: See more information on the goose3.Image class
        Note:
            Read only"""
        return self._top_image

    @property
    def tags(self):
        """list(str): List of article tags (non-metadata tags)

        Note:
            Read only"""
        return self._tags

    @property
    def opengraph(self):
        """dict: All opengraph tag data

        Note:
            Read only"""
        return self._opengraph

    @property
    def tweets(self):
        """list(str): A listing of embeded tweets in the article

        Note:
            Read only"""
        return self._tweets

    @property
    def movies(self):
        """list(Video): A listing of all videos within the article such as YouTube or Vimeo

        Returns:
            list(Video):  See more information on the goose3.Video class
        Note:
            Read only"""
        return self._movies
```

--------------------------------

### Goose.extract

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3.html

Extracts article content and metadata from a provided URL or raw HTML string.

```APIDOC
## extract(url=None, raw_html=None)

### Description
Parses the provided URL or raw HTML string to extract the main article content and associated metadata.

### Parameters
- **url** (str) - Optional - The URL of the page to fetch and parse.
- **raw_html** (str) - Optional - The raw HTML string to parse.

### Returns
- **Article** - An object containing the extracted article content and metadata.
```

--------------------------------

### goose3.Goose

Source: https://goose3.readthedocs.io/en/latest/_sources/code.rst.txt

The primary class for interacting with the Goose3 library to extract content from URLs.

```APIDOC
## class goose3.Goose

### Description
The main entry point for the Goose3 library. Use this class to initialize the extractor and perform content extraction from web pages.

### Methods
- **extract(url=None, raw_html=None)**: Extracts content from a given URL or raw HTML string.
```

--------------------------------

### Define LocallyStoredImage Class

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/image.html

The LocallyStoredImage class tracks metadata for images that have been downloaded and stored locally.

```python
class LocallyStoredImage:
    def __init__(self, src="", local_filename="", link_hash="", size=int(0), file_extension="", height=0, width=0):
        self.src = src
        self.local_filename = local_filename
        self.link_hash = link_hash
        self.bytes = size
        self.file_extension = file_extension
        self.height = height
        self.width = width
```

--------------------------------

### shutdown_network

Source: https://goose3.readthedocs.io/en/latest/code.html

Explicitly closes the network connection.

```APIDOC
## shutdown_network()

### Description
Closes the network connection used by the Goose instance.
```

--------------------------------

### Define AuthorPattern class

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/configuration.html

The AuthorPattern class is used to define patterns for extracting author information from HTML elements.

```python
__slots__ = ["attr", "value", "content", "tag", "subpattern"]

def __init__(self, *, attr=None, value=None, content=None, tag=None, subpattern=None):
    if (not attr and not value) and not tag:
        raise Exception("`attr` and `value` must be set or `tag` must be set")
    self.attr = attr
    self.value = value
    self.content = content
    self.tag = tag
    self.subpattern = subpattern

def __repr__(self):
    if self.tag:
        return f"AuthorPattern(tag={self.tag}, attr={self.attr}, value={self.value})"
    res = (
        f"AuthorPattern(attr={self.attr}, value={self.value} "
        f"content={self.content} subpattern={self.subpattern})"
    )
    return res
```

--------------------------------

### Goose.shutdown_network()

Source: https://goose3.readthedocs.io/en/latest/code.html

Shuts down the network connection used by the Goose instance.

```APIDOC
## Goose.shutdown_network()

### Description
Shuts down the network connection utilized by the Goose instance for fetching web content.
```

--------------------------------

### Define Image Metadata Class

Source: https://goose3.readthedocs.io/en/latest/_modules/goose3/image.html

The Image class stores metadata about an extracted image, including its source, dimensions, and confidence score.

```python
class Image:
    def __init__(self):
        # holds the Element node of the image we think is top dog
        self._top_image_node = None

        # holds the src of the image
        self._src = ""

        # how confident are we in this image extraction?
        # the most images generally the less confident
        self._confidence_score = float(0.0)

        # Height of the image in pixels
        self._height = 0

        # width of the image in pixels
        self._width = 0

        # what kind of image extraction was used for this?
        # bestGuess, linkTag, openGraph tags?
        self._extraction_type = "NA"

        # stores how many bytes this image is.
        self._bytes = int(0)

    @property
    def top_image_node(self):
        """etree: The most likely top image element node

        Note:
            Read only"""
        return self._top_image_node

    @property
    def src(self):
        """str: Source URL for the image

        Note:
            Read only"""
        return self._src

    @property
    def confidence_score(self):
        """float: The confidence score that this is the main image

        Note:
            Read only"""
        return self._confidence_score

    @property
    def height(self):
        """int: The image height in pixels

        Note:
            Read only"""
        return self._height

    @property
    def width(self):
        """int: The image width in pixels

        Note:
            Read only"""
        return self._width

    @property
    def extraction_type(self):
        """str: The extraction type used

        Note:
            Read only"""
        return self._extraction_type

    @property
    def bytes(self):
        """int: The size of the image in bytes

        Note:
            Read only"""
        return self._bytes

    def get_src(self):
        return self.src
```
Tokens:
Raw