General News Extractor

https://generalnewsextractor.readthedocs.io/zh-cn/latest/
Admin
A universal news website content extraction module that automatically extracts article title,...
Tokens:5,911
Snippets:56
Trust Score:9.6
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install GNE using Pipenv

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Use pipenv to install the General News Extractor library.

```bash
pipenv install gne
```

--------------------------------

### Install GeneralNewsExtractor

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/_sources/index.rst.txt

Install GNE using pip or pipenv. It is recommended to upgrade to the latest version.

```bash
pip install --upgrade gne
```

```bash
pipenv install gne
```

--------------------------------

### Install GNE using Pip

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Use pip to install or upgrade the General News Extractor library.

```bash
pip install --upgrade gne
```

--------------------------------

### Extract Main Content from HTML

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/_sources/index.rst.txt

Instantiate GeneralNewsExtractor and use the extract method to get structured data from HTML content. The input is HTML, and the output is a dictionary.

```python
from gne import GeneralNewsExtractor

extractor = GeneralNewsExtractor()
html = 'ç½‘ç«™æºä»£ç '
result = extractor.extract(html)
print(result)
```

--------------------------------

### Configuration File (.gne)

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Allows configuring GeneralNewsExtractor parameters via a YAML or JSON file. This provides an alternative to passing parameters directly to the `extract` method. Parameters in the `extract` method have higher precedence than those in the configuration file.

```APIDOC
## Configuration File (.gne)

### Description
Allows configuring GeneralNewsExtractor parameters via a YAML or JSON file. This provides an alternative to passing parameters directly to the `extract` method. Parameters in the `extract` method have higher precedence than those in the configuration file.

### Supported Formats
- YAML
- JSON

### YAML Configuration Example
```yaml
title:
    xpath: "//title/text()"
host: "https://www.xxx.com"
noise_node_list:
    - "//div[@class=\"comment-list\"]"
    - "//*[@style=\"display:none\"]"
body:
    xpath: "//div[@class=\"news-text\"]"
with_body_html: true
author:
    xpath: "//meta[@name=\"author\"]/@content"
publish_time:
    xpath: "//em[@id=\"publish_time\"]/text()"
use_visiable_info: false
```

### JSON Configuration Example
```json
{
    "title": {
        "xpath": "//title/text()"
    },
    "host": "https://www.xxx.com",
    "noise_node_list": [
        "//div[@class=\"comment-list\"]",
        "//*[@style=\"display:none\"]"
    ],
    "body": {
        "xpath": "//div[@class=\"news-text\"]"
    },
    "with_body_html": true,
    "author": {
        "xpath": "//meta[@name=\"author\"]/@content"
    },
    "publish_time": {
        "xpath": "//em[@id=\"publish_time\"]/text()"
    },
    "use_visiable_info": false
}
```

### Parameter Mapping
- `title`, `host`, `noise_node_list`, `with_body_html`, `author`, `publish_time`, `body`, `use_visiable_info` in the config file correspond to the parameters of the `extract` method.

### Precedence
- Parameters provided directly in the `extract` method call override the settings in the `.gne` configuration file.
```

--------------------------------

### JSON Configuration File for GNE

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Alternatively, configure GNE settings using a JSON file named '.gne' in the project root. This provides an alternative format for defining extraction parameters.

```json
{
    "title": {
        "xpath": "//title/text()"
    },
    "host": "https://www.xxx.com",
    "noise_node_list": ["//div[@class=\"comment-list\"]",
                        "//*[@style=\"display:none\"]"],
    "body": {
        "xpath": "//div[@class=\"news-text\"]"
    },
    "with_body_html": true,
    "author": {
        "xpath": "//meta[@name=\"author\"]/@content"
    },
    "publish_time": {
        "xpath": "//em[@id=\"publish_time\"]/text()"
    },
    "use_visiable_info": false
}
```

--------------------------------

### YAML Configuration File for GNE

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Configure GNE settings using a YAML file named '.gne' in the project root. This allows for persistent configuration of parameters like title XPath, host, and noise nodes.

```yaml
title:
    xpath: //title/text()
host: https://www.xxx.com
noise_node_list:
    - //div[@class="comment-list"]
    - //*[@style="display:none"]
body:
    xpath: //div[@class="news-text"]
with_body_html: true
author:
    xpath: //meta[@name="author"]/@content
publish_time:
    xpath: //em[@id="publish_time"]/text()
use_visiable_info: false
```

--------------------------------

### Basic HTML Extraction with GNE

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Instantiate GeneralNewsExtractor and extract content from an HTML string. The input must be the rendered HTML source code.

```python
from gne import GeneralNewsExtractor

extractor = GeneralNewsExtractor()
html = '网站源代码'
result = extractor.extract(html)
print(result)
```

--------------------------------

### Generate HTML with Visiability Information

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

JavaScript code to insert visibility and coordinate information into HTML elements. This is used by GNE for more accurate content extraction.

```javascript
function insert_visiability_info() {
    function get_body() {
        var body = document.getElementsByTagName('body')[0]
        return body
    }

    function insert_info(element) {
        is_visiable = element.offsetParent !== null
        element.setAttribute('is_visiable', is_visiable)
        if (is_visiable) {
            react = element.getBoundingClientRect()
            coordinate = JSON.stringify(react)
            element.setAttribute('coordinate', coordinate)
        }
    }

    function iter_node(node) {
        children = node.children
        insert_info(node)
        if (children.length !== 0) {
            for(const element of children) {
                iter_node(element)
            }
        }
    }

    function sizes() {
        let contentWidth = [...document.body.children].reduce(
          (a, el) => Math.max(a, el.getBoundingClientRect().right), 0) 
          - document.body.getBoundingClientRect().x;

        return {
          windowWidth:  document.documentElement.clientWidth,
          windowHeight: document.documentElement.clientHeight,
          pageWidth:    Math.min(document.body.scrollWidth, contentWidth),
          pageHeight:   document.body.scrollHeight,
          screenWidth:  window.screen.width,
          screenHeight: window.screen.height,
          pageX:        document.body.getBoundingClientRect().x,
          pageY:        document.body.getBoundingClientRect().y,
          screenX:     -window.screenX,
          screenY:     -window.screenY - (window.outerHeight-window.innerHeight),
        }
    }

    function insert_page_info() {
        page_info = sizes()
        node = document.createElement('meta')
        node.setAttribute('name', 'page_visiability_info')
        node.setAttribute('page_info', JSON.stringify(page_info))
        document.getElementsByTagName('head')[0].appendChild(node)
    }

    insert_page_info()
    body = get_body()
    iter_node(body)
}
insert_visiability_info()
```

--------------------------------

### Basic HTML Extraction with GeneralNewsExtractor

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Use this snippet to extract news information from HTML content. Ensure the input HTML is JavaScript-rendered. The 'extract' method processes the HTML and returns a result dictionary.

```python
from gne import GeneralNewsExtractor

extractor = GeneralNewsExtractor()
html = '你的目标网页正文'
result = extractor.extract(html)
print(result)
```

--------------------------------

### GNE GeneralNewsExtractor Class Definition

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/_sources/index.rst.txt

This defines the `GeneralNewsExtractor` class and its `extract` method. It outlines the parameters available for fine-tuning content extraction, including custom XPaths, host specification, and options for including HTML body and using visibility information.

```python
class GeneralNewsExtractor:
    def extract(self,
                html,
                title_xpath='',
                host='',
                author_xpath='',
                publish_time_xpath='',
                body_xpath='',
                noise_node_list=None,
                with_body_html=False,
                use_visiable_info=False)
```

--------------------------------

### Specify Title XPath for GNE Extraction

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/_sources/index.rst.txt

If automatic title detection fails, provide a custom XPath for the title using the `title_xpath` parameter. This is useful for non-standard title element structures.

```python
from gne import GeneralNewsExtractor

extractor = GeneralNewsExtractor()
html = 'ä½ çš„ç›®æ ‡ç½‘é¡µæ£æ–‡'
result = extractor.extract(html, title_xpath='//h5/text()')
print(result)
```

--------------------------------

### Enable Visiable Info for Higher Accuracy

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/_sources/index.rst.txt

Set `use_visiable_info=True` in the extract method to leverage special HTML attributes like `is_visiable` and `coordinate` for more accurate content extraction. This feature requires specific HTML structure generated by a JavaScript snippet.

```python
extractor.extract(html, use_visiable_info=True)
```

--------------------------------

### Extract News Content with GNE

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/_sources/index.rst.txt

Use this snippet to extract news content from HTML. Ensure the HTML is JavaScript-rendered. The `extract` method automatically attempts to find the title.

```python
from gne import GeneralNewsExtractor

extractor = GeneralNewsExtractor()
html = 'ä½ çš„ç›®æ ‡ç½‘é¡µæ£æ–‡'
result = extractor.extract(html)
print(result)
```

--------------------------------

### Generate Visiable Info Attributes in HTML

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/_sources/index.rst.txt

This JavaScript function iterates through DOM nodes, setting `is_visiable` and `coordinate` attributes based on element visibility and bounding box information. This is used by GNE for enhanced extraction accuracy.

```javascript
function insert_visiability_info() {
    function get_body() {
        var body = document.getElementsByTagName('body')[0]
        return body
    }

    function insert_info(element) {
        is_visiable = element.offsetParent !== null
        element.setAttribute('is_visiable', is_visiable)
        if (is_visiable) {
            react = element.getBoundingClientRect()
            coordinate = JSON.stringify(react)
            element.setAttribute('coordinate', coordinate)
        }
    }

    function iter_node(node) {
        children = node.children
        insert_info(node)
        if (children.length !== 0) {
            for(const element of children) {
                iter_node(element)
            }
        }
    }

    function sizes() {
        let contentWidth = [...document.body.children].reduce(
          (a, el) => Math.max(a, el.getBoundingClientRect().right), 0)
          - document.body.getBoundingClientRect().x;

        return {
          windowWidth:  document.documentElement.clientWidth,
          windowHeight: document.documentElement.clientHeight,
          pageWidth:    Math.min(document.body.scrollWidth, contentWidth),

```

--------------------------------

### Handling Noise Nodes to Improve Extraction

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Use the 'noise_node_list' parameter to exclude specific HTML elements, such as comment sections, that might interfere with accurate content extraction. Provide a list of XPath expressions for the nodes to be removed.

```python
result = extractor.extract(html, noise_node_list=['//div[@class="comment-list"]'])
```

--------------------------------

### Extracting Title with Custom XPath

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

If automatic title extraction fails, specify a custom XPath for the title element. This allows for more precise control over which element is identified as the title.

```python
from gne import GeneralNewsExtractor

extractor = GeneralNewsExtractor()
html = '你的目标网页正文'
result = extractor.extract(html, title_xpath='//h5/text()')
print(result)
```

--------------------------------

### GeneralNewsExtractor.extract

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Extracts news content from a given HTML source. It supports specifying XPaths for title, author, publish time, and body, as well as a list of noise nodes to exclude and options to include body HTML or use visual information.

```APIDOC
## GeneralNewsExtractor.extract

### Description
Extracts news content from a given HTML source. It supports specifying XPaths for title, author, publish time, and body, as well as a list of noise nodes to exclude and options to include body HTML or use visual information.

### Method Signature
```python
GeneralNewsExtractor.extract(
    html: str,
    title_xpath: str = '',
    host: str = '',
    author_xpath: str = '',
    publish_time_xpath: str = '',
    body_xpath: str = '',
    noise_node_list: list[str] = None,
    with_body_html: bool = False,
    use_visiable_info: bool = False
)
```

### Parameters
#### html (str) - Required
- The source code of the target website.

#### title_xpath (str) - Optional
- The XPath for the news title, used for targeted title extraction.

#### host (str) - Optional
- The domain name for images. If GNE extracts a relative image link like `/images/123.png`, it will prepend the host to form a complete URL, e.g., `https://www.kingname.info/images/123.png`.

#### author_xpath (str) - Optional
- The XPath for the article author, used for targeted author extraction.

#### publish_time_xpath (str) - Optional
- The XPath for the article publish time, used for targeted publish time extraction.

#### body_xpath (str) - Optional
- The XPath for the news body tag, used to narrow down the scope of body extraction and reduce noise.

#### noise_node_list (List[str]) - Optional
- A list of XPaths. Tags corresponding to the XPaths in this list will be directly removed during preprocessing to avoid interfering with news body extraction. For example, the XPath for the comment section on `guancha.cn` is `//div[@class="comment-list"]`.

#### with_body_html (bool) - Optional
- Defaults to `False`. If `False`, the extraction result does not include the HTML source code of the news body tag. When set to `True`, the result will include a `body_html` field containing the HTML source code of the news body tag.

#### use_visiable_info (bool) - Optional
- Indicates whether the HTML contains node coordinates and visual information.

### Request Example
```python
from gne import GeneralNewsExtractor

extractor = GeneralNewsExtractor()
html = 'Your target webpage content'
result = extractor.extract(html)
print(result)

# Example with custom title XPath
result_custom_title = extractor.extract(html, title_xpath='//h5/text()')
print(result_custom_title)

# Example with noise node list
result_with_noise_removal = extractor.extract(html, noise_node_list=['//div[@class="comment-list"]'])
print(result_with_noise_removal)
```

### Response
#### Success Response (200)
- The response is a dictionary containing extracted news information such as title, author, publish time, and body. If `with_body_html` is True, it also includes `body_html`.

#### Response Example
```json
{
  "title": "Extracted News Title",
  "author": "Author Name",
  "publish_time": "Publish Date",
  "body": "Extracted news content...",
  "body_html": "<p>HTML content of the news body...</p>"
}
```
```

--------------------------------

### ListExtractor Class Definition

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/_sources/index.rst.txt

Defines the ListExtractor class with its extract method. The 'element' parameter is a parsed DOM object, and 'feature' is an XPath or content string used to locate list items and extract their content.

```python
class ListExtractor:
    def extract(self, element: HtmlElement, feature)
```

--------------------------------

### Extract Content with Noise Node Exclusion

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Extract news content from HTML, excluding specified noise nodes using their XPath. This is useful for removing elements like comment sections.

```python
from gne import GeneralNewsExtractor
>>> html = '''经过渲染的网页 HTML 代码'''
>>> extractor = GeneralNewsExtractor()
>>> result = extractor.extract(html, noise_node_list=['//div[@class="comment-list"]'])
>>> print(result)
{"title": "xxxx", "publish_time": "2019-09-10 11:12:13", "author": "yyy", "content": "zzzz", "images": ["/xxx.jpg", "/yyy.png"]}
```

--------------------------------

### Extract List Page Content

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Use ListPageExtractor to extract content from list pages. Specify a feature using an XPath to identify elements within the list.

```python
from gne import ListPageExtractor
>>> html = '''经过渲染的网页 HTML 代码'''
>>> list_extractor = ListPageExtractor()
>>> result = list_extractor.extract(html, feature='列表中任意元素的 XPath")
>>> print(result)
```

--------------------------------

### Extract List Items from HTML

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/_sources/index.rst.txt

Use ListPageExtractor to extract items from a list page. Provide the HTML content and an XPath feature for the elements within the list.

```python
from gne import ListPageExtractor
html = '''ç»è¿‡æ¸²æŸ“çš„ç½‘é¡µ HTML ä»£ç '''
list_extractor = ListPageExtractor()
result = list_extractor.extract(html, feature='åˆ—è¡¨ä¸ä»»æ„å…ƒç´ çš„ XPath")
print(result)
```

--------------------------------

### ListExtractor.extract

Source: https://generalnewsextractor.readthedocs.io/zh-cn/latest/index.html

Extracts a list of items from a given HTML element. It requires an HtmlElement object and a feature (XPath or content) to identify the list items.

```APIDOC
## ListExtractor.extract

### Description
Extracts a list of items from a given HTML element. It requires an HtmlElement object and a feature (XPath or content) to identify the list items. This functionality is experimental and not recommended for production environments.

### Method Signature
```python
ListExtractor.extract(element: HtmlElement, feature: str)
```

### Parameters
#### element (HtmlElement) - Required
- A DOM tree object processed by `lxml.html.fromstring`.

#### feature (str) - Required
- An XPath or content of any item within the list. GNE will use this to automatically locate the list and return all its contents.

### Request Example
```python
from gne import ListExtractor
from lxml.html import fromstring

html_content = '<html><body><ul><li>Item 1</li><li>Item 2</li></ul></body></html>'
element = fromstring(html_content)

# Example using XPath to identify list items
list_items_xpath = '//li'
result_xpath = ListExtractor().extract(element, list_items_xpath)
print(result_xpath)

# Example using content to identify list items (if applicable)
# result_content = ListExtractor().extract(element, 'Item 1')
# print(result_content)
```

### Response
#### Success Response (200)
- A list of extracted items from the specified HTML element.

#### Response Example
```json
[
  "Item 1",
  "Item 2"
]
```
```
Tokens:
Raw