### Instantiate and Setup CrawlerConfig

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Get the singleton instance of CrawlerConfig and set up the configuration file path. This should be done once at the start of the application.

```python
cfg = CrawlerConfig.get_instance()
cfg.setup(<FILEPATH>)
```

--------------------------------

### Instantiate and Setup JsonConfig

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Obtain the singleton instance of JsonConfig and specify the path to the JSON configuration file. This setup step should be performed once.

```python
json = JsonConfig.get_instance()
json.setup(<FILEPATH>)
```

--------------------------------

### Install news-please using pip

Source: https://github.com/fhamborg/news-please/blob/master/README.md

Install the news-please library using pip. This command is for Python 3.8+.

```bash
pip install news-please
```

--------------------------------

### CLI Mode: Install and Run NewsPlease

Source: https://context7.com/fhamborg/news-please/llms.txt

Installs the news-please package using pip and runs it from the command line. Use the -c flag to specify a custom configuration directory.

```bash
# Install news-please
pip install news-please

# Run with default configuration (creates ~/news-please/config/)
news-please

# Run with custom config directory
news-please -c /path/to/config

# The config directory contains:
# - config.cfg: Main configuration file
# - sitelist.hjson: List of websites to crawl
```

--------------------------------

### Build and Upload Package to PyPI

Source: https://github.com/fhamborg/news-please/wiki/PyPI---How-to-upload-a-new-version

Execute these commands in the root directory after updating the version in setup.py. Ensure build and twine are installed.

```bash
python -m build
```

```bash
python -m twine upload dist/*
```

```bash
find dist -mindepth 1 -delete
```

```bash
find news_please.egg-info -mindepth 1 -delete
```

--------------------------------

### Install news-please via pip

Source: https://github.com/fhamborg/news-please/wiki/user-guide

Install the news-please package using pip. This command is for systems with Python 3.5+.

```bash
sudo pip install news-please
```

--------------------------------

### Configure News-Please Crawling

Source: https://context7.com/fhamborg/news-please/llms.txt

Set up filter hosts, date ranges, and callback functions for article extraction and WARC processing. Ensure the download directory exists before starting.

```python
FILTER_HOSTS = ['reuters.com', 'bbc.com']  # Empty list = all hosts
START_DATE = datetime.datetime(2024, 1, 1)
END_DATE = datetime.datetime(2024, 1, 31)
WARC_START_DATE = datetime.datetime(2024, 1, 1)
WARC_END_DATE = datetime.datetime(2024, 2, 1)

os.makedirs(DOWNLOAD_DIR_ARTICLES, exist_ok=True)

def on_article_extracted(article):
    """Callback for each successfully extracted article"""
    # Save to JSON file
    filename = hashlib.sha256(article.url.encode()).hexdigest()[:16]
    filepath = os.path.join(DOWNLOAD_DIR_ARTICLES, f"{filename}.json")

    with open(filepath, 'w', encoding='utf-8') as f:
        json.dump(article.__dict__, f, default=str, indent=2, ensure_ascii=False)

    print(f"Saved: {article.title[:50]}...")

def on_warc_completed(warc_path, passed, discarded, errors, total, processed):
    """Callback when a WARC file is fully processed"""
    print(f"WARC completed: {warc_path}")
    print(f"  Articles: passed={passed}, discarded={discarded}, errors={errors}")

# Start crawling
commoncrawl_crawler.crawl_from_commoncrawl(
    callback_on_article_extracted=on_article_extracted,
    callback_on_warc_completed=on_warc_completed,
    valid_hosts=FILTER_HOSTS,
    start_date=START_DATE,
    end_date=END_DATE,
    warc_files_start_date=WARC_START_DATE,
    warc_files_end_date=WARC_END_DATE,
    strict_date=True,
    reuse_previously_downloaded_files=True,
    local_download_dir_warc=DOWNLOAD_DIR_WARC,
    continue_after_error=True,
    number_of_extraction_processes=4,
    delete_warc_after_extraction=True,
    fetch_images=False,
    dry_run=False  # Set True to list files without processing
)
```

--------------------------------

### PostgreSQL Connection Configuration

Source: https://github.com/fhamborg/news-please/blob/master/README.md

Configuration parameters for connecting to a PostgreSQL database. Ensure psycopg2 is installed for production environments.

```ini
host = localhost
port = 5432
database = 'news-please'
# schema = 'news-please'
user = 'user'
password = 'password'
```

--------------------------------

### Website Object Configuration

Source: https://github.com/fhamborg/news-please/wiki/user-guide

Details on the parameters available for configuring the Website Object, which defines the start point, crawler, and heuristics for data collection.

```APIDOC
## Website Object Configuration

### Description
Configuration for the `base_urls` entries within the Website Object, allowing customization of crawling start points, crawlers, and heuristics.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **url** (string) - Required - The root URL to start crawling (e.g., "http://example.com").
- **crawler** (string) - Optional - The crawler used to collect data. Refer to [crawlers](crawlers-and-heuristics#markdown-header-crawlers) for implemented crawlers.
- **overwrite_heuristics** (dictionary, containing mixed types) - Optional - Overwrites default heuristics for article detection. Expects a dictionary with heuristic names as keys and conditions as values.
  - **bool**: Acceptable conditions are `True` and `False`. `False` disables the heuristic.
  - **string**: Acceptable conditions are simple strings like `"string_heuristic": "matched_value"`.
  - **float/int**: Acceptable conditions are strings with an equality operator (`<`, `>`, `<=`, `>=`, `=`) and a number (e.g., `"linked_headlines": "<=0.65"`). No spaces between operator and number.
- **pass_heuristics_condition** (string) - Optional - Overwrites the default boolean expression for evaluating heuristics. Can include heuristic names, boolean operators (`and`, `or`, `not`), and parentheses.
- **daemonize** (int) - Optional - If set, the crawler will start as a daemon. The value is the seconds to wait before scraping again. Only supported by `RSSCrawler`.
- **additional_rss_daemonize** (int) - Optional - If set, an additional `RSSCrawler` is spawned for the same target. The value is the seconds to wait before scraping again. Not supported by `RSSCrawler`.

### Request Example
```json
{
  "url": "http://example.com",
  "crawler": "ArticleNewsPlease",
  "overwrite_heuristics": {
    "og_type": True,
    "links_ratio": "<=0.5"
  },
  "pass_heuristics_condition": "og_type and links_ratio",
  "daemonize": 3600
}
```

### Response
#### Success Response (200)
This section is not detailed in the provided text.

#### Response Example
This section is not detailed in the provided text.
```

--------------------------------

### sitelist.hjson Configuration for Crawling

Source: https://context7.com/fhamborg/news-please/llms.txt

Example Hjson configuration file for defining websites to crawl. Specifies base URLs, crawler types (RssCrawler, SitemapCrawler, RecursiveCrawler, Download), and daemonization intervals.

```json
// sitelist.hjson - Define sites to crawl
{
  "base_urls": [
    {
      "url": "https://www.reuters.com/",
      "crawler": "RssCrawler",
      "daemonize": 3600
    },
    {
      "url": "https://www.bbc.com/news/",
      "crawler": "SitemapCrawler"
    },
    {
      "url": "https://www.theguardian.com/",
      "crawler": "RecursiveCrawler",
      "overwrite_heuristics": {
        "og_type": true,
        "linked_headlines": true
      }
    },
    {
      "crawler": "Download",
      "url": [
        "https://example.com/article1.html",
        "https://example.com/article2.html"
      ]
    }
  ]
}
```

--------------------------------

### Access JsonConfig Instance

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Retrieve the singleton instance of JsonConfig in any module after initial setup. This ensures consistent access to the parsed JSON configuration.

```python
json = JsonConfig.get_instance()
```

--------------------------------

### Get CrawlerConfig Configuration

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Retrieve a deep copy of the entire configuration as a 2D dictionary. Access specific options using section and option keys.

```python
config = cfg.config()
config[<section>][<option>] = <value>
```

--------------------------------

### Get CrawlerConfig Section

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Retrieve a deep copy of a specific configuration section as a 1D dictionary. This allows for focused access to related configuration options.

```python
section = cfg.section(<section>)
section[<option>] = <value>
```

--------------------------------

### Get Base URLs from JsonConfig

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Extract all URLs defined in the 'base_urls > url' section of the JSON configuration file. Returns the URLs as a Python list.

```python
print(json.get_url_array())
```

--------------------------------

### Configure Date Filter Settings

Source: https://github.com/fhamborg/news-please/wiki/pipeline

Configure the start and end dates for filtering articles and enable strict mode if articles without a date should be dropped. Dates can be None or in 'yyyy-mm-dd hh:mm:ss' format.

```ini
[DateFilter]
start_date = '1999-01-01 00:00:00'
end_date = '2999-12-31 00:00:00'

strict_mode = False
```

--------------------------------

### Get JsonConfig Configuration

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Retrieve a deep copy of the entire parsed JSON configuration. This provides access to all settings within the JSON file.

```python
json_config = json.config()
```

--------------------------------

### Extract Article from URL

Source: https://github.com/fhamborg/news-please/blob/master/README.md

Use this method to extract information from a single news article URL. Ensure the newsplease library is installed.

```python
from newsplease import NewsPlease
article = NewsPlease.from_url('https://www.nytimes.com/2017/02/23/us/politics/cpac-stephen-bannon-reince-priebus.html?hp')
print(article.title)
```

--------------------------------

### Get CrawlerConfig Option

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Retrieve a specific option's value after setting the current section. Ensure `set_section` is called before accessing an option.

```python
cfg.set_section(<section>)
option = cfg.option(<option>)
```

--------------------------------

### Crawl and extract articles from commoncrawl.org

Source: https://github.com/fhamborg/news-please/blob/master/README.md

To crawl and extract articles from the commoncrawl.org news archive, clone the news-please repository, adapt the configuration in the example script, and then execute it.

```python
python3 -m newsplease.examples.commoncrawl
```

--------------------------------

### CrawlerConfig Usage

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Demonstrates how to import and use the CrawlerConfig singleton class to set up and access configuration from a .cfg file.

```APIDOC
## CrawlerConfig Usage

### Description
This section details how to import and use the `CrawlerConfig` class, a singleton responsible for parsing configuration from `.cfg` files.

### Import
```python
from config import CrawlerConfig
```

### Initialization and Setup
Instantiate the singleton and set up the configuration file.

```python
# First instantiation (must be done once at the beginning)
cfg = CrawlerConfig.get_instance()
cfg.setup("<FILEPATH>")

# Further usage (in any file after the first instantiation)
cfg = CrawlerConfig.get_instance()
```

### Methods

*   **`get_instance()`**: Returns the singleton instance of `CrawlerConfig`.
*   **`setup(_filepath_)`**: Reads and parses the configuration file specified by `_filepath_`.
*   **`config()`**: Returns a deep copy of the entire configuration as a 2D dictionary.
    ```python
    config_data = cfg.config()
    # Accessing a value: config_data[<section>][<option>]
    ```
*   **`section(_section_)`**: Returns a copy of a specific section as a 1D dictionary.
    ```python
    section_data = cfg.section(<section>)
    # Accessing an option: section_data[<option>]
    ```
*   **`set_section(_section_)`**: Sets the current section for subsequent `option()` calls.
*   **`option(_option_)`**: Requires `set_section()` to be called first. Returns the value of a specific option within the currently set section.
    ```python
    cfg.set_section(<section>)
    option_value = cfg.option(<option>)
    ```
```

--------------------------------

### JsonConfig Usage

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Illustrates how to import and utilize the JsonConfig singleton class for parsing configuration from JSON files.

```APIDOC
## JsonConfig Usage

### Description
This section explains how to use the `JsonConfig` class, a singleton designed for parsing configuration from JSON files.

### Import
```python
from config import JsonConfig
```

### Initialization and Setup
Instantiate the singleton and set up the JSON configuration file.

```python
# First instantiation (must be done once at the beginning)
json_cfg = JsonConfig.get_instance()
json_cfg.setup("<FILEPATH>")

# Further usage (in any file after the first instantiation)
json_cfg = JsonConfig.get_instance()
```

### Methods

*   **`get_instance()`**: Returns the singleton instance of `JsonConfig`.
*   **`setup(_filepath_)`**: Reads and parses the JSON configuration file specified by `_filepath_`.
*   **`config()`**: Returns a deep copy of the entire parsed JSON configuration.
    ```python
    json_data = json_cfg.config()
    ```
*   **`load_json(_filepath_)`**: Loads a new JSON file, overwriting the current configuration. Generally not recommended for typical use.
    ```python
    json_cfg.load_json("../test.json")
    ```
*   **`get_url_array()`**: Retrieves all URLs from the `"base_urls" > "url"` section of the JSON file as a list.
    ```python
    urls = json_cfg.get_url_array()
    # Example output: ["http://examp.le", "http://te.st"]
    ```
```

--------------------------------

### Access Redis Storage Directly

Source: https://context7.com/fhamborg/news-please/llms.txt

Demonstrates how to interact with Redis storage for saving and retrieving articles. Requires Redis server to be running.

```python
from newsplease.pipeline.pipelines import RedisStorageClient, Collections

client = RedisStorageClient(host='localhost', port=6379, db=0)

# Get current version of an article
article = client.get_current_version('https://example.com/article')
if article:
    print(f"Title: {article['title']}")
    print(f"Version: {article['__version']}")

# Save an article
client.save_item(
    url='https://example.com/article',
    item={'title': 'Test', 'maintext': 'Content...'},
    collection=Collections.CurrentVersions,
    ttl=86400  # Expire after 24 hours
)
```

--------------------------------

### Run news-please from CLI

Source: https://github.com/fhamborg/news-please/wiki/user-guide

Execute news-please from the command line. Ensure your Elasticsearch server is running before execution.

```bash
news-please
```

--------------------------------

### Python Data Type Syntax in Config

Source: https://github.com/fhamborg/news-please/wiki/configuration

Demonstrates the correct Python syntax for boolean, dictionary, and string values in the configuration file to ensure proper parsing by ast.literal_eval.

```python
[test_section]

# Booleans
# (bools in python have to be uppercase first)
bool_string = true # This would become a string, because it is not uc-first.
bool_bool = True   # This would become a bool with value True.

# Dicts
dict_string = {"test_1" : True, "test_2" : true} # This would become a string, because of an ValueError: Malformed String. The reason is that test_2s true is written wrong.
dict_dict = {"test_1" : True, "test_2" : 1.1}    # This would become a dict where test_1 is bool (True) and test_2 is float (1.1)

# Strings
string_bool = True          # This would become a bool.
string_string = true        # This would become a string.
string_string_True = "True" # This would become a string as well. The quotation-marks will be stripped.
```

--------------------------------

### Import CrawlerConfig

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Import the CrawlerConfig class at the beginning of your program. It's a singleton and should only be instantiated once.

```python
from config import CrawlerConfig
```

--------------------------------

### Configure PostgreSQL Storage

Source: https://github.com/fhamborg/news-please/blob/master/README.md

Enable and configure PostgreSQL storage by adding PostgresqlStorage to ITEM_PIPELINES and adjusting database credentials in the configuration file.

```cfg
[Scrapy]
ITEM_PIPELINES = {
  'newsplease.pipeline.pipelines.ArticleMasterExtractor':100,
  'newsplease.pipeline.pipelines.PostgresqlStorage':350
}

[Postgresql]
```

--------------------------------

### Custom Configuration Path

Source: https://github.com/fhamborg/news-please/wiki/configuration

Specify a custom path for the configuration file using the -c parameter.

```bash
$ news-please -c /my/custom/path
```

```bash
$ news-please -c ~/somewhere/in/userdir
```

--------------------------------

### Import JsonConfig

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Import the JsonConfig class at the beginning of your program. It is a singleton class and should be instantiated only once.

```python
from config import JsonConfig
```

--------------------------------

### Redis Storage Configuration for News-Please

Source: https://github.com/fhamborg/news-please/blob/master/README.md

Configuration for enabling Redis storage in news-please. This includes adding the RedisStorage module to the item pipeline and setting connection details.

```cfg
[Scrapy]
ITEM_PIPELINES = {
  'newsplease.pipeline.pipelines.ArticleMasterExtractor':100,
  'newsplease.pipeline.pipelines.RedisStorage':350
}

[Redis]
host = localhost
port = 6379
db = 0

# You can add any redis connection parameter here
ssl_check_hostname = True
username = "news-please"
max_connections = 24
```

--------------------------------

### Run News-Please CLI

Source: https://github.com/fhamborg/news-please/blob/master/README.md

Execute the news-please crawler from the command line. Results are saved to the 'data' folder by default.

```bash
$ news-please
```

--------------------------------

### Load JSON Configuration File

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Manually load a JSON configuration file from a specified path. This method overwrites existing configuration and should typically not be used for standard operation.

```python
json.load_json("../test.json");
```

--------------------------------

### Configuration File Structure

Source: https://github.com/fhamborg/news-please/wiki/configuration

The config.cfg file is divided into sections with key-value pairs. Values are parsed using ast.literal_eval, requiring correct Python syntax for data types like booleans, dictionaries, and strings.

```ini
[section_name]

option_name = value
```

--------------------------------

### Configure Elasticsearch Storage

Source: https://github.com/fhamborg/news-please/blob/master/README.md

Enable and configure Elasticsearch storage by modifying the Scrapy ITEM_PIPELINES in config.cfg. Adjust host, port, and authentication settings as needed.

```cfg
[Scrapy]
ITEM_PIPELINES = {
  'newsplease.pipeline.pipelines.ArticleMasterExtractor':100,
  'newsplease.pipeline.pipelines.ElasticsearchStorage':350
}

[Elasticsearch]
host = localhost
port = 9200
...

# Credentials used  for authentication (supports CA-certificates):
use_ca_certificates = False           # True if authentification needs to be performed
ca_cert_path = '/path/to/cacert.pem'
client_cert_path = '/path/to/client_cert.pem'
client_key_path = '/path/to/client_key.pem'
username = 'root'
secret = 'password'
```

--------------------------------

### Run news-please from source

Source: https://github.com/fhamborg/news-please/wiki/user-guide

If you have checked out the source code, you can run news-please directly using the main Python script.

```python
python __main__.py
```

--------------------------------

### Configure Sitelist for Crawling

Source: https://github.com/fhamborg/news-please/wiki/user-guide

Define websites to be crawled using the 'base_urls' array in sitelist.hjson. Each entry specifies the URL, crawler type, and optional heuristic overrides.

```json
{
	 "base_urls" : [
		{
			"url": "http://www.faz.net/",
			"crawler": "RecursiveCrawler",
			"overwrite_heuristics": {
			  "meta_contains_article_keyword": true,
			  "og_type": true,
			  "linked_headlines": true,
			  "self_linked_headlines": false
			  },
			"pass_heuristics_condition": "meta_contains_article_keyword or (og_type and linked_headlines)"
		},
		{
			"url": "http://www.nytimes.com/",
			"crawler": "RssCrawler",
			"daemonize": 3600
		},
		...
		
	]
}
```

--------------------------------

### PostgreSQL Storage Configuration

Source: https://context7.com/fhamborg/news-please/llms.txt

Configure news-please to store articles in PostgreSQL with full versioning. This INI file specifies connection details and the Scrapy pipeline.

```ini
# config.cfg - PostgreSQL configuration
[Postgresql]
host = localhost
port = 5432
database = 'news-please'
user = 'newsplease'
password = 'secure_password'

[Scrapy]
ITEM_PIPELINES = {
    'newsplease.pipeline.pipelines.ArticleMasterExtractor': 100,
    'newsplease.pipeline.pipelines.PostgresqlStorage': 350
}
```

--------------------------------

### Direct URL Download Configuration

Source: https://github.com/fhamborg/news-please/wiki/user-guide

Configure news-please to download and extract specific articles directly. This involves listing URLs under the 'Download' crawler type.

```json
# Furthermore this is first of all the actual config file, but as default just filled with examples.
{
  # Every URL has to be in an array-object in "base_urls".
  # The same URL in combination with the same crawler may only appear once in this array.
  "base_urls" : [
    {
      "crawler": "Download",
      "url": [
        # Cubs win Championship ~03.11.2016
        "http://www.dailymail.co.uk/news/article-3899956/Chicago-Cubs-win-World-Series-epic-Game-7-showdown-Cleveland.html",
        "http://www.mirror.co.uk/sport/other-sports/american-sports/chicago-cubs-win-world-series-9185077",
        "https://www.theguardian.com/sport/2016/nov/03/world-series-game-7-chicago-cubs-cleveland-indians-mlb",
        "http://www.telegraph.co.uk/baseball/2016/11/03/chicago-cubs-break-108-year-curse-of-the-billy-goat-winning-worl/",
        "https://www.thesun.co.uk/sport/othersports/2106710/chicago-cubs-win-world-series-hillary-clinton-bill-murray-and-barack-obama-lead-celebrations-as-cubs-end-108-year-curse/",
        "http://www.bbc.com/sport/baseball/37857919"
      ],

      "overwrite_heuristics": {
        "meta_contains_article_keyword": true,
        "og_type": false,
        "linked_headlines": false,
        "self_linked_headlines": false
      }
    }

```

--------------------------------

### Access CrawlerConfig Instance

Source: https://github.com/fhamborg/news-please/wiki/config-parser

Retrieve the singleton instance of CrawlerConfig in any file after it has been initially set up. This ensures consistent access to configuration settings.

```python
cfg = CrawlerConfig.get_instance()
```

--------------------------------

### Redis Storage Configuration

Source: https://context7.com/fhamborg/news-please/llms.txt

Configure news-please to store articles in Redis with optional TTL and archive support. This INI file includes connection details and Scrapy pipeline settings.

```ini
# config.cfg - Redis configuration
[Redis]
host = localhost
port = 6379
db = 0
password =
username = news-please

# Optional settings
ssl = False
ssl_check_hostname = False
max_connections = 24
enable_archive = True
ttl = -1
dangerously_flush_db = False

[Scrapy]
ITEM_PIPELINES = {
    'newsplease.pipeline.pipelines.ArticleMasterExtractor': 100,
    'newsplease.pipeline.pipelines.RedisStorage': 350
}
```

--------------------------------

### PostgreSQL Storage Schema

Source: https://context7.com/fhamborg/news-please/llms.txt

SQL schema for creating 'CurrentVersions' and 'ArchiveVersions' tables in PostgreSQL for storing article data with full versioning.

```sql
-- init-postgresql-db.sql - Create required tables
CREATE TABLE IF NOT EXISTS CurrentVersions (
    id SERIAL PRIMARY KEY,
    date_modify TIMESTAMP,
    date_download TIMESTAMP,
    localpath TEXT,
    filename TEXT,
    source_domain TEXT,
    url TEXT UNIQUE,
    image_url TEXT,
    title TEXT,
    title_page TEXT,
    title_rss TEXT,
    maintext TEXT,
    description TEXT,
    date_publish TIMESTAMP,
    authors TEXT[],
    language VARCHAR(10),
    ancestor INTEGER,
    descendant INTEGER,
    version INTEGER
);

CREATE TABLE IF NOT EXISTS ArchiveVersions (
    id INTEGER,
    date_modify TIMESTAMP,
    date_download TIMESTAMP,
    localpath TEXT,
    filename TEXT,
    source_domain TEXT,
    url TEXT,
    image_url TEXT,
    title TEXT,
    title_page TEXT,
    title_rss TEXT,
    maintext TEXT,
    description TEXT,
    date_publish TIMESTAMP,
    authors TEXT[],
    language VARCHAR(10),
    ancestor INTEGER,
    descendant INTEGER,
    version INTEGER
);
```

--------------------------------

### Extract Articles from URL List File with NewsPlease

Source: https://context7.com/fhamborg/news-please/llms.txt

The `NewsPlease.from_file` function reads URLs from a text file, with one URL per line, and extracts articles from all of them. This simplifies batch processing of predefined URL lists by handling file input internally.

```python
from newsplease import NewsPlease
```

--------------------------------

### Query Articles from Elasticsearch

Source: https://context7.com/fhamborg/news-please/llms.txt

Connect to Elasticsearch and search for articles using keywords. Results are sorted by publication date in descending order, with a limit of 10.

```python
# Query articles from Elasticsearch
from elasticsearch import Elasticsearch

es = Elasticsearch(['http://localhost:9200'])

# Search for articles by keyword
results = es.search(
    index='news-please',
    body={
        'query': {
            'match': {
                'maintext': 'artificial intelligence'
            }
        },
        'sort': [{'date_publish': 'desc'}],
        'size': 10
    }
)

for hit in results['hits']['hits']:
    article = hit['_source']
    print(f"{article['date_publish']}: {article['title']}")
```

--------------------------------

### Elasticsearch Storage Configuration

Source: https://context7.com/fhamborg/news-please/llms.txt

Configure news-please to store articles in Elasticsearch with automatic versioning. This INI file sets connection details and the Scrapy pipeline for storage.

```ini
# config.cfg - Elasticsearch configuration
[Elasticsearch]
host = localhost
port = 9200
index_current = 'news-please'
index_archive = 'news-please-archive'

# Authentication (optional)
use_ca_certificates = False
ca_cert_path = /path/to/cacert.pem
client_cert_path = /path/to/client_cert.pem
client_key_path = /path/to/client_key.pem
username = 'elastic'
secret = 'password'

[Scrapy]
ITEM_PIPELINES = {
    'newsplease.pipeline.pipelines.ArticleMasterExtractor': 100,
    'newsplease.pipeline.pipelines.ElasticsearchStorage': 350
}
```

--------------------------------

### NewsArticle Object Attributes and Serialization

Source: https://context7.com/fhamborg/news-please/llms.txt

Demonstrates accessing attributes of a NewsArticle object and converting it to a dictionary for export. Use get_dict() for general dictionary conversion and get_serializable_dict() for dates as strings.

```python
from newsplease import NewsPlease

article = NewsPlease.from_url('https://example.com/news/article')

# All available attributes
print(f"title: {article.title}")              # Article headline
print(f"description: {article.description}")  # Lead paragraph/meta description
print(f"maintext: {article.maintext}")        # Full article text
print(f"authors: {article.authors}")          # List of author names
print(f"date_publish: {article.date_publish}")  # Publication datetime
print(f"date_download: {article.date_download}")  # Download datetime
print(f"date_modify: {article.date_modify}")  # Last modification datetime
print(f"language: {article.language}")        # Detected language code
print(f"image_url: {article.image_url}")      # Main article image URL
print(f"source_domain: {article.source_domain}")  # Domain name
print(f"url: {article.url}")                  # Original URL
print(f"filename: {article.filename}")        # Generated filename
print(f"title_page: {article.title_page}")    # HTML <title> tag
print(f"title_rss: {article.title_rss}")      # RSS feed title if available

# Convert to dictionary
article_dict = article.get_dict()

# Convert to serializable dictionary (dates as strings)
serializable = article.get_serializable_dict()

import json
print(json.dumps(serializable, indent=2))
```

--------------------------------

### NewsPlease.from_file - Extract from URL List File

Source: https://context7.com/fhamborg/news-please/llms.txt

Reads URLs from a text file (one URL per line) and extracts articles from all of them. Convenient for batch processing predefined URL lists without writing code to handle file I/O.

```APIDOC
## NewsPlease.from_file - Extract from URL List File

### Description
Reads URLs from a text file (one URL per line) and extracts articles from all of them. Convenient for batch processing predefined URL lists without writing code to handle file I/O.

### Method
POST

### Endpoint
/newsplease/from_file

### Parameters
#### Request Body
- **file_path** (string) - Required - The path to the text file containing URLs.
- **request_args** (object) - Optional - Dictionary of arguments to pass to the underlying HTTP request for each URL (e.g., timeout, headers).

### Request Example
```python
from newsplease import NewsPlease

# Assume 'urls.txt' contains one URL per line:
# https://www.nytimes.com/...
# https://www.bbc.com/...

articles = NewsPlease.from_file('urls.txt', request_args={"timeout": 10})

for url, article in articles.items():
    if article:
        print(f"Successfully extracted: {article.title}")
    else:
        print(f"Failed to extract: {url}")
```

### Response
#### Success Response (200)
- **articles** (object) - A dictionary where keys are the input URLs from the file and values are either Article objects or None if extraction failed for that URL.

#### Response Example
```json
{
  "https://www.nytimes.com/...": {
    "title": "NYT Article Title",
    "authors": ["Author NYT"],
    "date_publish": "2024-01-15",
    "language": "en",
    "maintext": "Text of NYT article...",
    "description": "Summary of NYT article.",
    "image_url": "http://example.com/nyt_img.jpg",
    "source_domain": "nytimes.com"
  },
  "https://www.bbc.com/...": null
}
```
```

--------------------------------

### News-Please Citation Format

Source: https://github.com/fhamborg/news-please/blob/master/README.md

BibTeX entry for citing the news-please project in academic work.

```bibtex
@InProceedings{Hamborg2017,
  author     = {Hamborg, Felix and Meuschke, Norman and Breitinger, Corinna and Gipp, Bela},
  title      = {news-please: A Generic News Crawler and Extractor},
  year       = {2017},
  booktitle  = {Proceedings of the 15th International Symposium of Information Science},
  location   = {Berlin},
  doi        = {10.5281/zenodo.4120316},
  pages      = {218--223},
  month      = {March}
}
```

--------------------------------

### Default Ignore File Extensions

Source: https://github.com/fhamborg/news-please/wiki/configuration

Specifies file extensions for URLs that should be ignored during recursive crawling. This prevents downloading non-article content like PDFs or images.

```regex
"(pdf)|(docx?)|(xlsx?)|(pptx?)|(epub)|(jpe?g)|(png)|(bmp)|(gif)|(tiff)|(webp)|(avi)|(mpe?g)|(mov)|(qt)|(webm)|(ogg)|(midi)|(mid)|(mp3)|(wav)|(zip)|(rar)|(exe)|(apk)|(css)"
```

--------------------------------

### Batch Extract Multiple Articles with NewsPlease

Source: https://context7.com/fhamborg/news-please/llms.txt

Utilize `NewsPlease.from_urls` for concurrent extraction of multiple articles, improving performance through parallel processing. The function returns a dictionary mapping URLs to their extracted article objects. Results can be iterated to process individual articles or exported to JSON files.

```python
from newsplease import NewsPlease

# Extract multiple articles in parallel
urls = [
    'https://www.reuters.com/world/article1',
    'https://www.theguardian.com/world/article2',
    'https://www.washingtonpost.com/politics/article3',
    'https://www.cnn.com/2024/01/15/tech/article4'
]

articles = NewsPlease.from_urls(urls, request_args={"timeout": 15})
```

```python
# Process results
for url, article in articles.items():
    if article:
        print(f"URL: {url}")
        print(f"  Title: {article.title}")
        print(f"  Date: {article.date_publish}")
        print(f"  Text length: {len(article.maintext) if article.maintext else 0} chars")
    else:
        print(f"Failed to extract: {url}")
```

```python
# Export all articles to JSON files
import json
import os

os.makedirs("articles", exist_ok=True)
for url, article in articles.items():
    if article:
        filename = f"articles/{article.source_domain}_{hash(url)}.json"
        with open(filename, "w") as f:
            json.dump(article.get_serializable_dict(), f, indent=2)
```

--------------------------------

### Crawler Fallbacks Configuration

Source: https://github.com/fhamborg/news-please/wiki/configuration

Defines fallback crawlers for when a site is incompatible with the primary crawler. The key is the failing crawler, and the value is the fallback crawler.

```python
fallbacks = {
    "RssCrawler": None,
    "RecursiveSitemapCrawler": "RecursiveCrawler",
    "SitemapCrawler": "RecursiveCrawler",
    "RecursiveCrawler": None,
    "Download": None
    }
```

--------------------------------

### Download Crawler URL Input

Source: https://github.com/fhamborg/news-please/wiki/crawlers-and-heuristics

The Download crawler accepts a list of URLs directly in the input configuration. This is primarily used for testing purposes when a specific page is required.

```json
"url":["http://example.com/1", "http://example.com/2"]
```

--------------------------------

### Extract Article from URL with NewsPlease

Source: https://context7.com/fhamborg/news-please/llms.txt

Use `NewsPlease.from_url` to crawl a single article from a URL and extract structured information. You can customize request arguments like timeout and headers, and control image fetching. The extracted article can be serialized to a dictionary for JSON export.

```python
from newsplease import NewsPlease

# Extract article from a single URL
article = NewsPlease.from_url('https://www.nytimes.com/2024/01/15/technology/ai-news.html')

# Access extracted information
print(f"Title: {article.title}")
print(f"Authors: {article.authors}")
print(f"Publish Date: {article.date_publish}")
print(f"Language: {article.language}")
print(f"Main Text: {article.maintext[:500]}...")
print(f"Description: {article.description}")
print(f"Image URL: {article.image_url}")
print(f"Source Domain: {article.source_domain}")
```

```python
# With custom request arguments (timeout, headers, etc.)
article = NewsPlease.from_url(
    'https://www.bbc.com/news/world-12345678',
    request_args={"timeout": 10, "headers": {"Accept-Language": "en-US"}},
    fetch_images=True
)
```

```python
# Export to JSON
import json
with open("article.json", "w") as f:
    json.dump(article.get_serializable_dict(), f, indent=2)
```

--------------------------------

### Extract Article from Raw HTML with NewsPlease

Source: https://context7.com/fhamborg/news-please/llms.txt

Use `NewsPlease.from_html` to extract article information directly from HTML content without making HTTP requests. Providing the original URL and download date can improve extraction accuracy. This method is useful when HTML is sourced from databases, caches, or custom crawlers.

```python
from newsplease import NewsPlease
import requests

# Fetch HTML yourself and extract
response = requests.get('https://example.com/news/article123')
html_content = response.text

article = NewsPlease.from_html(
    html_content,
    url='https://example.com/news/article123',  # Helps with date/title extraction
    download_date='2024-01-15 10:30:00',
    fetch_images=False  # Skip image downloading
)

print(f"Title: {article.title}")
print(f"Main text: {article.maintext}")
```

```python
# Process HTML from a database
def process_cached_articles(html_records):
    results = []
    for record in html_records:
        article = NewsPlease.from_html(
            record['html'],
            url=record['url'],
            download_date=record['crawl_date']
        )
        results.append({
            'id': record['id'],
            'title': article.title,
            'text': article.maintext,
            'authors': article.authors
        })
    return results
```

--------------------------------

### Extract Article from HTML

Source: https://github.com/fhamborg/news-please/blob/master/README.md

Extract information from raw HTML content. Optionally provide the original URL to improve date extraction accuracy.

```python
NewsPlease.from_html(html, url=None)
```

--------------------------------

### NewsPlease.from_urls - Batch Extract Multiple Articles

Source: https://context7.com/fhamborg/news-please/llms.txt

Crawls multiple articles concurrently using parallel processing for improved performance. Returns a dictionary mapping URLs to their extracted article objects. Ideal for batch processing news feeds or article lists.

```APIDOC
## NewsPlease.from_urls - Batch Extract Multiple Articles

### Description
Crawls multiple articles concurrently using parallel processing for improved performance. Returns a dictionary mapping URLs to their extracted article objects. Ideal for batch processing news feeds or article lists.

### Method
POST

### Endpoint
/newsplease/from_urls

### Parameters
#### Query Parameters
- **urls** (list of strings) - Required - A list of URLs of the news articles to extract.
- **request_args** (object) - Optional - Dictionary of arguments to pass to the underlying HTTP request for each URL (e.g., timeout, headers).

### Request Example
```python
from newsplease import NewsPlease

urls = [
    'https://www.reuters.com/world/article1',
    'https://www.theguardian.com/world/article2',
    'https://www.washingtonpost.com/politics/article3',
    'https://www.cnn.com/2024/01/15/tech/article4'
]

articles = NewsPlease.from_urls(urls, request_args={"timeout": 15})

for url, article in articles.items():
    if article:
        print(f"URL: {url}")
        print(f"  Title: {article.title}")
        print(f"  Date: {article.date_publish}")
        print(f"  Text length: {len(article.maintext) if article.maintext else 0} chars")
    else:
        print(f"Failed to extract: {url}")

import json
import os

os.makedirs("articles", exist_ok=True)
for url, article in articles.items():
    if article:
        filename = f"articles/{article.source_domain}_{hash(url)}.json"
        with open(filename, "w") as f:
            json.dump(article.get_serializable_dict(), f, indent=2)
```

### Response
#### Success Response (200)
- **articles** (object) - A dictionary where keys are the input URLs and values are either Article objects or None if extraction failed for that URL.

#### Response Example
```json
{
  "https://www.reuters.com/world/article1": {
    "title": "Article 1 Title",
    "authors": ["Author A"],
    "date_publish": "2024-01-15",
    "language": "en",
    "maintext": "Text of article 1...",
    "description": "Summary of article 1.",
    "image_url": "http://example.com/img1.jpg",
    "source_domain": "reuters.com"
  },
  "https://www.theguardian.com/world/article2": {
    "title": "Article 2 Title",
    "authors": ["Author B"],
    "date_publish": "2024-01-14",
    "language": "en",
    "maintext": "Text of article 2...",
    "description": "Summary of article 2.",
    "image_url": "http://example.com/img2.jpg",
    "source_domain": "theguardian.com"
  },
  "https://www.washingtonpost.com/politics/article3": null
}
```
```

--------------------------------

### Extract Articles from File

Source: https://github.com/fhamborg/news-please/blob/master/README.md

Extract information from a file containing a list of URLs, with each URL on a new line. Provide the path to the file.

```python
NewsPlease.from_file(path)
```

--------------------------------

### Extract Articles from a File

Source: https://context7.com/fhamborg/news-please/llms.txt

Reads a list of URLs from a file and extracts articles. Ensure the file contains one URL per line.

```python
urls_content = """
https://www.reuters.com/world/article1
https://www.bbc.com/news/article2
https://www.cnn.com/2024/01/article3
"""

with open("urls.txt", "w") as f:
    f.write(urls_content)

# Extract all articles from the file
articles = NewsPlease.from_file("urls.txt")

# Process results
for url, article in articles.items():
    if article and article.maintext:
        print(f"Extracted {len(article.maintext)} chars from {article.source_domain}")
```