Trafilatura

https://trafilatura.readthedocs.io/en/latest/
Admin
Trafilatura is a Python package and command-line tool for gathering text from the web through...
Tokens:95,216
Snippets:965
Trust Score:8.3
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install database client and libraries

Source: https://trafilatura.readthedocs.io/en/latest/_sources/tutorial-epsilla.rst.txt

Install the pyepsilla client for interacting with the Epsilla database and langchain with sentence_transformers for embedding models.

```bash
pip install -U pyepsilla
pip install -U langchain sentence_transformers
```

--------------------------------

### Install Optional Dependencies

Source: https://trafilatura.readthedocs.io/en/latest/_sources/installation.rst.txt

Commands to install individual optional packages or the full suite of additional functionality.

```bash
$ pip install cchardet  # single package only
$ pip install trafilatura[all]  # all additional functionality
```

--------------------------------

### Build Documentation from Source

Source: https://trafilatura.readthedocs.io/en/latest/index.html

Commands to install dependencies and generate HTML documentation using Sphinx.

```bash
pip install -r requirements.txt
sphinx-build -b html . _build/
```

--------------------------------

### Install Linux GUI dependencies

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-gui.rst.txt

Install required GTK development libraries for Linux systems.

```bash
sudo apt install libgtk-3-dev
```

--------------------------------

### Install Trafilatura GUI

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-gui.rst.txt

Install the specific version of Trafilatura that supports the graphical interface.

```bash
pip3 install -U trafilatura==1.8.1[gui]
```

--------------------------------

### Launch Trafilatura GUI

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-gui.rst.txt

Start the graphical interface from the terminal.

```bash
trafilatura_gui
```

--------------------------------

### Initialize Crawl Parameters

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/spider.html

Sets up `CrawlParameters`, initializes the URL store with known and to-do URLs, and optionally fetches the starting page.

```python
def init_crawl(
    start: str,
    lang: Optional[str] = None,
    rules: Optional[RobotFileParser] = None,
    todo: Optional[List[str]] = None,
    known: Optional[List[str]] = None,
    prune_xpath: Optional[str] = None,
) -> CrawlParameters:
    """Initialize crawl by setting variables, copying values to the
    URL store and retrieving the initial page if the crawl starts."""
    params = CrawlParameters(start, lang, rules, prune_xpath)

    # todo: just known or also visited?
    URL_STORE.add_urls(urls=known or [], visited=True)
    URL_STORE.add_urls(urls=params.filter_list(todo))
    URL_STORE.store_rules(params.base, params.rules)

    # visiting the start page if necessary
    if not todo:
        URL_STORE.add_urls(urls=[params.start], visited=False)
        params = crawl_page(params, initial=True)
    else:
        params.update_metadata(URL_STORE)

    return params
```

--------------------------------

### Connect to Epsilla and Setup Database

Source: https://trafilatura.readthedocs.io/en/latest/tutorial-epsilla.html

Connect to a local Epsilla server, load or create a database, and define a table for storing document embeddings.

```python
from pyepsilla import vectordb
client = vectordb.Client(
    # replace with a production server if not running a local docker container
    host='localhost',
    port='8888'
)

status_code, response = client.load_db(
    db_name="TrafilaturaDB",
    db_path="/tmp/trafilatura_store"
)
print(response)

client.use_db(db_name="TrafilaturaDB")

# creates a table called Trafilatura
client.drop_table('Trafilatura')
client.create_table(
  table_name="Trafilatura",
  table_fields=[
    {"name": "ID", "dataType": "INT"},
    {"name": "Doc", "dataType": "STRING"},
    {"name": "Embedding", "dataType": "VECTOR_FLOAT", "dimensions": 384}
  ]
)

```

--------------------------------

### Install Python Dependencies

Source: https://trafilatura.readthedocs.io/en/latest/tutorial-epsilla.html

Install the necessary Python packages for interacting with Epsilla and using embedding models.

```bash
$ pip install -U pyepsilla

```

```bash
$ pip install -U langchain sentence_transformers

```

--------------------------------

### Check Python Version

Source: https://trafilatura.readthedocs.io/en/latest/_sources/installation.rst.txt

Verify that a compatible version of Python is installed on your system.

```bash
$ python3 --version  # python can also work
Python 3.10.12       # version 3.6 or higher is fine
```

--------------------------------

### Install Trafilatura using Pip

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-r.rst.txt

Install the Trafilatura Python package using the pip package manager in your terminal.

```bash
$ pip install trafilatura
```

--------------------------------

### Initialize Crawl Parameters

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/spider.html

Sets up parameters for a focused crawl, including base URL, language, and robots.txt rules. Raises ValueError if the start URL is invalid.

```python
class CrawlParameters:
    """Store necessary information to manage a focused crawl."""
    __slots__ = ["start", "base", "lang", "rules", "ref", "i", "known_num", "is_on", "prune_xpath"]

    def __init__(
        self,
        start: str,
        lang: Optional[str] = None,
        rules: Optional[RobotFileParser] = None,
        prune_xpath: Optional[str] = None,
    ) -> None:
        self.start: str = start
        self.base: str = self._get_base_url(start)
        self.ref: str = self._get_reference(start)
        self.lang: Optional[str] = lang
        self.rules: Optional[RobotFileParser] = rules or get_rules(self.base)
        self.i: int = 0
        self.known_num: int = 0
        self.is_on: bool = True
        self.prune_xpath: Optional[str] = prune_xpath

    def _get_base_url(self, start: str) -> str:
        """Set reference domain for the crawl."""
        base: str = get_base_url(start)
        if not base:
            raise ValueError(f"cannot start crawl: {start}")
        return base

    def _get_reference(self, start: str) -> str:
        """Determine the reference URL."""
        return start.rsplit("/", 1)[0] if start.count("/") >= 3 else start
```

--------------------------------

### Start Epsilla Docker container

Source: https://trafilatura.readthedocs.io/en/latest/_sources/tutorial-epsilla.rst.txt

Use this command to pull the Epsilla vector database Docker image and run it locally on port 8888.

```bash
docker pull epsilla/vectordb
docker run --pull=always -d -p 8888:8888 epsilla/vectordb
```

--------------------------------

### Trafilatura CLI Options - Navigation

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-cli.rst.txt

Examples of navigation options for Trafilatura, including feed discovery, sitemap crawling, and website exploration.

```bash
--feed [FEED]
--sitemap [SITEMAP]
--crawl [CRAWL]
--explore [EXPLORE]
--probe [PROBE]
--archived
--url-filter URL_FILTER [URL_FILTER ...]
```

--------------------------------

### Basic HTML Extraction with Python

Source: https://trafilatura.readthedocs.io/en/latest/_sources/quickstart.rst.txt

Use this for basic text extraction from a URL. Ensure trafilatura is installed and imported.

```python
from trafilatura import fetch_url, extract

downloaded = fetch_url('https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/')
result = extract(downloaded)
print(result)
```

--------------------------------

### XML Generation and Validation Setup

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/xml.html

Imports and constants for XML processing, including TEI schema definitions, valid tags, attributes, and formatting rules. Used for generating and validating XML output.

```python
# pylint:disable-msg=E0611,I1101
"""
All functions related to XML generation, processing and validation.
"""

import csv
import logging

from html import unescape
from importlib.metadata import version
from io import StringIO
from json import dumps as json_dumps
from pathlib import Path
from typing import List, Optional

from lxml.etree import (_Element, Element, SubElement, XMLParser,
                        fromstring, tostring, DTD)

from .settings import Document, Extractor
from .utils import sanitize, sanitize_tree, text_chars_test


LOGGER = logging.getLogger(__name__)
PKG_VERSION = version("trafilatura")

# validation
TEI_SCHEMA = str(Path(__file__).parent / "data" / "tei_corpus.dtd")
TEI_VALID_TAGS = {'ab', 'body', 'cell', 'code', 'del', 'div', 'graphic', 'head', 'hi', 
                  'item', 'lb', 'list', 'p', 'quote', 'ref', 'row', 'table'}
TEI_VALID_ATTRS = {'rend', 'rendition', 'role', 'target', 'type'}
TEI_DTD = None  # to be downloaded later if necessary
TEI_REMOVE_TAIL = {"ab", "p"}
TEI_DIV_SIBLINGS = {"p", "list", "table", "quote", "ab"}

CONTROL_PARSER = XMLParser(remove_blank_text=True)

NEWLINE_ELEMS = {'code', 'graphic', 'head', 'lb', 'list', 'p', 'quote', 'row', 'table'}
SPECIAL_FORMATTING = {'del', 'head', 'hi', 'ref'}
WITH_ATTRIBUTES = {'cell', 'row', 'del', 'graphic', 'head', 'hi', 'item', 'list', 'ref'}
NESTING_WHITELIST = {"cell", "figure", "item", "note", "quote"}

META_ATTRIBUTES = [
    'sitename', 'title', 'author', 'date', 'url', 'hostname',
    'description', 'categories', 'tags', 'license', 'id',
    'fingerprint', 'language'
]

HI_FORMATTING = {'#b': '**', '#i': '*', '#u': '__', '#t': '`'}

MAX_TABLE_WIDTH = 1000

```

--------------------------------

### Initialize a focused crawler in Python

Source: https://trafilatura.readthedocs.io/en/latest/_sources/crawls.rst.txt

Use the focused_crawler function to start a crawl from a homepage URL. Customize the crawl depth and scope using parameters like max_seen_urls and max_known_urls.

```python
>>> from trafilatura.spider import focused_crawler

# perform the first iteration (will not work with this website, there are no internal links)
>>> to_visit, known_links = focused_crawler("https://example.org", max_seen_urls=1)
```

--------------------------------

### Connect to Epsilla and set up database

Source: https://trafilatura.readthedocs.io/en/latest/_sources/tutorial-epsilla.rst.txt

Connect to a local Epsilla server, load or create a database, and define a table schema for storing documents and their embeddings.

```python
from pyepsilla import vectordb
client = vectordb.Client(
    # replace with a production server if not running a local docker container
    host='localhost',
    port='8888'
)

status_code, response = client.load_db(
    db_name="TrafilaturaDB",
    db_path="/tmp/trafilatura_store"
)
print(response)

client.use_db(db_name="TrafilaturaDB")

# creates a table called Trafilatura
client.drop_table('Trafilatura')
client.create_table(
  table_name="Trafilatura",
  table_fields=[
    {"name": "ID", "dataType": "INT"},
    {"name": "Doc", "dataType": "STRING"},
    {"name": "Embedding", "dataType": "VECTOR_FLOAT", "dimensions": 384}
  ]
)
```

--------------------------------

### Install and Load Reticulate in R

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-r.rst.txt

Install the reticulate package from CRAN and load it into your R session to enable Python integration.

```R
> install.packages("reticulate")
> library(reticulate)
```

--------------------------------

### Sample URLs by domain via command-line

Source: https://trafilatura.readthedocs.io/en/latest/_sources/url-management.rst.txt

Use the --sample and --samplesize flags to perform domain-based sampling on a file of URLs.

```bash
$ courlan --inputfile urls.txt --outputfile samples-urls.txt --sample --samplesize 50
```

--------------------------------

### Install Trafilatura using Reticulate

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-r.rst.txt

Install the Trafilatura Python package within your R session using the reticulate::py_install() function.

```R
> library(reticulate)
> py_install("trafilatura")
```

--------------------------------

### Run GUI from source

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-gui.rst.txt

Alternative method to launch the interface by running the script directly from the cloned repository.

```bash
python trafilatura_gui/interface.py
```

--------------------------------

### Load custom configuration files in Python

Source: https://trafilatura.readthedocs.io/en/latest/settings.html

Shows how to load an external configuration file and apply it to the extraction process.

```python
# load the required functions
>>> from trafilatura import extract
>>> from trafilatura.settings import use_config

# load the new settings by providing a file name
>>> newconfig = use_config("myfile.cfg")

# use with a previously downloaded document
>>> extract(downloaded, config=newconfig)

# provide a file name directly (can be slower)
>>> extract(downloaded, settingsfile="myfile.cfg")
```

--------------------------------

### Download and Process Web Documents via CLI

Source: https://trafilatura.readthedocs.io/en/latest/_sources/tutorial0.rst.txt

Use wget to download files and the trafilatura command to convert them into XML-TEI format. Ensure the input directory contains the downloaded HTML files before running the processing command.

```bash
# download if necessary
$ wget --directory-prefix=download/ --wait 5 --input-file=mylist.txt
# process a directory with archived HTML files
$ trafilatura --input-dir download/ --output-dir corpus/ --xmltei --no-comments
```

--------------------------------

### Update and Reinstall Trafilatura

Source: https://trafilatura.readthedocs.io/en/latest/_sources/installation.rst.txt

Commands to ensure the latest version is installed or to force a reinstall from the source repository.

```bash
# to make sure you have the latest version
$ pip install --upgrade trafilatura
# latest available code base
$ pip install --force-reinstall -U git+https://github.com/adbar/trafilatura
```

--------------------------------

### Check Python Version

Source: https://trafilatura.readthedocs.io/en/latest/usage-r.html

Verify that a compatible version of Python (3.6 or higher) is installed and accessible from your system's terminal.

```bash
$ python3 --version
Python 3.10.12 # version 3.6 or higher is fine

```

--------------------------------

### Perform Bare Extraction

Source: https://trafilatura.readthedocs.io/en/latest/usage-python.html

Demonstrates basic extraction workflows including handling metadata requirements.

```python
>>> url = "https://web.archive.org/web/20210613232513/https://www.thecanary.co/feature/2021/05/19/another-by-election-headache-is-incoming-for-keir-starmer/"
>>> downloaded = fetch_url(url)

# content discarded since necessary metadata couldn't be extracted
>>> bare_extraction(downloaded, only_with_metadata=True)
>>>

# date found in URL, extraction successful
>>> bare_extraction(downloaded, only_with_metadata=True, url=url)
```

--------------------------------

### Run downloads via command-line

Source: https://trafilatura.readthedocs.io/en/latest/downloads.html

Use the trafilatura CLI to process URLs from a file with automatic threading and throttling.

```bash
# basic output as raw text with backup directory
$ trafilatura -i list.txt -o txtfiles/ --backup-dir htmlbackup/
```

--------------------------------

### Filter Crawl List

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/spider.html

Prepares the list of URLs to visit, excluding the start URL and ensuring URLs belong to the same reference domain.

```python
def filter_list(self, todo: Optional[List[str]]) -> List[str]:
    """Prepare the todo list, excluding invalid URLs."""
    if not todo:
        return []
    return [u for u in todo if u != self.start and self.ref in u]
```

--------------------------------

### Sample URLs by domain with Python

Source: https://trafilatura.readthedocs.io/en/latest/_sources/url-management.rst.txt

Use sample_urls to select a subset of URLs, balancing representation across different domains.

```python
>>> from courlan import sample_urls
>>> my_urls = ['…', '…', '…', ]  # etc.
>>> my_sample = sample_urls(my_urls, 50)
# optional: exclude_min=None, exclude_max=None, strict=False, verbose=False
```

--------------------------------

### Sample URLs by Domain with Python

Source: https://trafilatura.readthedocs.io/en/latest/url-management.html

Use the `sample_urls` function to select a subset of URLs, controlling the number of URLs from each domain to prevent overrepresentation.

```python
from courlan import sample_urls
my_urls = ['…', '…', '…', ]  # etc.
my_sample = sample_urls(my_urls, 50)
# optional: exclude_min=None, exclude_max=None, strict=False, verbose=False
```

--------------------------------

### Trafilatura CLI Usage Help

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-cli.rst.txt

Display comprehensive help information for the Trafilatura command-line interface, including all available arguments and their descriptions.

```bash
trafilatura [-h] [-i INPUTFILE | --input-dir INPUTDIR | -u URL]
                   [--parallel PARALLEL] [-b BLACKLIST] [--list]
                   [-o OUTPUTDIR] [--backup-dir BACKUP_DIR] [--keep-dirs]
                   [--feed [FEED] | --sitemap [SITEMAP] | --crawl [CRAWL] |
                   --explore [EXPLORE] | --probe [PROBE]] [--archived]
                   [--url-filter URL_FILTER [URL_FILTER ...]] [-f]
                   [--formatting] [--links] [--images] [--no-comments]
                   [--no-tables] [--only-with-metadata] [--with-metadata]
                   [--target-language TARGET_LANGUAGE] [--deduplicate]
                   [--config-file CONFIG_FILE] [--precision] [--recall]
                   [--output-format {csv,json,html,markdown,txt,xml,xmltei} |
                   --csv | --html | --json | --markdown | --xml | --xmltei]
                   [--validate-tei] [-v] [--version]
```

--------------------------------

### Trafilatura CLI Options - Extraction

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-cli.rst.txt

Examples of extraction customization options for Trafilatura, controlling formatting, links, images, and exclusion of comments or tables.

```bash
-f, --fast
--formatting
--links
--images
--no-comments
--no-tables
```

--------------------------------

### Configure urllib3 download pool

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/downloads.html

Creates a download pool using urllib3 or SOCKSProxyManager based on environment settings.

```python
def create_pool(**args: Any) -> Union[urllib3.PoolManager, Any]:
    "Configure urllib3 download pool according to user-defined settings."
    manager_class = SOCKSProxyManager if PROXY_URL else urllib3.PoolManager
    manager_args = {"proxy_url": PROXY_URL} if PROXY_URL else {}
    manager_args["num_pools"] = 50  # type: ignore[assignment]
    return manager_class(**manager_args, **args)  # type: ignore[arg-type]
```

--------------------------------

### Extract data using Python

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-api.rst.txt

Uses the requests library to send a POST request to the API. Ensure the requests library is installed in your Python environment.

```python
import requests

url = "https://trafilatura.mooo.com/extract-demo"

payload = {
	    "url": "https://example.org",
	    "args": { "output_format": "xml" }
    }
headers = {
	    "content-type": "application/json",
    }

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

--------------------------------

### Import Compression Libraries

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/utils.html

Imports optional libraries for response compression like gzip, zlib, brotli, and zstandard. Sets flags to indicate availability.

```python
try:
    import gzip
    HAS_GZIP = True
except ImportError:
    HAS_GZIP = False
```

```python
try:
    import zlib
    HAS_ZLIB = True
except ImportError:
    HAS_ZLIB = False
```

```python
try:
    import brotli  # type: ignore
    HAS_BROTLI = True
except ImportError:
    HAS_BROTLI = False
```

```python
try:
    import zstandard
    HAS_ZSTD = True
except ImportError:
    HAS_ZSTD = False
```

--------------------------------

### Discover links via sitemap or feed

Source: https://trafilatura.readthedocs.io/en/latest/_sources/tutorial0.rst.txt

Use the --sitemap or --feed options to search for links and list them without immediate processing.

```bash
$ trafilatura --sitemap "https://www.sitemaps.org/" --list
```

```bash
$ trafilatura --feed "https://www.dwds.de/" --list
```

--------------------------------

### Discover Feeds from Homepage

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-cli.rst.txt

Automatically detect and list web feeds starting from a given homepage URL. This is useful for discovering new content sources.

```bash
# automatically detecting feeds starting from the homepage
$ trafilatura --feed "https://www.dwds.de/" --list
```

--------------------------------

### Produce TEI file with validation

Source: https://trafilatura.readthedocs.io/en/latest/_sources/tutorial2.rst.txt

Use the `extract` function with `output_format='xmltei'` and `tei_validation=True` to generate and validate a TEI-compliant XML file from a URL. Ensure Trafilatura is installed.

```python
# load the necessary components
from trafilatura import fetch_url, extract

# download a file
downloaded = fetch_url('https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/')

# extract information as XML TEI and validate the result
result = extract(downloaded, output_format='xmltei', tei_validation=True)
```

--------------------------------

### Import External Libraries for Trafilatura

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/external.html

Imports necessary modules from third-party libraries like 'justext' and 'lxml' for text classification and HTML manipulation. Ensure these libraries are installed.

```python
# pylint:disable-msg=E0611,I1101
"""
Functions grounding on third-party software.
"""

import logging

from typing import Any, Tuple

# third-party
from justext.core import ParagraphMaker, classify_paragraphs, revise_paragraph_classification  # type: ignore
from justext.utils import get_stoplist, get_stoplists  # type: ignore
from lxml.etree import _Element, Element, strip_tags, tostring
from lxml.html import HtmlElement
```

--------------------------------

### Classify Text Language

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/utils.html

Identifies the language of a given text using an external component (py3langid). It prioritizes longer text for classification. Returns None if the language detector is not installed.

```python
def language_classifier(temp_text: str, temp_comments: str) -> Optional[str]:
    '''Run external component (if installed) for language identification'''
    if LANGID_FLAG is True:
        result, _ = (
            py3langid.classify(temp_text)
            if len(temp_text) > len(temp_comments)
            else py3langid.classify(temp_comments)
        )
    else:  # pragma: no cover
        LOGGER.warning('Language detector not installed, skipping detection')
        result = None
    return result  # type: ignore[no-any-return]
```

--------------------------------

### Load Sitemaps Module

Source: https://trafilatura.readthedocs.io/en/latest/usage-python.html

Initializes the sitemaps module for processing XML sitemaps.

```python
# load sitemaps module
>>> from trafilatura import sitemaps
```

--------------------------------

### Execute multi-threaded downloads with throttling

Source: https://trafilatura.readthedocs.io/en/latest/_sources/downloads.rst.txt

Utilize buffered_downloads for parallel processing with domain-aware throttling to improve efficiency.

```python
from trafilatura.downloads import add_to_compressed_dict, buffered_downloads, load_download_buffer

# list of URLs
mylist = ['https://www.example.org', 'https://www.httpbin.org/html']
# number of threads to use
threads = 4

# converted the input list to an internal format
url_store = add_to_compressed_dict(mylist)
# processing loop
while url_store.done is False:
    bufferlist, url_store = load_download_buffer(url_store, sleep_time=5)
    # process downloads
    for url, result in buffered_downloads(bufferlist, threads):
        # do something here
        print(url)
        print(result)
```

--------------------------------

### Extract text from URL using Trafilatura in Python

Source: https://trafilatura.readthedocs.io/en/latest/index.html

This Python snippet demonstrates how to download content from a URL and extract the main text using the trafilatura library. Ensure the library is installed before running.

```python
import trafilatura
downloaded = trafilatura.fetch_url('https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/')
trafilatura.extract(downloaded)
# outputs main content and comments as plain text ...
```

--------------------------------

### Fetch URL and Extract Text with Trafilatura (Python)

Source: https://trafilatura.readthedocs.io/en/latest/_sources/index.rst.txt

Use this snippet to download content from a URL and then extract the main text and comments using Trafilatura in Python. Ensure the trafilatura library is installed.

```python
import trafilatura
downloaded = trafilatura.fetch_url('https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/')
trafilatura.extract(downloaded)
```

--------------------------------

### Process a list of links via CLI

Source: https://trafilatura.readthedocs.io/en/latest/usage-cli.html

Use input files to process multiple URLs and specify output directories for raw text or XML results.

```bash
$ trafilatura -i list.txt -o txtfiles/              # output as raw text
$ trafilatura --xml -i list.txt -o xmlfiles/        # output in XML format
```

```bash
$ trafilatura --input-file links.txt --output-dir converted/ --backup-dir html-sources/ --xml
```

--------------------------------

### Navigation and Crawling Options

Source: https://trafilatura.readthedocs.io/en/latest/usage-cli.html

Lists command-line flags for web crawling, feed discovery, and URL filtering.

```bash
--feed [FEED]         look for feeds and/or pass a feed URL as input
--sitemap [SITEMAP]   look for sitemaps for the given website and/or enter a sitemap URL
--crawl [CRAWL]       crawl a fixed number of pages within a website starting from the given URL
--explore [EXPLORE]   explore the given websites (combination of sitemap and crawl)
--probe [PROBE]       probe for extractable content (works best with target language)
--archived            try to fetch URLs from the Internet Archive if downloads fail
--url-filter URL_FILTER [URL_FILTER ...] only process/output URLs containing these patterns (space-separated strings)
```

--------------------------------

### Set target language for extraction

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-python.rst.txt

Specify the target language using 2-letter ISO 639-1 codes. If the detected language does not match, no output will be generated. This feature requires the language identification component to be installed.

```python
result = extract(downloaded, url, target_language="de")
```

--------------------------------

### Crawl website via CLI

Source: https://trafilatura.readthedocs.io/en/latest/_sources/crawls.rst.txt

Execute a crawl on a specific website using the command-line interface.

```bash
$ trafilatura --crawl "https://www.example.org" > links.txt
```

--------------------------------

### Decompress Compressed Files

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/utils.html

Attempts to decompress binary content using a cascade of installed packages like gzip, zstandard, brotli, and zlib. It prioritizes magic numbers for identification and handles potential decompression errors.

```python
def handle_compressed_file(filecontent: bytes) -> bytes:
    """
    Don't trust response headers and try to decompress a binary string
    with a cascade of installed packages. Use magic numbers when available.
    """
    if not isinstance(filecontent, bytes):
        return filecontent

    # source: https://stackoverflow.com/questions/3703276/how-to-tell-if-a-file-is-gzip-compressed
    if HAS_GZIP and filecontent[:3] == b"\x1f\x8b\x08":
        try:
            return gzip.decompress(filecontent)
        except Exception:  # EOFError, OSError, gzip.BadGzipFile
            LOGGER.warning("invalid GZ file")
    # try zstandard
    if HAS_ZSTD and filecontent[:4] == b"\x28\xb5\x2f\xfd":
        try:
            return zstandard.decompress(filecontent)  # max_output_size=???
        except zstandard.ZstdError:
            LOGGER.warning("invalid ZSTD file")
    # try brotli
    if HAS_BROTLI:
        try:
            return brotli.decompress(filecontent)  # type: ignore[no-any-return]
        except brotli.error:
            pass  # logging.debug('invalid Brotli file')
    # try zlib/deflate
    if HAS_ZLIB:
        try:
            return zlib.decompress(filecontent)
        except zlib.error:
            pass

    # return content unchanged if decompression failed
    return filecontent
```

--------------------------------

### Send Robust HTTP Request with SSL Handling

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/downloads.html

Sends an HTTP GET request using a urllib3 PoolManager, handling potential SSLErrors by retrying with SSL disabled. It streams response content and checks against MAX_FILE_SIZE.

```python
def _send_urllib_request(
    url: str, no_ssl: bool, with_headers: bool, config: ConfigParser
) -> Optional[Response]:
    """Internal function to robustly send a request (SSL or not) and return its result."""
    try:
        pool_manager = _initiate_pool(config, no_ssl=no_ssl)

        # execute request, stop downloading as soon as MAX_FILE_SIZE is reached
        response = pool_manager.request(
            "GET",
            url,
            headers=_determine_headers(config),
            retries=_get_retry_strategy(config),
            preload_content=False,
        )
        data = bytearray()
        for chunk in response.stream(2**17):
            data.extend(chunk)
            if len(data) > config.getint("DEFAULT", "MAX_FILE_SIZE"):
                raise ValueError("MAX_FILE_SIZE exceeded")
        response.release_conn()

        # necessary for standardization
        resp = Response(bytes(data), response.status, response.geturl())
        if with_headers:
            resp.store_headers(response.headers)
        return resp

    except urllib3.exceptions.SSLError:
        LOGGER.warning("retrying after SSLError: %s", url)
        return _send_urllib_request(url, True, with_headers, config)
    except Exception as err:
        LOGGER.error("download error: %s %s", url, err)  # sys.exc_info()[0]

    return None
```

--------------------------------

### Sample URL lists

Source: https://trafilatura.readthedocs.io/en/latest/_sources/tutorial0.rst.txt

Extract a random sample of URLs from a list using shuf and head.

```bash
$ shuf myfile.txt | head -100 > myfile-random-sample.txt
```

--------------------------------

### Initialize JusText stoplist

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/external.html

Retrieves and combines stoplists from all available languages for JusText processing. Ensures a comprehensive stoplist is used for accurate text classification.

```python
def jt_stoplist_init() -> Tuple[str]:
    'Retrieve and return the content of all JusText stoplists'
    global JT_STOPLIST
    stoplist = set()
    for language in get_stoplists():
        stoplist.update(get_stoplist(language))
    JT_STOPLIST = tuple(stoplist)
    return JT_STOPLIST
```

--------------------------------

### List URLs with Sitemap and URL Filter

Source: https://trafilatura.readthedocs.io/en/latest/_sources/usage-cli.rst.txt

Use the --sitemap and --url-filter options to list URLs from a sitemap, optionally filtering by a specific pattern or protocol.

```bash
$ trafilatura --sitemap "https://www.sitemaps.org/" --list --url-filter "https://www.sitemaps.org/de"
$ trafilatura --sitemap "https://www.sitemaps.org/" --list --url-filter "protocol"
```

--------------------------------

### Execute Justext extraction logic

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/external.html

Initializes the Justext algorithm with a stoplist and processes the HTML tree to extract non-boilerplate paragraphs.

```python
# init
result_body = Element('body')
# determine language
if target_language in JUSTEXT_LANGUAGES:
    justext_stoplist = get_stoplist(JUSTEXT_LANGUAGES[target_language])
else:
    justext_stoplist = JT_STOPLIST or jt_stoplist_init()
# extract
try:
    paragraphs = custom_justext(tree, justext_stoplist)
except Exception as err:
    LOGGER.error('justext %s %s', err, url)
else:
    for paragraph in paragraphs:
        if paragraph.is_boilerplate:
            continue
        #if duplicate_test(paragraph) is not True:
        elem, elem.text = Element('p'), paragraph.text
        result_body.append(elem)
return result_body
```

--------------------------------

### Configure SOCKS proxy for downloads

Source: https://trafilatura.readthedocs.io/en/latest/_sources/downloads.rst.txt

Set the http_proxy environment variable to route all downloads through a SOCKS proxy.

```bash
# set socks proxy
export http_proxy=socks5://PROXYHOST:PROXYPORT

# with user and password
export http_proxy=socks5://USER:PASSWORD@PROXYHOST:PROXYPORT
```

--------------------------------

### Output Format Options

Source: https://trafilatura.readthedocs.io/en/latest/usage-cli.html

Lists command-line flags for specifying the desired output format of the extracted content.

```bash
--output-format {csv,json,html,markdown,txt,xml,xmltei}
                      determine output format
--csv                 shorthand for CSV output
--html                shorthand for HTML output
--json                shorthand for JSON output
--markdown            shorthand for MD output
--xml                 shorthand for XML output
--xmltei              shorthand for XML TEI output
--validate-tei        validate XML TEI output
```

--------------------------------

### Load Custom Configuration File in Python

Source: https://trafilatura.readthedocs.io/en/latest/_sources/settings.rst.txt

Override all standard settings by loading a new configuration file using `use_config()`. This custom configuration can then be applied to the `extract()` function.

```python
from trafilatura import extract
from trafilatura.settings import use_config

newconfig = use_config("myfile.cfg")

extract(downloaded, config=newconfig)

extract(downloaded, settingsfile="myfile.cfg")
```

--------------------------------

### Perform single and sequential downloads in Python

Source: https://trafilatura.readthedocs.io/en/latest/_sources/downloads.rst.txt

Use fetch_url for straightforward, single-threaded downloads. The function utilizes a connection pool for efficiency.

```python
from trafilatura.downloads import fetch_url

# single download
downloaded = fetch_url('https://www.example.org')

# sequential downloads using a list
mylist = ["https://www.example.org", "https://httpbin.org"]
for url in mylist:
    downloaded = fetch_url(url)
    # do something with it
```

--------------------------------

### Fetch URL with SSL Options and Configuration

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/downloads.html

Downloads a web page from a given URL, with options to disable SSL verification and use a custom configuration. It returns the decoded response content or None if an error occurs.

```python
def fetch_url(
    url: str,
    no_ssl: bool = False,
    config: ConfigParser = DEFAULT_CONFIG,
    options: Optional[Extractor] = None,
) -> Optional[str]:
    """Downloads a web page and seamlessly decodes the response.

    Args:
        url: URL of the page to fetch.
        no_ssl: Do not try to establish a secure connection (to prevent SSLError).

```

--------------------------------

### Implement Justext rescue fallback

Source: https://trafilatura.readthedocs.io/en/latest/_modules/trafilatura/external.html

Performs basic tree cleaning before attempting extraction via the Justext algorithm.

```python
def justext_rescue(tree: HtmlElement, options: Any) -> Tuple[_Element, str, int]:
    '''Try to use justext algorithm as a second fallback'''
    # additional cleaning
    tree = basic_cleaning(tree)
    # proceed
    temppost_algo = try_justext(tree, options.url, options.lang)
    temp_text = trim(' '.join(temppost_algo.itertext()))
    return temppost_algo, temp_text, len(temp_text)
```

--------------------------------

### Process XML files

Source: https://trafilatura.readthedocs.io/en/latest/tutorial1.html

Tokenize XML files and strip tags using SoMaJo and sed.

```bash
# tokenize a file
$ somajo-tokenizer --xml xmlfiles/filename.xml
# remove tags
$ somajo-tokenizer --xml xmlfiles/filename.xml | sed -e "s|</*.*>||g" -e "/^$/d"
```

--------------------------------

### Tokenize an XML File with somajo-tokenizer

Source: https://trafilatura.readthedocs.io/en/latest/_sources/tutorial1.rst.txt

Use the somajo-tokenizer command to tokenize an XML file. Ensure the XML files are in the specified directory.

```bash
# tokenize a file
$ somajo-tokenizer --xml xmlfiles/filename.xml
```
Tokens:
Raw