### Setup Virtual Environment and Install Dependencies

Source: https://github.com/mariadb/mariadb-documentation/blob/main/mariadb_pdf/README.md

Use these commands to set up a Python virtual environment and install project dependencies on Linux systems.

```console
virtualenv .venv
source .venv/bin/activate
pip3 install -r requirements.txt
```

--------------------------------

### Get Server Help

Source: https://github.com/mariadb/mariadb-documentation/blob/main/server/README.md

Run the server with the '--help' flag to display available command-line options and usage instructions.

```bash
./target/release/mariadb_kb_server --help
```

--------------------------------

### Build the Server Binary

Source: https://github.com/mariadb/mariadb-documentation/blob/main/server/README.md

Build the release binary for the mariadb_kb_server using Cargo. Ensure Rust is installed first.

```bash
cargo build --release
```

--------------------------------

### Run HELP Contents Generator

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Generates SQL files to populate MariaDB's `help_topic`, `help_keyword`, `help_category`, and `help_relation` tables. Requires `pip3 install -r requirements.txt`.

```bash
cd help_contents
pip3 install -r requirements.txt  # bs4, lxml, requests

# Generate HELP contents for MariaDB 10.11
python3 main.py -v 1011

# Generate for multiple versions at once
python3 main.py -v 1010 1011 1100

# Use a custom server port
python3 main.py -v 1011 --port 8080

# Custom output directory
python3 main.py -v 1011 --output ./sql_output/

# Quiet mode (suppress progress bar)
python3 main.py -v 1011 --quiet

# Output files created:
#   ./output/fill_help_tables1011.sql
```

--------------------------------

### Run the Server

Source: https://github.com/mariadb/mariadb-documentation/blob/main/server/README.md

Execute the compiled mariadb_kb_server binary. The output file is located in 'target/release/'.

```bash
./target/release/mariadb_kb_server(.exe)
```

--------------------------------

### Create New Directory

Source: https://github.com/mariadb/mariadb-documentation/blob/main/crawler/README.md

Use these commands to create a new directory for the project and navigate into it.

```bash
mkdir mariadb_kb
cd mariadb_kb
```

--------------------------------

### Clone, Build, and Run Crawler

Source: https://github.com/mariadb/mariadb-documentation/blob/main/crawler/README.md

Clone the mariadb_kb_crawler repository, build the release binary, and run the crawler with force and verbose flags. Ensure you are in the mariadb_kb_crawler directory before running.

```bash
git clone https://github.com/Icerath/mariadb_kb_crawler
cd mariadb_kb_crawler

cargo build --release

./target/release/mariadb_kb_crawler(.exe) --force --verbose
```

--------------------------------

### Run Main Script

Source: https://github.com/mariadb/mariadb-documentation/blob/main/mariadb_pdf/README.md

Execute the main Python script to generate PDF documentation. Ensure the MariaDB KB server is running.

```console
(python/python3) main.py
```

--------------------------------

### Build and Run MariaDB KB Crawler

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Build the Rust crawler and execute various crawl methods. Requires the Rust toolchain. Use `--force` to fetch directly from mariadb.com if the local server is not running.

```bash
cd crawler
cargo build --release
```

```bash
./target/release/mariadb_kb_crawler
```

```bash
./target/release/mariadb_kb_crawler --force
```

```bash
./target/release/mariadb_kb_crawler --force --verbose --outlog crawl.log
```

```bash
./target/release/mariadb_kb_crawler --resume --force
```

```bash
./target/release/mariadb_kb_crawler recent
```

```bash
./target/release/mariadb_kb_crawler --force --output /data/kb_archive
```

```bash
./target/release/mariadb_kb_crawler --port 8080
```

```bash
./target/release/mariadb_kb_crawler --clear --force
```

--------------------------------

### Import HELP Contents into MariaDB

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Imports the generated SQL file into a running MariaDB server and verifies the import using the `HELP` command.

```bash
# Import the generated SQL into a running MariaDB server
mysql -u root -p mysql < ./output/fill_help_tables1011.sql

# Verify by using the HELP command inside MariaDB
mysql -u root -p -e "HELP SELECT;"
# Returns: formatted help text for SELECT statement
```

--------------------------------

### Import Help Contents into MariaDB

Source: https://github.com/mariadb/mariadb-documentation/blob/main/help_contents/README.md

Imports the generated SQL help content file into a MariaDB database. Replace [filepath] with the actual path to the .sql file.

```console
mysql -u root -p mysql < [filepath]
```

--------------------------------

### Build and Run MariaDB KB Server

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Builds and runs the MariaDB KB server, which exposes the local archive over HTTP. It can be run on a default or custom port with a specified archive source.

```bash
# Build
cd server
cargo build --release

# Run on default port 7032 with archive at ../archive
./target/release/mariadb_kb_server

# Run on custom port with custom archive source
./target/release/mariadb_kb_server --port 8080 --source /data/kb_archive

# Output
# Listening on http://localhost:7032/
# Ctrl C to Exit.
```

--------------------------------

### Convert KB URL to Filesystem Path in Rust

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Utility functions to map MariaDB Knowledgebase URLs to local filesystem paths within the archive and vice-versa. URLs without extensions are mapped to `index.html`.

```rust
use std::path::PathBuf;

// url_to_path(root, url) -> PathBuf
let root = PathBuf::from("../archive");
let url = "https://mariadb.com/kb/en/select/";

// Result: ../archive/en/select/index.html
let path = url_to_path(&root, url);

// Reverse: path_to_url(path, archive_root) -> String
let reconstructed = path_to_url(&path, "../archive");
// Result: "https://mariadb.com/kb/en/select/"
assert_eq!(reconstructed, "https://mariadb.com/kb/en/select/");
```

--------------------------------

### Clone MariaDB KB Server Repository

Source: https://github.com/mariadb/mariadb-documentation/blob/main/crawler/README.md

Clone the mariadb_kb_server repository to set up the necessary server component. Further instructions are available in the server's repository.

```bash
git clone https://github.com/Icerath/mariadb_kb_server
```

--------------------------------

### Run MariaDB PDF Generator

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Generates PDFs from archived KB articles by reading kb_urls.csv, fetching articles, and rendering to PDF using wkhtmltopdf. Supports per-language output, custom ports, and options for HTML-only generation or quiet mode.

```bash
cd mariadb_pdf
pip3 install -r requirements.txt  # bs4, pdfkit, requests, toml

# Generate English PDF (default)
python3 main.py

# Generate PDF for specific languages
python3 main.py --langs en de fr

# Use a custom port and limit rows for testing
python3 main.py --port 8080 --numrows 50

# Generate HTML only (skip PDF rendering, faster for debugging)
python3 main.py --nopdf

# Custom output paths
python3 main.py --pdfpath /output/MariaDB.pdf --htmlpath /output/MariaDB.html

# Quiet mode (suppress progress output)
python3 main.py --quiet

# Use a custom config file
python3 main.py --config ./my_config.toml
```

--------------------------------

### Generate Help Contents for MariaDB Versions

Source: https://github.com/mariadb/mariadb-documentation/blob/main/help_contents/README.md

Generates help content SQL files for specified MariaDB versions. The output files are saved in the ./output directory.

```console
python main.py -v 1010 1011
```

--------------------------------

### Generate HELP SQL Table Data

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Coordinates the HELP SQL generation pipeline, including loading categories, fetching articles, converting HTML to plain text, and assembling INSERT statements for all four HELP tables.

```python
from src.generate_help_table import generate_help_table
from src.version import Version

version = Version.from_str("1011")  # MariaDB 10.11
# Requires mariadb_kb_server running on port 7032

sql_output: str = generate_help_table(
    version=version,
    concat_size=14600,   # Max chars per SQL field (avoids packet size limits)
    port=7032,
    verbose=True         # Show progress bar
)

# Write to file
from pathlib import Path
Path("./output/fill_help_tables1011.sql").write_text(sql_output, encoding="utf-8")

# The SQL contains:
# - LOCK TABLE statements
# - INSERT INTO help_category ...
# - INSERT INTO help_topic ... (with HELP_DATE and HELP_VERSION entries)
# - UPDATE help_topic SET description = CONCAT(...) for long entries
# - INSERT INTO help_keyword ...
# - INSERT INTO help_relation ...
# - UNLOCK TABLES
```

--------------------------------

### Serve KB Article as HTML via HTTP API

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Fetches an archived KB article as HTML using the /kb/<url> endpoint. Supports 'html', 'css', and 'png' content types. Returns 404 if the article is not found.

```bash
# Fetch the MariaDB SELECT article
curl http://localhost:7032/kb/en/select/
# Returns: full HTML of the archived SELECT documentation page

# Fetch a CSS stylesheet
curl http://localhost:7032/kb/static/main.css
# Returns: text/css content

# 404 if article not found in archive
curl -I http://localhost:7032/kb/en/nonexistent-article/
# HTTP/1.1 404 Not Found
```

--------------------------------

### Serve a KB article as HTML

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Returns the archived HTML for a given KB URL path. Supports 'html', 'css', and 'png' content types based on the file extension.

```APIDOC
## GET /kb/<url>

### Description
Serves an archived KB article as HTML. The endpoint supports different content types like HTML, CSS, and PNG based on the file extension in the URL.

### Method
GET

### Endpoint
/kb/<url>

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL path of the KB article to serve.

### Request Example
```bash
curl http://localhost:7032/kb/en/select/
```

### Response
#### Success Response (200)
- Returns the HTML content of the archived KB article or the requested static asset (CSS, PNG).

#### Error Response (404)
- Returns a 404 Not Found if the article is not found in the archive.

### Response Example
```bash
# Returns: full HTML of the archived SELECT documentation page
# Returns: text/css content for CSS files
```
```

--------------------------------

### MariaDB Version Dataclass for Comparison

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Parses a compact version string (e.g., "1011" → 10.11) and supports comparison for version-gating KB articles. Used to filter KB articles based on their minimum required version.

```python
from src.version import Version

v = Version.from_str("1011")
print(v.major)    # 10
print(v.minor)    # 11
print(repr(v))    # "10.11"

v2 = Version.from_str("1100")
print(v2 > v)     # True  (11.0 > 10.11)

# Version comparison is used to filter kb_urls.csv rows:
# Rows with "HELP Include" = "1" are always included.
# Rows with a version string are only included if that version <= target version.
v_gate = Version.from_str("1010")
print(v_gate <= v)  # True — article included for 10.11 if gated at 10.10
```

--------------------------------

### Generate Full PDF from HTML Pages

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Orchestrates the PDF generation pipeline, including reading HTML, merging, building TOC, and invoking wkhtmltopdf. Optionally repeats for accurate page numbers.

```python
from pdf.generate_pdf import generate_full_pdf
from setup.config import read_config
from setup.kb_urls import read_csv
from setup.languages import read_languages
from pathlib import Path

# Read config from config.toml and CLI args
config = read_config()

# Fetch kb_urls.csv from the running server
csv_items = read_csv(config.num_rows, config.port)

# Split items by language
language_csvs = read_languages(csv_items, config)

# Generate one PDF per language
for lang, lang_csv in language_csvs.items():
    output_dir = Path(f"output_{lang}")
    generate_full_pdf(lang_csv, output_dir, config)
    # Writes:
    #   output_en/output.html           (merged HTML)
    #   output_en/MariaDBServerKnowledgeBase.pdf  (final PDF)
```

--------------------------------

### PDF Generation Configuration (config.toml)

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

TOML configuration file for the PDF generator. Controls table-of-contents styling and wkhtmltopdf rendering options. Loaded automatically from ./config.toml.

```toml
[TOC]
main_font_size = "13px"
main_indent = "0.8em"
main_margin = "0.5em"

chapter_font_size = "18px"
chapter_indent = "3em"
chapter_margin = "1em"

[wkhtmltopdf]
dpi = 120
footer-font-size = 7
page-size = "A4"
margin-top = "0.6cm"
margin-bottom = "0.6cm"
margin-left = "2cm"
margin-right = "2cm"
footer-right = "[page]/[topage]"
zoom = 0.8
disable-smart-shrinking = true
dump-outline = "outline"
encoding = "UTF-8"
footer-line = ""
quiet = ""
disable-javascript = true

```

--------------------------------

### Access KB Page

Source: https://github.com/mariadb/mariadb-documentation/blob/main/server/README.md

Access a specific KB page by its URL. The server will respond with the original HTML content.

```http
localhost:[PORT]/kb/[url]
```

--------------------------------

### Download KB URLs CSV

Source: https://github.com/mariadb/mariadb-documentation/blob/main/server/README.md

Download the direct contents of the kb_urls.csv file, which lists available KB URLs.

```http
localhost:[PORT]/kb_urls.csv
```

--------------------------------

### Retrieve KB URLs Catalog via HTTP API

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Fetches the raw contents of kb_urls.csv via the /kb_urls.csv endpoint. This CSV maps KB article URLs to their PDF include flags, HELP categories, and keywords.

```bash
curl http://localhost:7032/kb_urls.csv
# Returns CSV with headers: URL,Header,Include,Depth,Duplicate slugs,...
# Example rows:
# https://mariadb.com/kb/en/select/,SELECT,1,2,,SQL Statements
# https://mariadb.com/kb/en/insert/,INSERT,1,2,,SQL Statements
```

--------------------------------

### Load and Filter KB URL Catalog

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Fetches and parses `kb_urls.csv` from a local server, returning a list of `CsvItem` objects. Filters to included entries and computes depth strings for TOC hierarchy.

```python
from setup.kb_urls import read_csv, CsvItem

# Fetch all included rows from server on port 7032
items: list[CsvItem] = read_csv(num_rows=-1, port=7032)

for item in items[:3]:
    print(item.url)        # https://mariadb.com/kb/en/select/
    print(item.header)     # SELECT
    print(item.depth)      # 2
    print(item.depth_str)  # 1.1
    print(item.include)    # 1
    print(item.slugs)      # ['https://mariadb.com/kb/en/select-statement/']

# Limit to first 20 rows (useful for testing)
test_items = read_csv(num_rows=20, port=7032)
```

--------------------------------

### List KB Subpages

Source: https://github.com/mariadb/mariadb-documentation/blob/main/server/README.md

Retrieve a list of all subpages for a given KB URL by appending '?list' to the URL.

```http
localhost:[PORT]/kb/[url]?list
```

--------------------------------

### List KB Category Sub-Pages via HTTP API

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Lists subdirectory names under a given KB path using the /kb/<url>?list endpoint. Returns a comma-separated list of child articles.

```bash
# List all sub-articles under the "en" locale root
curl "http://localhost:7032/kb/en/?list"
# Returns: select,insert,update,delete,create-table,...

# List sub-pages of the SQL-Statements category
curl "http://localhost:7032/kb/en/sql-statements/?list"
# Returns: data-manipulation,data-definition,transactions,...
```

--------------------------------

### List sub-pages of a KB category

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Returns a comma-separated list of subdirectory names under a given KB path, effectively listing child articles of a category.

```APIDOC
## GET /kb/<url>?list

### Description
Lists the sub-pages (child articles or directories) under a specified KB category path. The result is a comma-separated string of names.

### Method
GET

### Endpoint
/kb/<url>?list

### Parameters
#### Path Parameters
- **url** (string) - Required - The KB path for which to list sub-pages.

#### Query Parameters
- **list** - Required - This parameter triggers the listing of sub-pages.

### Request Example
```bash
curl "http://localhost:7032/kb/en/sql-statements/?list"
```

### Response
#### Success Response (200)
- Returns a comma-separated string of subdirectory names.

### Response Example
```bash
# Returns: select,insert,update,delete,create-table,...
# Returns: data-manipulation,data-definition,transactions,...
```
```

--------------------------------

### Check for Missing Files

Source: https://github.com/mariadb/mariadb-documentation/blob/main/server/README.md

Access the '/diff' endpoint to see files that are missing from the kb_urls.csv file.

```http
localhost:[PORT]/diff
```

--------------------------------

### StandardCrawler Implementation in Rust

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Performs a full, breadth-first crawl of the MariaDB Knowledgebase, scraping all linked sub-pages. This crawler is typically instantiated and run via CLI arguments.

```rust
// Instantiated via CLI args; simplified usage:
use app_args::AppArgs;
use standard_crawler::StandardCrawler;
use crawler::Crawler;

// args populated from CLI (--force, --output, --port, --resume, etc.)
let args = AppArgs::read();
let mut crawler = StandardCrawler::new(args);

// Crawl blocks until all discovered URLs are processed
crawler.crawl().expect("Crawl failed");
// Archive written to args.output (default: ../archive)
```

--------------------------------

### Retrieve the KB URLs catalog

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Returns the raw contents of the `kb_urls.csv` file, which maps KB article URLs to their metadata.

```APIDOC
## GET /kb_urls.csv

### Description
Retrieves the complete `kb_urls.csv` catalog. This CSV file maps Knowledge Base article URLs to various metadata, including PDF inclusion flags, HELP categories, and keywords. It is essential for both the PDF generator and the HELP contents generator.

### Method
GET

### Endpoint
/kb_urls.csv

### Response
#### Success Response (200)
- Returns the raw content of the `kb_urls.csv` file.

### Response Example
```bash
curl http://localhost:7032/kb_urls.csv
```

```csv
# Returns CSV with headers: URL,Header,Include,Depth,Duplicate slugs,...
# Example rows:
# https://mariadb.com/kb/en/select/,SELECT,1,2,,SQL Statements
# https://mariadb.com/kb/en/insert/,INSERT,1,2,,SQL Statements
```
```

--------------------------------

### Convert HTML KB Article to Plain Text

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Use this function to convert raw HTML content of a KB article into plain text suitable for the MariaDB HELP system. It handles line wrapping, escaping SQL-unsafe characters, and appending a URL.

```python
from src.format_to_text import format_to_text

# html is the raw HTML content of a KB article
html = open("archive/en/select/index.html").read()
text = format_to_text(html, name="select")

# text is ready to embed in a SQL INSERT statement:
# - Lines wrapped at 79 chars
# - Single quotes escaped as \'
# - Double curly quotes replaced with \"
# - Newlines encoded as \n literals
# - Trailing URL appended: "URL: https://mariadb.com/kb/en/select/"
print(text[:200])
# Syntax\n\nSELECT [ALL | DISTINCT | DISTINCTROW] [HIGH_PRIORITY]\n  [STRAIGHT_JOIN]
# ...
# URL: https://mariadb.com/kb/en/select/
```

--------------------------------

### Report articles missing from kb_urls.csv

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Compares articles in the archive against `kb_urls.csv` and reports those present in the archive but missing from the CSV.

```APIDOC
## GET /diff

### Description
Compares the set of articles available in the local archive against the list of URLs provided in `kb_urls.csv`. This endpoint identifies and returns a list of article paths that exist in the archive but are not cataloged in `kb_urls.csv`.

### Method
GET

### Endpoint
/diff

### Response
#### Success Response (200)
- Returns a list of archive paths that are not present in `kb_urls.csv`.

### Response Example
```bash
curl http://localhost:7032/diff
```

```
# Returns: list of archive paths not present in kb_urls.csv
# Useful for identifying newly crawled content not yet catalogued
```
```

--------------------------------

### Report Articles Missing from kb_urls.csv via HTTP API

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Compares archive contents against kb_urls.csv and reports articles present in the archive but not in the CSV using the /diff endpoint. Useful for identifying uncategorized content.

```bash
curl http://localhost:7032/diff
# Returns: list of archive paths not present in kb_urls.csv
# Useful for identifying newly crawled content not yet catalogued
```

--------------------------------

### Incremental Update Crawl with RecentCrawler

Source: https://context7.com/mariadb/mariadb-documentation/llms.txt

Performs an incremental crawl, refreshing only articles modified since the last update. Reads the 'last_updated' file to determine the cutoff date.

```rust
use recent_crawler::RecentCrawler;
use crawler::Crawler;
use std::path::PathBuf;

let root = PathBuf::from("../archive");
// Reads ../archive/last_updated to determine the cutoff date
let mut crawler = RecentCrawler::new(root).expect("Failed to read last_updated");
crawler.crawl().expect("Recent crawl failed");
// Only modified pages are refreshed; unchanged pages are untouched
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.