### Install forumscraper

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Install the forumscraper package using pip. This is the initial step before using the tool.

```bash
pip install forumscraper
```

--------------------------------

### Gather All Forum URLs from SMF

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Retrieve a list of all forum URLs from an SMF installation. This example also gathers tags and boards.

```python
forums = ex.smf.get_board(
    "https://www.simplemachines.org/community/index.php",
    output=outputs.only_urls_forums,
)  # only get a list of urls to all forums
threads["urls"]["forums"]
threads["urls"]["boards"]
threads["urls"]["tags"]  # tags and boards are also gathered
```

--------------------------------

### Platform-Specific Scrapers: XenForo 2 Example

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Instantiate platform-specific scrapers like `xenforo2` directly for efficiency when scraping known forum types. This example shows XenForo 2 with file output, thread scraping, and specific request configurations, including retrying failures.

```python
import os
import sys
from forumscraper import xenforo2, smf2, phpbb, Outputs

# XenForo 2 with file output and 8 threads
os.makedirs("xenforo_out", exist_ok=True)
os.chdir("xenforo_out")

failures = []
xen = xenforo2(
    output=Outputs.write_by_hash | Outputs.threads,
    max_workers=8,
    requests={"timeout": 30, "retry": 3, "retry_delay": 10,
               "wait": 0.4, "wait_random": 0.4},
)
guess_result = xen.guess(
    "https://xenforo.com/community/",
    logger=sys.stdout,
    failed=failures,
    undisturbed=True,
)
# Retry failures using already-identified scraper
if guess_result:
    scraper = guess_result["scraper"]
    for entry in failures:
        parts = entry.split(" ")
        if len(parts) == 4 and parts[1] == "failed":
            scraper.get_thread(parts[0], state=guess_result)
```

--------------------------------

### Platform-Specific Scrapers: SMF 2 Example

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Instantiate platform-specific scrapers like `smf2` directly for efficiency. This example shows SMF 2 scraping a specific board and returning only data, threads, and users, with a configured timeout.

```python
import os
import sys
from forumscraper import xenforo2, smf2, phpbb, Outputs

# SMF 2 — scrape a specific board, return data only
smf = smf2(output=Outputs.data | Outputs.threads | Outputs.users)
result = smf.get_board(
    "https://www.simplemachines.org/community/index.php",
    requests={"timeout": 60},
)
if result:
    print(result["data"]["threads"][0])
```

--------------------------------

### Get help and view options

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Display the help message for forumscraper, which lists all available options and their short aliases. This is useful for discovering and understanding command-line arguments.

```bash
forumscraper --help
```

--------------------------------

### Get forumscraper version

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Display the currently installed version of the forumscraper tool.

```bash
forumscraper --version
```

--------------------------------

### Scrape XenForo Community with Failure Logging

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Scrape the XenForo community starting from the root URL, logging progress to standard output and recording any failed URLs. The `undisturbed` option is set to true.

```python
failures = []
files = xen.guess(
    "https://xenforo.com/community/",
    logger=sys.stdout,
    failed=failures,
    undisturbed=true
)
```

--------------------------------

### Attempt to Get SMF Thread (Returns None)

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

An example demonstrating that calling `get_thread` on an SMF forum URL might return `None`.

```python
ex.smf.get_thread(
    "https://www.simplemachines.org/community/index.php?topic=578496.0",
    output=outputs.only_urls_forums,
)  # returns none
```

--------------------------------

### Configure Forum Scraper Instance

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Configures a ForumExtractor instance with specific request settings like timeout, retries, and worker count. This setup is global for the scraper instance.

```python
xen = xenforo2(
    output=outputs.write_by_hash | outputs.threads,
    undisturbed=true,
    requests={
        "timeout": 30,
        "retry": 3,
        "retry_delay": 10,
        "wait": 0.4,
        "wait_random": 0.4,
        "max_workers": 8
    }
)
```

--------------------------------

### Scrape Forum Data and URLs

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Retrieve thread and URL lists from a forum. This example uses specific request settings for retries.

```python
forum = ex.get_forum(
    "https://xenforo.com/community/forums/off-topic.7/",
    output=outputs.data | outputs.urls | outputs.threads,
    requests={ "retry": 0 }
)  # get list of all threads and  urls from forum
forum["data"]["threads"]  # access the results
forum["urls"]["threads"]  # list of urls to found threads
forum["urls"]["forums"]  # list of urls to found forums
```

--------------------------------

### Run Full Discovery Pipeline

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Execute the `getall` script from the `discover/` directory to scrape builtwith.com for forum URLs and then validate them using `forumscraper-findroot`.

```bash
# From the discover directory:
cd discover

# Run full discovery pipeline: scrapes builtwith.com for forum URLs,
# then runs forumscraper-findroot on each to find valid board roots.
# Outputs ~593 verified forum root URLs after ~2 hours.
./getall

# Results are stored in files ending with "-root-found"
ls *-root-found
```

--------------------------------

### Run getall Script

Source: https://github.com/tuvimen/forumscraper/blob/master/discover/README.md

Execute the 'getall' script from the 'discover' directory to initiate the process of discovering forum engine root URLs.

```bash
cd discover
./getall
```

--------------------------------

### Download with additional content types

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Include additional content types such as user profiles, reactions, and board/forum/tag metadata in the download.

```bash
# Include user profiles, reactions, board/forum/tag metadata
forumscraper --users --reactions --boards --forums --tags \
  --directory DIR URL
```

--------------------------------

### Basic download to a directory

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Download forum content to a specified directory, naming files by thread/user ID.

```bash
forumscraper --directory DIR https://xenforo.com/community/forums/off-topic.7/
# Output files: 12345, 67890, m-100, m-200 (threads by id, users prefixed with m-)
```

--------------------------------

### Download forums with default settings

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Download forum data from specified URLs into a directory. Files are named by thread/user IDs by default. Ensure the directory exists before running.

```bash
forumscraper --directory DIR URL1 URL2 URL3
```

--------------------------------

### Initialize XenForo2 Scraper with Custom Settings

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Configure the XenForo2 scraper with specific settings for maximum workers, output format, and detailed request options including wait times and retries.

```python
os.mkdir("xenforo")
os.chdir("xenforo")

xen = xenforo2(
    max_workers=8,
    output=outputs.write_by_id | outputs.threads,
    requests={
        "timeout": 30,
        "retry": 3,
        "retry_delay": 10,
        "wait": 0.4,
        "wait_random": 0.4
    }
)
# specifies global config, writes output in files by their id (beginning with m- in case of users) in current directory
# ex.xenforo.v2 is an initialized instance of xenforo2 with the same settings as ex
# output by default is set to outputs.write_by_id|outputs.threads anyway
```

--------------------------------

### Use a Proxy Server

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Specify a proxy server to use for all requests.

```bash
forumscraper --proxy PROXY URL
```

--------------------------------

### Download using different scrapers for URLs

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Specify different scrapers for subsequent URLs in a single command. This allows for targeted scraping of various forum types within one execution.

```bash
forumscraper URL1 smf URL2 URL3 .thread URL4 xenforo2.forum URL5 URL6
```

--------------------------------

### Download with specific thread count and failure logging

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Download forum data using a specified number of threads and log any failures to a designated file. Note the order of options: failure log file is specified before the directory.

```bash
forumscraper --failures failures.txt --threads 8 --directory DIR URL1 URL2 URL3
```

--------------------------------

### Combining Output Flags

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Demonstrates how to combine different output flags to specify the desired scraping behavior and output format.

```python
output = Outputs.write_by_id | Outputs.threads | Outputs.users
```

```python
output = Outputs.data | Outputs.threads
```

```python
output = Outputs.only_urls_threads  # URL-collection mode
```

--------------------------------

### Multi-threaded download with failure logging

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Perform multi-threaded downloads and log any failed URLs to a file.

```bash
forumscraper --failures failures.txt --threads 8 --directory DIR \
  https://xenforo.com/community/
# Spawns 8 worker threads; failed URLs written to failures.txt
```

--------------------------------

### Identify Forum and Scrape Threads

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Initializes the scraper by guessing the forum structure from a URL and populates a 'failures' list with any failed requests during the initial identification. The identified scraper instance is then extracted.

```python
failures = []
files = xen.guess("https://xenforo.com/community/", logger=sys.stdout, failed=failures)
scraper = files["scraper"]
```

--------------------------------

### Scrape Top-Level Board and Subforums

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Use `get_board` to process the main index page of a forum, recursing into all sub-boards and forums. This is the entry point for downloading an entire forum. Configure output types, worker count, and request parameters like timeout and wait times.

```python
import os
from forumscraper import Extractor, Outputs

os.makedirs("full_forum", exist_ok=True)
os.chdir("full_forum")

ex = Extractor(
    output=Outputs.write_by_id | Outputs.threads | Outputs.users,
    max_workers=8,
    requests={"timeout": 60, "wait": 0.5, "wait_random": 0.5},
)

result = ex.get_board("https://www.simplemachines.org/community/index.php")

if result:
    print(f"Thread files written: {len(result['files']['threads'])}")
```

--------------------------------

### Identify forum engine

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Identify the forum software at a given URL or find the root board URL from any page on the site.

```bash
# Identify the forum software at a URL
forumscraper all.identify https://some-forum.org/threads/example.123

# Find the root board URL from any page on the site
forumscraper all.findroot https://some-forum.org/threads/example.123
# Output: https://some-forum.org/\thttps://some-forum.org/threads/example.123
```

--------------------------------

### Detect Forum Engine

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Use `identify` to download a page and fingerprint its HTML to return an initialized `ForumExtractor` instance for the detected platform, or `None` if unrecognized. This avoids re-identification on subsequent calls.

```python
from forumscraper import Extractor

ex = Extractor()

scraper = ex.identify("https://some-forum.org/threads/topic.123/")
if scraper:
    print(scraper.__class__.__name__)  # e.g. "xenforo2"
    # Use the returned scraper directly to avoid re-identification on subsequent calls
    result = scraper.get_thread("https://some-forum.org/threads/topic.123/")
```

--------------------------------

### Initialize Extractor and Guess URL

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Initializes the Extractor with custom headers and timeout, then uses the guess method to identify and process a URL. Subsequent calls to guess can include state and a different timeout.

```python
ex = forumscraper.Extractor(headers={"Referer":"https://xenforo.com/community/"},timeout=20)
state = ex.guess('https://xenforo.com/community/threads/selling-and-buying-second-hand-licenses.131205/',timeout=90)

html = requests.get('https://xenforo.com/community/threads/is-it-possible-to-set-up-three-websites-with-a-second-hand-xenforo-license.222507/').text
ex.guess('https://xenforo.com/community/threads/is-it-possible-to-set-up-three-websites-with-a-second-hand-xenforo-license.222507/',html,state,timeout=40)
```

--------------------------------

### Configure Gzip Compression for JSON Files

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Enable gzip compression for all written JSON files by providing `gzip.compress` as the `compress_func` during Extractor initialization.

```python
import gzip
import lzma
from forumscraper import Extractor, Outputs

# gzip compression
ex = Extractor(
    output=Outputs.write_by_id | Outputs.threads,
    compress_func=gzip.compress,
)
ex.get_board("https://xenforo.com/community/")
# Creates gzip-compressed files: "12345", "m-678", etc.
```

--------------------------------

### Enable Redirections

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Allow the scraper to follow HTTP redirections.

```bash
forumscraper --location URL
```

--------------------------------

### Ignore SSL Errors and Set Timeout/User-Agent

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Download content while ignoring SSL certificate errors, set a custom request timeout, and specify a user-agent string.

```bash
forumscraper --insecure --timeout 60 --user-agent 'why are we still here?'
```

--------------------------------

### Compressed output

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Enable compressed output for JSON files, using gzip compression.

```bash
forumscraper --compression gzip --directory DIR URL
# Creates .gz-compressed JSON files
```

--------------------------------

### Configure LZMA Compression for JSON Files

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Enable lzma compression for all written JSON files by providing `lzma.compress` as the `compress_func` during Extractor initialization.

```python
# lzma compression
ex2 = Extractor(
    output=Outputs.write_by_hash | Outputs.threads,
    compress_func=lzma.compress,
)
ex2.get_forum("https://phpbb.com/community/viewforum.php?f=46")
```

--------------------------------

### Download forums using SHA256 hash for filenames

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Download forum data, naming files using the SHA256 hash of their source URLs instead of IDs. This is useful for avoiding filename collisions or when IDs are not desired.

```bash
forumscraper --names hash --directory DIR URL
```

--------------------------------

### Hash-named output files

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Name output files using the SHA-256 hash of their source URL instead of their numeric ID.

```bash
forumscraper --names hash --directory DIR https://xenforo.com/community/
# Output files: a3f9c1..., b72e44..., etc.
```

--------------------------------

### Request tuning options

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Configure request parameters such as wait times, retries, timeouts, proxies, and custom headers.

```bash
forumscraper \
  --wait 0.8 --wait-random 0.4 \
  --retry 5 --retry-delay 120 \
  --timeout 60 --insecure \
  --proxy http://127.0.0.1:8080 \
  --header "Referer: https://example.com" \
  --cookie "auth=8f82ab" \
  --user-agent "MyBot/1.0" \
  --directory DIR URL
```

--------------------------------

### Use Extractor.guess() for Auto-Detection

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Utilizes the Extractor.guess() method to automatically detect the page type and forum software, then dispatches to the appropriate scraping function. This is the recommended entry point for most use cases.

```python
import sys
import forumscraper
from forumscraper import Extractor, Outputs

ex = Extractor(
    output=Outputs.data | Outputs.threads | Outputs.users,
    requests={"timeout": 60, "retry": 0},
)
failures = []

# Scrape a thread
thread = ex.guess(
    "https://xenforo.com/community/threads/selling-second-hand-licenses.131205/",
    failed=failures,
    undisturbed=True,
)
if thread:
    post = thread["data"]["threads"][0]
    print(post.keys())  # thread data fields
    print(f"Users found: {len(thread['data']['users'])}")

# Scrape from a board, collecting all threads
board = ex.guess(
    "https://www.simplemachines.org/community/index.php",
    output=forumscraper.Outputs.data | forumscraper.Outputs.threads,
    failed=failures,
)
# Retry failed URLs using the identified scraper (avoids re-identification)
if board:
    scraper = board["scraper"]
    for entry in failures:
        parts = entry.split(" ")
        if len(parts) == 4 and parts[1] == "failed":
            scraper.get_thread(parts[0], state=board)
```

--------------------------------

### Extractor.get_board()

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Processes the main index page of a forum, recursing into all sub-boards and forums to download the entire forum structure.

```APIDOC
## Extractor.get_board()

### Description
Processes the main index page of a forum, recursing into all sub-boards and forums. This is the entry point for downloading an entire forum.

### Method
```python
ex.get_board("https://www.simplemachines.org/community/index.php")
```

### Parameters
- `url` (string): The URL of the main index page of the forum.

### Request Example
```python
import os
from forumscraper import Extractor, Outputs

os.makedirs("full_forum", exist_ok=True)
os.chdir("full_forum")

ex = Extractor(
    output=Outputs.write_by_id | Outputs.threads | Outputs.users,
    max_workers=8,
    requests={"timeout": 60, "wait": 0.5, "wait_random": 0.5},
)
result = ex.get_board("https://www.simplemachines.org/community/index.php")
```

### Response
#### Success Response (200)
- `files` (dict): Contains information about written files, such as threads.

#### Response Example
```json
{
  "files": {
    "threads": ["thread_1.json", "thread_2.json"]
  }
}
```
```

--------------------------------

### Add Custom Headers

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Include custom HTTP headers in requests. This option can be specified multiple times.

```bash
--header "Key: Value"
```

```bash
--header 'User: Admin' --header 'Pass: 12345'
```

--------------------------------

### Add Custom Cookies

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Provide custom cookies for requests. This option can be specified multiple times.

```bash
--cookie "Key=Value"
```

```bash
--cookie 'auth=8f82ab' --cookie 'PHPSESSID=qw3r8an829'
```

--------------------------------

### Find Forum Root URL

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Use `findroot` to navigate up the link structure from any URL on a forum site to find the board root URL. Returns the root URL string or `None` on failure. An instance of `Extractor` is required.

```python
from forumscraper import Extractor

ex = Extractor()

root = ex.findroot("https://some-forum.org/viewtopic.php?t=12345")
print(root)  # e.g. "https://some-forum.org/index.php"

# Use in batch mode (CLI equivalent):
# forumscraper all.findroot URL
```

--------------------------------

### xenforo2.get_tag()

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Traverses all threads listed under a forum tag page using the xenforo2 scraper.

```APIDOC
## xenforo2.get_tag()

### Description
Traverses all threads listed under a forum tag page.

### Method
```python
scraper.get_tag("https://xenforo.com/community/tags/addon/")
```

### Parameters
- `url` (string): The URL of the forum tag page.

### Request Example
```python
from forumscraper import xenforo2, Outputs

scraper = xenforo2(
    output=Outputs.only_urls_threads,
    requests={"timeout": 30},
)
result = scraper.get_tag("https://xenforo.com/community/tags/addon/")
```

### Response
#### Success Response (200)
- `urls` (dict): Contains a list of thread URLs found under the tag.

#### Response Example
```json
{
  "urls": {
    "threads": ["https://xenforo.com/community/threads/example-addon-thread.54321/"]
  }
}
```
```

--------------------------------

### Scrape a User Profile

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Use `get_user` to fetch a single user profile page and extract profile data. Configure output types and request timeout.

```python
from forumscraper import invision, Outputs

scraper = invision(
    output=Outputs.data | Outputs.users,
    requests={"timeout": 30},
)

result = scraper.get_user("https://invisioncommunity.com/profile/12345-someuser/")
if result:
    user = result["data"]["users"][0]
    print(user)
```

--------------------------------

### Process Pre-downloaded HTML with Extractor

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Pass pre-fetched HTML content to the Extractor's guess method to avoid redundant HTTP requests. The state object can be reused across calls.

```python
import requests as req
from forumscraper import Extractor, Outputs

ex = Extractor(output=Outputs.data | Outputs.threads, requests={"timeout": 60})

url = "https://xenforo.com/community/threads/is-it-possible.222507/"
html = req.get(url).text

state = ex.guess(
    "https://xenforo.com/community/threads/selling-licenses.131205/"
)
# Reuse state and pass pre-fetched HTML for next call
ex.guess(url, html, state, timeout=40)

print(f"Total threads: {len(state['data']['threads'])}")
```

--------------------------------

### Combine Multiple Options for Debugging

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

A combination of options to limit scraping to a single page within a thread and forum, useful for debugging.

```bash
forumscraper --thread-pages-max 1 --pages-max 1 --pages-forums-max 1 --pages-threads-max 1 URL1 URL2 URL3
```

--------------------------------

### Platform-specific scrapers

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Instantiates platform-specific scraper classes directly for known forum types, bypassing auto-identification for efficiency.

```APIDOC
## Platform-specific scrapers

### Description
Each platform has one or two versioned `ForumExtractor` subclasses that can be instantiated directly for known-type forums. Bypassing auto-identification avoids re-downloading for identification and is more efficient when scraping many URLs from the same known forum.

Available scrapers: `invision`, `phpbb`, `smf1`, `smf2`, `smf` (identify), `xenforo1`, `xenforo2`, `xenforo` (identify), `xmb`, `hackernews`, `stackexchange`, `vbulletin`.

### Example Usage (XenForo 2)
```python
import os
import sys
from forumscraper import xenforo2, Outputs

os.makedirs("xenforo_out", exist_ok=True)
os.chdir("xenforo_out")

failures = []
xen = xenforo2(
    output=Outputs.write_by_hash | Outputs.threads,
    max_workers=8,
    requests={"timeout": 30, "retry": 3, "retry_delay": 10,
               "wait": 0.4, "wait_random": 0.4},
)
xen.guess(
    "https://xenforo.com/community/",
    logger=sys.stdout,
    failed=failures,
    undisturbed=True,
)
```

### Example Usage (SMF 2)
```python
from forumscraper import smf2, Outputs

smf = smf2(output=Outputs.data | Outputs.threads | Outputs.users)
smf.get_board(
    "https://www.simplemachines.org/community/index.php",
    requests={"timeout": 60},
)
```
```

--------------------------------

### Initialize Extractor for Auto-Identification

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Initializes the Extractor class with specified output flags, worker count, and request configurations. Use this for auto-identifying forum engines and scraping all available data.

```python
import forumscraper
from forumscraper import Extractor, Outputs

ex = Extractor(
    output=Outputs.write_by_id | Outputs.threads | Outputs.users,
    max_workers=4,
    requests={
        "timeout": 90,
        "retry": 3,
        "retry_delay": 30,
        "wait": 0.5,
        "wait_random": 0.3,
    }
)

# Auto-identify forum engine and page type, scrape everything
result = ex.guess("https://xenforo.com/community/threads/forum-data-breach.180995/")

# result is None on failure, or a dict:
# result["data"]["threads"]  -> list of thread dicts
# result["data"]["users"]    -> list of user dicts
# result["files"]["threads"] -> list of created file paths
# result["urls"]["threads"]  -> list of scraped URLs
# result["scraper"]          -> the identified ForumExtractor instance
# result["scraper-method"]   -> the function used (e.g. ex.smf.v2.get_board)

if result:
    print(f"Scraped {len(result['data']['threads'])} threads")
    print(f"Identified scraper: {result['scraper'].__class__.__name__}")
```

--------------------------------

### Scrape All Threads in a Forum Section

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Use `get_forum` to traverse all pages of a forum listing and scrape each thread. Supports recursion into subforums with `pages_max_depth`. Configure output types, page limits, thread limits, and request parameters.

```python
from forumscraper import Extractor, Outputs

ex = Extractor(
    output=Outputs.data | Outputs.urls | Outputs.threads,
    pages_max=5,              # traverse at most 5 listing pages
    pages_threads_max=10,     # process at most 10 threads per page
    pages_max_depth=2,        # recurse up to 2 subforum levels deep
    max_workers=4,
    requests={"timeout": 60, "retry": 0},
)

result = ex.get_forum("https://xenforo.com/community/forums/off-topic.7/")

if result:
    print(f"Threads scraped: {len(result['data']['threads'])}")
    print(f"Thread URLs found: {len(result['urls']['threads'])}")
    print(f"Forum URLs found: {len(result['urls']['forums'])}")
```

--------------------------------

### Scrape a Single Thread with xenforo2

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Uses the xenforo2 scraper to download all pages of a specific thread, extracting post content, author metadata, and reaction data. This method creates files named by ID for threads and users.

```python
import os
from forumscraper import xenforo2, Outputs

os.makedirs("output", exist_ok=True)
os.chdir("output")

scraper = xenforo2(
    output=Outputs.write_by_id | Outputs.threads | Outputs.users | Outputs.reactions,
    requests={"timeout": 30, "retry": 3, "retry_delay": 10},
)

result = scraper.get_thread(
    "https://xenforo.com/community/threads/forum-data-breach.180995/"
)
# Creates files: "180995" (thread JSON), "m-<userid>" (user JSONs)

if result:
    print(f"Files written: {result['files']['threads']}")
    print(f"User files: {result['files']['users']}")
```

--------------------------------

### Configure Retries and Delay

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Set the number of retries for non-fatal failures and the delay between retries. Setting `--retry 0` disables retries.

```bash
forumscraper --retry 5 --retry-delay 120 URL
```

--------------------------------

### Force overwrite of existing files

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Use the --force option to overwrite existing files if they are found and not empty. This ensures that the latest data is always downloaded.

```bash
forumscraper --force --directory DIR URL
```

--------------------------------

### Retry Failed Requests

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Iterates through a list of failed requests, attempts to re-download them if they match a specific format, and appends results to the provided state. This is useful for retrying downloads after initial failures.

```python
for i in failures:  # try to download failed one last time
    x = i.split(" ")
    if len(x) == 4 and x[1] == "failed":
        scraper.get_thread(x[0], state=files)  # use of already identified class
```

--------------------------------

### Mixed scrapers and URL types

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Use mixed scrapers and URL types in a single command by specifying the scraper and function between URLs.

```bash
forumscraper \
  https://forum-a.com \
          # uses default all.guess
  smf https://smf-forum.org \
    # forces SMF scraper, guess function
  .thread https://phpbb.org/t/1  # forces thread function on auto-identified scraper
```

--------------------------------

### Extractor.findroot()

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Finds the forum root URL from any given page URL on a forum site.

```APIDOC
## Extractor.findroot()

### Description
Given any URL on a forum site, navigates up the link structure to find the board root URL. Returns the root URL string or `None` on failure.

### Method
```python
ex.findroot("https://some-forum.org/viewtopic.php?t=12345")
```

### Parameters
- `url` (string): Any URL on the forum site.

### Request Example
```python
from forumscraper import Extractor

ex = Extractor()
root = ex.findroot("https://some-forum.org/viewtopic.php?t=12345")
```

### Response
#### Success Response (200)
- `root` (string | None): The detected root URL of the forum, or `None` if not found.

#### Response Example
```json
"https://some-forum.org/index.php"
```
```

--------------------------------

### Gather Thread URLs from SMF Forum

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Extract only the URLs of threads from a Simple Machines Forum (SMF) board without scraping the thread data itself.

```python
threads = ex.smf.get_forum(
    "https://www.simplemachines.org/community/index.php?board=1.0",
    output=outputs.only_urls_threads,
)  # gather only urls to threads without scraping data
threads["urls"]["threads"]
threads["urls"]["forums"]  # is also created
```

--------------------------------

### Extractor.identify()

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Detects the forum engine of a given URL and returns an initialized scraper instance.

```APIDOC
## Extractor.identify()

### Description
Downloads a page and fingerprints its HTML to return an initialized `ForumExtractor` instance for the detected platform, or `None` if unrecognized.

### Method
```python
ex.identify("https://some-forum.org/threads/topic.123/")
```

### Parameters
- `url` (string): The URL of a page on the forum site.

### Request Example
```python
from forumscraper import Extractor

ex = Extractor()
scraper = ex.identify("https://some-forum.org/threads/topic.123/")
```

### Response
#### Success Response (200)
- `scraper` (ForumExtractor | None): An initialized scraper instance for the detected forum platform, or `None` if the platform is unrecognized.

#### Response Example
```python
# If xenforo2 is detected
"xenforo2"
```
```

--------------------------------

### Scrape a Tag/Label Page

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Use `get_tag` to traverse all threads listed under a forum tag page. Configure output types and request timeout.

```python
from forumscraper import xenforo2, Outputs

scraper = xenforo2(
    output=Outputs.only_urls_threads,
    requests={"timeout": 30},
)

result = scraper.get_tag("https://xenforo.com/community/tags/addon/")
if result:
    for url in result["urls"]["threads"]:
        print(url)
```

--------------------------------

### Set Request Wait Times

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Control the delay between requests. Use `--wait` for a fixed delay and `--wait-random` for a random delay within a range.

```bash
forumscraper --wait 0.8 --wait-random 0.4 URL
```

--------------------------------

### Collect only URLs

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Collect only thread or forum URLs without scraping full content, saving them to a specified file.

```bash
# Only collect thread URLs from a forum page
forumscraper --only-urls-threads --output thread-urls.txt \
  https://www.simplemachines.org/community/index.php?board=1.0

# Only collect forum/subforum URLs from a board
forumscraper --only-urls-forums --output forum-urls.txt \
  https://www.simplemachines.org/community/index.php
```

--------------------------------

### invision.get_user()

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Fetches a single user profile page and extracts profile data using the invision scraper.

```APIDOC
## invision.get_user()

### Description
Fetches a single user profile page and extracts profile data.

### Method
```python
scraper.get_user("https://invisioncommunity.com/profile/12345-someuser/")
```

### Parameters
- `url` (string): The URL of the user profile page.

### Request Example
```python
from forumscraper import invision, Outputs

scraper = invision(
    output=Outputs.data | Outputs.users,
    requests={"timeout": 30},
)
result = scraper.get_user("https://invisioncommunity.com/profile/12345-someuser/")
```

### Response
#### Success Response (200)
- `data` (dict): Contains user profile data.

#### Response Example
```json
{
  "data": {
    "users": [
      {
        "username": "someuser",
        "user_id": 12345,
        "join_date": "2020-01-01T12:00:00Z",
        "posts": 100
      }
    ]
  }
}
```
```

--------------------------------

### Log output and failures to files

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Configure forumscraper to log general output to a specified file and failures to another. This is useful for debugging and monitoring the scraping process.

```bash
forumscraper --log stdout --failed FILE --directory DIR URL1 URL2 URL3
```

--------------------------------

### Scrape with Extractor and Handle Failures

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Uses the Extractor's guess method to scrape a URL, specifying output formats and handling potential failures. Failed URLs can be retried using the identified scraper instance.

```python
failures = []
results = ex.guess('https://www.simplemachines.org/community/index.php',output=forumscraper.Outputs.urls|forumscraper.Outputs.data,failed=failures,undisturbed=True)
#results['scraper-method'] points to ex.smf.v2.get_board

scraper = results['scraper'] #points to ex.smf.v2

for i in failures: #try to download failed one last time
    x = i.split(' ')
    if len(x) == 4 and x[1] == 'failed':
        scraper.get_thread(x[0],state=results) #save results in 'results'
```

--------------------------------

### Extractor.get_forum()

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Scrapes all threads within a specified forum section, supporting recursion into subforums up to a defined depth.

```APIDOC
## Extractor.get_forum()

### Description
Traverses all pages of a forum listing, scraping each thread found. Supports recursion into subforums via `pages_max_depth`.

### Method
```python
ex.get_forum("https://xenforo.com/community/forums/off-topic.7/")
```

### Parameters
- `url` (string): The URL of the forum section to scrape.
- `pages_max_depth` (int, optional): The maximum depth to recurse into subforums. Defaults to 2.

### Request Example
```python
from forumscraper import Extractor, Outputs

ex = Extractor(
    output=Outputs.data | Outputs.urls | Outputs.threads,
    pages_max=5,
    pages_threads_max=10,
    pages_max_depth=2,
    max_workers=4,
    requests={"timeout": 60, "retry": 0},
)
result = ex.get_forum("https://xenforo.com/community/forums/off-topic.7/")
```

### Response
#### Success Response (200)
- `data` (dict): Contains scraped thread data.
- `urls` (dict): Contains lists of thread and forum URLs found.

#### Response Example
```json
{
  "data": {
    "threads": [
      {
        "title": "Example Thread Title",
        "author": "ExampleUser",
        "date": "2023-10-27T10:00:00Z",
        "content": "Thread content..."
      }
    ]
  },
  "urls": {
    "threads": ["https://xenforo.com/community/threads/example-thread.12345/"],
    "forums": ["https://xenforo.com/community/forums/example-subforum.678/"]
  }
}
```
```

--------------------------------

### Scrape Thread Data with Custom Requests

Source: https://github.com/tuvimen/forumscraper/blob/master/README.md

Extract thread data, including threads and users, from a given URL. Overrides global request settings with specific ones for this operation.

```python
import os
import sys
from forumscraper import extractor, outputs, xenforo2

ex = extractor(requests={\"timeout\": 90})

thread = ex.guess(
    "https://xenforo.com/community/threads/forum-data-breach.180995/",
    output=outputs.data | outputs.threads | outputs.users,
    requests={
        "timeout": 60,
        "retry": 0,
    }
)  # automatically identify forum and type of page and save results
thread["data"]["threads"][0]  # access the result
thread["data"]["users"]  # found users are also saved into an array
```

--------------------------------

### Collect Thread URLs from PhpBB Forum

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Use the phpbb function with Outputs.only_urls_threads to extract only thread URLs from a specified forum section.

```python
ph = phpbb(output=Outputs.only_urls_threads)
result = ph.get_forum("https://phpbb.com/community/viewforum.php?f=46")
if result:
    for url in result["urls"]["threads"]:
        print(url)
```

--------------------------------

### Limit scraping scope for debugging

Source: https://context7.com/tuvimen/forumscraper/llms.txt

Limit the scope of scraping for debugging purposes, such as downloading a maximum number of pages per thread or forum.

```bash
# Download only 1 page per thread, 1 page per forum, 1 forum, 1 thread per page
forumscraper \
  --thread-pages-max 1 --pages-max 1 \
  --pages-forums-max 1 --pages-threads-max 1 \
  --directory DIR URL
```

--------------------------------

### State Object Structure for Scraper Results

Source: https://context7.com/tuvimen/forumscraper/llms.txt

The state dictionary returned by scraper methods accumulates data, URLs, and file paths. It can be passed between calls to maintain context and results.

```python
# Returned state structure:
state = {
    "data": {
        "boards": [],    # list of board data dicts
        "tags": [],      # list of tag data dicts
        "forums": [],    # list of forum data dicts
        "threads": [],   # list of thread data dicts
        "users": [],     # list of user data dicts
    },
    "urls": {
        "threads": [],   # source URLs of scraped threads
        "users": [],     # source URLs of scraped users
        "reactions": [], # reaction endpoint URLs
        "forums": [],    # source URLs of scraped forums
        "tags": [],      # source URLs of scraped tags
        "boards": [],    # source URLs of scraped boards
    },
    "files": {
        "boards": [],    # paths to written board JSON files
        "tags": [],      # paths to written tag JSON files
        "forums": [],    # paths to written forum JSON files
        "threads": [],   # paths to written thread JSON files
        "users": [],     # paths to written user JSON files
    },
    "visited": set(),        # set of all visited URLs (shared with session)
    "scraper": None,         # identified ForumExtractor instance (ForumExtractorIdentify only)
    "scraper-method": None,  # function reference used by guess()
}
```

--------------------------------

### Outputs enum for output behavior

Source: https://context7.com/tuvimen/forumscraper/llms.txt

The Outputs enum controls scraper collection and storage. Flags can be combined using the bitwise OR operator.

```python
from forumscraper import Outputs
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.