### Setup GermEval Dataset

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/errors.md

Instructions for setting up the GermEval dataset by creating the necessary directory and downloading the required TSV files.

```bash
mkdir -p data/germeval18
# Download from: https://sites.google.com/view/germeval2018-hatespeech/home
# Place train.tsv and test.tsv in data/germeval18/
```

--------------------------------

### Start MongoDB Service

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/errors.md

Commands to start the MongoDB service on different operating systems or via Docker.

```bash
# Linux
sudo service mongod start
```

```bash
# macOS
brew services start mongodb-community
```

```bash
# Docker
docker run -d -p 27017:27017 mongo
```

--------------------------------

### WikiExtractor Command Line Usage

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/wiki_extraction.md

Example of how to run WikiExtractor from the command line. Adjust output directory, file size, and number of processes as needed.

```bash
python WikiExtractor.py -b 100M -o output --processes 8 input.xml.bz2
```

--------------------------------

### Train a RoBERTa Model for Masked Language Modeling

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/README.md

This example shows how to configure and train a RoBERTa model for masked language modeling. It defines the model configuration, initializes the model, sets up training arguments including output directory and batch size, and then starts the training process using the Trainer API.

```python
from transformers import RobertaConfig, RobertaForMaskedLM, Trainer, TrainingArguments

config = RobertaConfig(
    vocab_size=50_265,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6
)
model = RobertaForMaskedLM(config=config)

training_args = TrainingArguments(
    output_dir="output",
    max_steps=500_000,
    per_device_train_batch_size=256
)

trainer = Trainer(model=model, args=training_args, ...)
trainer.train()
```

--------------------------------

### WikiExtractor Input XML Structure Example

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/wiki_extraction.md

Illustrates the expected Mediawiki XML export format for input dumps.

```xml
<mediawiki>
  <siteinfo>
    <sitename>Wikipedia</sitename>
  </siteinfo>
  <page>
    <title>Article Title</title>
    <ns>0</ns>
    <id>12345</id>
    <revision>
      <id>67890</id>
      <text>Article content with [[links]] and {{templates}}</text>
    </revision>
  </page>
</mediawiki>
```

--------------------------------

### Initialize SoMaJo Tokenizer

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/errors.md

Initializes the SoMaJo tokenizer. An error will occur if the specified language model is not installed or cannot be found.

```python
from somajo import SoMaJo
tokenizer = SoMaJo("de_CMC")
# Error if "de_CMC" model not found
```

--------------------------------

### Start TPU VM with ctpu

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/Training.md

Use this command to provision a TPU VM. Ensure to specify the zone, TensorFlow version, and desired TPU size. Preemptible instances are recommended for cost savings.

```bash
$ctpu up --zone=europe-west4-a  --tf-version=1.15 -preemptible --name=tpu-testv13 --tpu-size=v3-8 (In the overall Google Cloud Shell @ Project Quickpiq)
```

--------------------------------

### Initialize and Run Trainer

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/model_training.md

Instantiate the `Trainer` with your model, training arguments, data collator, and training dataset. Use `trainer.train()` to start the training process and `trainer.save_model()` to save the trained model.

```python
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)

trainer.train()
trainer.save_model(save_dir)
tokenizer.save_pretrained(save_dir)
```

--------------------------------

### Initialize and Shutdown Ray Cluster

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Basic setup and teardown for Ray, a distributed computing framework. Used for managing parallel tasks.

```python
ray.init(num_cpus=THREADS)
# ... work ...
ray.shutdown()
```

--------------------------------

### TSV Format Example

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/types.md

Example of a Tab-Separated Values (TSV) file used for benchmark datasets. Each line represents a record with fields separated by tabs.

```text
id	coarse_label	fine_label	comment
1	OFFENSE	HS	This is hateful speech
2	OTHER	—	This is normal
```

--------------------------------

### Main CC-Net Processing Script Setup

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/text_processing.md

Initializes the German tokenizer and sets up command-line argument parsing for processing gzip files containing JSON lines. This script filters for German content and outputs processed sentences.

```python
import gzip
import orjson
from somajo import SoMaJo
from tqdm import tqdm
import argparse

# Initialize German tokenizer
tokenizer = SoMaJo("de_CMC")

# Command line argument for input file
parser = argparse.ArgumentParser()
parser.add_argument('filename')
args = parser.parse_args()
input_filename = args.filename
```

--------------------------------

### Reading Files with Error Handling

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/errors.md

This example demonstrates reading files from an input directory. It includes a check to ensure the directory exists, preventing FileNotFoundError.

```python
files = read_file(IN_DIR, skip_files=skip_files)
# Raises FileNotFoundError if IN_DIR doesn't exist
```

--------------------------------

### Train Transformer Model

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/INDEX.md

Minimal example for training a transformer model. This involves training a tokenizer, training the model, and then evaluating its performance.

```bash
# 1. Train tokenizer
python src/01_tokenize_01.py

# 2. Train model
python src/02_train_01.py

# 3. Evaluate
python src/germeval18_benchmark.py
```

--------------------------------

### Check for Document Start Line

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/text_processing.md

Determines if a given line marks the beginning of a document in WikiExtractor output. It checks if the line starts with the '<doc' tag.

```python
def is_doc_start_line(line: str) -> bool:
```

```python
line = '<doc id="123" title="Example">'
print(is_doc_start_line(line))  # True

line = "This is content"
print(is_doc_start_line(line))  # False
```

--------------------------------

### Optional Parameter Types

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/types.md

Provides examples of `Optional` type hints for parameters that can be `None`, commonly used for directory paths, file lists, or output files.

```python
from typing import Optional

# Common optional parameters
path: Optional[str] = None        # Sentence_Extraction.TRANSFER_DIR
skip_files: Optional[List[str]] = None
output_file: Optional[str] = None
```

--------------------------------

### Define Classification Labels

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/benchmarking.md

Examples of how to define the label list for classification tasks. Supports both binary and multi-class classification.

```python
# Custom binary classification
label_list = ["NEGATIVE", "POSITIVE"]
```

```python
# Multi-class classification
label_list = ["NEGATIVE", "NEUTRAL", "POSITIVE"]
```

--------------------------------

### GermEval 2018 Benchmark Training Configuration

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Configuration parameters for training and model setup for the GermEval 2018 benchmark. These include epochs, batch size, evaluation frequency, and model paths.

```python
# Training
n_epochs = 1
batch_size = 32
evaluate_every = 15
repeats = 31

# Model
lang_model = "/home/phmay/data/nlp/checkpoints_256/model-electra"
do_lower_case = True

# Task
label_list = ["OTHER", "OFFENSE"]
metric = "f1_macro"
max_seq_len = 128
```

--------------------------------

### ByteLevelBPE Tokenizer Training

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/tokenization.md

Trains a ByteLevelBPE tokenizer. This example demonstrates how to specify the training data path using glob patterns and define special tokens relevant to ByteLevelBPE.

```python
from tokenizers import ByteLevelBPETokenizer
from pathlib import Path

folder_path = "/path/to/training/data"
paths = [str(x) for x in Path(folder_path).glob("**/*.txt")]

tokenizer = ByteLevelBPETokenizer()
tokenizer.train(
    files=paths,
    vocab_size=50_000,
    min_frequency=2,
    special_tokens=[
        "<s>",
        "<pad>",
        "</s>",
        "<unk>",
        "<mask>",
    ]
)

tokenizer.save(".", "germanBERT-CC")
```

--------------------------------

### Model Training API (RoBERTa)

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/INDEX.md

Provides classes for a complete RoBERTa model pretraining pipeline from scratch, including configuration, dataset preparation, and training setup.

```APIDOC
## Model Training API (RoBERTa)

### Description
Provides classes for a complete RoBERTa model pretraining pipeline from scratch, including configuration, dataset preparation, and training setup.

### Classes
- `RobertaConfig`: Configuration for RoBERTa models.
- `RobertaTokenizerFast`: Fast tokenizer for RoBERTa.
- `RobertaForMaskedLM`: RoBERTa model for Masked Language Modeling.
- `LineByLineTextDataset`: Dataset class for line-by-line text.
- `DataCollatorForLanguageModeling`: Data collator for language modeling tasks.
- `Trainer`: The main training class.
- `TrainingArguments`: Arguments for configuring the training process.
```

--------------------------------

### JSON Lines Format Example

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/types.md

Example of JSON Lines (JSONL) format, where each line is a valid JSON object. Used for processing text data with associated metadata.

```json
{"raw_content": "Text here", "language": "de", "url": "...", ...}
{"raw_content": "More text", "language": "en", "url": "...", ...}
```

--------------------------------

### Example Usage of add2mongo

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/deduplication.md

Demonstrates how to use the `add2mongo` function to insert metadata from a Common Crawl gzip file into MongoDB. Ensure the `Path` object points to the correct file.

```python
from pathlib import Path
from Hashing.Insert2Mongo import add2mongo

cc_file = Path("/data/CC-2019-09-head-000.gz")
add2mongo(cc_file)
print("Metadata inserted")
```

--------------------------------

### SentencePiece Tokenizer Training

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/tokenization.md

Trains a SentencePiece tokenizer using the SentencePiece library. This example shows how to specify the input data file, model prefix, and vocabulary size.

```python
import sentencepiece as spm

spm.SentencePieceTrainer.train(
    input='data/Splitted.txt',
    model_prefix='m',
    vocab_size=30000
)
```

--------------------------------

### Reduce Processes for Memory Issues

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/wiki_extraction.md

When encountering 'Out of Memory' errors with multiple processes, reduce the `--processes` value or split the dump. This example shows how to use fewer processes.

```bash
# Use fewer processes
python WikiExtractor.py -b 100M -o output --processes 4 dump.xml.bz2
```

--------------------------------

### Initialize Data Collator for MLM

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/model_training.md

Sets up a data collator for masked language modeling. It dynamically creates training examples by masking tokens with a specified probability and handles padding.

```python
from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=True,
    mlm_probability=0.15,
)
```

--------------------------------

### Get Data Directories

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/text_processing.md

Lists all subdirectories within a specified root directory. It returns a list of directory names, not their full paths.

```python
def get_data_dirs(root_dir: str) -> List[str]:
```

```python
from src.process_wiki_files import get_data_dirs

dirs = get_data_dirs("/data/wiki")
# Returns: ['AA', 'AB', 'AC', ...]
```

--------------------------------

### Example Usage of add_cc_month

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/deduplication.md

Shows how to use the `add_cc_month` function to process all relevant gzip files within a specified Common Crawl month directory. The function returns a boolean indicating the success of the batch processing.

```python
from pathlib import Path
from Hashing.Insert2Mongo import add_cc_month

cc_month_dir = Path("/data/CC-2019-09/")
success = add_cc_month(cc_month_dir)
print(f"Processing {'succeeded' if success else 'failed'}")
```

--------------------------------

### Combined Text Processing Pipeline

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/text_processing.md

This sequence of commands outlines a full pipeline for preparing text data, starting with Wikipedia extraction, followed by processing Wikipedia files, and then CC-Net files. The output is ready for subsequent deduplication and training steps.

```bash
# 1. Extract Wikipedia
python src/WikiExtractor.py -b 100M -o data/wiki dewiki-*.xml.bz2

# 2. Process extracted files
python src/process_wiki_files.py

# 3. Process CC-Net
python src/process_cc_net_files.py data/cc-net-*.gz

# 4. Combined output ready for deduplication and training

```

--------------------------------

### Set up Basic Logging

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Configure the Python logging module to display INFO level messages and above. Useful for basic debugging.

```python
import logging
logging.basicConfig(level=logging.INFO)
```

--------------------------------

### Compile Regular Expression

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/errors.md

Compiles a regular expression pattern. This example shows a valid pattern for HTML tags.

```python
html_tag_patten = re.compile('<[^<>]+>')
# Typo in variable name but pattern is valid
```

--------------------------------

### Configure Google Cloud Storage Client

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Set up a Google Cloud Storage client to interact with buckets and blobs. Ensure you have the necessary permissions.

```python
from google.cloud import storage

storage_client = storage.Client()
bucket = storage_client.bucket("bucket-name")
blob = bucket.blob("path/to/file")
```

--------------------------------

### Download and Run Benchmark

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/benchmarking.md

Instructions for downloading the GermEval 2018 dataset and running the evaluation script. Ensure the dataset files are placed in the correct directory before execution.

```bash
# Download dataset
mkdir -p data/germeval18
# Place train.tsv and test.tsv in data/germeval18/

# Run evaluation
python src/germeval18_benchmark.py
```

--------------------------------

### Initialize Device Settings for GPU/TPU

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Set up device settings, including options for GPU acceleration and automatic mixed precision. This is crucial for leveraging hardware acceleration.

```python
device, n_gpu = initialize_device_settings(use_cuda=True, use_amp=None)
```

--------------------------------

### Set up TPU Cluster

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Use this command to provision a TPU cluster on Google Cloud. Specify zone, TensorFlow version, preemptibility, name, and TPU size.

```bash
ctpu up --zone=europe-west4-a \
        --tf-version=1.15 \
        --preemptible \
        --name=tpu-testv13 \
        --tpu-size=v3-8
```

--------------------------------

### Connect to TPU Cluster and Initialize System

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/Training.md

This Python code initializes the connection to the TPU cluster and makes the TPU devices available for use. It's essential for any TPU-based training.

```python
resolver = tf.distribute.cluster_resolver.TPUClusterResolver()
tf.config.experimental_connect_to_cluster(resolver)
```

```python
tf.tpu.experimental.initialize_tpu_system(resolver)
#print("All devices: ", tf.config.list_logical_devices('TPU'))
```

--------------------------------

### XML Format Example (Wikipedia Extractor)

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/types.md

Default XML output format from the Wikipedia Extractor module. Includes document metadata and text content.

```xml
<doc id="12345" revid="67890" url="https://..." title="Article Title">
Text content here.
Multiple paragraphs.
</doc>
```

--------------------------------

### Set Google Cloud Credentials and Project

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Configure environment variables for Google Cloud Storage access. Ensure the path to your credentials JSON file is correct.

```python
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/path/to/credentials.json"
os.environ["GCLOUD_PROJECT"] = "project-id"
```

--------------------------------

### Prepare Text Dataset

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/model_training.md

Creates a dataset from a text file for training. Each line in the file is treated as a separate document. Ensure `block_size` matches the tokenizer's `max_len`.

```python
from transformers import LineByLineTextDataset

dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path=text_corpus_file,
    block_size=tokenizer_max_len,
)
```

--------------------------------

### Connect to MongoDB

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/errors.md

Establishes a connection to a MongoDB instance. An error will be raised if the MongoDB server is not running at the specified address.

```python
client = MongoClient('mongodb://localhost:27017/')
# Error raised here if MongoDB not running
```

--------------------------------

### Define Special Tokens for Tokenizer

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/tokenization.md

Lists the special tokens to be included during the ByteLevelBPE tokenizer training. These tokens have specific roles like sequence start, padding, and unknown.

```python
special_tokens = [
    "<s>",      # Start of sequence
    "<pad>",    # Padding token
    "</s>",     # End of sequence
    "<unk>",    # Unknown token
    "<mask>",   # Masked token (for MLM)
]
```

--------------------------------

### ByteLevelBPE Special Tokens

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Defines a list of special tokens to be included in the vocabulary for the ByteLevelBPE tokenizer. These tokens have specific meanings like start, padding, end, unknown, and mask.

```python
special_tokens = [
    "<s>",      # Start
    "<pad>",    # Padding
    "</s>",     # End
    "<unk>",    # Unknown
    "<mask>",   # Mask
]
```

--------------------------------

### Complete Model Training Workflow

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/model_training.md

This snippet outlines the entire process of training a masked language model. It covers configuration, data loading, model creation, and the training execution itself. Ensure all necessary libraries are imported before running.

```python
# 1. Configure model
config = RobertaConfig(
    vocab_size=50_265,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)

# 2. Load tokenizer
tokenizer = RobertaTokenizerFast.from_pretrained(
    "de-wiki-talk",
    max_len=512,
)

# 3. Create model
model = RobertaForMaskedLM(config=config)

# 4. Prepare dataset
dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path='corpus.txt',
    block_size=512,
)

# 5. Create data collator
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=True,
    mlm_probability=0.15,
)

# 6. Configure training
training_args = TrainingArguments(
    output_dir='model_output',
    overwrite_output_dir=True,
    max_steps=500_000,
    per_device_train_batch_size=256,
    save_steps=50_000,
    save_total_limit=3,
)

# 7. Train model
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)

trainer.train()
trainer.save_model('model_output')
tokenizer.save_pretrained('model_output')
```

--------------------------------

### Build Pre-training Dataset

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/model_cards/electra-base-german-uncased.md

Command to build the TensorFlow dataset for pre-training. Ensure to replace `<corpus_dir>` with the actual directory containing your corpus and `<dir>/vocab.txt` with the path to your vocabulary file. This command utilizes the `no-strip-accents` branch of the Electra repository.

```bash
python build_pretraining_dataset.py --corpus-dir <corpus_dir> --vocab-file <dir>/vocab.txt --output-dir ./tf_data --max-seq-length 512 --num-processes 8 --do-lower-case --no-strip-accents
```

--------------------------------

### Load Model Checkpoint

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/benchmarking.md

Load a pre-trained language model and its tokenizer from a specified directory. Ensure the directory contains model weights, configuration, and tokenizer files.

```python
# Load model trained by 02_train_01.py
lang_model = "/home/phmay/data/nlp/checkpoints_256/model-electra"

tokenizer = AutoTokenizer.from_pretrained(lang_model)
language_model = LanguageModel.load(lang_model)
```

--------------------------------

### Configure RoBERTa Pretraining

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/model_training.md

Sets up essential configuration variables for RoBERTa pretraining, including save directory, corpus file path, and hyperparameters like tokenizer max length, vocabulary size, batch size, and training steps.

```python
save_dir = "de-wiki-talk"
text_corpus_file = '/home/phmay/data/ml-data/gtt/dewiki-talk-20200620-split/xaa'

# Hyperparameters
tokenizer_max_len = 512
vocab_size = 50_265
batch_size = 8              # For testing; use 8_000+ for production
max_steps = 20_000          # For testing; use 500_000+ for production
```

--------------------------------

### TFRecord TFExample Class Definition

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/types.md

Python class definition for TFExample, representing a TensorFlow Example structure used for BERT pretraining tasks. It outlines the expected fields for token IDs, masks, and labels.

```python
class TFExample:
    """TensorFlow Example for MLM task"""
    input_ids: List[int]           # Token IDs [CLS] ... [SEP]
    input_mask: List[int]          # Attention mask (1 for real, 0 for padding)
    segment_ids: List[int]         # Segment IDs
    masked_lm_positions: List[int] # Positions of masked tokens
    masked_lm_ids: List[int]       # True token IDs for masked positions
    masked_lm_weights: List[float] # Loss weights (1.0 or 0.0)
```

--------------------------------

### Set UTF-8 Encoding for Special Characters

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/wiki_extraction.md

To resolve issues with special characters like German umlauts, ensure UTF-8 encoding is used by setting the locale environment variables. This example demonstrates setting `LC_ALL` and `LANG`.

```bash
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
python WikiExtractor.py ...
```

--------------------------------

### Initialize RoBERTaForMaskedLM Model

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/model_training.md

Initializes a RoBERTa model for masked language modeling from scratch using the provided configuration. This model is ready for pretraining without any pre-existing weights.

```python
from transformers import RobertaForMaskedLM

model = RobertaForMaskedLM(config=config)
```

--------------------------------

### Flexible Path Parameter Types

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/types.md

Demonstrates the use of `Union[str, Path]` for flexible path parameters, allowing either a string or a `pathlib.Path` object.

```python
from pathlib import Path

# Flexible path parameter types found in codebase
path: Union[str, Path]            # Can be string or Path object
```

--------------------------------

### Configure Training Arguments

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/model_training.md

Use `TrainingArguments` to define hyperparameters and settings for your training process. Specify output directories, training steps, batch sizes, and checkpointing behavior.

```python
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir=save_dir,
    overwrite_output_dir=True,
    max_steps=max_steps,
    per_device_train_batch_size=batch_size,
    save_steps=10_000,
    save_total_limit=2,
)
```

--------------------------------

### Initialize TPU System in Python

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Connect to and initialize a TPU system using TensorFlow. This is required before using TPUs for distributed training.

```python
import tensorflow as tf

# Connect to TPU
resolver = tf.distribute.cluster_resolver.TPUClusterResolver()
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)

# Training with TPU strategy
strategy = tf.distribute.TPUStrategy(resolver)
with strategy.scope():
    # Define model
    pass
```

--------------------------------

### Union Types for File Paths, Optional Parameters, and Return Values

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/types.md

Demonstrates the usage of Python's Union type for defining file paths, optional parameters, and return values that can be of multiple types or None.

```python
# File paths
path: Union[str, Path]

# Optional parameters
Optional[str] = Union[str, None]

# Return values
Union[int, None]                   # Can return int or None
Union[List[str], None]
```

--------------------------------

### Train and Save ByteLevelBPE Tokenizer

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/tokenization.md

Initializes and trains a ByteLevelBPE tokenizer using specified files, vocabulary size, and special tokens. The trained tokenizer and its configuration are then saved to disk.

```python
from tokenizers import ByteLevelBPETokenizer

tokenizer = ByteLevelBPETokenizer()
tokenizer.train(
    files=[text_corpus_file],
    vocab_size=vocab_size,
    min_frequency=2,
    special_tokens=special_tokens
)

tokenizer.save_model(save_dir)
tokenizer.save(save_dir + "/tokenizer.json")
```

--------------------------------

### Configure ByteLevelBPE Tokenizer Training

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/tokenization.md

Defines configuration variables for training a ByteLevelBPE tokenizer, including save directory, corpus file path, and vocabulary size.

```python
save_dir = "de-wiki-talk"
text_corpus_file = '/home/phmay/data/ml-data/gtt/dewiki-talk-20200620-split/xaa'
vocab_size = 50_265
```

--------------------------------

### BERT Pretraining Data Creation

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/tokenization.md

Command-line script to create pretraining data for BERT. It specifies input and output files, vocabulary file, and various parameters for sequence length, masking, and whole word masking.

```bash
python create_pretraining_data.py \
  --input_file=Model/merged_wiki.txt \
  --output_file=tmp/tf_examples.tfrecord \
  --vocab_file=Model/bert_german-vocab.txt \
  --do_lower_case=True \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --masked_lm_prob=0.15 \
  --random_seed=12345 \
  --dupe_factor=5 \
  --do_whole_word_mask=True
```

--------------------------------

### is_doc_start_line()

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/text_processing.md

Checks if a given line marks the beginning of a document in the WikiExtractor output.

```APIDOC
## is_doc_start_line()

### Description
Checks if a given line marks the beginning of a document in the WikiExtractor output.

### Signature
```python
def is_doc_start_line(line: str) -> bool
```

### Parameters
#### Path Parameters
- **line** (str) - Yes - Line from Wikipedia extracted file

### Returns
`bool` — True if line starts with `<doc`
```

--------------------------------

### Track Progress with tqdm

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Use the `tqdm` library to display a progress bar for iterators. This helps in monitoring long-running processes.

```python
from tqdm import tqdm
for line in tqdm(iterator):
    process(line)
```

--------------------------------

### Desktop Configuration for Sentence Extraction

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/sentence_extraction.md

Use this configuration on a local desktop environment. It specifies paths for vocabulary files, input data, temporary directories, TensorFlow output, and the BERT model.

```python
VOCAB_FILE = "/media/data/48_BERT/german-transformer-training/src/vocab.txt"
THREADS = 8
IN_DIR = "/media/data/48_BERT/german-transformer-training/data/head"
TMP_DIR = "/media/data/48_BERT/german-transformer-training/data/tmp"
TF_OUT_DIR = "/media/data/48_BERT/german-transformer-training/data/tf_rec"
BERT_PATH = "../../01_BERT_Code/bert/"
```

--------------------------------

### Initialize RobertaConfig

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/types.md

Initializes a RobertaConfig object from Hugging Face Transformers with specified parameters for model architecture.

```python
from transformers import RobertaConfig

config = RobertaConfig(
    vocab_size: int = 50265
    max_position_embeddings: int = 514
    num_attention_heads: int = 12
    num_hidden_layers: int = 6
    type_vocab_size: int = 1
    hidden_size: int = 768          # Derived from other params
    intermediate_size: int = 3072
    hidden_act: str = "gelu"
    hidden_dropout_prob: float = 0.1
    attention_probs_dropout_prob: float = 0.1
    layer_norm_eps: float = 1e-12
)
```

--------------------------------

### Run BERT Pretraining Data Generation Command

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/sentence_extraction.md

Executes subprocess calls to TensorFlow's create_pretraining_data.py script for parallelized data generation. Requires BERT model path, input file, output directory, and an index.

```python
def run_command(cmd_vars: Tuple[str, str, str, int]) -> None:
    # ... (implementation details omitted for brevity)
    pass
```

```python
from multiprocessing import Pool
from src.Sentence_Extraction import run_command

cmd_vars = [
    ("/path/to/bert/", "/tmp/data.txt", "/tmp/tf_records", 0),
    ("/path/to/bert/", "/tmp/data2.txt", "/tmp/tf_records", 1),
]

with Pool(processes=8) as pool:
    pool.map(run_command, cmd_vars)
```

--------------------------------

### Detect Deployment Environment

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Detects the deployment environment based on the system's hostname to apply specific configurations. This is useful for differentiating between local development and server environments.

```python
import socket
if socket.gethostname() == "philipp-desktop":
    # Desktop configuration
else:
    # Server configuration
```

--------------------------------

### Recommended Error Handling with Logging

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/errors.md

This recommended approach uses explicit logging for SyntaxError and other unexpected exceptions, improving debuggability for production environments. It logs warnings for parse errors and re-raises unexpected exceptions.

```python
try:
    scraped = eval(line)
except SyntaxError as e:
    logging.warning(f"Parse error on line {i}: {e}")
    continue
except Exception as e:
    logging.error(f"Unexpected error: {e}")
    raise
```

--------------------------------

### Initialize TrainingArguments

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/types.md

Initializes a TrainingArguments object from Hugging Face Transformers to configure the training process, including output directory and learning rate.

```python
from transformers import TrainingArguments

args = TrainingArguments(
    output_dir: str                          # Checkpoint directory
    overwrite_output_dir: bool = False
    do_train: bool = True
    do_eval: bool = False
    do_predict: bool = False
    evaluation_strategy: str = "no"
    eval_steps: Optional[int] = None
    learning_rate: float = 5e-5
    per_device_train_batch_size: int = 8
    per_device_eval_batch_size: int = 8
    num_train_epochs: float = 3.0
    max_steps: int = -1
    warmup_steps: int = 0
    warmup_ratio: float = 0.0
    weight_decay: float = 0.0
    save_total_limit: Optional[int] = None
    save_steps: float = 500
    logging_steps: int = 500
    save_strategy: str = "steps"
    seed: int = 42
)
```

--------------------------------

### GermEval 2018 Benchmark Configuration

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/benchmarking.md

Defines configuration parameters for evaluating German transformer models on the GermEval 2018 task. Includes settings for training epochs, batch size, evaluation frequency, model checkpoint, repeats, and model selection.

```python
n_epochs = 1
batch_size = 32
evaluate_every = 15
cp_num = 650_000  # Model checkpoint number
repeats = 31      # Number of evaluation repeats
do_lower_case = True

# Model selection
lang_model = "/home/phmay/data/nlp/checkpoints_256/model-electra"
label_list = ["OTHER", "OFFENSE"]
metric = "f1_macro"
```

--------------------------------

### Initialize RoBERTaConfig

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/model_training.md

Defines the configuration for the RoBERTa model architecture. Ensure `max_position_embeddings` is set to `tokenizer_max_len + 2` to avoid CUDA errors. `type_vocab_size=1` is standard for pretraining.

```python
config = RobertaConfig(
    vocab_size=vocab_size,
    max_position_embeddings=tokenizer_max_len + 2,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)
```

--------------------------------

### Desktop Environment Configuration

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Configuration variables for the desktop environment, including file paths, thread count, and BERT model path. These are typically used for development and testing on a single GPU machine.

```python
VOCAB_FILE = "/media/data/48_BERT/german-transformer-training/src/vocab.txt"
THREADS = 8
IN_DIR = "/media/data/48_BERT/german-transformer-training/data/head"
TRANSFER_DIR = "/media/data/48_BERT/german-transformer-training/data/download"
TMP_DIR = "/media/data/48_BERT/german-transformer-training/data/tmp"
TF_OUT_DIR = "/media/data/48_BERT/german-transformer-training/data/tf_rec"
BERT_PATH = "../../01_BERT_Code/bert/"
```

--------------------------------

### Training Hyperparameters

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Sets essential hyperparameters for the training process. Batch size and max_steps should be tuned based on available hardware and desired training duration.

```python
tokenizer_max_len = 512
batch_size = 8              # Development: 8, Production: 256+
max_steps = 20_000          # Development: 20K, Production: 500K+
learning_rate = 2e-5        # From BERT paper
num_warmup_steps = 3000     # 10% of total steps typical
weight_decay = 0.01         # L2 regularization
```

--------------------------------

### MongoDB Connection Configuration for Deduplication

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Sets up the MongoDB connection URI and database/collection names for deduplication tasks.

```python
mongo_uri = 'mongodb://localhost:27017/'
database = 'common_crawl'
collection = 'main'
```

--------------------------------

### Project Structure Overview

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/PROJECT_OVERVIEW.md

This snippet displays the directory structure of the german-transformer-training project, highlighting key source code, data deduplication utilities, model cards, and documentation files.

```tree
german-transformer-training/
├── src/                          # Main source code directory
│   ├── 01_tokenize_01.py        # ByteLevelBPE tokenizer training
│   ├── 01_tokenize_electra.py   # ELECTRA tokenizer training
│   ├── 02_train_01.py           # RoBERTa model pretraining
│   ├── Tokenizer.py             # BERT tokenizer utilities
│   ├── Sentence_Extraction.py   # Sentence splitting and text processing
│   ├── WikiExtractor.py         # Wikipedia dump extraction
│   ├── WikiExtractor-talk.py    # Wikipedia talk pages extraction
│   ├── germeval18_benchmark.py  # Text classification benchmarking
│   ├── process_cc_net_files.py  # Common Crawl cc_net processing
│   ├── process_wiki_files.py    # Wikipedia file processing
│   └── process_opensubtitles_files.py  # Subtitle data processing
├── Hashing/                      # Data deduplication utilities
│   ├── Dedup.py                 # Duplicate removal using MongoDB
│   └── Insert2Mongo.py          # Metadata insertion to MongoDB
├── model_cards/                 # Model documentation
│   └── electra-base-german-uncased.md
├── README.md                    # Project overview and dataset references
├── Training.md                  # TPU training instructions
└── data-prep.md                # Data preparation guidelines
```

--------------------------------

### Wiki Processing Configuration

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/text_processing.md

Defines input and output directories for WikiExtractor processed data. Ensure these paths are correctly set for your project structure.

```python
INPUT_DIR = "../data/wiki/"        # WikiExtractor output
OUTPUT_DIR = "../output/wiki/"     # Processed sentences
```

--------------------------------

### Configuration Classes

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/README.md

Available configuration classes for model architecture, training parameters, and dataset handling.

```APIDOC
## Configuration Classes

### `RobertaConfig`

**Description**: Defines the architecture for RoBERTa models.

### `TrainingArguments`

**Description**: Specifies parameters for the training process.

### `LineByLineTextDataset`

**Description**: Handles loading text datasets line by line.

### `DataCollatorForLanguageModeling`

**Description**: Prepares batches for language modeling tasks.

### FARM Classes

**Description**: Various classes from the FARM framework are available for benchmarking purposes.
```

--------------------------------

### Initialize TPUClusterResolver

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/Training.md

This Python code is required at the beginning of your script to connect to the TPU cluster. No environment variables need to be set if running within Cloud Shell.

```python
$ tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver()
```

--------------------------------

### CC-Net Processing Configuration

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/text_processing.md

Shows how to access the input filename from command-line arguments for CC-Net processing. The output filename is derived from the input.

```python
# Command line argument
input_filename = args.filename
# Output: {input_filename}-out.gz
```

--------------------------------

### Local Model Path Configuration

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/benchmarking.md

Specifies the local path to a pre-trained language model for use in the benchmark.

```python
lang_model = "/home/phmay/data/nlp/checkpoints_256/model-electra"
```

--------------------------------

### Capture TPU Profile

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/Training.md

This command is used for analyzing TPU performance. It requires the TPU name and a monitoring level to be specified.

```bash
capture_tpu_profile --tpu=$TPU_NAME  --monitoring_level=2
```

--------------------------------

### SoMaJo Tokenizer Configuration

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/configuration.md

Initializes the SoMaJo tokenizer for German text, with an option to split camel case words.

```python
tokenizer = SoMaJo("de_CMC", split_camel_case=True)
```

--------------------------------

### Check Model Path Existence

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/errors.md

Before loading a model, verify that the specified path exists to prevent FileNotFoundError. This is crucial for ensuring model checkpoints are accessible.

```python
from pathlib import Path
assert Path(lang_model).exists(), f"Model not found: {lang_model}"
```

--------------------------------

### add_cc_month()

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/deduplication.md

Batch processes all gzip files in a directory for a Common Crawl month using multiprocessing. It iterates through specified files, spawns a separate process for each using `add2mongo()`, and logs completed files.

```APIDOC
## add_cc_month()

### Description
Batch processes all gzip files in a directory for a Common Crawl month using multiprocessing.

### Parameters
#### Path Parameters
- **path** (Path) - Required - Directory containing gzip files

### Returns
`bool` — True if all files processed successfully.

### Behavior
1. Iterates through all files in directory
2. Filters for `.gz` files containing 'head' in filename
3. Spawns separate process for each file using `add2mongo()`
4. Logs processed files to `finished.txt`
5. Waits for all processes to complete

### Multiprocessing Details
- One process per gzip file
- Each process runs independently
- Blocks until all processes finish with `p.join()`

### Output
File `finished.txt` with one filename per line, tracking completed files.

### Example Usage
```python
from pathlib import Path
from Hashing.Insert2Mongo import add_cc_month

cc_month_dir = Path("/data/CC-2019-09/")
success = add_cc_month(cc_month_dir)
print(f"Processing {'succeeded' if success else 'failed'}")
```
```

--------------------------------

### split()

Source: https://github.com/german-nlp-group/german-transformer-training/blob/master/_autodocs/api-reference/sentence_extraction.md

Distributed Ray task for splitting text into sentences using the SoMaJo German tokenizer. It processes a list of text documents and writes the tokenized sentences to a file.

```APIDOC
## split()

### Description
Distributed Ray task for splitting text into sentences using the SoMaJo German tokenizer. It processes a list of text documents and writes the tokenized sentences to a file.

### Method
`@ray.remote def split(list_of_text: List[str], thread_number: int, TMP_DIR: str) -> int`

### Parameters
#### Path Parameters
- **list_of_text** (List[str]) - Required - List of text documents/paragraphs to split
- **thread_number** (int) - Required - Unique identifier for this thread's output file
- **TMP_DIR** (str) - Required - Temporary directory path where split output files are written

### Returns
- **int** - The thread_number, used for tracking completion in Ray distributed execution.

### Behavior
- Uses `SoMaJo("de_CMC", split_camel_case=True)` for German sentence tokenization
- Writes one sentence per line with leading spaces for Byte-Pair Encoding (BPE) preprocessing
- Separates documents with blank lines
- Outputs to file: `TMP_DIR/Splitted_{thread_number:05d}.txt`
- Handles space_after, first_in_sentence, and last_in_sentence token attributes correctly

### Example Usage
```python
import ray
from src.Sentence_Extraction import split

ray.init(num_cpus=8)

texts = [
    "Das ist ein Satz. Das ist ein anderer Satz.",
    "Ein neues Dokument. Mit mehreren Sätzen.",
]

result = split.remote(texts, thread_number=1, TMP_DIR="/tmp/output")
thread_id = ray.get(result)
print(f"Completed thread {thread_id}")

ray.shutdown()
```
```