### Complete Training Pipeline

Source: https://context7.com/climatebert/language-model/llms.txt

An end-to-end example combining model initialization, vocabulary augmentation, and tokenizer setup.

```python
import math
import random
from sklearn.model_selection import train_test_split
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForMaskedLM,
    AutoModelForPreTraining,
    Trainer,
    TrainingArguments,
    DataCollatorForLanguageModeling
)
from transformers_domain_adaptation import DataSelector, VocabAugmentor

# Step 1: Load base model
base_model = "distilroberta-base"
tokenizer = AutoTokenizer.from_pretrained(base_model, use_fast=True)
model = AutoModelForMaskedLM.from_pretrained(base_model)

# Step 2: Augment tokenizer with climate vocabulary
augmentor = VocabAugmentor(tokenizer=tokenizer, cased=True, target_vocab_size=50_500)
new_tokens = augmentor.get_new_tokens('./fine_tuning_texts.txt')
tokenizer.add_tokens(new_tokens)
model.resize_token_embeddings(len(tokenizer))
```

--------------------------------

### Initialize and Train Model

Source: https://context7.com/climatebert/language-model/llms.txt

Initializes the Hugging Face Trainer with the model, training arguments, datasets, data collator, and tokenizer. Subsequently, it starts the training process and saves the final model to a specified directory. A confirmation message is printed upon completion.

```python
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer
)

trainer.train()
trainer.save_model("model/climatebert-final")
print("Training complete!")
```

--------------------------------

### Import Required Libraries

Source: https://github.com/climatebert/language-model/blob/main/tokenizer_augmentation.ipynb

Initializes the necessary transformers, datasets, and domain adaptation modules.

```python
from transformers import AutoTokenizer
from transformers import Trainer, TrainingArguments
from transformers import AutoModelForMaskedLM
from transformers import DataCollatorForLanguageModeling
from sklearn.model_selection import train_test_split
from datasets import load_dataset

import random
import math
from transformers_domain_adaptation import DataSelector, VocabAugmentor
```

--------------------------------

### Execute Training Loop

Source: https://context7.com/climatebert/language-model/llms.txt

Initialize the Trainer, run the training process, and evaluate the model performance.

```python
# Initialize trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer
)

# Start training
trainer.train()

# Evaluate model performance
eval_results = trainer.evaluate()
print(f"Evaluation loss: {eval_results['eval_loss']}")
print(f"Perplexity: {math.exp(eval_results['eval_loss'])}")

# Save final model
trainer.save_model("model/climatebert-final")
```

--------------------------------

### Initialize Model and Tokenizer

Source: https://github.com/climatebert/language-model/blob/main/language_modeling.ipynb

Loads the pretrained model and tokenizer from the specified directory.

```python
card = "model/distilroberta-base-augmented"
tokenizer = AutoTokenizer.from_pretrained(card, use_fast=True)
model = AutoModelForPreTraining.from_pretrained(card)
```

--------------------------------

### Load Base Model and Tokenizer

Source: https://context7.com/climatebert/language-model/llms.txt

Initialize the foundation model and tokenizer for domain adaptation, defaulting to DistilRoBERTa.

```python
# Load base model and tokenizer
card = "distilroberta-base"

tokenizer = AutoTokenizer.from_pretrained(card, use_fast=True)
model = AutoModelForMaskedLM.from_pretrained(card)

print("Current vocab size:", len(tokenizer))
# Output: Current vocab size: 50265
```

--------------------------------

### Initialize Trainer

Source: https://github.com/climatebert/language-model/blob/main/language_modeling.ipynb

Creates the Trainer instance with the model, arguments, and datasets.

```python
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer
)
```

--------------------------------

### Initialize Base Model and Tokenizer

Source: https://github.com/climatebert/language-model/blob/main/tokenizer_augmentation.ipynb

Loads the pre-trained distilroberta-base model and its corresponding tokenizer.

```python
card = "distilroberta-base"

tokenizer = AutoTokenizer.from_pretrained(card, use_fast=True) 
model = AutoModelForMaskedLM.from_pretrained(card)
```

--------------------------------

### Initialize Data Collator

Source: https://github.com/climatebert/language-model/blob/main/language_modeling.ipynb

Sets up the data collator for masked language modeling.

```python
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)
```

--------------------------------

### Configure Training Arguments

Source: https://context7.com/climatebert/language-model/llms.txt

Define hyperparameters and training settings such as batch size, learning rate, and mixed precision.

```python
training_args = TrainingArguments(
    output_dir="model/climatebert",
    overwrite_output_dir=False,
    per_device_train_batch_size=24,      # Adjust based on GPU memory
    per_device_eval_batch_size=24,       # Adjust based on GPU memory
    evaluation_strategy="epoch",
    save_strategy="epoch",
    fp16=True,                           # Enable mixed precision training
    dataloader_num_workers=8,            # Adjust based on CPU cores
    load_best_model_at_end=True,
    gradient_accumulation_steps=42,      # Effective batch size = 24 * 42 = 1008
    num_train_epochs=12,                 # Training epochs
    learning_rate=0.0005,
    weight_decay=0.01,
)
```

--------------------------------

### Configure Training Arguments

Source: https://github.com/climatebert/language-model/blob/main/language_modeling.ipynb

Sets hyperparameters and training configuration for the Trainer.

```python
# For further arguments, see Hugging Face docs

training_args = TrainingArguments(
    output_dir="model/xyz",
    overwrite_output_dir=False,     # Attention!
    per_device_train_batch_size=24, # Adjust depending on machine
    per_device_eval_batch_size=24,  # Adjust depending on machine
    evaluation_strategy="epoch",
    save_strategy="epoch",
    fp16=True,                      # Adjust depending on machine
    dataloader_num_workers=8,       # Adjust depending on machine
    load_best_model_at_end=True,
    gradient_accumulation_steps=42, # Adjust depending on machine
    num_train_epochs=12,            # After 12 epochs the models didn't improve any further on our corpus 
    learning_rate=0.0005,
    weight_decay=0.01,
)
```

--------------------------------

### Execute Training and Evaluation

Source: https://github.com/climatebert/language-model/blob/main/language_modeling.ipynb

Runs the training process, evaluates the model, and saves the final weights.

```python
trainer.train()
```

```python
trainer.evaluate()
```

```python
trainer.save_model("model/xyz")
```

--------------------------------

### Import Required Libraries

Source: https://context7.com/climatebert/language-model/llms.txt

Load necessary modules from Hugging Face Transformers, scikit-learn, and the domain adaptation package.

```python
from transformers import AutoTokenizer
from transformers import Trainer, TrainingArguments
from transformers import AutoModelForMaskedLM
from transformers import AutoModelForPreTraining
from transformers import DataCollatorForLanguageModeling
from sklearn.model_selection import train_test_split
from datasets import load_dataset

import random
import math
from transformers_domain_adaptation import DataSelector, VocabAugmentor
```

--------------------------------

### Prepare Training Corpus

Source: https://context7.com/climatebert/language-model/llms.txt

Read text data from a file and split it into training and validation sets for model training.

```python
# Read corpus from file
with open('./corpus.txt', 'r') as file:
    corpus = [line.rstrip("\n") for line in file.readlines()]

# Perform train-test split
train_text, val_text = train_test_split(corpus, test_size=0.15, random_state=42)

# Save train corpus
with open('corpus/train_corpus.txt', 'w') as file:
    for paragraph in train_text:
        file.write(paragraph + "\n")

# Save validation corpus
with open('corpus/val_corpus.txt', 'w') as file:
    for paragraph in val_text:
        file.write(paragraph + "\n")

print(f"Training samples: {len(train_text)}")
print(f"Validation samples: {len(val_text)}")
```

--------------------------------

### Import Libraries

Source: https://github.com/climatebert/language-model/blob/main/language_modeling.ipynb

Imports necessary modules from transformers, datasets, and torch libraries.

```python
from transformers import AutoTokenizer
from transformers import Trainer, TrainingArguments
from transformers import AutoModelForPreTraining
from transformers import DataCollatorForLanguageModeling

from datasets import load_dataset

import torch
# torch.cuda.empty_cache()
```

--------------------------------

### Load Datasets for Training

Source: https://context7.com/climatebert/language-model/llms.txt

Load training and validation text files using the Hugging Face datasets library.

```python
# Load text datasets
datasets = load_dataset(
    "text",
    data_files={
        "train": 'corpus/train_corpus.txt',
        "validation": 'corpus/val_corpus.txt'
    }
)

print(f"Training samples: {len(datasets['train'])}")
print(f"Validation samples: {len(datasets['validation'])}")
```

--------------------------------

### Load Datasets for Training

Source: https://context7.com/climatebert/language-model/llms.txt

Loads training and validation datasets from text files. Specify the correct paths for 'train_corpus.txt' and 'val_corpus.txt'.

```python
datasets = load_dataset("text", data_files={
    "train": 'corpus/train_corpus.txt',
    "validation": 'corpus/val_corpus.txt'
})
```

--------------------------------

### Configure Hugging Face Trainer

Source: https://context7.com/climatebert/language-model/llms.txt

Defines the training arguments for the Hugging Face Trainer. Key parameters include output directory, batch size, evaluation and save strategies, mixed precision (fp16), number of epochs, learning rate, weight decay, and loading the best model at the end.

```python
training_args = TrainingArguments(
    output_dir="model/climatebert",
    per_device_train_batch_size=24,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    fp16=True,
    num_train_epochs=12,
    learning_rate=0.0005,
    weight_decay=0.01,
    load_best_model_at_end=True,
)
```

--------------------------------

### Configure Data Collator

Source: https://context7.com/climatebert/language-model/llms.txt

Initialize the data collator for masked language modeling with a specified masking probability.

```python
# Initialize data collator with 15% masking probability
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=True,
    mlm_probability=0.15
)
```

--------------------------------

### Load Datasets

Source: https://github.com/climatebert/language-model/blob/main/language_modeling.ipynb

Loads training and validation corpora from text files using the Hugging Face datasets library.

```python
datasets = load_dataset("text", data_files={"train": 'corpus/train_corpus.txt',         # Path to txt file with training corpus (selected or not)
                                            "validation": 'corpus/val_corpus.txt'})     # Path to txt file with validation corpus
```

```python
print(len(datasets['train']))
print(len(datasets['validation']))
```

--------------------------------

### Select Data for Domain Adaptation

Source: https://github.com/climatebert/language-model/blob/main/tokenizer_augmentation.ipynb

Batches the training data and uses DataSelector to filter content based on diversity metrics.

```python
# Batch data due to memory limits

batch_size_fine_tuning = 200
batch_size_corpus = 100000
train_text_copy = train_text.copy()
random.shuffle(train_text_copy)

selected_corpus = []
j = 1

for i in range(0, len(train_text_copy), batch_size_corpus):
    print(f"Processing batch {j} of {math.ceil(len(train_text_copy) / batch_size_corpus)}...")
    j += 1
    train_text_batch = train_text_copy[i:i + batch_size_corpus]
    
    selector = DataSelector(
        keep=0.7,
        tokenizer=tokenizer,
        #similarity_metrics=['euclidean'],
        diversity_metrics=[
            "type_token_ratio",
            "entropy",
        ],
    )
    
    selector.fit(random.sample(fine_tuning_texts, batch_size_fine_tuning))

    selected_corpus.extend(selector.transform(train_text_batch))
```

--------------------------------

### Tokenize Datasets

Source: https://context7.com/climatebert/language-model/llms.txt

Apply the tokenizer to the dataset with parallel processing for efficient data preparation.

```python
# Define tokenization function
def tokenize_function(samples):
    return tokenizer(samples["text"], truncation=True)

# Tokenize datasets with parallel processing
tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=16,  # Adjust based on available CPU cores
    remove_columns=["text"]
)

print(f"Tokenized training samples: {len(tokenized_datasets['train'])}")
print(f"Tokenized validation samples: {len(tokenized_datasets['validation'])}")
```

--------------------------------

### Split Corpus into Training and Validation Sets

Source: https://github.com/climatebert/language-model/blob/main/tokenizer_augmentation.ipynb

Reads the raw corpus file and splits it into training and validation subsets for file storage.

```python
with open('./corpus.txt', 'r') as file:
    corpus = [line.rstrip("\n") for line in file.readlines()]    

train_text, val_text = train_test_split(corpus, test_size=0.15, random_state=42)

with open('corpus/train_corpus.txt', 'w') as file:
    for paragraph in train_text:
        file.write(paragraph + "\n")
        
with open('corpus/val_corpus.txt', 'w') as file:
    for paragraph in val_text:
        file.write(paragraph + "\n")
```

--------------------------------

### Tokenize Dataset

Source: https://context7.com/climatebert/language-model/llms.txt

Defines and applies a tokenization function to the loaded datasets. This prepares the text data for model training by converting it into numerical tokens. `batched=True` and `remove_columns` optimize processing.

```python
def tokenize_function(samples):
    return tokenizer(samples["text"], truncation=True)

tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=16, remove_columns=["text"])
```

--------------------------------

### Load Augmented Model

Source: https://context7.com/climatebert/language-model/llms.txt

Load a previously saved augmented tokenizer and model, ensuring the embedding layer matches the vocabulary size.

```python
# Load augmented model and tokenizer
card = "model/distilroberta-base-augmented"
tokenizer = AutoTokenizer.from_pretrained(card, use_fast=True)
model = AutoModelForPreTraining.from_pretrained(card)

# Ensure embeddings match vocabulary size
model.resize_token_embeddings(len(tokenizer))

print(f"Model vocabulary size: {len(tokenizer)}")
```

--------------------------------

### Save Augmented Model

Source: https://context7.com/climatebert/language-model/llms.txt

Saves the tokenizer and model to a specified directory after augmentation. Ensure the directory exists or is created.

```python
tokenizer.save_pretrained("model/climatebert-augmented")
model.save_pretrained("model/climatebert-augmented")
```

--------------------------------

### Tokenize Dataset

Source: https://github.com/climatebert/language-model/blob/main/language_modeling.ipynb

Defines the tokenization function and applies it to the dataset.

```python
def tokenize_function(samples):
    return tokenizer(samples["text"], truncation=True)
```

```python
tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=16, remove_columns=["text"]) # Adjust num_proc depending on machine
```

--------------------------------

### Augment Vocabulary and Save Model

Source: https://github.com/climatebert/language-model/blob/main/tokenizer_augmentation.ipynb

Expands the tokenizer vocabulary based on fine-tuning texts, resizes model embeddings, and saves the updated artifacts.

```python
new_vocab_size = 50_500                 # Adjust to needs

augmentor = VocabAugmentor(
    tokenizer=tokenizer, 
    cased=True,                         # Depends on model
    target_vocab_size=new_vocab_size
)

new_tokens = augmentor.get_new_tokens('./fine_tuning_texts.txt')

tokenizer.add_tokens(new_tokens)

model.resize_token_embeddings(len(tokenizer))

print("New vocab size: " + str(len(tokenizer)))

tokenizer.save_pretrained("model/distilroberta-base-augmented")
model.save_pretrained("model/distilroberta-base-augmented")
```

--------------------------------

### Check Current Vocabulary Size

Source: https://github.com/climatebert/language-model/blob/main/tokenizer_augmentation.ipynb

Prints the current number of tokens in the tokenizer.

```python
print("Current vocab size: " + str(len(tokenizer)))
```

--------------------------------

### Load Downstream Task Texts

Source: https://github.com/climatebert/language-model/blob/main/tokenizer_augmentation.ipynb

Reads the text file containing the final downstream task data.

```python
with open('./fine_tuning_texts.txt', 'r') as file:
    fine_tuning_texts = [line.rstrip("\n") for line in file.readlines()]
```

--------------------------------

### Filter Data with DataSelector

Source: https://context7.com/climatebert/language-model/llms.txt

Select relevant training samples based on similarity to downstream task texts to improve training efficiency.

```python
# Load fine-tuning texts for data selection
with open('./fine_tuning_texts.txt', 'r') as file:
    fine_tuning_texts = [line.rstrip("\n") for line in file.readlines()]

# Configure batch processing for memory efficiency
batch_size_fine_tuning = 200
batch_size_corpus = 100000
train_text_copy = train_text.copy()
random.shuffle(train_text_copy)

selected_corpus = []
j = 1

# Process corpus in batches
for i in range(0, len(train_text_copy), batch_size_corpus):
    print(f"Processing batch {j} of {math.ceil(len(train_text_copy) / batch_size_corpus)}...")
    j += 1
    train_text_batch = train_text_copy[i:i + batch_size_corpus]

    # Initialize data selector with diversity metrics
    selector = DataSelector(
        keep=0.7,  # Keep 70% of most relevant data
        tokenizer=tokenizer,
        diversity_metrics=[
            "type_token_ratio",
            "entropy",
        ],
    )

    # Fit selector on downstream task texts
    selector.fit(random.sample(fine_tuning_texts, batch_size_fine_tuning))

    # Transform and select relevant corpus samples
    selected_corpus.extend(selector.transform(train_text_batch))

# Save selected corpus
with open('corpus/train_selected_corpus.txt', 'w') as file:
    for paragraph in selected_corpus:
        file.write(paragraph + "\n")

print(f"Selected corpus size: {len(selected_corpus)}")
```

--------------------------------

### Augment Tokenizer and Resize Embeddings

Source: https://context7.com/climatebert/language-model/llms.txt

Extract new domain-specific tokens from text files and update the model's embedding layer to accommodate the expanded vocabulary.

```python
new_tokens = augmentor.get_new_tokens('./fine_tuning_texts.txt')

# Add new tokens to tokenizer
tokenizer.add_tokens(new_tokens)

# Resize model embeddings to match new vocabulary
model.resize_token_embeddings(len(tokenizer))

print("New vocab size:", len(tokenizer))
# Output: New vocab size: 50500

# Save augmented tokenizer and model
tokenizer.save_pretrained("model/distilroberta-base-augmented")
model.save_pretrained("model/distilroberta-base-augmented")
```

--------------------------------

### Augment Tokenizer Vocabulary

Source: https://context7.com/climatebert/language-model/llms.txt

Expand the tokenizer vocabulary to include domain-specific terms using VocabAugmentor.

```python
# Define target vocabulary size
new_vocab_size = 50_500  # Adjust based on domain needs

# Initialize vocabulary augmentor
augmentor = VocabAugmentor(
    tokenizer=tokenizer,
    cased=True,  # Use True for cased models, False for uncased
    target_vocab_size=new_vocab_size
)
```

--------------------------------

### Resize Token Embeddings

Source: https://github.com/climatebert/language-model/blob/main/language_modeling.ipynb

Adjusts the model's token embeddings to match the tokenizer's vocabulary size.

```python
model.resize_token_embeddings(len(tokenizer))
```

--------------------------------

### Save Selected Corpus

Source: https://github.com/climatebert/language-model/blob/main/tokenizer_augmentation.ipynb

Writes the filtered corpus to a new text file.

```python
with open('corpus/train_selected_corpus.txt', 'w') as file:
    for paragraph in selected_corpus:
        file.write(paragraph + "\n")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.