### Install Project Dependencies

Source: https://github.com/leia-llm/leia/blob/main/README.md

Installs necessary Python packages for the project using pip. It includes installing requirements, the 'packaging' library, 'flash-attn' for optimized attention mechanisms, and the project itself in editable mode.

```bash
pip install -r requirements.txt
pip install packaging
pip install flash-attn --no-build-isolation
pip install -e .
```

--------------------------------

### Iterate through examples and predictions in Python

Source: https://context7.com/leia-llm/leia/llms.txt

This Python snippet iterates through examples and their corresponding predictions, printing the question, predicted answer, and correct answer. It assumes `result.examples` and `result.predictions` are iterable and contain dictionaries with 'question', 'choices', and 'answer' keys.

```python
for example, prediction in zip(result.examples, result.predictions):
    question = example["question"]
    choices = example["choices"]
    correct_answer = example["answer"]
    predicted_answer = prediction

    print(f"Question: {question}")
    print(f"Predicted: {choices[predicted_answer]}, Correct: {choices[correct_answer]}")
```

--------------------------------

### Iterate Through Fixed-Length Examples in Python

Source: https://context7.com/leia-llm/leia/llms.txt

This Python code snippet demonstrates how to iterate through a dataset, access input IDs, and perform processing for training. It's intended for fixed-length datasets.

```python
for example in train_dataset:
    input_ids = example["input_ids"]  # torch.Tensor of shape (2048,)
    # Process example for training
    break
```

--------------------------------

### LEIA Model Training Script with Custom Dataset in Python

Source: https://context7.com/leia-llm/leia/llms.txt

This Python script provides a complete example of training a LEIA model. It utilizes custom datasets and the LeiaTrainer, configuring training arguments including entity augmentation parameters. It demonstrates parsing arguments and setting the random seed.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, set_seed
from datasets import load_from_disk
from leia.data import LeiaConstantLengthDataset, LeiaDataCollator
from leia.trainer import LeiaTrainer
from leia.training_args import LeiaTrainingArguments

# Parse arguments
parser = HfArgumentParser(LeiaTrainingArguments)
(args,) = parser.parse_args_into_dataclasses()

set_seed(args.seed)
```

--------------------------------

### Implement Custom Question Answering Task using Generation in Python

Source: https://context7.com/leia-llm/leia/llms.txt

This Python code defines a `CustomQATask` class inheriting from `GenerationTask`, suitable for open-ended question answering. It processes free-form text generation. The implementation includes defining task datasets, formatting examples, specifying generation requests with stop sequences and maximum length, and processing results for exact and contains matches. Dependencies include `leia.tasks.base` and `datasets`.

```python
from leia.tasks.base import GenerationTask, GenerationRequest, TaskResult
from datasets import Dataset
import re

class CustomQATask(GenerationTask):
    """Example: Question answering task with free-form generation"""

    def _get_train_dataset(self) -> Dataset | None:
        return None

    def _get_task_dataset(self) -> Dataset:
        return Dataset.from_dict({
            "question": ["What is the capital of France?", "Who wrote Hamlet?"],
            "answer": ["Paris", "William Shakespeare"]
        })

    def _example_to_text(self, example: dict) -> str:
        return f"Q: {example['question']}\nA:"

    def _example_to_target(self, example: dict) -> str:
        return f" {example['answer']}\n"

    def _create_requests(self, example: dict, context: str) -> list[GenerationRequest]:
        return [GenerationRequest(
            context=context,
            stop_sequences=["\n", "Q:"],  # Stop at newline or next question
            max_generation_length=50
        )]

    def _process_results(self, example: dict, results: list[str]) -> dict:
        prediction = results[0].strip()
        answer = example["answer"].lower()

        # Simple exact match after normalization
        exact_match = prediction.lower() == answer

        # Check if answer is contained in prediction
        contains_match = answer in prediction.lower()

        return {
            "exact_match": float(exact_match),
            "contains_match": float(contains_match),
            "prediction": prediction
        }

# Evaluate with generation
task = CustomQATask(
    model=model,
    accelerator=accelerator,
    tokenizer=tokenizer,
    batch_size=1,
    max_length=2048,
    max_generation_length=50,
    num_fewshot_samples=2,
    use_dynamic_generation_length=True  # Adjust length per example
)

result = task.run()
print(f"Exact Match: {result.metrics['exact_match']:.2%}")
print(f"Contains Match: {result.metrics['contains_match']:.2%}")
```

--------------------------------

### LeiaConstantLengthDataset: Streaming Dataset with Dynamic Entity Insertion (Python)

Source: https://context7.com/leia-llm/leia/llms.txt

This Python code defines a streaming dataset class, LeiaConstantLengthDataset, for creating fixed-length training examples from a preprocessed Wikipedia corpus. It dynamically inserts English entity names alongside target language entities during iteration, controlled by probabilities and insertion strategies. This requires the 'datasets' and 'transformers' libraries.

```python
from datasets import load_from_disk
from leia.data import LeiaConstantLengthDataset
from transformers import AutoTokenizer

# Load preprocessed Wikipedia dataset
wikipedia_dataset = load_from_disk("data/wikipedia/ja_dataset")

# Initialize tokenizer and get special token IDs
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer.add_special_tokens({"additional_special_tokens": ["<translate>", "</translate>"]})

# Create streaming dataset with entity augmentation
train_dataset = LeiaConstantLengthDataset(
    dataset=wikipedia_dataset,
    dataset_size=len(wikipedia_dataset),
    max_length=2048,  # Fixed sequence length
    max_num_examples=100000,  # Total examples to generate
    entity_name_start_token_id=tokenizer.vocab["<translate>"],
    entity_name_end_token_id=tokenizer.vocab["</translate>"],
    entity_name_insertion_prob=0.5,  # Insert English names for 50% of entities
    entity_name_insertion_strategy="right",  # Insert after entity: "東京<translate>Tokyo</translate>"
    no_separator_tokens=False,  # Use <translate> tokens
    shuffle=True,
    seed=42
)

```

--------------------------------

### Implement Custom Multiple Choice Task using Log-likelihood in Python

Source: https://context7.com/leia-llm/leia/llms.txt

This Python code defines a `CustomMultipleChoiceTask` class inheriting from `LoglikelihoodTask`. It is designed for multiple-choice evaluation using log-likelihood scoring. Key methods include defining datasets, formatting examples to text and target, creating log-likelihood requests for each choice, and processing results to determine accuracy. Dependencies include `leia.tasks.base`, `datasets`, `transformers`, and `accelerate`.

```python
from leia.tasks.base import LoglikelihoodTask, LogLikelihoodRequest, TaskResult
from datasets import Dataset
from transformers import PreTrainedModel, PreTrainedTokenizerBase
from accelerate import Accelerator
import numpy as np

class CustomMultipleChoiceTask(LoglikelihoodTask):
    """Example: Custom multiple-choice task using log-likelihood scoring"""

    def _get_train_dataset(self) -> Dataset | None:
        # Return training set for few-shot examples, or None
        return None

    def _get_task_dataset(self) -> Dataset:
        # Load your evaluation dataset
        return Dataset.from_dict({
            "question": ["What is the capital of Japan?", "What color is the sky?"],
            "choices": [["Tokyo", "Beijing", "Seoul"], ["Blue", "Green", "Red"]],
            "answer": [0, 0]
        })

    def _example_to_text(self, example: dict) -> str:
        # Format question and choices as prompt
        text = f"Question: {example['question']}\n"
        for i, choice in enumerate(example["choices"]):
            text += f"{i}. {choice}\n"
        text += "Answer:"
        return text

    def _example_to_target(self, example: dict) -> str:
        # Return correct answer for few-shot examples
        return f" {example['answer']}\n"

    def _create_requests(self, example: dict, context: str) -> list[LogLikelihoodRequest]:
        # Create log-likelihood request for each choice
        requests = []
        for i, choice in enumerate(example["choices"]):
            requests.append(LogLikelihoodRequest(
                context=context,
                continuation=f" {i}"
            ))
        return requests

    def _process_results(self, example: dict, results: list[float]) -> dict:
        # Select choice with highest log-likelihood
        prediction = int(np.argmax(results))
        correct = int(example["answer"])
        return {
            "accuracy": float(prediction == correct),
            "prediction": prediction
        }

# Use the custom task
task = CustomMultipleChoiceTask(
    model=model,
    accelerator=accelerator,
    tokenizer=tokenizer,
    batch_size=1,
    max_length=2048,
    num_fewshot_samples=3
)

result = task.run()
print(f"Accuracy: {result.metrics['accuracy']:.2%}")
```

--------------------------------

### Evaluate Models on Cross-Lingual Tasks (Bash Script)

Source: https://context7.com/leia-llm/leia/llms.txt

Sets up and runs an evaluation script for trained LEIA models on cross-lingual understanding and reasoning tasks. It utilizes `accelerate launch` to distribute the evaluation process and outputs metrics and predictions for each task. The script configures essential parameters such as model path, tasks to evaluate, and few-shot learning settings.

```bash
# Set evaluation parameters
export MODEL_NAME_OR_PATH="runs/leia_ja"  # Path to trained model
export TASKS="xcodah_ja,xcsqa_ja,xnli_ja"  # Comma-separated task names
export NUM_FEWSHOT_SAMPLES="0,4,2"  # Few-shot examples per task
export OUTPUT_DIR="evaluation_results"

# Run evaluation
./evaluate.sh

# The script runs:
# accelerate launch --num_processes 1 evaluate.py \
#     --model_name_or_path ${MODEL_NAME_OR_PATH} \
#     --tasks ${TASKS} \
#     --num_fewshot_samples ${NUM_FEWSHOT_SAMPLES} \
#     --output_dir ${OUTPUT_DIR} \
#     --use_flash_attention_2

# Outputs:
# - evaluation_results/xcodah_ja_metrics.json
# - evaluation_results/xcodah_ja_predictions.jsonl
# - evaluation_results/xcsqa_ja_metrics.json
# - evaluation_results/xcsqa_ja_predictions.jsonl
# - evaluation_results/xnli_ja_metrics.json
# - evaluation_results/xnli_ja_predictions.jsonl

```

--------------------------------

### Load and Prepare Model and Tokenizer (Python)

Source: https://context7.com/leia-llm/leia/llms.txt

Loads a pre-trained model and tokenizer, adds special tokens for entity boundaries, and resizes token embeddings. Initializes new token embeddings to the average of existing embeddings. This sets up the model for tasks requiring specific entity recognition.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from datasets import load_from_disk
from leia.datasets import LeiaConstantLengthDataset
from leia.data_collator import LeiaDataCollator
from leia.trainer import LeiaTrainer

# Assuming 'args' is a parsed argument object

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
if tokenizer.pad_token_id is None:
    tokenizer.pad_token_id = tokenizer.eos_token_id

model = AutoModelForCausalLM.from_pretrained(
    args.model_name_or_path,
    use_flash_attention_2=args.use_flash_attention_2
)

# Add special tokens for entity boundaries
num_new_tokens = tokenizer.add_special_tokens({
    "additional_special_tokens": ["<translate>", "</translate>"]
})
model.resize_token_embeddings(len(tokenizer))

# Initialize new token embeddings to average of existing embeddings
embeddings_data = model.get_input_embeddings().weight.data
embeddings_avg = embeddings_data[:-num_new_tokens].mean(dim=0, keepdim=True)
embeddings_data[-num_new_tokens:] = embeddings_avg

# Load augmented Wikipedia dataset
wikipedia_dataset = load_from_disk(args.wikipedia_dataset_dir)

# Create streaming dataset with dynamic entity insertion
max_num_examples = (args.max_steps * args.gradient_accumulation_steps *
                   args.per_device_train_batch_size * args.world_size)

train_dataset = LeiaConstantLengthDataset(
    wikipedia_dataset,
    dataset_size=len(wikipedia_dataset),
    max_length=args.max_length,
    max_num_examples=max_num_examples,
    entity_name_start_token_id=tokenizer.vocab["<translate>"],
    entity_name_end_token_id=tokenizer.vocab["</translate>"],
    entity_name_insertion_prob=args.entity_name_insertion_prob,
    entity_name_insertion_strategy=args.entity_name_insertion_strategy,
    no_separator_tokens=args.no_separator_tokens,
    shuffle=True,
    seed=args.seed
)

# Data collator with optional entity loss masking
data_collator = LeiaDataCollator(
    tokenizer=tokenizer,
    max_length=args.max_length,
    disable_entity_name_token_loss=args.disable_entity_name_token_loss
)

# Parse evaluation tasks if provided
tasks = args.tasks.split(",") if args.tasks else []
num_fewshot_samples = [int(n) for n in args.num_fewshot_samples.split(",")] if args.num_fewshot_samples else None

# Train with custom LEIA trainer (supports evaluation on cross-lingual tasks)
trainer = LeiaTrainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
    data_collator=data_collator,
    tasks=tasks,
    num_fewshot_samples=num_fewshot_samples,
    task_kwargs={"max_length": args.max_length}
)

trainer.train()
trainer.save_state()
trainer.save_model()

```

--------------------------------

### Build Wikipedia Dataset with Entity Augmentation (Bash)

Source: https://context7.com/leia-llm/leia/llms.txt

This script automates the process of downloading Wikipedia and Wikidata dumps, extracting text and links, resolving redirects and Wikidata IDs, and finally building a tokenized dataset augmented with English entity names. It requires specific environment variables to be set and utilizes various Python scripts and external tools like wikiextractor.

```bash
# Set environment variables
export LANG="ja"
export MODEL_NAME_OR_PATH="meta-llama/Llama-2-7b-hf"
export WIKIDATA_DATA_DIR="data/wikidata"
export WIKIPEDIA_DATA_DIR="data/wikipedia"
export WIKIDATA_DUMP_DATE="20230703"
export WIKIPEDIA_DUMP_DATE="20230701"
export WIKIPEDIA_DATASET_DIR="${WIKIPEDIA_DATA_DIR}/${LANG}_dataset"

# Create directories
mkdir -p ${WIKIDATA_DATA_DIR}
mkdir -p ${WIKIPEDIA_DATA_DIR}

# Download dumps
wget https://dumps.wikimedia.org/wikidatawiki/entities/${WIKIDATA_DUMP_DATE}/wikidata-${WIKIDATA_DUMP_DATE}-all.json.bz2 -P ${WIKIDATA_DATA_DIR}
wget https://dumps.wikimedia.org/${LANG}wiki/${WIKIPEDIA_DUMP_DATE}/${LANG}wiki-${WIKIPEDIA_DUMP_DATE}-pages-articles-multistream.xml.bz2 -P ${WIKIPEDIA_DATA_DIR}

# Extract Wikipedia text with links preserved
wikiextractor \
    ${WIKIPEDIA_DATA_DIR}/${LANG}wiki-${WIKIPEDIA_DUMP_DATE}-pages-articles-multistream.xml.bz2 \
    -o ${WIKIPEDIA_DATA_DIR}/${LANG} \
    --html-safe "" \
    --links \
    --no-templates

# Extract Wikipedia redirects
python scripts/extract_wikipedia_redirects.py \
    --dump_file ${WIKIPEDIA_DATA_DIR}/${LANG}wiki-${WIKIPEDIA_DUMP_DATE}-pages-articles-multistream.xml.bz2 \
    --output_file ${WIKIPEDIA_DATA_DIR}/${LANG}wiki-redirects.tsv

# Extract Wikidata interlanguage links
python scripts/extract_wikidata_ids.py \
    --dump_file ${WIKIDATA_DATA_DIR}/wikidata-${WIKIDATA_DUMP_DATE}-all.json.bz2 \
    --output_dir ${WIKIDATA_DATA_DIR} \
    --languages "en,${LANG}"

# Preprocess Wikipedia corpus (parse HTML, resolve links)
python scripts/preprocess_wikipedia.py \
    --wikiextractor_output_dir "${WIKIPEDIA_DATA_DIR}/${LANG}" \
    --redirect_file "${WIKIDATA_DATA_DIR}/${LANG}wiki-redirects.tsv" \
    --wikidata_id_file "${WIKIDATA_DATA_DIR}/${LANG}-wikidata-ids.tsv" \
    --output_dir "${WIKIPEDIA_DATA_DIR}/${LANG}_preprocessed"

# Build final tokenized dataset with entity augmentation metadata
python scripts/build_wikipedia_dataset.py \
    --model_name ${MODEL_NAME_OR_PATH} \
    --preprocessed_dataset_dir "${WIKIPEDIA_DATA_DIR}/${LANG}_preprocessed" \
    --wikidata_id_file "${WIKIDATA_DATA_DIR}/en-wikidata-ids.tsv" \
    --output_dir ${WIKIPEDIA_DATASET_DIR}

# Preview the augmented dataset
python scripts/preview_dataset.py \
    --model_name ${MODEL_NAME_OR_PATH} \
    --dataset_dir ${WIKIPEDIA_DATASET_DIR}
```

--------------------------------

### Preview Augmented Wikipedia Dataset

Source: https://github.com/leia-llm/leia/blob/main/README.md

Generates a preview of the built Wikipedia dataset, which is augmented with English entity names. This script helps in verifying the structure and content of the training data before fine-tuning.

```bash
python scripts/preview_dataset.py \
    --model_name ${MODEL_NAME_OR_PATH} \
    --dataset_dir ${WIKIPEDIA_DATASET_DIR}
```

--------------------------------

### Build Japanese Wikipedia Corpus with English Entities

Source: https://github.com/leia-llm/leia/blob/main/README.md

Scripts to download and process Japanese Wikipedia data, augmenting it with English entity names for LLM fine-tuning. This process involves downloading dumps, extracting text, and linking entities using Wikidata.

```bash
# Target language
export LANG="ja"

# Model to be fine-tuned
export MODEL_NAME_OR_PATH="meta-llama/Llama-2-7b-hf"

# Directories for storing Wikidata and Wikipedia data
export WIKIDATA_DATA_DIR="data/wikidata"
export WIKIPEDIA_DATA_DIR="data/wikipedia"

# Dump dates for Wikidata and Wikipedia
export WIKIDATA_DUMP_DATE="20230703"
export WIKIPEDIA_DUMP_DATE="20230701"

# Directory for storing the training dataset
export WIKIPEDIA_DATASET_DIR="${WIKIPEDIA_DATA_DIR}/${LANG}_dataset"

# Create directories for Wikidata and Wikipedia data
mkdir -p ${WIKIDATA_DATA_DIR}
mkdir -p ${WIKIPEDIA_DATA_DIR}

# Download Wikidata and Wikipedia dumps
wget https://dumps.wikimedia.org/wikidatawiki/entities/${WIKIDATA_DUMP_DATE}/wikidata-${WIKIDATA_DUMP_DATE}-all.json.bz2 -P ${WIKIDATA_DATA_DIR}
wget https://dumps.wikimedia.org/${LANG}wiki/${WIKIPEDIA_DUMP_DATE}/${LANG}wiki-${WIKIPEDIA_DUMP_DATE}-pages-articles-multistream.xml.bz2 -P ${WIKIPEDIA_DATA_DIR}

# Process Wikipedia dump using WikiExtractor
wikiextractor \
    ${WIKIPEDIA_DATA_DIR}/${LANG}wiki-${WIKIPEDIA_DUMP_DATE}-pages-articles-multistream.xml.bz2 \
    -o ${WIKIPEDIA_DATA_DIR}/${LANG} \
    --html-safe "" \
    --links \
    --no-templates

# Extract Wikipedia redirect information
python scripts/extract_wikipedia_redirects.py \
    --dump_file ${WIKIPEDIA_DATA_DIR}/${LANG}wiki-${WIKIPEDIA_DUMP_DATE}-pages-articles-multistream.xml.bz2 \
    --output_file ${WIKIPEDIA_DATA_DIR}/${LANG}wiki-redirects.tsv

# Extract inter-language link data from Wikidata dump
python scripts/extract_wikidata_ids.py \
    --dump_file ${WIKIDATA_DATA_DIR}/wikidata-${WIKIDATA_DUMP_DATE}-all.json.bz2 \
    --output_dir ${WIKIDATA_DATA_DIR} \
    --languages "en,${LANG}"

# Preprocess Wikipedia corpus
python scripts/preprocess_wikipedia.py \
    --wikiextractor_output_dir "${WIKIPEDIA_DATA_DIR}/${LANG}" \
    --redirect_file "${WIKIPEDIA_DATA_DIR}/${LANG}wiki-redirects.tsv" \
    --wikidata_id_file "${WIKIDATA_DATA_DIR}/${LANG}-wikidata-ids.tsv" \
    --output_dir "${WIKIPEDIA_DATA_DIR}/${LANG}_preprocessed"

# Build Wikipedia dataset for training
python scripts/build_wikipedia_dataset.py \
    --model_name ${MODEL_NAME_OR_PATH} \
    --preprocessed_dataset_dir "${WIKIPEDIA_DATA_DIR}/${LANG}_preprocessed" \
    --wikidata_id_file "${WIKIDATA_DATA_DIR}/en-wikidata-ids.tsv" \
    --output_dir ${WIKIPEDIA_DATASET_DIR}
```

--------------------------------

### Train LEIA Model

Source: https://github.com/leia-llm/leia/blob/main/README.md

Initiates the fine-tuning process for the LLM using the previously built augmented Wikipedia dataset. The script saves the trained model checkpoints to a specified output directory.

```bash
# Name for this training run
export RUN_NAME="leia_${LANG}"
# Output directory for saving the model checkpoint files
export OUTPUT_DIR="runs/leia_${LANG}"

# Start training
./train.sh
```

--------------------------------

### Train Language Models with Entity-Augmented Wikipedia Data via Bash Script

Source: https://context7.com/leia-llm/leia/llms.txt

This bash script outlines the process for training language models on a Wikipedia corpus with dynamic English entity insertion, utilizing DeepSpeed ZeRO-3 for distributed training. It sets various training parameters and launches the training script.

```bash
# Set training parameters
export RUN_NAME="leia_ja"
export OUTPUT_DIR="runs/leia_ja"
export MODEL_NAME_OR_PATH="meta-llama/Llama-2-7b-hf"
export WIKIPEDIA_DATASET_DIR="data/wikipedia/ja_dataset"

# Optional: Configure entity insertion
export ENTITY_NAME_INSERTION_STRATEGY="right"  # Options: left, right, replace, none
export ENTITY_NAME_INSERTION_PROB="0.5"  # Probability of inserting English names
export NO_SEPARATOR_TOKENS="false"  # Whether to use <translate> tokens
export LEARNING_RATE="5e-6"
export SEED="42"

# Launch distributed training
./train.sh

# The script runs:
# accelerate launch --num_processes 8 --use_deepspeed --zero_stage 3 train.py \
#     --run_name ${RUN_NAME} \
#     --output_dir ${OUTPUT_DIR} \
#     --model_name_or_path ${MODEL_NAME_OR_PATH} \
#     --wikipedia_dataset_dir ${WIKIPEDIA_DATASET_DIR} \
#     --entity_name_insertion_strategy right \
#     --entity_name_insertion_prob 0.5 \
#     --per_device_train_batch_size 1 \
#     --gradient_accumulation_steps 256 \
#     --learning_rate 5e-6 \
#     --max_steps 50 \
#     --max_length 2048 \
#     --bf16 \
#     --use_flash_attention_2
```

--------------------------------

### LeiaTrainingArguments Configuration in Python

Source: https://context7.com/leia-llm/leia/llms.txt

This Python code defines the LeiaTrainingArguments dataclass, which extends HuggingFace's TrainingArguments. It incorporates LEIA-specific parameters for configuring entity name insertion strategies, probabilities, and loss masking during training.

```python
from dataclasses import dataclass, field
from transformers import TrainingArguments

@dataclass
class LeiaTrainingArguments(TrainingArguments):
    model_name_or_path: str | None = field(default=None)
    use_flash_attention_2: bool = field(default=True)

    # Dataset configuration
    wikipedia_dataset_dir: str | None = field(default=None)

    # Entity insertion strategy
    entity_name_insertion_strategy: str = field(default="right")
    # Options:
    #   - "left": Insert before entity ("Tokyo" -> "<translate>Tokyo</translate>東京")
    #   - "right": Insert after entity ("東京" -> "東京<translate>Tokyo</translate>")
    #   - "replace": Replace entity ("東京" -> "<translate>Tokyo</translate>")
    #   - "none": No insertion

    entity_name_insertion_prob: float = field(default=0.5)  # 0.0 to 1.0
    disable_entity_name_token_loss: bool = field(default=False)  # Mask English token loss
    no_separator_tokens: bool = field(default=False)  # Omit <translate> tokens

    max_length: int = field(default=2048)

    # Evaluation tasks during training
    tasks: str | None = field(default=None)  # Comma-separated: "xcodah_ja,xcsqa_ja"
    num_fewshot_samples: str | None = field(default=None)  # Comma-separated: "0,4"
```

--------------------------------

### Programmatic Evaluation with Custom Tasks (Python)

Source: https://context7.com/leia-llm/leia/llms.txt

Evaluates LEIA models programmatically using the evaluation API. This Python script demonstrates how to load a model and tokenizer, prepare them with `accelerator`, and then use the `get_task` function to instantiate and run a specific evaluation task, such as XWinograd. It allows for detailed control over evaluation parameters like batch size, few-shot samples, and maximum samples.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from accelerate import Accelerator
from leia.tasks import get_task

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "runs/leia_ja",
    trust_remote_code=True,
    torch_dtype=torch.float16,
    use_flash_attention_2=True
)

accelerator = Accelerator()
model = accelerator.prepare(model)

tokenizer = AutoTokenizer.from_pretrained("runs/leia_ja", trust_remote_code=True)
tokenizer.pad_token_id = tokenizer.eos_token_id

# Evaluate on XWinograd (Japanese commonsense reasoning)
task_cls = get_task("xcodah_ja")
task = task_cls(
    model=model,
    accelerator=accelerator,
    tokenizer=tokenizer,
    batch_size=1,
    max_length=2048,
    num_fewshot_samples=4,  # 4-shot evaluation
    max_samples=None  # Evaluate on full dataset
)

result = task.run()

# Access metrics and predictions
print(result.metrics)  # {"accuracy": 0.72, "num_examples": 500}
print(len(result.examples))  # 500
print(len(result.predictions))  # 500

```

--------------------------------

### Evaluate LEIA Model Performance

Source: https://github.com/leia-llm/leia/blob/main/README.md

Runs the evaluation script to assess the performance of the fine-tuned LEIA model on specified cross-lingual tasks. It uses the trained model checkpoints and task configurations for evaluation.

```bash
# Model path to be evaluated
export MODEL_NAME_OR_PATH=${OUTPUT_DIR}
# Tasks to be evaluated
export TASKS="xcodah_${LANG},xcsqa_${LANG}"
# Number of fewshot samples for each task
export NUM_FEWSHOT_SAMPLES="0,4"

./evaluate.sh
```

--------------------------------

### LeiaDataCollator for Batch Collation in Python

Source: https://context7.com/leia-llm/leia/llms.txt

The LeiaDataCollator class from the LEIA project collates batches of data for training. It supports optional masking of loss computation for English entity names, allowing them to be retained as input context but not directly optimized.

```python
from leia.data import LeiaDataCollator
from transformers import AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer.add_special_tokens({"additional_special_tokens": ["<translate>", "</translate>"]})
tokenizer.pad_token_id = tokenizer.eos_token_id

# Create collator with entity name loss disabled
collator = LeiaDataCollator(
    tokenizer=tokenizer,
    max_length=2048,
    disable_entity_name_token_loss=True  # Don't compute loss on English tokens
)

# Example batch of input sequences
examples = [
    {"input_ids": torch.randint(0, 32000, (2048,))},
    {"input_ids": torch.randint(0, 32000, (1500,))}
]

# Collate batch
batch = collator(examples)
# Returns:
# {
#     "input_ids": torch.Tensor of shape (2, 2048),  # Padded batch
#     "attention_mask": torch.Tensor of shape (2, 2048),
#     "labels": torch.Tensor of shape (2, 2048)  # -100 for pad, special tokens, and English entities
# }

# Labels mask padding, <translate> tokens, and optionally English entity tokens
# This allows the model to see English names as context but not optimize for generating them
```

--------------------------------

### Load TSV Mappings with Python

Source: https://context7.com/leia-llm/leia/llms.txt

Loads tab-separated mapping files using the `load_tsv_mapping` function from `leia.utils`. This function is used to load mappings such as Wikipedia redirects (title to canonical title) and Wikidata IDs (title to Wikidata ID). It takes the file path and an optional `value_type` argument. The loaded mappings are typically dictionaries used for entity resolution.

```python
from leia.utils import load_tsv_mapping

# Load Wikipedia redirect mapping (title -> canonical_title)
redirects = load_tsv_mapping(
    "data/wikipedia/jawiki-redirects.tsv",
    value_type=str
)
# Returns: {"東京都": "東京", "ニューヨーク市": "ニューヨーク"}

# Load Wikidata ID mapping (title -> wikidata_id)
wikidata_ids = load_tsv_mapping(
    "data/wikidata/ja-wikidata-ids.tsv",
    value_type=str
)
# Returns: {"東京": "Q1490", "ニューヨーク": "Q60"}

# Use for entity resolution
entity_title = "東京都"
canonical_title = redirects.get(entity_title, entity_title)  # "東京"
wikidata_id = wikidata_ids.get(canonical_title)  # "Q1490"
```

--------------------------------

### Normalize Wikipedia Titles with Python

Source: https://context7.com/leia-llm/leia/llms.txt

Normalizes Wikipedia article titles using the `normalize_wikipedia_title` function from `leia.utils`. This function ensures consistency by capitalizing the first character and replacing underscores with spaces, which is crucial for accurate entity matching. It accepts various title formats as input.

```python
from leia.utils import normalize_wikipedia_title

# Normalize various Wikipedia title formats
title1 = normalize_wikipedia_title("tokyo")  # "Tokyo"
title2 = normalize_wikipedia_title("new_york_city")  # "New york city"
title3 = normalize_wikipedia_title("united_states_of_america")  # "United states of america"

# First character capitalized, underscores replaced with spaces
assert title1 == "Tokyo"
assert title2 == "New york city"
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.